Title: Feint Behaviors and Strategies: Formalization, Implementation and Evaluation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Feint Behaviors

Multi-Player Games

Multi-Agent Reinforcement Learning
Abstract: Feint behaviors refer to a set of deceptive behaviors in a nuanced manner, which enable players to obtain temporal and spatial advantages over opponents in competitive games. Such behaviors are crucial tactics in most competitive multi-player games (e.g., boxing, fencing, basketball, motor racing, etc.). However, existing literature does not provide a comprehensive (and/or concrete) formalization for Feint behaviors, and their implications on game strategies. In this work, we introduce the first comprehensive formalization of Feint behaviors at both action-level and strategy-level, and provide concrete implementation and quantitative evaluation of them in multi-player games. The key idea of our work is to (1) allow automatic generation of Feint behaviors via Palindrome-directed templates, combine them into meaningful behavior sequences via a Dual-Behavior Model; (2) concertize the implications from our formalization of Feint on game strategies, in terms of temporal, spatial, and their collective impacts respectively; and (3) provide a unified implementation scheme of Feint behaviors in existing MARL frameworks. The experimental results show that our design of Feint behaviors can (1) greatly improve the game reward gains; (2) significantly improve the diversity of Multi-Player Games; and (3) only incur negligible overheads in terms of time consumption.

Feint Behaviors and Strategies: Formalization, Implementation and Evaluation

Abstract

Feint behaviors refer to a set of deceptive behaviors in a nuanced manner, which enable players to obtain temporal and spatial advantages over opponents in competitive games. Such behaviors are crucial tactics in most competitive multi-player games (e.g., boxing, fencing, basketball, motor racing, etc.). However, existing literature does not provide a comprehensive (and/or concrete) formalization for Feint behaviors, and their implications on game strategies. In this work, we introduce the first comprehensive formalization of Feint behaviors at both action-level and strategy-level, and provide concrete implementation and quantitative evaluation of them in multi-player games. The key idea of our work is to (1) allow automatic generation of Feint behaviors via Palindrome-directed templates, combine them into meaningful behavior sequences via a Dual-Behavior Model; (2) concertize the implications from our formalization of Feint on game strategies, in terms of temporal, spatial and their collective impacts respectively; and (3) provide a unified implementation scheme of Feint behaviors in existing MARL frameworks. The experimental results show that our design of Feint behaviors can (1) greatly improve the game reward gains; (2) significantly improve the diversity of Multi-Player Games; and (3) only incur negligible overheads in terms of time consumption.

1 Introduction

In most real-world Multi-Player Games (e.g., boxing, basketball, motor racing, and etc.), players exhibit complex behaviors, which imply hard-to-be-quantified interactions. Simulating these games requires to model the players' behaviors into action spaces as the elements (i.e., which we denote such a decision as 'action-level"), and explore strategies based on these elements [34, 35]. As one of the most common behaviors in real-world games, Feint behaviors represent a class of tactic behaviors, which are used to mislead opponents to obtain temporally-strategic advantages 1 . Such behaviors generally exhibit nuanced differences with normal behaviors, in terms of action movements 2 ; but these behaviors can help the agent to obtain strategic advantages to a considerable extent 3 . However, no existing literature provides a comprehensive formalization (and/or concrete modeling of) Feint behaviors, at either action level or strategy level, and we justify this claim below.

To the best of our knowledge, [34] is the first work to mention Feint behaviors, as a proof-ofconcept to construct animations for nuanced behaviors in two-player boxing games, based on the unpredictability introduced in stochastic and simultaneous decision making.

A more recent work [35] learn control strategies to animate nuanced agent behaviors like Feint in gaming environments, from motion clips using Deep Reinforcement Learning.

Summary. Prior attempts do not provide detailed formalization to address the action-level characteristic of Feint and do not give Feint behavior generation guidelines. As for the strategy-level formalization, existing learning-based works either neglect Feint behaviors or assume that Feint are just glitches from other behaviors, by inducing similar impacts implicitly. 4

Overview. Our work provides the first comprehensive and concrete formalization of Feint behaviors in action-level and strategy-level. We first present an automatic approach to generate Feint behaviors using Palindrome-directed Templates based on our observation on Feint characteristics, and provide Dual-Behavior Model to examine the design for the combination of Feint behaviors and normal actions. Based on the action-level formalization, we then model the Feint behavior impacts on strategy-level in terms of the temporal, spatial, and their collective impacts under a learnable scheme. Then, we provide a concrete and unified implementation to incorporate the action-level and strategylevel formalization in common Multi-Player Reinforcement Learning (MARL) frameworks; so that we can showcase the effectiveness of our formalization 5 .

Results. To properly examine the effectiveness of our formalization, we extensively construct a complex and physics-based boxing game as abstraction of some animation-related works [34, 35]. We use a two-player and a six-player scenario with 4 commonly used MARL models (MADDPG [20], MASAC [11, 13], MATD3 [2], and MAD3PG [4, 6]) to extensively evaluate our formalization. We also evaluate our formalization of Feint in a strategic real-game, Alpha Star, to examine the diversity gain introduced by our formalization. The results show that our formalization of Feint can significantly increase the gaming rewards in all scenarios with all 4 MARL models. We also extensively examine our approaches: for the Diversity Gain, our method can increase the exploitation of the search space by 1.98X, measured by the Exploitability metrics; and our implementation scheme only incur less than 5% overheads in terms of per game episode time consumption. We conclude that our formalization of Feint behaviors is effective and practical, significantly increasing players' game rewards and making Multi-Player Games more interesting.

2 Background

2.1 Feint Behaviors in the Real-World and Simulated Games

Feint behaviors are common for human players, as a set of active actions to obtain strategic advantages in real-world games. Examples can include sports games such as boxing, basketball, and motor racing [10, 9, 12], and electronic games such as King of Fighters and Starcraft [33, 5]. Feint behaviors are not simple deceptive behaviors as their goal is to not to gain rewards for themselves but to create temporal and spatial advantages for some short-term follow-up actions. In addition, Feint behaviors have nuanced action formalizations. Though Feint is undoubtedly important in many real-world games, there still lacks a comprehensive formalization of Feint in Multi-Player Game simulations using Non-Player Characters (NPCs). There are only a limited amount of works to tackle this issue. [34] is an early example of incorporating Feint as a proof-of-concept, which focuses on constructing animations for nuanced game strategies for more unpredictability from NPCs. More recently, [36] learn multi-level control strategies of agents via deep reinforcement learning from a set of motion clips including nuanced behaviors like Feint (i.e. in animating combat scenes). However, these prior works (1) lack concrete formalizations of Feint behavior characteristics, which cannot fully unveil the variety of Feint behaviors in the action level; and (2) lack comprehensive explorations of Feint behaviors implications on game strategies, which neglects the potential impacts of fusing effective Feint behaviors into strategies; and (3) solely focus on Two-Player Games, which can not be effectively generalized to multi-player scenarios.

2.2 MARLModels at Strategy-Level in Multi-Player Game Simulations

Multi-Agent Reinforcement Learning (MARL) aims to learn optimal policies for agents in a multiagent environment, which consists of various agent-agent and agent-environment interactions 6 . Many single-agent Reinforcement Learning methods (e.g. DDPG [18], SAC [11], PPO [29] and TD3 [8], D4PG [4]) can not be directly used in multi-agent scenarios, since the rapidly-changing multi-agent environment can cause highly unstable learning results (evidenced by [20]). Thus, recent efforts

on MARL model designs aim to address such an issue. [7] proposes Counterfactual Multi-Agent (COMA) policy gradients, which uses centralised critic to estimate the Q-function and decentralised actors to optimize agents' policies. [20] proposes Multi-Agent Deep Deterministic Policy Gradient (MADDPG), which decreases the variance in policy gradient and instability of Q-function of DDPG in multi-agent scenarios. [13] proposes Multi-Agent Actor-attention Critic (MAAC), which applies attention entropy mechanism to enable effective and scalable policy learning. These models can have varied impacts within a diverse set of scenarios. [6] introduces Multi-agent Distributed Deep Deterministic Policy Gradient (MAD3PG), which extends the D4PG to multi-agent scenarios with distributed critics to enable distributed tracking. [2] proposes Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3), which integrates twin delayed Q-learning and addressing the overestimation bias in Q-values in a multi-agent setting. Though different MARL models have different design details, they all share the same high-level learning structure. Thus, our goal is to provide a unified scheme to fuse our formalization of Feint behaviors into game simulations that can be learned using common MARL models, enabling effective Feint behaviors impacts regardless of specific design choices of MARL models.

3 Formalizing Feint Behavior

We introduce our formalization of Feint behaviors in action level regarding (1) how to automatically generate Feint behavior with templates from common attack behaviors; and (2) how can the generated Feint behaviors be synergistically combined with follow-up high-reward actions. We first introduce our methodology to automatically generate Feint behaviors, by exploiting our newly-revealed insight called Palindrome-directed Generation of Feint Templates . Next, we illustrate key design choices on how to combine the generated Feint behaviors with follow-up actions in a Double-Behavior Model , which forms the foundation for the designs of Feint-accounted strategy designs in Section 4 7 .

3.1 Feint Behavior Characteristics and Templates

Since Feint behaviors aim to provide deceptive attacks, they are naturally expected to be derived from a subset of existing attack behaviors. Based on our exploration, we derive two key findings from an extensive amount of attack behaviors. First, most attack behaviors can be decomposed into three action sequences, which are Stretch-out Sequence (Sequence 1), Reward Sequence (Sequence 2), and Retract Sequence (Sequence 3) (an example shown in the first row in Figure 1). We elaborate on each action sequence in detail.

Sequence 1 leads the agent's movements to the Reward Sequence (in boxing, approaching the opponents before actually hitting them); Sequence 2 contains actions that gain game rewards (in boxing, physical contact with the opponents); and Sequence 3 retracts an agent's movements to a relative rest position (in boxing, retracting back to a preparation position for next behaviors). Second, body movements in Sequence 1 and Sequence 3 usually have semi-symmetric yet reverse-order action patterns in the timeline. A behavior usually starts and ends in a similar physical state due to physical restrictions (e.g., bones and muscles stretching restrictions for a humanoid).

The above three-stage decomposition of attack behaviors has motivated a series of constraints, to deliver proper design of Feint generators. To satisfy the above two requirements, we propose a Feint behavior template generator called Palindrome-directed Generation of Feint Templates , by extracting subsets of semi-symmetrical actions from an attack behavior and synthesizing them as a Feint behavior. The general method to generate these templates are (1) by extracting subsets of unit actions from an attack behavior, a Feint behavior can be considered as a semi-finished real attack behavior. This ensures the high similarity of a generated Feint behavior with an attack behavior, thus opponents can be deceived; and (2) by synthesizing semi-symmetric action sections, the overall movements can be connected smoothly and the naturalness of humanoid actions can be guaranteed. Within our proposed template generator Palindrome-directed Generation of Feint Templates , there are two key adjustable parameters in practice: (1) sequence composition positions for Feint templates; and (2) sequence length for Feint templates. We describe our rationale in appendix B. Figure 1 demonstrates 3 templates for generating Feint behavior templates in boxing games.

3.2 Feint Behavior in Consecutive Game Steps

Standalone Feint behaviors are meaningless in competitive games since the Feint behaviors themselves do not gain rewards. Only by effectively combining Feint behaviors with intended follow-up actions

Sequence 1 - Strech-Out

Sequence 2 - Reward

Sequence 3 - Retract

Similarphysical state

Actions before the similarstate inSequence1

Actions afterthe similarstate inSequence3

FromSequnece1+Sequnece3:

Thereverse

Thereverse

ActionsequenceextractedfromSequence3

From Sequence 1 + Sequence 1 Reverse

FromSequence 3+Sequence3Reverse

can showcase their effectiveness. Thus, we define an effective Feint cycle as a Dual-Behavior Model , which jointly considers a Feint behavior and its intended follow-up behavior (can be a single action or an action sequence). Our formalization for standalone Feint behaviors (Section 3.1) already provides a large number of possible Feint behaviors. However, not all these morphologically reasonable Feint actions can be directly combined with all high-reward follow-up actions in combating scenarios. Therefore, certain constraints are demanded to construct effective combinations of Feint behaviors and follow-up actions. Hereby, we introduce two major considerations and then propose relevant restrictions, to enable naturalistic and suitable combinations of Feint behaviors and follow-up actions.

AchosenAttackBehaviortogenerate

Feint

TheintendedAttackBehavior

Backward

Stretchi out

Damage

Retract

Search

Stretch out

Damage

Retract

GenerateFeintBehavior

using templates

Agent

Retract

Reward

Retract

Agent-FeintBehavior

Agent-Follow-upOffensiveBahevior

Opponent

AfullDefenseBehaviortowardsFeint

Stretch out

Notabletofinish

Opponent-defendtowardsFeint

Opponent ↓ not able to acomplish another behavior

Temporal

Spatial

SufferfromAgent'sReward

Advantage

Advantage

Timeline

(1) Physical Constraints: Physical constraints need to be accounted for when synthesizing Feint behaviors and follow-up actions. The ending physical state for a Feint behavior must be a state that is physically possible for an agent to perform the follow-up high-reward actions. For example, if an agent in boxing games finishes Feint behavior with the left foot forward, but the following attack behavior starts with the right foot forwarded, the synthesis of these two behavior is inappropriate since this combination is physically unrealistic.

To ensure that the combinations of Feint behaviors and follow-up actions obey the physical constraints, we use a Reverse Search Principle which decides the intended follow-up actions (behavior) first and then use the starting physical state of this behavior to search and compose proper Feint behaviors (a more detailed description combined with strategy is described in Section 5). By first selecting an intended follow-up high-reward behavior, the end physical state of the Feint behavior is constrained to be close to the starting physical state of the follow-up behavior. Thus the composition of possible Feint behaviors using the Palindrome-directed templates should aim to start and end at a physical state that is close to the follow-up behavior.

(2) Effectiveness: The effectiveness of the incorporation of Feint behaviors is evaluated by whether the following attack actions can successfully gain rewards from the opponent. A successful Feint behavior would usually enable an agent to gain temporal and spatial advantages when performing the follow-up behaviors. Thus, the two design parameters introduced in Section 3.2 play crucial roles in combining Feint with follow-up behaviors. The abstraction of an ideal Dual-Behavior model that can enable an agent with temporal and spatial advantage is illustrated in Figure 2 and a corresponding example is provided in Figure 6. An effective Feint behavior creates temporal advantages that make the opponents to defend in a wrong direction and enable temporal advantages to allow the follow-up high-reward behavior to successfully gain rewards on the opponents.

To ensure the consistency and correctness of the understanding, we provide a detailed demonstration for successful and unsuccessful Feint cases in Appendix C.

4 Formalizing Feint Behaviors in Strategies

To effectively fuse the Feint beahviors with Dual-Behavior Model into game interactions, we provide the strategy-level formalization of Feint behaviors. We use Multi-Agent Reinforcement Learning (MARL) schemes to discuss our formalization of Feint behaviors in the strategy level, as MARL provides flexibility in exposing multiple adjustable parameters in learnable policy models. As discussed in generating Feint behaviors (Section 3.1) and composing them in the Dual-Behavior Models (Section 3.2), the key considerations for effective Feint cycle is to enable temporal and spatial advantages for an agent. Thus, our strategy-level formalization centers on how to address the temporal, spatial, and their collective impacts of Feint behaviors with Dual-Behavior Models. A more concrete introduction for fusing of Feint into the MARL frameworks is presented in Section 5.

4.1 The Basic Formalization: Derivation and Limitations

Under commonly used MARL schemes [25, 19], we define a K -agent Non-transitive Active Markov Game Model as a tuple ⟨ K, S , A , P, R, Π , Θ ⟩ : S is the state space; A = {A i } K i =1 is the set of action space for each agent, where there are no dominant actions; P performs state transitions of current state by agents' actions: P : S × A 1 ×A 2 × ... ×A K → P ( S ) , where P ( S ) denotes the set of probability distribution over state space S ; R = { R i } K i =1 is the set of reward functions for each agent; Π = { π i } K i =1 is the set of policy models for each agent; and Θ = { Θ i } K i =1 is the set of policy parameters for each agent's policy model. For simplicity, we use i to denote an agent from our description perspective and use -i to refer to all other agents.

We first summarize two major limitations of existing works to justify that they cannot deliver a sufficient formalization of Feint behaviors in Multi-Player Games. Since there is no prior formalization, we discuss relevant works and derive the key features to discuss them in detail.

➊ The basic formalization on temporal impacts is insufficient for Multi-Player Games. Multi-Player Games require agents to account for complex future planning for decision-making, which is critical for deceptive behaviors like Feint [22, 24, 26]. Several works simplify the temporal impacts of deceptive game strategies in different gaming scenarios. [22] uses a discount factor γ to calculate the reward for an agent i following actions as ∑ ∞ t =0 γ t R i ( s t , a i t , a -i t ) for agent i . However, such a method suffers from the "short-sight" issue [24], since the weights for future actions' rewards shrink exponentially with time, which are not suitable for all gaming situations (discussed in [26]). More recently, [14] applies a long-term average reward for an agent i , to equalize the rewards of all future actions as 1 T ∑ T t =0 R i ( s t , a i t , a -i t ) . However, such a method is restricted by the "far-sight" issue, since there are no differentiation between near-future and far-future planning. The mismatch between abstraction granularity heavily saddles with the design of Feint , because they use relatively static representations (e.g. static γ and T ). Therefore, they cannot be aware of any potential changes of strategies in different phases of a game. Hence, the temporal dimension is simplified hereby.

➋ The basic formalization of spatial impacts are generally in simplified 2-player scenarios only, which cannot be effectively generalized to Multi-Player Game scenarios. Prior works, which attempt to fuse Feint into complete game scenarios, only consider two-player scenarios [36, 31]. However, in Multi-Player (more then two player) Games, gaming strategies (especially deceptive strategies) yield spatial impacts on other agents. Such impacts have been overlooked by all prior works. This is because an agent, who launches the Feint behaviors, can impact not only the target agent but also other agents in the scenario. Therefore, the influences of such an action needs to account for spatial impacts [19]. Moreover, with a new dimension accounted, the interactions between them also raise a potential issue for their mutually collective impacts.

4.2 Our Formalization in a Generalized Game Model

Therefore, to deliver an effective formalization of Feint in Multi-Player Games, it is essential to consider the temporal, spatial and their collective impacts comprehensively. We first discuss the Temporal Dimension, then we elaborate our considerations on Spatial Dimension, and finally we summarize the design for the collective impacts from both temporal and spatial dimensions.

4.2.1 Temporal Dimension: Influence Time

To formalize the temporal impacts of Feint behaviors based on our Palindrome-directed Templates and the Dual-Behavior Model, we use a Dynamic Short-Long-Term manner to emulate them, which differ from the prior works' formalization (Section 4.1). The short-term period refers to a complete Dual-Behavior Model (Section 3.2), including a Feint behavior followed by an intended high-reward behavior led by the Feint. The long-term period is the time steps after this Feint cycle. The rationale behind such a design choice is that: the purpose of Feint is to obtain strategic advantages against the opponent in the temporal dimension, aiming to benefit the follow-up high-reward behavior. Hence, the Dynamic Short-Long-Term temporal impacts of Feint shall be (1) the actions that follow Feint beahviors (e.g. actual attacks) in a short-term period of time should have a strong correlation to Feint ; (2) the actions in the long-term periods explicitly or implicitly depend on the effect of the Feint and its following actions; and (3) for different Dual-Behavior models in different gaming scenarios, the threshold that divides short-term and long-term should be dynamically adjusted to enable sufficient flexibility in strategy making.

For Dynamic Short-Long-Term , we use the time-step length of a Dual-Behavior Model t s as the short-term planning threshold. The short-term (the Dual-Behavior) starts at time step t 0 with actions of a Feint behavior { a i t 0 , ..., a i t 0 + t f } and actions of a high-reward behavior { a i t 0 + t f +1 , ..., a i t 0 + t s } sampled from Feint policy π ′ ( t f denotes the Feint behavior length). We use two different set of scheduler weights on Feint behaviors α Feint = { α t 0 , ..., α t 0 + t f } and the following attack behavior α attack = { α t s f , ..., α t 0 + t s } . α Feint are small weights as regularizers since the Feint behaviors themselves do not intend to gain rewards, while α attack are large weights to emphasize the importance of the intended behaviors that Feint lead to. Thus, we formalize the short-time reward as

, since the purpose of Feint policy π ′ i is to actively find effective combinations of Feint behaviors and high-reward behaviors in Dual-Behavior Models that can benefit in a short-term period. We then consider long-term planning after the short-term planning threshold t s : we use a set of discount factor β = { β t 0 + t s +1 , ..., β T } on the long-term average reward calculation (proposed by [14]), to distinguish these reward from short-term rewards:

where T denotes the end time of the long-term planning threshold.

Finally, we put them together to formalize the Short-Long-Term reward calculation mechanism, when an agent i plans to perform a Feint action at time t 0 with a short-term planning threshold t s and the end time T as:

where λ short and λ long are weights for dynamically balancing the weight of short-term and long-term rewards for different gaming scenarios. λ short and λ long are initially set as 0 . 67 and 0 . 33 and are adjusted to achieve better performance with the iterations of training.

4.2.2 Spatial Dimension: Influence Range

The spatial advantage of Feint behaviors refers to deceiving the opponents (i.e., deviating the opponents' actions from their policies while exploring new possible advantage game states). In a Multi-Player Game (i.e. usually more than two players), the strict one-to-one relationship between two agents is not realistic, since an agent can impact both the target agent and other agents. Therefore, the influences on all other agents shall maintain different levels [19]. Therefore, our work includes the spatial dimension of Feint impacts by fusing spatial distributions (i.e., joint state-action space distribution). The key idea of this design is to model the influence range of Feint behaviors as the divergence of occupancy measures during policy learning. More specifically, we incorporate Behavioral Diversity from [19], to mathematically calculate and maximize the diversity gain of Feint behaviors on the influence range.

We follow the occupancy measure introduced in [19], the distribution of state-action pairs ρ π ( s, a ) = ρ π ( s ) π ( a | s ) , to measure a joint policy π = ( π i , π -i ) . ρ π ( s ) can be calculated by the normalizedweighted sum of game state visit probabilities for the joint policy π = ( π i , π -i ) of all agents. Game state s can be parameterized by a set of physical properties. We demonstrate a set of commonly used parameters in boxing games [35]: the relative positions p ( i, -i ) , relative moving orientations o ( i, -i ) , the linear velocities l _ vel ( i, -i ) , and angular velocities a _ vel ( i, -i ) . s can thus be composed in a vector s = ( p ( i, -1) , o ( i, -i ) , l _ vel ( i, -i ) , a _ vel ( i, -i )) . The spatial domain influence of Feint policy can be naturally represented by the occupancy measure. When players apply Feint behaviors to deceive opponents, the resulting spatial impact is to exploit new possibilities to achieve more advantageous state transitions from the current state. When a Feint policy π ′ i is added, we aim to maximize the effective influence range under the influence distribution of Feint . Specifically, we maximize the divergence of the new distribution of joint policy π ′ = ( π ′ i , π -i ) introduced by Feint from the distribution of the old one π = ( π i , π -i ) without Feint . By using Behavior Diversity [19], such a maximization problem at state s can be formalized as:

where we use the divergence as the reward Rew spatial and the general f -divergence is used to measure the discrepancy of two distributions. [19] introduces multiple ways to approximately calculate such divergence for policy models represented by neural networks.

4.3 Collective Impacts: Influence Degree

Solely relying on the Temporal Dimension and Spatial Dimension overlooks the interactions between them, and these two dimensions are expected to have mutual influences for realistic modeling [19]. Therefore, we consider the influence degree for the collective impacts.

We formulate it for a Feint policy π ′ i in Multi-Player Games which performs a full Feint cycle (i.e., a complete Dual-Behavior Model and long-term actions) that starts at t 0 and ends at T as:

where temporal impacts Rew temporal (Section 4.2.1) for agent i are are aggregated on spatial domain considering all agents and spatial impacts Rew spatial (Section 4.2.2) are aggregated on the temporal domain for all the states in the time period. µ 1 and µ 2 denote the weights of aggregated temporal impacts and spatial impacts respectively, enabling flexible adaption to different gaming scenarios. They are initially set as 0 . 5 .

In addition to the collective impacts of Feint itself in terms of temporal domain and spatial domain, our formalized impacts of Feint can also result in response diversity of opponents, since different related opponents (spatial domain) at different time steps (temporal domain) can have diverse response. Such Response Diversity can be used as a reward factor that makes the final reward calculation more comprehensive [25, 19]. Thus, to incorporate the Response Diversity together with our final reward calculation model, we refer to [19] to characterize the diversity gain incurred by our collective

impacts formalization. For an agent who maintains a pool of policy P i = { π 1 i , . . . , π i M } while opponents maintain a pool of policy P -i = { π 1 -i , . . . , π N -i } , an empirical payoff matrix A P i × P -i can be calculated element-wise using the reward function Rew collective ( π k i , π j -i ) for ( k, j ) entry. Thus the Response Diversity gain for a Feint policy π M +1 can be measured by follows:

where D ( a M +1 || A P i × P -i ) represents the diversity gain of the Feint action on current policy space. We follow the method in [19] for the quantification of diversity gain, which uses a practical and differential lower bound for feasible computation.

5 Proof-of-concept Implementation

To provide a unified implementation scheme of Feint into most MARL frameworks, we choose to implement on the training iteration level and avoid changing the MARL models themselves. We create an additional policy model (e.g., MADDPG [20], MASAC [11, 13], MAD3PG [4, 6], MATD3 [2], etc.) for each agent as the Feint policy, which works together with the regular policy models for agents but is trained and inferenced differently.

ImaginaryPlay

AccumulatedGame

FeintPolicyModel

Update

forDual-Behavior

Reward

Inference√

Model Period

Iterate

Actions of a Feint behavior

High-reward Actions

Long-term Actions

Action

个

Feint Reward

Feint

Evaluation

Reward

When

Accumulate

larger

Dual-Behavior

Compare

Model in

Progress

Regular Reward

Regular

IsDual-Behavior

Model in

Yes

Evaluation

↑

Reward

larger

Interact

with

Progress

Actions of the same time step length

Environment

Yes

个

No

Isfeasible

toInitiate

Inference

Dual-Behavior

Model Not in

Progress

When

Accumulate

Feint?

No

RegularPolicy

Action

Accumulated Game

Reward

Model

for Regular Training

本

Settings

Update

Figure 3 illustrates the full process of our implementation of Feint in game iterations. We implement the Feint behavior generation in an imaginary play module in training iterations (i.e., game steps). The imaginary play module decides whether an agent should initiate a Feint behavior, composes a Dual-Behavior Model using Palindrom-directed templates, and utilizes the Feint reward calculation to evaluate the quality of the generated action sequence in the Dual-Behavior model.

In our experimental settings, Feint behavior templates can be pre-computed only once before training, providing a fast lookup for composing Dual-Behavior models in gaming iterations. Algorithm 1 in Appendix E illustrates the pseudo-code for pre-computing available Feint behavior templates given a set of available attack behaviors B . For each pair of attack behaviors ( Behavior i , Behavior j ) in B , we check whether there are physically satisfiable states illustrated in Figure 1. If any of the three conditions are satisfied, we cut out the available actions ( Avail i and Avail j ) and store their compositions as an available template candidate. Note that the stored templates do not enforce the Feint behaviors to contain the exact same action sequences. Instead, tuples ( a k , Avail i , Avail j ) are served as keys for quick lookup and action restrictions in composing Dual-Behavior models during gaming iterations.

During gaming iterations, the imaginary play will only be activated when no Dual-Behavior Model is in progress and the current physical state s c of an agent is close to a physical state s r where it is physically realistic for the agent to perform a high-reward action a target , while the possibility of performing a target is relatively low according to its regular policy model (i.e., action a target are highly likely to be diminished by other agents current actions). Thus, the purpose of Feint behavior

is to lead the agent to a state s r where the agent can maximize the game environment reward by performing the intended high-reward action a target (i.e., other agents are deceived by Feint to perform other actions which cannot effectively diminish the high-reward actions performed by the agent).

When the imaginary play is activated, available Dual-Behavior models can be generated using the last known action a t and the intended high-reward action a target . Algorithm 2 in Appendix E shows the pseudo-code for composing available Dual-Behavior models with backward searches. The intended high-reward action a target provides constraints on the ending actions of the Feint behaviors, while the last known action a t provides constraints on the starting action of Feint behaviors. Thus, the available Dual-Behavior models can be quickly computed by doing a one-pass search from the pre-computed Feint behavior templates. The composed Dual-Behavior models can thus constrain the available action choices for the Feint policy model (set other actions' possibilities to 0) in corresponding templates and use a reflection frame to compose a (semi-)palindrome leading to the agent's physical state s r . After having available Dual-Behavior models which are composed of Feint behavior and followed by high-reward actions, the short-term reward can be calculated by Equation 1. After this Dual-Behavior action sequence, the imaginary play would play a few steps to incorporate the long-term reward (using Equation 1). The collective reward (Section 4.2.2) can thus be calculated. This reward is then compared to an accumulated reward from an imaginary play using only the agent's regular policy model in the same number of time steps. If the Feint collective reward is higher, the action sequence of the dual action model will be applied in the following real-game steps. When a Dual-Behavior Model is in progress, the actions will not be sampled from the regular policy models.

In the real game steps, where all the agents' actions interact with the environment and the real game rewards are calculated, our formalization of Feint only changes the way to update the Feint policy models for agents. The Feint policy models are updated only when corresponding Dual-Behavior Models finish and are updated using the accumulated real game rewards for that period. The regular policy models are updated as usual settings (e.g., after some fixed steps - an episode).

6 Experimental Studies

6.1 Methodology

Testbed Implementations. Due to the lack of a general benchmark, we selectively implement two scenarios under a customized manner. They consist of a "1 vs 1" and a "3 vs 3" multi-player free fight games, based on widely-provided testbeds. We provide additional details on how our testbeds are designed and implemented in appendix D.

Evaluation Metrics. Our main evaluation objective is the gaming rewards. We first examine the gaming outcomes when using the MADDPG, MASAC, MATD3, and MAD3PG MARL models, by comparing the per episode gaming rewards of agents across all scenarios 8 . We also evaluate other metrics, and report our results in appendix F.

6.2 Major Results

Figure 4 shows the game reward comparisons of using Feint behaviors or not in the Two-Player scenario (Section 6.1) for 4 MARL models. The first row shows the baseline results where all agents are trained normally, while the second row shows the results where the player labeled with "Good" incorporates Feint behaviors. In most of the baseline results (e.g., using MADDPG, MAD3PG, and MATD3), the two players' rewards tend to progress to a similar level when after enough training iterations. For MASAC, the "Good" player seems to gain higher rewards than its opponents when the training iterations are large, but the advantage is not stable and such a phenomenon can likely be the instability of the MASAC algorithm itself. For all the results where Feint behaviors are incorporated, we can see a significant advantage gain for the "Good" player. Thus, our formalization of incorporating Feint behaviors can effectively improve the actual game rewards in two-player combating scenarios.

To further evaluate the effectiveness of our formalization of Feint behaviors in multi-player scenarios, Figure 5 shows the game reward comparisons in Six-Player scenario (Section 6.1) for 4 MARL models. The first row shows the baseline results while the second row shows the results where the player labeled with "Good 3" incorporates Feint behaviors. In all baseline results, all 6 players seem to achieve similar levels of rewards after enough training iterations. In comparison, in all results where the "Good 3" player incorporates Feint , it gains significantly more rewards than the

MADDPG without Feint

MAD3PG without Feint

MASAC without Feint

MATD3 without Feint

200

400

400

200

200

MOH

200

200

200

400

400

200

009-

00000000900009000000000000000

Good 1

Adv 1

400

0

10000200003000040000500006000070000

Adv 1

Good 1

400

10000200003000040000500006000070000

Good 1

Adv 1

600

1000020000

3000040000500006000070000

Good 1

Adv 1

MADDPG with Feint

MAD3PG with Feint

MASAC with Feint

MATD3 with Feint

600

600

300

600

w/

400

00

200

00F

200

Game Rewards

200

100

s

200

0

OUE

M

200

200

100

200

400

1000020000

3000040000500006000070000

Adv 1

Good 1

400

0

1000020000

3000040000500006000070000

Good 1

200

0

1000020000

3000040000500006000070000

Good 1

400

1000020000

3000040000500006000070000

Adv 1

Good 1

opponents as well as its teammates. This result shows that our formalization of Feint can not only gain higher rewards towards the direct opponents, but also gain advantages among teammates who do not incorporate Feint . Another interesting observation is that there are no more symmetric patterns in the players' rewards, showing that the gaming interactions in multi-player scenarios have enough complexity (Note that the scenario is not designed to be a zero-sum game).

MADDPG without Feint

MAD3PG without Feint

MASAC without Feint

MATD3 without Feint

100

100

100

150

Rewards

-100

100

Rewards

200

Game Rewat

100

200

200

300

Adv 1

Adv 2

300

Adv 2

Adv 1

250

200

Adv 2

Adv 1

400

Good 1

Good2

Good 3

Adv 3

400

Good 3

Adv3

Good 1

Good 2

300

Good 3

Good 1

Good 2

-300

Good 3

Adv 3

Good 1

Good 2

2000040000

6000080000100000120000140000

000000001000004000000000900000000

350

2000040000

6000080000100000120000140000

000000001000004000000000900000000

MADDPG with Feint

MAD3PG with Feint

MASAC with Feint

MATD3 with Feint

400

100

400

200

2U0

100

200

Rerwa

200

200

300

-400

600

Good 2

Good 1

Good 3

Adv 3

-600

400

Good 2

Adv3

Good 3

Adv 2

Good 1

500

400

Good 3

Good 2

ood 1

400

Good 2

Good 1

0

2000040000

6000080000100000120000140000

2000040000

6000080000100000120000140000

2000040000

600008000010000012000140000

2000040000

6000080000100000120000140000

7 Conclusions and Main Implications

This work introduces the first comprehensive formalization, implementation and quantitative evaluations of Feint in Multi-Player Games. We provide automatic generation of Feint behaviors using Palindrome-directed Templates and synergistically combine Feint with follow-up actions in DualBehavior Model. The decision choices on the action-level are fused into strategy-level formalizations in game interactions. We provide a concrete implementation scheme to incorporate Feint into common MARL frameworks. The results show that our design of Feint can (1) greatly improve the reward gains from the game; (2) significantly improve the diversity of Multi-Player Games; and (3) only incur negligible overheads in terms of the time consumption. We conclude that our formalization of Feint is effective and practical, to make Multi-Player Games more interesting.

The successful formation of Feint behaviors and strategies imply the potential unsafety of existing machine learning models (and, clearly, future ones also). Therefore, the wide adoption of machine learning models certainly demand the consideration of this work (and its variants), for building responsible Artificial Intelligence; and/or leveraging them for the future society in a responsible way.

References

Johannes Ackermann, Codacy Badger, and Ted Jiang. Johannesack/tf2multiagentrl, November 2020.

Johannes Ackermann, Volker Gabler, Takayuki Osa, and Masashi Sugiyama. Reducing overestimation bias in multi-agent domains using double centralized critics. arXiv preprint arXiv:1910.01465 , 2019.

Kai Arulkumaran, Antoine Cully, and Julian Togelius. Alphastar: an evolutionary computation perspective. In Manuel López-Ibáñez, Anne Auger, and Thomas Stützle, editors, Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO 2019, Prague, Czech Republic, July 13-17, 2019 , pages 314-315. ACM, 2019.

Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy P. Lillicrap. Distributed distributional deterministic policy gradients. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018.

Lucas Critch and David Churchill. Sneak-attacks in starcraft using influence maps with heuristic search. In 2021 IEEE Conference on Games (CoG), Copenhagen, Denmark, August 17-20, 2021 , pages 1-8. IEEE, 2021.

Dongyu Fan, Haikuo Shen, and Lijing Dong. Multi-agent distributed deep deterministic policy gradient for partially observable tracking. In Actuators , volume 10, page 268. MDPI, 2021.

Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 , pages 2974-2982. AAAI Press, 2018.

Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 1582-1591. PMLR, 2018.

Iris Güldenpenning, Mustafa Alhaj Ahmad Alaboud, Wilfried Kunde, and Matthias Weigelt. The impact of global and local context information on the processing of deceptive actions in game sports. German Journal of Exercise and Sport Research , 48(3):366-375, 2018.

Iris Güldenpenning, Wilfried Kunde, and Matthias Weigelt. How to trick your opponent: A review article on deceptive actions in interactive sports. Frontiers in psychology , 8:917, 2017.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 1856-1865. PMLR, 2018.

Ray Hyman. The psychology of deception. Annual review of psychology , 40(1):133-154, 1989.

Shariq Iqbal and Fei Sha. Actor-attention-critic for multi-agent reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 2961-2970. PMLR, 2019.

Dong-Ki Kim, Matthew Riemer, Miao Liu, Jakob N. Foerster, Michael Everett, Chuangchuang Sun, Gerald Tesauro, and Jonathan P. How. Influencing long-term behavior in multiagent reinforcement learning. CoRR , abs/2203.03535, 2022.

Marc Lanctot, Vinícius Flores Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 4190-4203, 2017.

Jehee Lee and Kang Hoon Lee. Precomputing avatar behavior from human motion data. Graph. Model. , 68(2):158-174, 2006.

Seyoung Lee, Sunmin Lee, Yongwoo Lee, and Jehee Lee. Learning a family of motor skills from a single motion clip. ACM Trans. Graph. , 40(4):93:1-93:13, 2021.

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings , 2016.

Xiangyu Liu, Hangtian Jia, Ying Wen, Yaodong Yang, Yujing Hu, Yingfeng Chen, Changjie Fan, and Zhipeng Hu. Unifying behavioral and response diversity for open-ended learning in zero-sum games. CoRR , abs/2106.04958, 2021.

Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 6379-6390, 2017.

David Matz. Ancient Roman Sports, AZ: Athletes, Venues, Events and Terms . McFarland, 2019.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR , abs/1312.5602, 2013.

Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multiagent populations. arXiv preprint arXiv:1703.04908 , 2017.

Abhishek Naik, Roshan Shariff, Niko Yasui, and Richard S. Sutton. Discounted reinforcement learning is not an optimization problem. CoRR , abs/1910.02140, 2019.

Nicolas Perez Nieves, Yaodong Yang, Oliver Slumbers, David Henry Mguni, Ying Wen, and Jun Wang. Modelling behavioural diversity for learning in open-ended games. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 8514-8524. PMLR, 2021.

Chris Nota and Philip S. Thomas. Is the policy gradient a gradient? In Amal El Fallah Seghrouchni, Gita Sukthankar, Bo An, and Neil Yorke-Smith, editors, Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, AAMAS '20, Auckland, New Zealand, May 9-13, 2020 , pages 939-947. International Foundation for Autonomous Agents and Multiagent Systems, 2020.

Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. AMP: adversarial motion priors for stylized physics-based character control. ACM Trans. Graph. , 40(4):144:1144:20, 2021.

Sebastian Risi and Mike Preuss. Behind deepmind's alphastar ai that reached grandmaster level in starcraft ii. KI-Künstliche Intelligenz , 34(1):85-86, 2020.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017.

Hubert P. H. Shum, Taku Komura, and Shuntaro Yamazaki. Simulating interactions of avatars in high dimensional state space. In Eric Haines and Morgan McGuire, editors, Proceedings of the 2008 Symposium on Interactive 3D Graphics, SI3D 2008, February 15-17, 2008, Redwood City, CA, USA , pages 131-138. ACM, 2008.

Oswin So, Kyle Stachowicz, and Evangelos A. Theodorou. Multimodal maximum entropy dynamic games. CoRR , abs/2201.12925, 2022.

Stefano Corazza and Nazim Kareemi. Mixamo, 2022.

ToalhaNerd Team. Toalha nerd-king of fighter xv: Trailer de ryo sakazaki e robert garcia! 2021.

Kevin Wampler, Erik Andersen, Evan Herbst, Yongjoon Lee, and Zoran Popovic. Character animation in two-player adversarial games. ACM Trans. Graph. , 29(3):26:1-26:13, 2010.

Jungdam Won, Deepak Gopinath, and Jessica K. Hodgins. Control strategies for physically simulated characters performing two-player competitive sports. ACM Trans. Graph. , 40(4):146:1146:11, 2021.

Jungdam Won, Deepak Gopinath, and Jessica K. Hodgins. Control strategies for physically simulated characters performing two-player competitive sports. ACM Trans. Graph. , 40(4):146:1146:11, 2021.

Barbara Yersin, Jonathan Maïm, Julien Pettré, and Daniel Thalmann. Crowd patches: populating large-scale virtual environments for real-time applications. In Eric Haines, Morgan McGuire, Daniel G. Aliaga, Manuel M. Oliveira, and Stephen N. Spencer, editors, Proceedings of the 2009 Symposium on Interactive 3D Graphics, SI3D 2009, February 27 - March 1, 2009, Boston, Massachusetts, USA , pages 207-214. ACM, 2009.

A Conceptual Clarifications

Since this work spans multiple disciplines, there are a few clarifications to ensure the consistent understanding between our work and prior arts.

A.1 Differences between actions and behaviors in our formalization

To provide a unified definition of Feint behavior in both continuous and discrete action space, we highlight the difference between the terms action and behavior used in our formalization. We use action as the minimal unit movement in a unit time step, such as a unit step movement along the X and Y axis in a 2D board game, raising arms for a certain distance in a boxing game, turning steering wheels while applying brakes for a certain degree in a racing game, etc. This definition of action coincides with the commonly used definition of action in general MARL environments, which is intuitive to understand, simulate, and build our formalization of Feint upon it. One may argue that in some game simulations, combat movements like a cross punch are simply considered as one action, but one can always divide those movements into several unified unit-time-step actions to create a unified alignment in terms of time step in games. In terms of behavior , we refer to it as a combination of several actions in a sequence (e.g., a cross punch in boxing games). Thus, Feint can be naturally defined as a behavior that uses a sequence of actions to deceive opponents and lead to large reward actions in the near future. We describe our observation of Feint behaviors' characteristics and introduce our formalization at the action level in Section 3.

A.2 Modeling Behaviors at Action-Level in Game Animation and Simulation

Modeling characters' behaviors (series of actions) in games can be divided into two categories based on the main purpose: animation-driven modeling or simulation-driven modeling, though animation and simulation are inherently closely correlated. Animation-driven methods mainly focus on modeling the behaviors themselves, with goals of producing a variety of nuanced and coherent action sequences. The interactions with the environment (whether physics-based or not) are generally considered after the modeling of the behaviors and are generally simplified to showcase the behaviors themselves. Patch-based generation is a direct way for such methods, which directly compose behaviors by combining pre-defined action sequences [35]. This approach is widely adopted in the industry due to its high production efficiency, supported by an extensive amount of animation libraries (e.g. Mixamo [32]) [16, 30, 37]. However, in recent years, Learning-based generation dominates the field as they can automatically produce animated behaviors to mimic the styles of learned actions from the training inputs [17, 27]. On the other hand, simulation-driven modeling usually considers the full interactions with the environment in the first hand. These methods generally formalize the behavior modeling process using Reinforcement Learning (RL) based frameworks to fully explore the complicated space of physics-based action modeling [35]. In our work, we use a animation-driven modeling with strong physical constraints to describe our observations of Feint behavior characteristics and use the general simulation-driven modeling in MARL schemes for learnable formalization of Feint in action and strategy levels.

B Feint Behavior Generator and the Resulting Templates

Under the above three-stage decomposition of attack behaviors, there are abundant possibilities to compose Feint behaviors from the three action sequences. However, to ensure physically realistic generation, we summarize two requirements that Feint behaviors must follow: (1) Feint behaviors should follow semi-symmetrical patterns to effectively deceive opponents and return to a rest position for follow-up moves. In boxing, a human player must retract the stretched-out limbs to the relatively rest position, before stretching out to perform an actual attack action. This is because the retraction requires recharging the force to contracted muscles; and (2) transitions between adjacent actions in different behaviors are expected to be smooth, as humanoid body movements must provide continuous movements.

To satisfy the above two requirements, we propose a Feint behavior template generator called Palindrome-directed Generation of Feint Templates , by extracting subsets of semi-symmetrical actions from an attack behavior and synthesizing them as a Feint behavior. The general method to generate these templates are (1) by extracting subsets of unit actions from an attack behavior, a Feint behavior can be considered as a semi-finished real attack behavior. This ensures the high similarity of a generated Feint behavior with an attack behavior, thus opponents can be deceived; and (2) by synthesizing semi-symmetric action sections, the overall movements can be connected smoothly and the naturalness of humanoid actions can be guaranteed. Within our proposed template generator Palindrome-directed Generation of Feint Templates , there are two key adjustable parameters in practice: (1) sequence composition positions for Feint templates; and (2) sequence length for Feint templates. We provide the rationales for these two key design choices.

(1) Sequence composition positions for Feint templates: Determining which position to extract the subsets of action sequences needs to ensure that the extracted actions are semi-symmetrical and allow physically realistic connections. To this end, we can have three templates with different restrictions to exploit the composing patterns: (A) For template ➊ , if there are similar physical states, which refer to the positions of all joints and stretching angles are similar (as shown in ➊ of Figure 1), actions before the first similar state and after the second similar state can be extracted and directly synthesized as a Feint behavior (shown in ➊ of Figure 1); (B) For template ➋ , by cutting once at any time point in Sequence 1, action sequences before the selected point and the corresponding reversion can be synthesized as a Feint behavior (shown in ➋ of Figure 1); and (C) For template ➌ , similar to the second situation, by cutting once at any time point in sequence 3, action sequences after the selected point and the corresponding reversion can be synthesized as a Feint behavior (as shown in ➌ of Figure 1). With these considerations, the Feint behavior generation templates guarantee the naturalness of continuous movements via semi-symmetrical patterns.

(2) Sequence length for Feint templates: The choices for the length of extracted action sequences in each template can vary greatly, since multiple actions in an attack behavior can be extracted based on different time ranges. The available choices can be any time length that results in action sequences that satisfy the physical requirements discussed above (e.g. morphologically reasonable Template ➋ or Template ➌ in Figure 1). Note that it is also possible to construct nested Feint behaviors, given a large number of feasible extraction positions. We formalize this choice as a learnable parameter that needs to combine Feint behaviors with their intended follow-up actions (Section 3.2), and the learning adjustment is described in Section 5.

C Demonstration of Feint Behaviors

C.1 Demonstration of Feint Behaviors in Dual-Beahvior Models

To explain the generation of physically realistic Feint behavior in a Dual-Behavior Model in detail, we use humanoid models: when selecting the corresponding actions (i.e. from Feint behaviors and then an attack behavior), the starting position (jointly connected body) of the second action should be the same as the ending position of the starting action. With such a principle, the joints of a character's body can perform natural movements during the transition between these two behaviors. Figure 6 demonstrates a physically realistic combination of a Feint behavior and a follow-up attack behavior. When checking the end of NPC A's Feint behavior and the beginning of the Agent's (left white agent) real attack, both the upper and lower body parts of NPC A perform the same postures (the left arm raised and the right arm charged, performing a punch for the upper body, and the left foot forward for lower body).

Figure 6 provides a detailed example of a successful Feint behavior in a Dual-Behavior Model. We refer to the Agent as the white player on the left and its Opponent as the black player on the right, and describe the Feint behavior from the Agent perspective. The agent first performs a Feint behavior which is fake punch towards its opponent's head, which leads the opponent to defend towards its head. However, the agent connects such Feint behavior with a follow-up hook towards the opponent's waist. Due to the temporal advantage gained by the quick Feint behavior and the spatial advantage gained by deceiving the opponents to defend to wrong directions, the opponent would be knocked down by the follow-up behavior of the agent. Thus, a successful Feint behavior is performed in this Dual-Behavior Model.

Opponent'sresponsetotheFeint-afullfordefendforitshead

TemporalAdvantage

Agent-Followed-upattack(Hooktowards the oppoent'swaist)

Opponentabouttofinishheaddefending

Opponent won't able to accomplish a defend for its waist in time

C.2 Demonstration of Successful and Unsuccessful Feint Behaviors

To enable a successful Feint behavior in a Dual-Behavior Model, the temporal and spatial advantages should be properly formalized. The advantages of combining Feint behaviors with follow-up highreward actions stem from an appropriate time difference, incurred by Feint behaviors to mislead the opponents' actions. If the length of a Feint behavior is too short, the following attack actions might not gain much advantage compared to actions combinations without Feint behaviors; and if the length of a Feint action is too long, the process to perform a Feint behaviors can leave sufficient time for the opponent to react and even attack back. We provide examples for these scenarios in Figure 7, Figure 8, and Figure 9. We refer to the left white player as NPC A and describe the Feint from its perspective, and the right black agent NPC B is considered as its opponent.

We use the timeline of the Dual-Behavior Model in Figure 2 to analyze and evaluate the three Feint behaviors. We use three key time points that are highlighted in Figure 7, Figure 8, and Figure 9 to explain the action sequences, in which t B 1 indicates the end of defense behavior while t A 2 indicates the estimated start of reward in the second action sequence for NPC A and t B 2 indicates the estimated start of reward in second action for NPC B. The three consequences mainly differ in these three key time points.

1) Very short Feint behaviors t A 2 < t B 1 : The action sequence of simulation is shown in Figure 7, in which the Feint behaviorsduration is extremely short and the estimated start of reward in second action for NPC A ( t A 2 ) happens when NPC B is still in the first defense action (thus t A 2 < t B 1 ). As the sequence shows, the second real action of NPC A would not benefit much since NPC B is still in defense.

NPC A's first behavior - Feint (too short)

NPC A's second behavior -real attack

NPC B's first behavior - step back as defense

NPC A's second behavior - real attack (no effective reward since NPC B is still defending)

tA2 NPCB'sfirstbehavior(continue)-stepbackasdefense

tB1

Proper length Feint behaviors t B 1 < t A 2 < t B 2 : The action sequence of simulation is shown in Figure 8, in which the Feint behaviors have a moderate duration. The key difference of this duration is that the estimated start of reward in the second behavior for NPC A happens after the end of the defense behavior of NPC B and before the estimated start of reward in the second behavior for NPC B, thus showing the temporal advantages introduced in Section 3.2. With such temporal advantages, NPC A gains preemptive advantage over NPC B, inflicting rewards from NPC B (at time t A 2 in Figure 8) before NPC B's reward inflicting of second behavior starting (at time t B 2 in Figure 8). When NPC A hits NPC B at t A 2 , the ongoing action of NPC B will be interrupted and NPC B would be knocked down.

3) Very long Feint behaviors t A 2 > t B 2 : The action sequence of simulation is shown in Figure 9, in which the Feint actions duration is too long and the estimated start of reward in second behavior for NPC A ( t A 2 ) happens after the estimated start of damage in second action for NPC B ( t B 2 ). This condition has the opposite consequence of a moderate length Feint behaviors, in which NPC B can inflict rewards on NPC A before NPC A's reward inflicting of the second behavior starts. When NPC B hits NPC A at t A 2 , the ongoing action of NPC A will be interrupted and NPC A would be knocked down.

Thus, the choice of the time duration for Feint actions highly depends on the action combinations and the estimation of opponents' actions, proving our observation in Section 3. Thus the learning to formalize such a choice in the strategy learning scheme (Section 4) is important to construct effective Feint behaviors with corresponding Dual-Behavior Models.

NPC A's first behavior -Feint

NPC B's first behavior - stepback as defense

NPC A's second behavior - real attack

tB1

tA2

tB2

NPCB'sfirstbehavior(continue)

NPCB's second behavior-knocked downbyNPC A

NPC A's first behavior - Feint (too long)

NPCB'sfirstbehavior-stepbackasdefense

NPC A's first behavior (continue) - Feint NPC A's second behavior,- real attack (interrupted by NPC B)

tB2

tA2

NPCB'ssecondaction-realattack(effectivereward)

D Testbed Implementations

Our main testbed game environment is a multi-player boxing game, which is based on OpenAI's open-source environment Multi-Agent Particle Environment [23], but with heavy additional implementation to create a physically realistic scenario.This game resembles intense free fight scenarios in ancient Roman free fight scenarios [21], where interactions are intense and Feint is expected to be effective. We incorporate common boxing behaviors (action sequences) in boxing games. following the methodology in some animation and simulation works [34, 36]. This handcrafted scenario contains complex physics-based interaction systems and fine-grained time steps to enable learning and generating Feint behaviors. A detailed description of the reward gaining system, environment parameters, and agent settings is presented in Appendix D.1. We also modify and extend a strategic real-world game, AlphaStar [3], which is widely used as the experimental testbed in recent studies of Reinforcement Learning studies [28, 19]. We make extra efforts to emulate a six-player game, where players are free to have convoluted interactions with each other. And we implement Feint as dynamically generated policies, based on the 888 regular gaming policies.

D.1 Details of Boxing Game Scenario

Our testbed game scenario is emulates a complex boxing game by modeling all the detailed combat behaviors except building the graphical rendering process. The reason we neglect the rendering process is that our main goal is to evaluate the effectiveness of formalization of Feint behaviors in multi-player games, and the building a real-time graphical rendering with such complex humanoid interactions would be a graphics paper itself. We fully emulate all the behavior details in our game simulation, thus our constructed game simulation is detailed enough to evaluate our formalization of Feint behaviors. We provide a detailed description of the game scenario here.

We follow a similar boxing game scenario construction approach as [34, 35], and model the full set of Mixamo [32] 22 behaviors (action sequences) which contain over 250 available full body actions (illustrated in Figure 10). We extensively construct a gaming environment based on Multi-Agent Particle System [23] to incorporate these behaviors, which then can be seamlessly integrated with common MARL models.

Short Punch

MediumPunch

Long Punch

Short Hook

Medium Hook

Long Hook

QuickPunch

Cross Punch

Elbow Punch

OffensiveBehaviors

StepBackwardsStep Sidewards

6LongBlock

Short Block

Head Hit

Center Hit

Stomach Hit

DefenseBehaviors

3

ReactionBehaviors

Warm Up

Pivot Around

Idle

Walk

TransitionBehaviors

The players can move around in a 2D plane. We use a vector to model the physical state of players, which stores and tracks the body movements of a player. This vector tracks the positions of body parts: left and right limbs, the left and right legs, and the center body, which is used to select available combat behaviors (the transitions of body movements must be smooth as mentioned in Section 3.1 and Section 3.2). With this setting, Feint behaviors can be naturally generated and incorporated into suitable Dual-Behavior Models. We follow the exact Mixamo dataset to model the length of the behaviors (the length of action sequences) and rewards the behaviors (e.g., a successful long punch would gain more rewards than a short punch.) Specifically, we measure the number of frames contained in all behaviors and normalize them to define unit time steps for action space and thus get the action sequence lengths for all behaviors. An example of game rewards and action sequence length of 5 behaviors are provided in Figure 11.

D.2 Experimental Procedure

We choose 4 commonly used MARL models: MADDPG [20], MASAC [11, 13], MATD3 [2], and MAD3PG [4, 6] and incorporate them into testbed scenarios. Our implementation is based on [1], which provides a unified MARL frameworks for the above models. We aim to test whether Feint behaviors can be uniformly and effectively learned using all these commonly used MARL models and how can Feint affect the game rewards for agents. Note that our purpose is to verify the effectiveness of our formalization of Feint behaviors and not to compare or modify the MARL models themselves.

Example

Behaviors

Short Punch

Short Hook

MediumPunch

Long Punch

Cross Punch

GameReward

7

8

12

20

23

ActionSequence

5

6

6

13

15

Length

We create two test scenarios, the first one with two players (one player per team) and the second one with six players (3 players per team). For all of these scenarios, we first train the agents without Feint as baselines using the 4 models. Then for the two-player scenario, we incorporate Feint on one player (shown as the Good player in Figure 4). For the six-player scenario, we select 1 agent in the Good team (labeled as Good 3 in Figure 5), to incorporate our formalization of Feint , and keep all other 3 agents regular. The reason for this design in the six-player scenario is that we want to not only test how Feint behaviors can affect the reward gain against direct opponents, but also see whether Feint can bring advantages for a player among its teammates. All the players are rewarded independently and the notion of the "Good" and "Adv" team does not mean that teammates have a shared reward (i.e., not explicit constraints that force them to cooperate). Note that all players have identical capabilities and are rewarded using the same mechanisms, thus Feint can be incorporated on any player. Our labeling choice here is to provide to a consistent way to track and analysis the game rewards. All experiments for the two-player scenario are trained for 75,000 game iterations and all experiments for the six-player scenario are trained for 150,000 game iterations.

E Implementation Details

We provide the pseudo-code for computing Feint behavior templates and composing Dual-Behavior Models for our proof-of-concept implementation discussed in Section 5.

Algorithm 1 Precompute Available Feint Behavior Templates

̸

Algorithm 2 Backward Search to Compose Dual Behavior Models 1: Input: Last action a t , beginning of the high-rewards behavior a target , precomputed Feint templates T 2: Output: Set of available Dual Behavior Models D 3: Initialize empty set of Dual Behavior Models: D = ∅ 4: for each ( a k , Avail i = { a p | p ∈ [0 , k -1] } , Avail j = { a q | q ∈ [ k +1 , max ] } ) in T do 5: ▷ Here, max is the maximum index of action in Avail j 6: if ( a t ∈ Avail i ) and ( a target ∈ Avail j ) then 7: Select actions from a t to a k in Avail i : Select i = { a m | m ∈ [ t, k -1] } 8: Select actions from a k to a target in Avail j : Select j = { a n | n ∈ [ k +1 , max ] } 9: Update D : D ← D ∪ { ( Select i , a k , Select j ) } 10: end if 11: end for 12: return D

F Additional Experimental Results

We report the effects of Feint on ➊ diversity gain of policy space; and ➋ overhead of computation load. We examine the effects of Feint actions on how Feint can improve the diversity of gaming policies (Section 4.3). We also perform overhead analysis, incurred by fusing Feint formalization in strategy learning.

F.1 Diversity Gain

To examine the impacts on the policy diversity in AlphaStar games, we perform a comparative study between MARL training with and without Feint . Specifically, We use Exploitability and Population Efficacy (PE) to measure the diversity gain in the policy space. Exploitability [15] measures the distance of a joint policy chosen by the multiple agents to the Nash equilibrium, indicating the gains of players compared to their best response. The mathematical expression of Exploitability is expressed as:

where π i stands for the policy of agent i and π -i stands for the joint policy of other agents. Rew i denotes our formalized Reward Calculation Model (Section 4.3). Thus, small Exploitability values show that the joint policy is close to Nash Equilibrium, showing higher diversity. In addition, we also use Population Efficacy (PE) [19] to measure the diversity of the whole policy space. PE is a generalized opponent-free concept of Exploitability by looking for the optimal aggregation in the worst cases, which is expressed as:

where π i stands for the policy of agent i and π -i stands for the joint policy of other agents. α denotes an optimal aggregation where agents owning the population optimizes towards. Rew i denotes our formalized Reward Calculation Model (Section 4.3) and opponents can search over the entire policy space. PE gives a more generalized measurement of diversity gain from the whole policy space.

Figure 12 shows the experimental results for evaluating diversity gains. From the figure, we obtain two observations. First, agents that can dynamically perform Feint actions (Agent 1, 2, and 3) achieve lower Exploitability (around 4 . 9 × 10 -2 ) compared to agents who perform regular actions (around 9 . 7 × 10 -2 ) and have higher PE (lower negative PE - around 5 . 3 × 10 -2 ) than those who only perform regular actions (around 1 . 2 × 10 -2 ). This result shows that our formalized Feint can effectively increase the diversity and effectiveness of policy space. Second, agents with Feint have slightly higher variations in both metrics. This is because Feint naturally incurs more randomness (e.g. succeed or not) in games, resulting in higher variations in metrics.

Agent1

Agent1

一

Agent2

Agent2

10

Agent3

Agent4

Negative Population Effectivity

Agent3

Agent4

Agent5

Agent5

Agent6

Agent6

10

10

10

0

20

40

60

80

100

120

140

0

20

40

60

80

100

120

140

Gaming Iterations

Gaming Iterations

F.2 Overhead Analysis

Figure 13 shows the results of our overhead analysis. We make two observations. First, fusing Feint in MARL training do incur some overhead increment in terms of running time. This is because the

formalization and fusion of Feint in MARL incur additional calculation load. Secondly, in both MADDPGmodels and MAAC models, the increased overhead is generally lower than 5% , which still indicates that our proposed formalization of Feint actions can have enough feasibility and scalability on fusing with MARL models. Note that even we use two policy models for each agent in our implementation, our designs restrict that only one model is inferenced in each game step (Section 5), thus the overhead is low.

1 VS 1

3 VS 3

6.00%

Overhead increment of Feint in MADDPG

3.50%

Overhead increment of Feint in MAD3PG

4.25%

Overhead increment of Feint in MASAC

3.75%

Overhead increment of Feint in MATD3

%05's

5.00%

3.00%

3.75%

4.00%

3.50%

4.50%

1.00%

2.50%

2.00%

M

3.50%

3.25%

3.25%

3.00%

1.50%

3.00%

2.75%

3.00%

1.00%

2.75%

%09Z

2.50%

0.50%

2.50%

2.25%

2.00%

1000020000

3000040000500006000070000

1000020000

3000040000500006000070000

2.25%

1000020000

3000040000500006000070000

Trsining Episodes

10000200003000040000500006000070000

4.00%

Overhead increment of Feint in MADDPG

4.00%

Overhead increment of Feint in MAD3PG

4.00%

Overhead increment of Feint in MASAC

4.00%

Overhead increment of Feint in MATD3

L.50%

3.50%

3.50%

3.00%

3.00%

3.00%

3.00%

%00Z

2.50%

2.50%

2.00%

2.50%

2.00%

2.00%

2.50%

1.50%

2000040000

600008000010000012000014000

2000040000600008000100000120000140000

000000000000000000009000000000

1.50%

20000400006000080000100000120000140000

G Limitations

We want to point out that our current design aims to provide a generalizable and as-automatic-aspossible approach to concretely implement Feint behaviors into MARL models as a proof-of-concept. We acknowledge that there are possible ways to optimize this template selection stage in different gaming scenarios, for example, adding human-knowledge guidance or learning heuristics. However, we believe that the variations at this implementation level do not harm our main proof-of-concept contribution.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract accurately describes our main ideas, paper structures, and contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitation discussed in the Discussions.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the theorems, formulas, and proofs in the paper are well stated and referenced.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Implementation and experimental details are introduced in Section 5 and Section D.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Our main contribution is a unified formalization of Feint behaviors which could be widely adapted to most commonly used MARL frameworks. We have provided a detailed description of the codebase our implementation is based on in Section 6.1.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details are provided in Section D.1

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our experiment shows that Feint behaviors could significantly increase agents' reward and game diversity, by comparing the agents' rewards and game diversity to other agents' values in all our plots. These plots show clear differences which proves the effectiveness of our formalization and implementation. The exact reward values are not are main concern thus we don't include error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details are described in Section D.1

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We fully follow NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Well-cited in Section 6.1

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to generate feint behaviors and strategies so that the agent can obtain temporal and spatial advantages when competing with opponents. Specifically, this paper first describes the characteristics of feint behaviors at action-level and proposes a Feint behavior template generator called Palindrome-directed Generation that extracts subsets of semi-symmetrical actions from an offensive behavior and synthesizes them as a Feint behavior. Then, the paper proposes a dual-behavior model, which considers the physical constraint and effectiveness when constructing effective combinations of feint behaviors and follow-up actions. Moreover, the paper formalizes the feint implications at strategy-level and proposes rewards reflecting the temporal, spatial, and collective impacts. Finally, the paper provides an implementation scheme of feint behaviors that can be integrated into the existing MARL frameworks.\n\nSTRENGTHS:\n1. This paper provides the formalization of feint behaviors and the corresponding palindrome-directed feint behavior template generator. The analysis of feint behaviors and their combination with follow-up actions makes sense.\n\n2. The proposed feint behavior implementation scheme can be easily integrated into existing MARL frameworks.\n\n3. The paper is organized well and the presentation is easy to follow.\n\nWEAKNESSES:\n1. This paper claims that most offensive behaviors can be decomposed into three action sequences, which are Stretch-out Sequence (Sequence 1), Reward Sequence (Sequence 2), and Retract Sequence (Sequence 3) and it proposes a feint behavior template generator based on palindrome structure. This design should be effective for many feint behaviors composed at the action level. However, more complex feint behaviors belonging to the strategy level may not consist of the above three parts and thus the palindrome structure will not work. This paper may specify more clearly the scope of feint behaviors to which the proposed scheme can be applied.\n\n2. The discussion in the experiment section is relatively weak. For example, according to Appendix E, when applying the feint behavior implementation to MARL frameworks, the implementation trains a feint policy model to generate feint behaviors. The experiment results show that the feint behaviors can increase task returns. However, it is unclear whether the actions generated by the feint policy model are really feint behaviors. It is also possible that the feint policy model trained based on $Rew_{collective}$ manages to find certain good actions to beat opponents. This paper may have more discussion about this based on a clear definition of feint behaviors.\n\nQUESTIONS:\n1. When combining feint behaviors with follow-up actions, this paper proposes to first select an intended follow-up high-reward behavior, and the end physical state of the feint behavior is constrained to be close to the starting physical state of the follow-up behavior. How to implement this process for MARL framework and what is the time complexity when using the palindrome-directed feint behavior template?\n\n2. In section 4.2.1, the paper proposes that \"the Dynamic Short-Long-Term temporal impacts of Feint shall be (1)...(2)...(3)...\". How do the following design of $Rew_{temporal}$ achieve this?\n\n3. In Equation (1) and (2), should $\\alpha_t$ and $\\beta_t$ be inside $\\sum_t$?\n\nLIMITATIONS:\nPlease see the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a comprehensive formalization of feint behaviors in competitive multi-player games. The paper introduces a method for the automatic generation of feint behaviors using Palindrome-directed templates and combines them with high-reward actions in a Dual-Behavior Model. This formalization is incorporated into Multi-Agent Reinforcement Learning (MARL) frameworks. The authors conducted extensive evaluations using various MARL models in both two-player and six-player scenarios, demonstrating that their formalization significantly improves game rewards.\n\nSTRENGTHS:\n* The paper introduces a new formalization of feint behaviors, in which automatic generation of Feint behaviors via Palindrome-directed templates, and combine them with intended high-reward actions in a Dual-Behavior Model\n* The methodology addresses Feint implications on game strategies in terms of the temporal, spatial, and their collective impacts via the implementation in existing MARL frameworks.\n* In the experiments using various MARL models in both two-player and six-player scenarios, the authors demonstrate that their formalization significantly improves game rewards.\n\nWEAKNESSES:\n* The paper may lack clarity and consistency in the definitions and notations of key concepts, such as the reward functions in Section 4.2.1 and the Policy Occupancy Measure for model-free policies in Section 4.2.2. Additionally, the inconsistency in describing the payoff matrix dimensions in Section 4.2.3 creates confusion about the interaction between different agents' policies.\n\n* The paper also does not provide sufficient methodological details for calculating crucial measures, such as the Policy Occupancy Measure for model-free policies and the structure of the new policy occupancy measure.  \n\n* The evaluation results focus primarily on gaming rewards without a thorough analysis of the types and distributions of observed Feint behaviors. A more comprehensive assessment, including the underlying reasons for focusing on gaming rewards and the detailed impact of Feint behaviors, would enhance the understanding of the study's contributions.\n\nQUESTIONS:\n1. The current framework of Palindrome-directed Generation of Feint Templates may not capture certain Feint behaviors. For example, while linear movements can be reversed to return to the same position, planar movements often involve feints that result in a change of location, which may not be adequately represented by this method. Additionally, there may be other types of Feint behaviors that this framework cannot express. It would be beneficial to clarify these limitations and discuss any potential Feints that fall outside the scope of this approach.\n\n2. In Section 4.2.1, the paper introduces two reward functions, R^i, in Eqs (1) and (2). However, it is unclear how these reward functions are determined. If the R^i in both equations are the same, it seems this reward is simply proposing a weighted combination of short-term and long-term rewards. If they are not the same, the authors should use distinct notations to clarify the difference between the two. Additionally, the definition of R^i is not provided, making it difficult to understand the practical implementation of these reward functions. To improve clarity, the authors can provide concrete examples illustrating how R^i is defined and calculated in both the short-term and long-term contexts.  \n\n3. In Section 4.2.2, it is unclear how the Policy Occupancy Measure is calculated for model-free policies, such as those used in deep RL. The paper may not provide a detailed explanation of how to derive this measure without a model of the environment. \n\n4. (Section 4.2.3) In the discussion of the payoff matrix, it is stated as P_i ​ ×P_i, but then it is described as an M×N matrix. This seems inconsistent. Shouldn't it be P_i ​ ×P_j ​ instead, representing the policy spaces of different agents? Please clarify how M and N correspond to the policy spaces of agents i and j respectively, and ensure that the definition and usage of the payoff matrix align with the interaction between different agents' policies.\n\n5. (Section 4.2.3) To accurately calculate the divergence between the payoff matrix and the vector a_{M+1}, it is necessary that a_{M+1} also be represented as an M×N matrix. The paper should clarify how a_{M+1}, representing the new policy occupancy measure after introducing the Feint policy, is structured as an M×N matrix, and provide detailed steps on how these values are computed to maintain coherence with the payoff matrix.\n\n6. In Section 5.1, the authors refer to Appendix D for Testbed Implementations. It is crucial to specify whether these implementations are original or derived from prior research in the main text.\n\n7. In Section 5.2, it appears that only gaming rewards are evaluated. To provide a comprehensive assessment, it is essential to detail the underlying reasons for this focus. Additionally, the results should include a thorough analysis of the types and distributions of observed Feint behaviors. Such detailed results would offer a clearer understanding of how Feint behaviors impact the overall performance and diversity in gaming scenarios. \n\n8. The citation format used in the manuscript does not adhere to the NeurIPS guidelines, making it difficult to read. For instance, in lines 20-21, \"and explore strategies based on them Wampler et al. [2010], Won et al. [2021a]\" should be revised to either [1, 2] or (Wampler et al. 2010; Won et al. 2021a) for better readability and compliance with the NeurIPS format.\n\nLIMITATIONS:\nThe authors have not adequately addressed the limitations and potential negative societal impact of their work. While the paper presents a novel approach to generating and integrating Feint behaviors in MARL frameworks, it lacks a detailed discussion on the limitations of the proposed methods, such as the scope of applicability and potential challenges in real-world implementations. To improve, the authors should include a section discussing these aspects, providing a balanced view of the strengths and weaknesses of their approach, and consider potential negative outcomes and mitigation strategies.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a comprehensive approach to formalizing and implementing feint behaviors in multiplayer games. It introduces a new method for the automatic generation of feint behaviors using Palindrome-directed templates and combines these with high-reward actions in a Dual-Behavior Model. The paper further explores the implications of feint behaviors on game strategies, focusing on temporal and spatial impacts, and provides a unified implementation scheme. Experimental results demonstrate significant improvements in game reward gains, diversity, and minimal time overheads when incorporating feint behaviors.\n\nSTRENGTHS:\n* The introduction of Palindrome-directed templates for generating feint behaviors is a novel concept that adds value to the field of game strategy formalization.\n* The paper offers the first detailed formalization of feint behaviors at both the action and strategy levels, addressing a gap in existing literature.\n* The integration of feint behaviors into common MARL frameworks and the use of diverse experimental scenarios enhance the practical applicability of the proposed methods.\n* The implementation scheme is designed to be adaptable across various MARL models, which increases its versatility and potential for widespread adoption.\n\nWEAKNESSES:\n* While the paper evaluates several MARL models, it would benefit from a broader comparison with more diverse baseline strategies and models to establish the relative performance gains more comprehensively, such as [1-2]. \n* The paper does not sufficiently address potential scalability issues when applying the proposed methods to larger and more complex game environments beyond the tested scenarios.\n* The focus is primarily on multiplayer games, and the applicability of the proposed methods to other domains with different characteristics is not explored in depth.\n* It is recommended to apply appropriate smoothing to the curves to enhance the readability of the figures. This is particularly important for Fig 4, where almost all the curves without feint are nearly overlapping.\n\n[1] Yu, Chao, et al. \"The surprising effectiveness of ppo in cooperative multi-agent games.\" Advances in Neural Information Processing Systems 35 (2022): 24611-24624.\n[2] Yang, Tianpei, et al. \"ASN: action semantics network for multiagent reinforcement learning.\" Autonomous Agents and Multi-Agent Systems 37.2 (2023): 45.\n\nQUESTIONS:\n* How does the performance of the proposed feint behavior generation method compare with other state-of-the-art techniques in terms of computational efficiency and scalability?\n* Can the proposed methods be extended or adapted to non-gaming domains, such as robotics or real-world strategic planning? If so, what modifications would be necessary?\n* How does the complexity of the Palindrome-directed templates affect the learning curve and training time of the MARL models?\n\nLIMITATIONS:\nThe author mentioned \"Limitation discussed in the Discussions,\" but there is no section called \"Discussion\" in the manuscript. I also couldn't find any discussion about limitations in other parts of the document.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the first comprehensive formalization of Feint behaviors in multi-player games. The authors present a novel approach to automatically generate Feint behaviors using Palindrome-directed templates and combine them with intended high-reward actions in a Dual-Behavior Model. The formalization addresses both action-level and strategy-level aspects of Feint behaviors, considering temporal, spatial, and collective impacts.\n\nThe authors provide a unified implementation scheme to incorporate Feint behaviors into common Multi-Agent Reinforcement Learning (MARL) frameworks. They evaluate their approach using multiple MARL models in a custom boxing game scenario and a strategic real-game simulation.\n\nThe results demonstrate that incorporating Feint behaviors can significantly increase game rewards and improve the diversity of multi-player games. The authors show that their method outperforms existing approaches in several scenarios, with minimal computational overhead.\n\nWhile the work presents a novel and potentially impactful approach to modeling deceptive behaviors in games, it is limited by its focus on specific game scenarios and lack of theoretical analysis. Nevertheless, this paper contributes a valuable framework for enhancing the realism and strategic depth of AI agents in multi-player games.\n\nSTRENGTHS:\nThis paper demonstrates significant originality by providing the first comprehensive formalization of Feint behaviors in multi-player games. The authors address a notable gap in the literature, as previous works have only touched on Feint behaviors superficially or as proof-of-concept. The formalization at both action and strategy levels shows a depth of thinking that goes beyond existing approaches.\n\nThe quality of the work is evident in the thorough development of the formalization, from the Palindrome-directed templates for generating Feint behaviors to the Dual-Behavior Model for combining them with high-reward actions. The authors have clearly put considerable effort into creating a robust framework that can be applied across different game scenarios.\n\nIn terms of clarity, the paper is generally well-structured, guiding the reader through the complexities of Feint behavior formalization in a logical manner. The use of illustrative examples, particularly in the boxing game scenario, helps to ground the abstract concepts in concrete applications.\n\nThe significance of this work lies in its potential to enhance the realism and strategic depth of AI agents in multi-player games. By incorporating Feint behaviors, the authors have shown significant improvements in game rewards and diversity, which could lead to more engaging and challenging game AI. Furthermore, the unified implementation scheme for common MARL frameworks suggests broad applicability of this approach.\n\nWEAKNESSES:\nWhile the paper presents a novel approach, there are several areas where it could be strengthened:\n\n1. The primary experimental validation is conducted on a custom boxing game scenario. While this provides a good test case, it may not fully demonstrate the generalizability of the approach. The authors should consider including experiments from a wider range of game types (e.g., strategy games, team sports) to show the broad applicability of their formalization.\n\n2. While the paper provides a detailed formalization, it lacks rigorous theoretical analysis or proofs of the properties of the proposed approach. For instance, the authors could provide theoretical bounds on the performance improvements or convergence guarantees for their method.\n\n3. The paper primarily compares the performance of agents with and without Feint behaviors. However, it would be valuable to see comparisons against other existing methods for modeling deceptive or strategic behaviors in games.\n\n4. The formalization and implementation details are quite complex, which could make it challenging for others to replicate or build upon this work. The authors could consider providing a simplified version or pseudocode of key algorithms to improve accessibility.\n\n5. While the paper focuses on game simulations, it doesn't adequately address how this approach might be applied to or impact real-world game design or AI systems beyond simulations.\n\nQUESTIONS:\n1. How well does your formalization of Feint behaviors generalize to other types of games beyond the boxing scenario? Could you provide examples or preliminary results from applying your approach to significantly different game genres (e.g., strategy games, team sports)?\n\n2. Have you considered any theoretical analysis of your approach? For instance, can you provide any bounds on the expected performance improvements or convergence guarantees when incorporating Feint behaviors?\n\n3. Your paper primarily compares agents with and without Feint behaviors. Have you considered comparing your approach to other methods for modeling deceptive or strategic behaviors in games? If so, which methods and why were they not included in the current paper?\n\n4. The implementation of your approach seems quite complex. Could you provide a simplified version or pseudocode of the key algorithms to make it easier for others to understand and implement your method?\n\n5. How do you envision your formalization of Feint behaviors being applied in real-world game design or AI systems beyond simulations? Are there any specific applications or domains where you think this approach could have significant impact?\n\n6. Your Palindrome-directed templates for generating Feint behaviors are intriguing. Could you elaborate on how these templates were developed and whether there are any limitations to this approach for generating diverse Feint behaviors?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}