Feint Behaviors and Strategies: Formalization, Implementation and Evaluation
Authors: Junyu Liu
Xiangjun Peng
Venue: NeurIPS 2024 poster (2024)
Keywords: Feint Behaviors
Multi-Player Games
Multi-Agent Reinforcement Learning
Primary area: reinforcement_learning
TL;DR: The first comprehensive formalization of Feint behaviors at action-level and strategy-level, and provide concrete implementation and quantitative evaluation in Multi-Player games.

Abstract: Feint behaviors refer to a set of deceptive behaviors in a nuanced manner, which enable players to obtain temporal and spatial advantages over opponents in competitive games. Such behaviors are crucial tactics in most competitive multi-player games (e.g., boxing, fencing, basketball, motor racing, etc.). However, existing literature does not provide a comprehensive (and/or concrete) formalization for Feint behaviors, and their implications on game strategies. In this work, we introduce the first comprehensive formalization of Feint behaviors at both action-level and strategy-level, and provide concrete implementation and quantitative evaluation of them in multi-player games. The key idea of our work is to (1) allow automatic generation of Feint behaviors via Palindrome-directed templates, combine them into meaningful behavior sequences via a Dual-Behavior Model; (2) concertize the implications from our formalization of Feint on game strategies, in terms of temporal, spatial, and their collective impacts respectively; and (3) provide a unified implementation scheme of Feint behaviors in existing MARL frameworks. The experimental results show that our design of Feint behaviors can (1) greatly improve the game reward gains; (2) significantly improve the diversity of Multi-Player Games; and (3) only incur negligible overheads in terms of time consumption.

Feint Behaviors and Strategies: Formalization, Implementation and Evaluation
Junyu Liu Brown University jiw_junyubroomm.om edu
Xiangjun Peng The Chinese University of Hong Kong xjpeng0cse cuhk. edu.
Abstract
Feint behaviors refer to a set of deceptive behaviors in nuanced manner, which enable players to obtain temporal and spatial advantages over opponents in com- petitive games. Such behaviors are crucial tactics in most competitive multi- multi-player games (e.g., boxing, fencing, basketball, motor racing, etc.). However, existing lit- erature does not provide comprehensive (and/or concrete) formalization for Feint behaviors, and their implications on game strategies. this work, we introduce the first comprehensive formalization of Feint behaviors at both action-level and strategy-level. and provide concrete implementation and quantitative evaluation them in multi-player games. The key idea of our work is to (I) allow automatic generation of Feint behaviors via Palindrome-directte templates, combine them into meaningful behavior sequences via the implications from our formalization Feint on game strategies, in terms temporal, spatial and their collective impacts respectively; and (3) provide unified implementation scheme of Feint behaviors in existing MARL frameworks. The experimental results show that design of Feint behaviors can (1) greatly im- prove the game reward gains; (2) significantly improve the diversity of Multi-Player Games; and (3) only incur negligible overheads in terms of time consumption.
Model; (2) concertize
Introduction
In most real-world Multi-Player Games (e.g., boxing, basketball, motor racing, and etc.), players exhibit complex behaviors, which imply hard-to-be-quantffie interactions. Simulating these games requires to model the players' behaviors into action spaces as the elements (i.e., which we denote such a decision as "action-level"), and explore strategies based on these elements [34] 35 As one of the most common behaviors in real-world games, Feint behaviors represent class of tactic behaviors, which are used to mislead opponents to obtain temporally-stratee generally exhibit nuanced differences with normal behaviors, in terms of action movements] but these behaviors can help the agent to obtain strategic advantages considerable exten However, no existing literature provides a comprehensive formalization (and/or concrete modeling of) Feint behaviors, at either action level or strategy level, and we justify this claim below.
advantages" Such behaviors
To the best of our knowledge, concept to construct animations for nuanced behaviors two-player boxing games, based the unpredictability introduced in stochastic and simultaneous decision making.
34 is the first work to mention Feint behaviors, as proof-of-
A more recent work learn control strategies to animate nuanced agent behaviors like Feint in gaming environments, from motion clips using Deep Reinforcement Learning.
Note that the resulting advantages may exhibited temporally and/or spatially. Examples include fake punch in boxing, early-braking in motor racing, and etc.
And, clearly, Feint behaviors can also increase the diversity those games, according to
26
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Summary. Prior attempts do not provide detailed formalization to address the action-level char- acteristic of Feint and do not give Feint behavior generation guidelines. As for the strategy -lvell formalization, existing learning-based works either neglect Feint behaviors or assume that Feint are just glitches from other behaviors, by inducing similar impacts implicitly
Overview, Our work provides the first comprehensive and concrete formalization of Feint behaviors in action-level and stralegy-level. We first present an automatic approach to generate Feint behaviors using Palindrome-directee Templates based on our observation on Feint characteristics, and provide Dual-Behavior Model to examine the design for the combination of Feint behaviors and normal actions. Based on the action-level formalization, we then model the Feint behavior impacts on strategy-level in terms of the temporal, spatial, and their collective impacts under a learnable scheme. Then, we provide a concrete and unified implementation to incorporate the action-level and strategy- level formalization in common Multi-Plan Platirr Reinforcement Learning (MARL) frameworks; so that we can showcase the effectiveness of our formalizatior
Results. To properly examine the effectiveness of our formalization, we extensively construct complex and sessicssbased boxing game as abstraction of some animation-related works [34 35 We use a two-player and six-player scenario with 4 commonly used MARL models (MADDPG 20
to extensively evaluate our formalization. We also evaluate our formalization of Feint in strategic real-game, Alpha Star, to examine the diversity gain introduced by our formalization. The results show that our formalization of Feint can significantly increase the gaming rewards in all scenarios with all 4 MARL models. We also extensively examine our approaches: for the Diversity Gain, our method can increase the exploitation of the search space by 1.98X, measured by the Exploitability metries; and our implementation scheme only incur less than 5% overheads in terms of per game episode time consumption. We conclude that our formalization of Feint behaviors is effective and practical, significantly increasing players' game rewards and making Multi-Player Games more interesting.
MASAC 13. MATD3 [2]. and MAD3PG 4 6
2 Background
2.1
Feint Behaviors in the Real- World and Simulated Games
Feint behaviors are common for human players, as a set of active actions to obtain strategic advantages in real-world games. Examples can include sports games such boxing, basketball, and motor racing
[10 9 [2]. and electronic games such as King of Fighters and Starcraft [3 5]. Feint behaviors are not simple deceptive behaviors as their goal is not to gain rewards for themselves but create temporal and spatial advantages for some short-term follow-up actions. In addition, Feint behaviors have nuanced action formalizations. Though Feint is undoubtedly important in many real-world games, there still lacks a comprehensive formalization of Feint in Multi-Player Game simulations using Mon-Player Characters (NPCs). There are only limited amount of works tackle this issue. 34] is an early example of incorporating Feint as a animations for nuanced game strategies for more unpredictability from NPCs. More recently, (36) learn multi-level control strategies of agents via deep reinforcement learning from set of motion clips including nuanced behaviors like Feint (i.e. in animating combat scenes). However, these prior works (1) lack concrete formalizations of Feint behavior characteristics, which cannot fully unveil the variety of Feint behaviors the action level; and (2) lack comprehensive explorations of Feint behaviors implications on game strategies, which neglects the potential impacts of fusing effective Feint behaviors into strategies; and (3) solely focus on Two-Player Games, which can not be effectively generalized to multi-player scenarios.
which focuses on constructing
2.2 MARL Models at Strategy-Level in Multi- Maatii Game Simulations
Multi- Aalti-Ageettii Reinforcement Learning (MARL) aims to learn optimal policies for agents in multi- agent environment, which consists of various agent-agent and agent-environnee single-agent Reinforcement I Learning methods (e.g. DDPG [8]. SAC [T], PPO (29 and TD3 (8) D4PG 4 can not be directly used in multi-agent scenarios, since the rapidly-changing multi-agent environment can cause highly unstable learning results (evidenced by 20 ). Thus, recent efforts
interactions Many
*As shown in this work, existing learning- baaedd based approaches can not effectively model Feint behaviors in strategy-Ieved -evel, since Feint behaviors require intricate planning which is an active process.
'To deliver unitied definition of Feint behavior in both continuous and discrete action space, highlight the difference in appendix
'Note that these efforts can establish Feint upon prior arts (as covered in appendix A.2 and we have justified the novelty of our approach in appendix A.2
2

---

on MARL model designs aim to address such an issue. 77 proposes Counterfactual Multi-Agent (COMA) policy gradients, which uses centralised critic to estimate the Q-function and decentralised actors to optimize agents' policies. [20] proposes Multi-Agent Deep Deterministic Policy Gradient (MADDPG), which decreases the variance in policy gradient and instability of Q-function of DDPG in multi-agent scenarios. [13] proposes Multi-Agent Actor-attention Critic (MAAC). which applies attention entropy mechanism to enable effective and scalable policy learning. These models can have varied impacts within a diverse set of scenarios. [6] introduces Multi-agent Distributed Deep Deterministic Policy Gradient (MAD3PG), which extends the D4PG to multi-agent scenarios with distributed crities to enable distributed tracking. [2] proposes Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3), which integrates twin delayed Q-learning and addressing the overestimation bias in Q-values in multi-agent setting. Though different MARL models have different design details, they all share the same high-level learning structure. Thus, our goal is to provide unified scheme fuse our formalization of Feint behaviors into game simulations that can be learned using common MARL models, enabling effective Feint behaviors impacts regardless of specific design choices of MARL models.
3 Formalizing Feint Behavior
We introduce our formalization of Feint behaviors in action level regarding (1) how to automatically generate Feint behavior with templates from common attack behaviors; and (2) how can the generated Feint behaviors be synergistically combined with follow-up high-reward actions. We first introduce our methodology to automatically generate Feint behaviors, by exploiting our newly-revealed insight called Palindrome-directed Generation of Feint Templates. Next, we illustrate key design choices on how to combine the generated Feint behaviors with follow-up actions in a Double-Behavee Model, which forms the foundation for the designs of Feint-accounted strategy designs in Section
3.1 Feint Behavior Characteristics and Templates
Since Feint behaviors aim to provide deceptive attacks, they are naturally expected to be derived from subset of existing attack behaviors. Based on our exploration, we derive two key findings from an extensive amount of attack behaviors. First, most attack behaviors can be decomposed into three action sequences, which are Stretch-out Sequence (Sequence 1), Reward Sequence (Sequence 2), and Retract Sequence (Sequence 3) (an example shown in the first row in Figure D). We elaborate on each action sequence in detail.
Sequence 1 leads the agent's movements to the Reward Sequence (in boxing, approaching the opponents before actually hitting them); Sequence 2 contains actions that gain game rewards (in boxing, physical contact with the opponents); and Sequence 3 retracts an agent's movements to a relative rest position (in boxing, retracting back to preparation position for next behaviors). Second, body movements in Sequence and Sequence 3 usually have semi-symmetric yet reverse-order action patterns in the timeline. A behavior usually starts and ends in a similar physical state due to physical restrictions (e.g., bones and muscles stretching restrictions for a humanoid).
The above three-stage decomposition of attack behaviors has motivated series of constraints, to deliver proper design of Feint generators. To satisfy the above two requirements, we propose a Feint behavior template generator called Palindrome-directt extracting subsets of symmttymmttiiaal actions from an attack behavior and synthesizing them as a Feint behavior. The general method to generate these templates are (1) by extracting subsets of unit actions from an attack behavior, a Feint behavior can be considered as a semi- comi-finsshed real attack behavior. This ensures the high similarity of generated Feint behavior with an attack behavior, thus opponents can be deceived; and (2) by synthesizing semi-symmetric action sections, the overall movements can be connected smoothly and the naturalness of humanoid actions can be guaranteed. Within our proposed template generator Palindrome-directed Generation of Feint Templates, there two key adjustable parameters in practice: (I) sequence composition positions for Feint templates; and (2) sequence length for Feint templates. We describe our rationale in appendix B Figure| | demonstrates 3 templates for generating Feint behavior templates in boxing games.
Generation of Feint Templates, by
3.2 Feint Behavior in Consecutive Game Steps
Standalone Feint behaviors are meaningless in competitive games since the Feint behaviors themselves do not gain rewards. Only by effectively combining Feint behaviors with intended follow-up actions
We choose boxing game as an example to concretely explain our insights for Feint behaviors in this section but our formalization is unified abstraction of common games and can be easily adapted to other games including basketball, fencing, motor racing, etc.
3

---

Sequence Strech-Ou Out
Sequence
Sequence
Simlar physical state
Actiona before the aimilrr state in Sequeence
Actions ober the simber state in Sequuince
From Sequnece 1 Sequnece 3:
Action extracted
reverse
Action extracted
From Sequence
Sequence Reverse
From Sequence 3 Sequence Reverse
Figure I:
An example of Palindrome-directed Generation Templates of Feint behaviors. The first row shows an action sequence of a cross-punch behavior. Three examples of templates are shown as @, and @ to demonstrate physically realistic generation of Feint behaviors.
can showcase their effectiveness. Thus, we define an effective Feint cycle as Dual-Behavior Model, which jointly considers a Feint behavior and intended follow-up behavior (can be single action or an action sequence). Our formalization for standalone Feint behaviors (Section 3.1) already provides a large number of possible Feint behaviors. However, not all these morphologically reasonable Feint actions can be directly combined with all high-reward follow-up actions in combating scenarios. Therefore, certain constraints are demanded to construct effective combinations of Feint behaviors and follow-up actions. Hereby, introduce two major considerations and then propose relevant restrictions, to enable naturalistic and suitable combinations of Feint behaviors and follow-up actions.
A chosen Attack Behavior to generate
intended Attack Behavior
Feint
Backward Search
Stretch out
Damage
Retract
Stretch out
Damage
Retract
Ganarata Faint Behavior using templates
Agent
Stretch out
Retract
Stretch out
Reward
Retract
Agent Foint Behavlor
Agant Follow-up Offensiive Bahevior
Opponent
Defense Behavior towards Feint
Stretch
Not able finish
Opponent delend towards Feint
Oppornnt not
Temporal Advantage
Spatial Advantage
Suffer from Agent's Reward
Timeline
Figure 2: Dual-action Model
high-level abstraction and demonstration of internal stage transitions
Physical constraints need be accounted for when synthesizing Feint behaviors and follow-up actions. The ending physical state for Feint behavior must be state that is physically possible for an agent to perform the follow-up high-reward actions. For example, an agent in boxing games finishes Feint behavior with the left foot forward, but the following attack behavior starts with the right foot forwarded, the synthesis of these two behavior is inappropriate since this combination is physically unrealistic.
(1) Physical Constraints:

---

To ensure that the combinations of Feint behaviors and follow-up actions obey the physical constraints, we use a Reverse Search Principle which decides the intended follow-up actions (behavior) first and then use the starting physical state of this behavior to search and compose proper Feint behaviors (a more detailed description combined with strategy is described in Section [5) By first selecting an intended follow-up high-reward behavior, the end physical state of the Feint behavior is constrained to be close to the starting physical state of the follow-up behavior. Thus the composition of possible Feint behaviors using the Palindrome-ddrretter templates should aim to start and end at a physical state that is close to the follow-up behavior.
(2) Effectiveness:: The effectiveness of the incorporation of Feint behaviors is evaluated by whether the following attack actions can successfully gain rewards from the opponent. A successful Feint behavior would usually enable an agent to gain temporal and spatial advantages when performing the follow-up behaviors. Thus, the two design parameters introduced in Section 3.2 play crucial roles in combining Feint with follow-up behaviors. The abstraction of an ideal enable an agent with temporal and spatial advantage is illustrated in Figure 22 [annd a corresponding example is provided in Figure 6 An effective Feint behavior creates temporal advantages that make the opponents to defend in a wrong direction and enable temporal advantages to allow the follow-up high-reward behavior to successfully gain rewards on the opponents.
model that can
To ensure the consistency and correctness of the understanding, we provide a detailed demonstration for successful and unsuccessful Feint cases in Appendix C
4 Formalizing Feint Behaviors in Strategies
To effectively fuse the Feint beahviors with Dual-Behavior Model into game interactions, we provide the strategy-level formalization of Feint behaviors. We use Multi-Agent Reinforcement Learning (MARL) schemes to discuss our formalization of Feint behaviors in the strategy level, as MARL provides flexibility in exposing multiple adjustable parameters in learnable policy models. As discussed in generating Feint behaviors (Section 3.1 and composing them in the Dual- Behavie Models (Section (.2).. the key considerations for effective Feint cycle is to enable temporal and spatial advantages for agent. Thus, our strategy-level formalization centers on how address the temporal, spatial, and their collective impacts of Feint behaviors with Dual-Behavior Models. A more concrete introduction for fusing of Feint into the MARL frameworks is presented in Section 5
4.1 The Basic Formalization: Derivation and Limitations
Under commonly used MARL schemes (25 19], we define K Kaagent Non-trunsitive Active Markov K Game Model as a tuple (K, S, AA, P. R. II, θ): S is the state space; A = {-A,) is the set of action space for each agent, where there are dominant actions; P performs state transitions of current
state by agents' actions: P S x A1 x Ag x X Ak + P(S), where P(S) denotes the set of probability distribution over state space S: R {R.)
K is the set of reward functions for each agent; 11 = ] is the set of policy models for each agent; and θ {O: is the set of policy parameters for each agent's 's policy model. For simplicity, we use to denote an agent from our description perspective and use -i to refer to all other agents.
We first summarize two major limitations of existing works to justify that they cannot deliver a suffi- cient formalization of Feint behaviors in Multi- Games. Since there is no prior formalization, we discuss relevant works and derive the key features to discuss them in detail.
The basic formalization on temporal impacts is insufficient for Multi-Player Games. Multi-Player Games require agents account for complex future planning for decision-making. which is critical for deceptive behaviors like Feint [22 24 26). Several works simplify the temporal impacts of deceptive game strategies in different gaming scenarios. [22) uses discount factor 7 to calculate
the reward for an agent i following actions as E > R (st, ai a, ") for agent i. However, such method suffers from the "short-sight" issue (24), since the weights for future actions' rewards shrink exponentially with time, which are not suitable for all gaming situations (discussed in (26)). More recently, 14] applies a long-term average reward for an agent i, to equalize the rewards of all future actions as T
1 Ei= R'(st, of. a) However, such a method is restricted by the "far-sight" issue, since there are no differentiation between near-future and far-future planning. The mismatch between abstraction granularity heavily saddles with the design of Feint, because they use relatively static representations (e.g. static 7 and T). Therefore, they cannot be aware of any potential changes of strategies in different phases of a game. Hence, the temporal dimension is simplified hereby.
5

---

The basic formalization of spatial impacts are generally in simplified 2- 2-player scenarios only, which cannot be effectively generalized to Multi-Player Game scenarios. Prior works, which attempt to fuse Feint into complete game scenarios, only consider two-player scenarios [36 31 However, in Multi-Player (more then two player) Games, gaming strategies (especially deceptive strategies) yield spatial impacts on other agents. Such impacts have been overlooked by all prior works. This is because an agent, who launches the Feint behaviors, can impact not only the target agent but also other agents in the scenario. Therefore, the influences of such an action needs to account for spatial impacts [9]. Moreover, with a new dimension accounted, the interactions between them also raise a potential issue for their mutually collective impacts.
4.2 Our Formalization in a Generalized Game Model
Therefore, deliver effective formalization of Feint in Multi-Player Games, it is essential to consider the temporal, spatial and their collective impacts comprehensively. We first discuss the Temporal Dimension, then we elaborate our considerations on Spatial Dimension, and finally we summarize the design for the collective impacts from both temporal and spatial dimensions.
4.2.1 Temporal Dimension:
Influence Time
To formalize the temporal impacts of Feint behaviors based on our celindrome-directer Templates and the hhal-Behavior Model, we use a Dynamic Short-Long- Long- Term manner to emulate them, which differ from the prior works' formalization (Section 4. p The short-term period refers to a complete Dual-Behavior Model (Section 3.2). including a Feint behavior followed by an intended high-reward behavior led by the Feint. The long-term period is the time steps after this Feint cycle. The rationale behind such a design choice is that: the purpose of Feint is to obtain strategic advantages against the opponent in the temporal dimension, aiming to benefit the follow-up high-reward behavior. Hence, the Dynamic Short-Long Term temporal impacts of Feint shall be (1) the actions that follow Feint beahviors (e.g. actual attacks) in a short-term period of time should have a strong correlation to Feint (2) the actions in the long-term periods explicitly or implicitly depend on the effect of the Feint and its following actions; and (3) for different -eal-Behavior models in different gaming scenarios, the threshold that divides short-term and long- term-termm should be dynamically adjusted to enable sufficient flexibility in strategy making.
For Dynamic Short-Long Term, we use the time-step length of a Dual-Behavior Model L, as the short-term planning threshold. The short-term (the of a Feint behavior [at..-.,,(iytt } and actions of a high-reward behavior { (1igtty+1--- +i,+ ..,, sampled from Feint policy n' (lf denotes the Feint behavior length). We use two different set of scheduler weights on Feint behaviors a'Feint = :att...... (to-t, } and the following attack behavior Qattack {at,f.... f........ atott..: ). a'Feint are small weights as regularizers since the Feint behaviors themselves do not intend to gain rewards, while Oattack are large weights to emphasize the importance
starts at time step ln with actions
of the intended behaviors that Feint lead to. Thus, we formalize the short-time reward as
tmtatty Σ
t=to+
Rewshort (Te to tg, to a)
o'Fentt R² os
Gattack, RR (st, al, (1)
tmt,
since the purpose of Feint policy oद is to actively find effective combinations of Feint behaviors and high-reward behaviors in Dual-Behavior Models that can benefit in a short-term period. We then consider long-term planning after the short-term planning threshold f,: we use a set discount factor / = {Beutt+t1,- .... Br } on the long- term average reward calculation (proposed by [14), to distinguish these reward from short-term rewards:
T Σ t=t++t+1
Rewlong (T to T,
B, a
(2)
where T' denotes the end time of the long-term planning threshold.
Finally, we put them together to formalize the Short-Long-Term reward calculation mechanism, when an agent plans to perform a Feint action at time to with a short-term planning threshold t, and the end time T as:
ReWtemporal (T,, lo, t L, T,,,3) Ashort Rews (T to 1 L, a))
+ Aoong Rewiong ( to t,p T. T)))
(3)

---

where Ashort and Atong are weights for dynamically balancing the weight of short-term and long- ann-tterm rewards for different gaming scenarios. Ashort and Atong are initially set as 0.67 and 0.33 and are adjusted to achieve better performance with the iterations of training.
4.2.2 Spatial Dimension: Influence Range
The spatial advantage of Feint behaviors refers to deceiving the opponents (i.e., deviating the opponents" actions from their policies while exploring new possible advantage game states). In a Multi-Player Game (i.e. usually more than two players), the strict one-to-one relationship between two agents is not realistic, since an agent can impact both the target agent and other agents. Therefore, the influences on all other agents shall maintain different levels [1]]. Therefore, our work includes the spatial dimension of Feint impacts by fusing spatial distributions (i.e., joint state-action space distribution). The key idea of this design is to model the influence range of Feint behaviors as the divergence of oceupancy measures during policy learning, More specifically, we incorporate Behavioral Diversity from [19] mathematically calculate and maximize the diversity gain of Feint behaviors on the influence range.
We follow the occupancy measure introduced in [9], the distribution of state-action pairs Pr (8, a) P+ (T s), to measure joint policy π = ( T-i)- ) Pre (s) can be calculated by the normalized- weighted sum of game state visit probabilities for the joint policy π (T,, - of all agents. Game state 8 can be parameterized by a set of physical properties. We demonstrate a set of commonly used parameters in boxing games [55: the relative positions -i), relative moving orientations o(i, -i), the linear velocities / vel (i, -), and angular velocities a vel(i, -i). s can thus be composed in vector (p(i. - 1), oti, -)), vel(i, -i), a_vel(v -)). The spatial domain influence of Feint policy can be naturally represented by the occupancy measure. When players apply Feint behaviors to deceive opponents, the resulting spatial impact is to exploit new possibilities advantageous state transitions from the current state. When Feint policy πद is added, we aim to maximize the effective influence range under the influence distribution of Feint Specifically, we maximize the divergence of the new distribution of joint policy π (T/- ==;) introduced by Feint from the distribution of the old one π = (m, π-;) without Feint By using Behavior Diversity [19] such a maximization problem at state can be formalized as:
achieve more
maz,; Rewapatial (T s) = (p(, (s) (s).
where we use the divergence as the reward ReWspatial and the general f-divergence is used to measure the discrepancy of two distributions. [19 introduces multiple ways to approximately calculate such
divergence for policy models represented by neural networks.
4.3 Collective Impacts: Influence Degree
Solely relying the Temporal Dimension and Spatial Dimension overlooks the interactions between them, and these two dimensions are expected to have mutual influences for realistic modeling [19 Therefore, consider the influence degree for the collective impacts.
We formulate for a Feint policy n/ in Multi- Multi-Player Games which performs a full Feint cycle (i.e., a complete
Model and long-term actions) that starts at to and ends at T as:
ReW ( (T n-.) M1
Σ
t,: ReWtesppoal (T, to, t,, T, 3)
ST
Σ
+ 122
Rewssptial (Te -= 8)
where temporal impacts ReWtemporal (Section 4.2.1) for agent i are are aggregated on spatial domain considering all agents and spatial impacts Rewspatial (Section 4.2.2) are aggregated on the temporal domain for all the states in the time period. μ4 and #2 denote the weights of aggregated temporal impacts and spatial impacts respectively, enabling flexible adaption to different gaming scenarios. They are initially set as 0.5.
In addition to the collective impacts of Feint itself in terms of temporal domain and spatial domain, our formalized impacts of Feint can also result in response diversity of opponents, since different related opponents (spatial domain) at different time steps (temporal domain) can have diverse response. Such Response Diversity can be used as a reward factor that makes the final reward calculation more comprehensive 25 19.. Thus, to incorporate the Response Diversity together with our final reward calculation model, we refer to [9] to characterize the diversity gain incurred by our collective
7

---

impacts formalization. For an agent who maintains pool of policy P; = ........ T. while opponents maintain a pool of policy P-,= -=-...., an empirical payoff matrix x can be calculated element-wise using the reward function Reweallective ( T²,) for (k, entry. Thus the Response Diversity gain for Feint policy MM+ can be measured by follows:
HoW (T M D(aM+1 || Ap.xP..)
(6)
M+1 = (ReWeallectiee (T:, x-,
M+1
where D(aM+1 |I Ap represents the diversity gain of the Feint action on current policy space.
We follow the method in [19] for the quantification of diversity gain, which uses practical and differential lower bound for feasible computation.
5 Proof-of-concept Implementation
To provide unified implementation scheme of Feint into most MARL frameworks, choose to implement on the training iteration level and avoid changing the MARL models themselves. We create an additional policy model (e.g.. MADDPG (20), MASAC [I MAD3PG 4 6), MATD3 (2].
etc.) for each agent as the Feint policy, which works together with the regular policy models for agents but is trained and inferenced differently.
Imaginary Play
Feint Policy Model
Ubdee
Aporeulazd Faeaadd Tuak-Setaddy Mola
Roole
Setwen Fule botwver
Acare
Action
Panan] Sudlaatinn
hood
Accuralde
saun
Rapúa Sewarz
5: Rehevie Model
Rlverd
Interact with Environment
Progresa
Accumande
fessibbie
Paggrsn
Regular Policy
Accuroleedd Revard Rogukr Training
Model
Action
Updake
Figure 3: Illustration of Feint behavior implementation in game iterations
Figure 3 illustraats the full process of our implementation of Feint game iterations. We implement the Feint behavior generation in imaginary play module in training iterations (i.c., game steps). The imaginary play module decides whether an agent should initiate a Feint behavior, composes a Dual-Bchavior Model using Palindrom-ddrected templates, and utilizes the Feint reward calculation to evaluate the quality of the generated action sequence in the Dual-Behare maal-Behavior model.
In our experimental settings, Feint behavior templates can be pre-computed only once before training. providing fast lookup for composing Dual-Behave models gaming iterations. Algorithm [ Appendix illustrates the pseudo-code for pre-computing available Feint behavior templates given a set of available attack behaviors B. For each pair of attack behaviors (Behavior, Behavior,) in B, we check whether there are physically satisfiable states illustrated in Figure [ If any of the three conditions are satisfied, we cut out the available actions (Avail, and Avail,) and store their compositions as available template candidate. Note that the stored templates do not enforce the Feint behaviors to contain the exact same action sequences. Instead, tuples (a,, Avail,, Avail,) are served as keys for quick lookup and action restrictions in composing gaming iterations.
[
models during
During gaming iterations, the imaginary play will only be activated when no Gual-Behavior Model in progress and the current physical state 8, an agent close to a physical state 8, where physically realistic for the agent to perform high-reward action Otarget, while the possibility of performing atarget relatively low according highly likely to be diminished by other agents current actions). Thus, the purpose of Feint behavior
regular policy model (i.e., action Otarget are

---

is to lead the agent to a state s, where the agent can maximize the game environment reward by performing the intended high-reward action atarget (i.e., other agents are deceived by Feint to perform other actions which cannot effectively diminish the high-reward actions performed by the agent).
When the imaginary play is activated, available Dual-Behavior models can be generated using the last known action ag and the intended high-reward action atarget. Algorithm in Appendix [ shows the pseudo-code for composing available Dual- Hual-Behavior models with backward searches. The intended high-reward action Ogarget provides constraints on the ending actions of the Feint behaviors, while the last known action ae provides constraints on the starting action of Feint behaviors. Thus, the available models can be quickly computed by doing a one-pass search from the pre-computed Feint behavior templates. The composed Dual-Behavins models can thus constrain the available action choices for the Feint policy model (set other actions' possibilities to 0) in corresponding templates and use a reflection frame to compose (semi-)palindree state 8,. After having available Dual- Seaa-Bee models which are composed of Feint behavior and followed by high-reward actions, the short-term reward can be calculated by Equation IT After this Dual-Behavio action sequence, the imaginary play would play few steps to incorporate the long-term reward (using Equation D. The collective reward (Section 4.2.2 can thus be calculated. This reward is then compared to accumulated reward from an imaginary play using only the agent's 's regular policy model in the same number of time steps. If the Feint collective reward is higher, the action sequence of the dual action model will be applied in the following real game steps. When Beha-Be Model is in progress, the actions will not be sampled from the regular policy models.
leading to the agent's 's physical
In the real game steps, where all the agents' actions interact with the environment and the real game rewards are calculated, our formalization of Feint only changes the way to update the Feint policy maal-Behavior
models for agents. The Feint policy models are updated only when corresponding Models finish and are updated using the accumulated real game rewards for that period. The regular policy models are updated usual settings (e.g., after some fixed steps an episode).
Experimental Studies
6.1 Methodology
Testbed Implementations. Due to the lack of a general benchmark, we selectively implement two scenarios under customized manner. They consist of a "I vs 1" and a "3 vs 3" multi-player free fight games, based on widely-providee testbeds. We provide additional details on how our testbeds are designed and implemented in appendix D
Evaluation Metries. Our main evaluation objective is the gaming rewards. We first examine the gaming outcomes when using the MADDPG, MASAC, MATD3, and MAD3PG MARL models, by comparing the per episode gaming rewards of agents across all scenario [ metrics, and report our results in appendix F
We also evaluate other
6.2 Major Results
Figure 4 shows the game reward comparisons of using Feint behaviors or not in the Two-Player scenario (Section(6.) 6.1) for 4 MARL models. The first row shows the baseline results where all agents are trained normally, while the second row shows the results where the player labeled with "Good" incorporates Feint behaviors. In most of the baseline results (e.g., using MADDPG, MAD3PG, and MATD3), the two players' rewards tend progress to a similar level when after enough training iterations. For MASAC, the "Good" player seems to gain higher rewards than its opponents when the training iterations are large, but the advantage is not stable and such a phenomenon can likely be the instability of the MASAC algorithm itself. For all the results where Feint behaviors are incorporated, we can see significant advantage gain for the "Good" player. Thus, our formalization of incorporating Feint behaviors can effectively improve the actual game rewards in two-player combating scenarios.
To further evaluate the effectiveness of our formalization of Feint behaviors in multi-player scenarios, Figure 5 shows the game reward comparisons in Six-Player scenario (Section 6.1] for 4 MARL models. The first row shows the baseline results while the second row shows the results where the player labeled with "Good 3" incorporates Feint behaviors. In all bascline results, all players seem to achieve similar levels of rewards after enough training iterations. In comparison, in all results where the "Good 3" player incorporates Feint, it gains significantly more rewards than the
'Not that these rewards are the actual game rewards (the reward that returned by the gaming environment), which are not the rewards that policy models used to select actions update parameters

---

Figure 4: Comparison of Game Reward when using Feint and not using Feint in a VS 1 scenario.
opponents as well as its teammates. This result shows that our formalization of Feint can not only gain higher rewards towards the direct opponents, but also gain advantages among teammates who do not incorporate Feint Another interesting observation is that there are no more symmetric patterns in the players' rewards, showing that the gaming interactions in multi Malli-ppaaerrree scenarios have enough complexity (Note that the scenario is not designed to be zero-sum game).
Figure 5: Comparison of Game Reward when using Feint and not using Feint a VS 3 scenario.
7 Conclusions and Main Implications
This work introduces the first comprehensive formalization, implementation and quantitative evalu- ations of Feint Multi-Player Games. We provide automatic generation of Feint behaviors using Palindrome-directed Templates and synergistically combine Feint with follow-up actions in Dual- Behavior Model. The decision choices on the action-level are fused into strategy-level formalizations in game interactions. We provide concrete implementation scheme to incorporate Feint into common MARL frameworks. The results show that our design of Feint can (1) greatly improve the reward gains from the game; (2) significantly improve the diversity of Multi-Player Games; and (3) only incur negligible overheads in terms of the time consumption. We conclude that our formalization of Feint is effective and practical, to make Multi-Player Games more interesting.
The successful formation of Feint behaviors and strategies imply the potential unsafety of existing machine learning models (and, clearly, future ones also). Therefore, the wide adoption of machine learning models certainly demand the consideration of this work (and its variants), for building responsible Artificial Intelligence; and/or leveraging them for the future society in responsible way.
10

---

References
[1] Johannes Ackermann, Codacy Badger, and Ted Jiang. Johaamesad//22nulliggnnntt November 2020.
[2] Johannes Ackermann, Volker Gabler, Takayuki Osa, and Masashi Sugiyama. Reducing overestimation bias in multi-agent domains using double centralized critics. arXiv preprint 1rXiv:1910.01465, 2019.
[3] Kai Arulkumaran, Antoine Cully, and Julian Togelius. Alphastar: an evolutionary computation perspective. In Manuel fépez-Ibáñez, Anne Auger, and Thomas Stützle, editors, Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO 2019, Prague, Czech Republic, July 13-17, 2019, pages 314-315. ACM, 2019.
[4] Gabriel Barth-Maron Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy P. Lillicrap. Distributed distributional de- terministic policy gradients. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
[5] Lucas Critch and David Churchill. Sneak-attacks in starcraft using influence maps with heuristic search. In 202/ IEEE Conference Games (CoG), Copenhagen, Denmark, August 17-20, 2021, pages 1-8. IEEE, 2021.
[6] Dongyu Fan, Haikuo Shen, and Lijing Dong. Multi-agent distributed deep deterministic policy gradient for partially observable tracking. Actuators, volume 10, page 268. MDPI, 2021.
[7] Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Sheila A. Mcllraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelli- gence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18) and the 8th AAAI Symposium on Educational Advances Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 2974-2982. AAAI Press, 2018.
[8] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1582-1591. PMLR, 2018
[9] Iris Güldennenning, Mustafa Alhaj Ahmad Alaboud, Wilfried Kunde, and Matthias Weigelt. The impact of global and local context information on the processing of deceptive actions in game sports. German Journal of Exercise and Sport Research, 4863):366-375, 2018.
[10) Iris Güldenpenning, Wilfried Kunde, and Matthias Weigelt. How to trick your opponent: A review article on deceptive actions in interactive sports. Frontiers in psychology, 8:917, 2017.
[11] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässsan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1856-1865. PMLR, 2018.
[12] Ray Hyman. The psychology of deception. Annual review of psychology, 40(1):133-154, 1989.
[13] Shariq Iqbal and Fei Sha. Actor-attentionncritte for multi-agent reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2961-2970, PMLR, 2019.
[14] Dong-Ki Ki Kim, Matthew Riemer, Miao Liu, Jakob N. Foerster, Michael Everett, Chuangchuang Sun, Gerald Tesauro, and Jonathan How. Influencing long-term behavior in multiagent reinforcement learning. CoRR, abs/2203.03533. 2022.

---

[15] Mare Lanctot, Vinícius Flores Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified theoreth approach to multiagent reinforcement learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4190-4203, 2017.
[16] Jehee Lee and Kang Hoon Lee. Precomputing avatar behavior from human motion data. Graph. Model, 68(2):158-174 2006.
[17] Seyoung Lee, Sunmin Lee, Yongwoo Lee, and Jehee Lee. Learning a family of motor skills from a single motion clip. ACM Trans. Graph., 40(4):93:1-933)) 2021.
[18] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Repre- sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[19] Xiangyu Liu, Hangtian Jia, Ying Wen, Yaodong Yang. Yujing Hu, Yingfeng Chen, Changjie Fan, and Zhipeng Hu. Unifying behavioral and response diversity for open- open-ended learning in zero-sum games. CoRR, abs/2106.04g0 04958, 2021.
[20] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-commptttivve environments. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30; Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 3379-6390, 2017.
[21] David Matz. Ancient Roman Sports, AZ: Athletes, Venues, Events and Terms. McFarland, 2019.
[22] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
[23] Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi- agent populations. arXiv preprint 1rXiv:1703.049003 2017.
[24] Abhishek Naik, Roshan Shariff, Niko Yasui, and Richard Sutton. Discounted reinforcement learning is not an optimization problem. CoRR, abs/1910.02140, 2019.
[25] Nicolas Perez Nieves, Yaodong Yang. Oliver Slumbers, David Henry Mguni, Ying Wen, and Jun Wang. Modelling behavioural diversity for learning in open-ended games. In Marina Mcila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 1, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 85 9514 8524. PMLR, 2021.
[26] Chris Nota and Philip Thomas. Is the policy gradient gradient? In Amal El Fallah Seghrouchni, Gita Sukthankar, Bo An, and Neil Yorke-Smith, editors, Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 20, Auckland. New Zealand, May 9-13, 2020, pages 339-947, International Foundation for Autonomous Agents and Multiagent Systems, 2020.
[27] Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. AMP: adversarial motion priors for stylized physics-basee character control. ACM Trans. 40(4). 144:20, 2021.
40(4): 144:1-
[28] Sebastian Risi and Mike Preuss. Behind deepmind's 's alphastar ai that reached grandmaster level in starcraft ii. Ki-Kinstleche Intelligenz, 34(1):85-86, 2020.
[29] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/17007.634.4 2017.
12

---

[30] Hubert P. H. Shum, Taku Komura, and Shuntaro Yamazaki. Simulating interactions of avatars in high dimensional state space. In Eric Haines and Morgan McGuire, editors, Proceedings of the 2008 Symposium on Interactive 3D Graphics, SI3D 2008, February 15-17, 2008, Redwood City, CA, USA, pages 131-138, ACM, 2008
[31] Oswin So, Kyle Stachowicz, and Evangelos A. Theodorou. Multimodal maximum entropy dynamic games. CoRR, 0tbs/2201.12925, 2022.
[32] Stefano Corazza and Nazim Kareemi. Mixamo, 2022.
[33] ToalhaNerd Team. Toalha nerd-king of fighter xv: Trailer de ryo sakazaki e robert garcia! 2021.
[34] Kevin Wampler, Erik Andersen, Evan Herbst, Yongjoon Lee, and Zoran Popovic. Character animation in two-player adversarial games. ACM Trans. Graph., 29(3):26:1 2010.
[35] Jungdam Won, Deepak Gopinath, and Jessica K. Hodgins. Control strategies for physically sim- ulated characters performing two-player competitive sports. ACM Trans. Graph., 40(4): 146:1- 146:11, 2021.
[36] Jungdam Won, Deepak Gopinath, and Jessica K. Hodgins. Control strategies for physically sim- ulated characters performing two-player competitive sports. ACM Trans. Graph., 40(4): 146:11 1, 2021.
[37] Barbara Yersin, Jonathan Maim, Julien Pettré, and Daniel Thalmann. Crowd patches: populating large-scale virtual environments for real-time applications. In Eric Haines, Morgan McGuire, Daniel G. Aliaga, Manuel M. Oliveira, and Stephen N. Spencer, editors, Proceedings of the 2009 Symposium on Interactive 3D Graphics, SI3D 2009, February 27 March 1, 2009, Boston, Massachusetts, USA, pages 207-214. ACM, 2009.
13

---

A
Conceptual Clarifications
Since this work spans multiple disciplines, there are few clarifications ensure the consistent understanding between our work and prior arts.
A.1 Differences between actions and behaviors in our formalization
To provide unified definition of Feint behavior in both continuous and discrete action space, highlight the difference between the terms action and behavior used in our formalization. We use action as the minimal unit movement in a unit time step, such as unit step movement along the X and Y axis in 2D board game, raising arms for certain distance in a boxing game. turning steering wheels while applying brakes for certain degree in a racing game, etc. This definition of action coincides with the commonly used definition of action in general MARL environments, which is intuitive understand, simulate, and build our formalization of Feint upon it. One may argue that in some game simulations, combat movements like a cross punch are simply considered as one action, but one can always divide those movements into several unified unit-time-step actions to create a unified alignment in terms of time step in games. In terms of behavior, we refer to it as combination of several actions sequence (e.g., cross punch in boxing games). Thus, Feint can be naturally defined as a behavior that uses sequence of actions to deceive opponents and lead to large reward actions in the near future. We describe our observation Feint behaviors' characteristics and introduce our formalization at the action level in Section 3
A.2 Modeling Behaviors at Action-Level in Game Animation and Simulation
Modeling characters' behaviors (series of actions) in games can be divided into two categories based on the main purpose: animation-driven modeling or simulation-driven modeling, though animation and simulation are inherently closely correlated. Animation- on modeling the behaviors themselves, with goals of producing a variety of nuanced and coherent action sequences. The interactions with the environment (whether physics-based or not) are generally considered after the modeling of the behaviors and are generally simplified to showcase the behaviors themselves. Patch-based generation a direct way for such methods, which directly compose behaviors by combining pre-defined action sequences [35] This approach is widely adopted in the industry due to its high production efficiency, supported by an extensive amount of animation libraries (e.g. Mixamo (32) [16] 30 371. However, in recent years, Learning-bast based generation
methods mainly focus
dominates the field as they can automatically produce animated behaviors to mimic the styles of learned actions from the training inputs 17 277. On the other hand, simulation-driven modeling usually considers the full interactions with the environment in the first hand. These methods generally formalize the behavior modeling process using Reinforcement Learning (RL) based frameworks to fully explore the complicated space of physics-based action modeling (35). In our work, we use animation-driven modeling with strong physical constraints to describe our observations of Feint behavior characteristics and use the general simulation-driven modeling in MARL schemes for learnable formalization of Feint in action and strategy levels.
14

---

B Feint Behavior Generator and the Resulting Templates
Under the above three-stage decomposition of attack behaviors, there are abundant possibilities to compose Feint behaviors from the three action sequences. However, to ensure physically realistic generation, we summarize two requirements that Feint behaviors must follow: (1) Feint behaviors should follow semi-symmetrice patterns to effectively deceive opponents and return to rest position for follow-up moves. boxing, a human player must retract the stretched-out limbs to the relatively rest position, before stretching out to perform actual attack action. This is because the retraction requires recharging the force to contracted muscles; and (2) transitions between adjacent actions in different behaviors expected to be smooth, as humanoid body movements must provide continuous movements.
To satisfy the above two requirements, we propose a Feint behavior template generator called Palindrome-diretted Generation of Feint Templates, by extracting subsets of cmmi-symmerricl actions from an attack behavior and synthesizing them as a Feint behavior. The general method to generate these templates are (1) by extracting subsets of unit actions from an attack behavior, Feint behavior can be considered as a semi-finishen real attack behavior. This ensures the high similarity of a generated Feint behavior with an attack behavior, thus opponents can be deceived; and (2) by synthesizing semi-symmetric action sections, the overall movements can be connected smoothly and the naturalness of humanoid actions can be guaranteed. Within our proposed template generator Palindrome-directed Generation of Feint Templates, there are two key adjustable parameters in practice: (I) sequence composition positions for Feint templates; and (2) sequence length for Feint templates. We provide the rationales for these two key design choices.
(1) Sequence composition positions for Feint templates: Determining which position to extract the subsets of action sequences needs to ensure that the extracted actions are semi- comi-symmttrical and allow physically realistic connections. To this end, we can have three templates with different restrictions to exploit the composing patterns: (A) For template 1, if there are similar physical states, which refer to the positions of all joints and stretching angles are similar (as shown in of Figure T). actions before the first similar state and after the second similar state can be extracted and directly synthesized as a Feint behavior (shown in of Figure I : (B) For template @, by cutting once at any time point in Sequence 1, action sequences before the selected point and the corresponding reversion can be synthesized as a Feint behavior (shown @ Figure DD: and (C) For template I, similar to the second situation, by cutting once any time point sequence 3, action sequences after the selected point and the corresponding reversion can be synthesized as a Feint behavior (as shown in @ of Figure T. With these considerations, the Feint behavior generation templates guarantee the naturalness of continuous movements via semi-symmetrical patterns.
(2) Sequence length for Feint templates: The choices for the length of extracted action sequences in each template can vary greatly, since muitiple actions in an attack behavior can extracted based on different time ranges. The available choices can be any time length that results in action sequences that satisfy the physical requirements discussed above (e.g. morphologically reasonable Template @ or Template @ in Figure Note that it is also possible to construct nested Feint behaviors, given a large number of feasible extraction positions. We formalize this choice as a learnable parameter that needs to combine Feint behaviors with their intended follow-up actions (Section 3.2 and the learning adjustment is described in Section 5
15

---

C Demonstration of Feint Behaviors
C.1 Demonstration of Feint Behaviors in Dual-Beahvior Models
To explain the generation of physically realistic Feint behavior in
Model in detail. we use humanoid models: when selecting the corresponding actions (i.e. from Feint behaviors and then attack behavior), the starting position (jointly connected body) of the second action should be the same as the ending position of the starting action. With such principle, the joints a character's body can perform natural movements during the transition between these demonstrates a physically realistic combination of Feint behavior and follow-up attack behavior. When checking the end of NPC A's Feint behavior and the beginning of the Agent's (left white agent) real attack, both the upper and lower body parts of NPC A perform the same postures (the left arm raised and the right arm charged, performing punch for the upper body, and the left foot forward for lower body).
behaviors. Figure
providdes detailed example of successful Feint behavior in Dual-Behavior Model. We refer to the Agent as the white player on the left and its Opponent as the black player on the right, and describe the Feint behavior from the Agent perspective. The agent first performs Feint behavior which is fake punch towards its opponent's head, which leads the opponent to defend towards its head. However, the agent connects such Feint behavior with follow-up hook towards the opponent's waist. Due to the temporal advantage gained by the quick Feint behavior and the spatial advantage gained by deceiving the opponents defend to wrong directions, the opponent would be knocked down by the follow-up behavior of the agent. Thus, successful Feint behavior is performed in this Dual- Behavior Model.
Agent Feint Behavior (Fake punch towards oppoent's head)
Opponent's ' response to the Feint full for defend for its head
Temporal Advantage
Agent - Followed-up attack (Hook towards the oppoent's waist)
Opponent about finish head defending
Opponent won't able accomplish a defend for its waist in time
Figure 6: Dual-action Model snapshots of the full process
C.2 Demonstration of Successful and Unsuccessful Feint Behaviors
To enable a successful Feint behavior in Dual-Behavior Model, the temporal and spatial advantages should be properly formalized. The advantages of combining Feint behaviors with follow-up high- reward actions stem from an appropriate time difference, incurred by Feint behaviors to mislead the opponents' actions. If the length of Feint behavior too short, the following attack actions might not gain much advantage compared to actions combinations without Feint behaviors; and if the length of Feint action too long, the process to perform Feint behaviors can leave sufficient time for the opponent to react and even attack back. We provide examples for these scenarios in Figure 7 Figure & and Figure 9 We refer to the left white player as NPC and describe the Feint from its perspective, and the right black agent NPC B is considered its opponent.
16

---

We use the timeline of the Dual-Behavio Model in Figure 2 to analyze and evaluate the three Feint behaviors. We use three key time points that are highlighted in Figure Figure [ and Figure to explain the action sequences, in which LB, indicates the end of defense behavior while LA, indicates the estimated start of reward in the second action sequence for NPC and L, indicates the estimated start of reward in second action for NPC B. The three consequences mainly differ in these three key time points.
1) Very short Feint behaviors Ag < tB The action sequence of simulation is shown in Figure) 7 which the Feint behaviorsduration is extremely short and the estimated start of reward in second action for NPC A (ta,) happens when NPC B is still in the first defense action (thus Ê < tp, ). As the sequence shows, the second real action of NPC A would not benefit much since NPC is still in defense.
NPC A's first behavior Feint (too short)
NPC A's second behavior real attack
NPC B's first behavior step back defense NPC A's second behavior real attack (no effective reward since NPC B still defending)
LA B's first behavior (continue) step back as defense
[B
Figure 7: Demonstration of unsuccessful Feint behavior when its too short
2) Proper length Feint behaviors tB, < tA < t, The action sequence of simulation is shown in Figure in which the Feint behaviors have moderate duration. The key difference of this duration is that the estimated start of reward in the second behavior for NPC A happens after the end of the defense behavior of NPC and before the estimated
of reward in the second behavior for NPC B, thus showing the temporal advantages introduced in Section 3.2 With such temporal advantages, NPC A gains preemptive advantage over NPC inflicting rewards from NPC B (at time 4 in Figure []] before NPC B' 's reward inflicting of second behavior starting (at time t2 in Figure 8). When NPC hits NPC B at 422 the ongoing action of NPC B will be interrupted and NPC would be knocked down.
3) Very long Feint behaviors A, t,, The action sequence of simulation is shown in Figure in which the Feint actions duration is too long and the estimated start of reward in second behavior for NPC A (LAz) happens after the estimated start of damage in second action for NPC B (L This condition has the opposite consequence of a moderate length Feint behaviors, in which NPC B can inflict rewards on NPC before NPC A's reward inflicting of the second behavior starts. When NPC B hits NPC A at A2, the ongoing action of NPC A will be interrupted and NPC A would be knocked down.
Thus, the choice of the time duration for Feint actions highly depends on the action combinations and the estimation of opponents' actions, proving our observation in Section Thus the learning formalize such a choice the strategy learning scheme (Section 4) is important to construct effective Feint behaviors with corresponding Qual-Behavior Models.
17

---

NPC A's first behavior
Feint
NPC first behavior step back as defense NPC A's second behavior - real attack
LB NPC B's first behavior (continue)
1Az
182
NPC B's second behavior knocked down by NPC A
Figure 8: Demonstration of successful Feint behavior with proper length
NPC A's first behavior Feint (too long)
NPC B's first behavior step back defense
NPC A's first behavior (continue)
Feint
NPC A's second behavior real attack (interrupted NPC B)
NPC B's second action real attack (effective reward)
Figure 9; Demonstration of unsuccessful Feint behavior when its too long
D
Testbed Implementations
Our main testbed game environment multi-player boxing game, which is based on OpenAl's open-source environment Multi-Agent Particle Environment 233, but with heavy additional imple- mentation to create a physically realistic acenario.This game resembles intense free fight scenarios in ancient Roman free fight scenarios (2I), where interactions are intense and Feint expected be effective. We incorporate common boxing behaviors (action sequences) in boxing games. following the methodology some animation and simulation works [34 36] This handcrafted scenario contains complex physics- bassed interaction systems and fine- tne-grained time steps to enable learning and generating Feint behaviors. A detailed description of the reward gaining system, environment parameters, and agent settings is presented in Appendix D.I We also modify and extend a strategic real-world game, AlphaStar [3], which is widely used as experimental testbed recent studies of Reinforcement Learning studies [28 [9 We make extra efforts to emulate six-player game, where players are free to have convoluted interactions with each other. And we implement Feint as dynamically generated policies, based on the 888 regular gaming policies.
18

---

D.1 Details of Boxing Game Scenario
Our testbed game scenario emulates a complex boxing game by modeling all the detailed combat behaviors except building the graphical rendering process. The reason neglect the rendering process is that our main goal is to evaluate the effectiveness of formalization of Feint behaviors in multi-player games, and the building real-time graphical rendering with such complex humanoid interactions would be a graphics paper itself. We fully emulate all the behavior details in our game simulation, thus constructed game simulation is detailed enough evaluate formalization of Feint behaviors. We provide detailed description the game scenario here.
We follow similar boxing game scenario construction approach [34 35]. and model the full set of Mixamo [32] 22 behaviors (action sequences) which contain over 250 available full body actions (illustrated in Figure T0 We extensively construct gaming environment based Multi-Agent
Particle System 23 to incorporate these behaviors, which then can be seamlessly integrated with
common MARL models.
Short Punon
Medium Punch
Long Punch
Short Hook
Mediaan Huck
Long Hook
Cutck Punch
Cross Punch
Kidk
Ebow Punch
Offensive Behaviors
Step Backww
Step Sidzwards
Short
Head
Center
Kieney Hil
Defense Behaviors
Reaction Behaviors
Wem
Plvot-Around
Transition Behaviors
Figure 10: The full set of 22 behavior (action sequences) of boxing game from Mixamo.
The players move around in 2D plane. We use vector to model the physical state of players, which stores and tracks the body movements of a player. This vector tracks the positions of body parts: left and right limbs, the left and right legs, and the center body, which is used to select available combat behaviors (the transitions of body movements must be smooth as mentioned in Section [.]] be naturally generated and incorporated into suitable Dual-Behavior Models. We follow the exact Mixamo dataset to model the length of the behaviors (the length of action sequences) and rewards the behaviors (e.g., successful long punch would gain more rewards than a short punch.) Specifically, we measure the number of frames contained in all behaviors and normalize them to define unit time steps for action space and thus get the action sequence lengths for all behaviors. An example of game rewards and action sequence length of behaviors provided in Figure TI
and Section (..2.. With this setting, Feint behaviors
D.2 Experimental Procedure
We choose 4 commonly used MARL models: MADDPG 20), MASAC I3) MATD3 (2, and MAD3PG A 6 and incorporate them into testbed scenarios. Our implementation is based on [ which provides a unified MARL frameworks behaviors can be uniformly and effectively learned using all these commonly used MARL models and how can Feint affect the game rewards for agents. Note that our purpose is to verify the effectiveness of our formalization of Feint behaviors and not to compare modify the MARL models themselves.
the above models. We aim test whether Feint
19

---

Example Behaviors
Short Punch
Short Hook
Medium Punch
Punch
Cress Punch
Game
8
12
20
Action Sequence Length
5
6
9
13
Figure 11: Demonstration of the game rewards and action sequence lengths of Mixamo behaviors.
We create two test scenarios, the first with players (one player per team) and the second with six players (3 players per team). For all of these scenarios, we first train the agents without Feint baselines using the models. Then for the two-player scenario, we incorporate Feint one player (shown as the Good player in Figure 4). For the six-player scenario, we select agent in the Good team (labeled as Good in Figure 5). incorporate our formalization of Feint, and keep all other 3 agents regular. The reason for this design in the six six-player scenario is that we want to only test how Feint behaviors can affect the reward gain against direct opponents, but also see whether Feint can bring ad advantages for player among its teammates. All the players are rewarded independently and the notion of the "Good" and "Adv" team does not mean that teammates have shared reward (i.e., not explicit constraints that force them to cooperate). Note that all players have identical capabilities and are rewarded using the same mechanisms, thus Feint can be incorporated on any player. Our labeling choice here is to provide to consistent way to track and analysis the game rewards. All experiments for the two-player scenario are trained 75,000 game iterations and experiments for the six-player scenario are trained for 150,000 game iterations.
20

---

E Implementation Details
We provide the pseudo-code for computing Feint behavior templates and composing mnal-Behavior Models for our proof-of-concept implementation discussed in Section 5
Algorithm 1 Precompute Available Feint Behavior Templates
1: Input: Set of behaviors B {Behavior:Ji [1,n|], where each Behavior,
Ai, } is sequence of actions.
2: Output: Set of physically available Feint behavior templates T.
3: Compose a set of behavior pairs: Baair = (Beehawior; Behavior,) i, ii, E [1,
4: Initialize empty set of templates: T=
5: for each [Beharior,, Behavior,) E Bpair do
6: Compute the common action set: Acomoo {ax lax E Behavior, and ag E Behavior }
7: if' Acommmn + O then
8:
for each 0Ą Acommon do Available actions from Behavior; Avail, = {aom am E Behavior, m <k} Available actions from Behaviory; Avail, {anlan E Behavior n > k}
9:
11: end for
Update T: T + + T {(ak, Avoil, Availy)}
13:
end if
end for
15: return T
Algorithm Backward Search Compose Dual Behavior Models
1: Input: Last action beginning of the high-rewards behavior Otarget, precomputed Feint templates T'
2: Output: Set of available Dual Behavior Models D
3: Initialize empty set of Dual Behavior Models: 0
4: for each (4x, Avail; {aplp E [0, 1]]. Avail, {aylg E [k + 1, max]}) in T do
5:
Here, max is the maximum index of action in Avail,
6: if (ot Avail,) and (Atarget E Avail,) then
7: Select actions from a, to dg Avail;: Select, (am/m E [t, k - 1}}
8: Select actions from Otarget in Availy: Select, = {an/n E [ max))
9: Update D: D + DU {(Select, ag, Select,))
10: end if
11: end for
12: return D
21

---

F
Additional Experimental Results
We report the effects of Feint on 1 diversity gain of policy space; and overhead of computation load. We examine the effects of Feint actions on how Feint can improve the diversity of gaming policies (Section 4.3) strategy learning.
We also perform overhead analysis, incurred by fusing Feint formalization in
F.1 Diversity Gain
To examine the impacts the policy diversity in AlphaStar games, perform comparative study between MARL training with and without Feint Specifically, We use Exploitability and Population Efficacy (PE) measure the diversity gain in the policy space. Exploitability [I5] measures the distance of a joint policy chosen by the multiple agents to the Nash equilibrium, indicating the gains of players compared to their best response. The mathematical expression of Exploitability is expressed as:
N Exp((T)= = Σ (maz Rew, x., - Rew; π xi))
(8)
where Ty stands for the policy of agent and .. stands for the joint policy of other agents. Rew; denotes our formalized Reward Calculation Model (Section 4.3). Thus, small Exploitability values show that the joint policy is close to Nash Equilibrium, showing higher diversity. In addition, we also use Population Efficacy (PE) [19] to measure the diversity of the whole policy space. PE is generalized opponent- concept of Exploitability by looking for the optimal aggregation in the worst cases, which is expressed as:
PE((+ = minr- 1 a=1 a=>=00 or Rew, (m T-1)
where n, stands for the policy of agent and T- stands for the joint policy of other agents. a denotes an optimal aggregation where agents owning the population optimizes towards. Rew; denotes our formalized Reward Calculation Model (Section 4.3) and opponents can search over the entire policy space. PE gives a more generalized measurement of diversity gain from the whole policy space.
Figure 12 shows the experimental results for evaluating diversity gains. From the figure, obtaín two observations. First, agents that can dynamically perform Feint actions (Agent 1, 2, and 3) achieve lower Exploitability (around 4.9 x 10 2) compared agents who perform regular actions (around 9.7 x 10 and have higher PE (lower negative PE around 5.3 x 10 3) than those who only perform regular actions (around 1.2 10-3 2). This result shows that our formalized Feint can effectively increase the diversity and effectiveness of policy space. Second, agents with Feint have slightly higher variations in both metrics. This is because Feint naturally incurs more randomness (e.g. succeed or not) in games, resulting in higher variations in metrics.
Agent
Agect
Agel2
Aguma
Agrit
30"
142
87
Ganning Ravalions
Gaming Rerations
Figure 12: Diversity gain for agents, in terms of the exploitablity and the negative population efficacy.
F.2 Overhead Analysis
Figure shows the results of our overhead analysis. make two observations. First, fusing Feint in MARL training do incur some overhead increment in terms of running time. This is because the
22

---

formalization and fusion of Feint in MARL incur additional calculation load. Secondly, in both MADDPG models and MAAC models, the increased overhead is generally lower than 5%, which still indicates that our proposed formalization of Feint actions can have enough feasibility and scalability on fusing with MARL models. Note that even we use two policy models for each agent our implementation, our designs restrict that only one model is inferenced in each game step (Section 5 thus the overhead is low.
VS
3
Figure 13: Overhead of Feint the VS and 3 VS scenarios using MARL models.
G Limitations
We want to point out that our current design aims to provide generalizable and as-automatic-as- possible approach to concretely implement Feint behaviors into MARI models proof-of-concept. We acknowledge that there are possible ways to optimize this template selection stage in different gaming scenarios, for example, adding human-kkowleee knowledge guidance or learning heuristics. However, we believe that the variations at this implementation level do not harm our main proof-of-concept contribution.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract accurately describes main ideas, paper structures, and contri- butions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitation discussed in the Discussions.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

All the theorems, formulas, and proofs in the paper are well stated and referenced.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Implementation and experimental details are introduced Section and Section D.I
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
25

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Our main contribution is unified formalization of Feint behaviors which could be widely adapted to most commonly used MARL frameworks. We have provided detailed description of the codebase our implementation is based in Section 6.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
mublic/guider CodeSubmissiooooooo...... sisiinPolicy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips wwtpccccccccccueeeeeeeeeeeeppeee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Details are provided in Section D.I
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our experiment shows that Feint behaviors could significantly increase agents' reward and game diversity, by comparing the agents' rewards and game diversity other agents' values in all our plots. These plots show clear differences which proves the effec- tiveness of our formalization and implementation. The exact reward values not main concern thus we don't 't include error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details are described Section D.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: We fully follow NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Well-cited in Section 6.1
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "q3GJHyNkMR", "reviewer": "Reviewer_Wu2K", "summary": "This paper introduces the first comprehensive formalization of Feint behaviors in multi-player games. The authors present a novel approach to automatically generate Feint behaviors using Palindrome-directed templates and combine them with intended high-reward actions in a Dual-Behavior Model. The formalization addresses both action-level and strategy-level aspects of Feint behaviors, considering temporal, spatial, and collective impacts.\n\nThe authors provide a unified implementation scheme to incorporate Feint behaviors into common Multi-Agent Reinforcement Learning (MARL) frameworks. They evaluate their approach using multiple MARL models in a custom boxing game scenario and a strategic real-game simulation.\n\nThe results demonstrate that incorporating Feint behaviors can significantly increase game rewards and improve the diversity of multi-player games. The authors show that their method outperforms existing approaches in several scenarios, with minimal computational overhead.\n\nWhile the work presents a novel and potentially impactful approach to modeling deceptive behaviors in games, it is limited by its focus on specific game scenarios and lack of theoretical analysis. Nevertheless, this paper contributes a valuable framework for enhancing the realism and strategic depth of AI agents in multi-player games.", "questions": "1. How well does your formalization of Feint behaviors generalize to other types of games beyond the boxing scenario? Could you provide examples or preliminary results from applying your approach to significantly different game genres (e.g., strategy games, team sports)?\n\n2. Have you considered any theoretical analysis of your approach? For instance, can you provide any bounds on the expected performance improvements or convergence guarantees when incorporating Feint behaviors?\n\n3. Your paper primarily compares agents with and without Feint behaviors. Have you considered comparing your approach to other methods for modeling deceptive or strategic behaviors in games? If so, which methods and why were they not included in the current paper?\n\n4. The implementation of your approach seems quite complex. Could you provide a simplified version or pseudocode of the key algorithms to make it easier for others to understand and implement your method?\n\n5. How do you envision your formalization of Feint behaviors being applied in real-world game design or AI systems beyond simulations? Are there any specific applications or domains where you think this approach could have significant impact?\n\n6. Your Palindrome-directed templates for generating Feint behaviors are intriguing. Could you elaborate on how these templates were developed and whether there are any limitations to this approach for generating diverse Feint behaviors?", "limitations": "n/a", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper demonstrates significant originality by providing the first comprehensive formalization of Feint behaviors in multi-player games. The authors address a notable gap in the literature, as previous works have only touched on Feint behaviors superficially or as proof-of-concept. The formalization at both action and strategy levels shows a depth of thinking that goes beyond existing approaches.\n\nThe quality of the work is evident in the thorough development of the formalization, from the Palindrome-directed templates for generating Feint behaviors to the Dual-Behavior Model for combining them with high-reward actions. The authors have clearly put considerable effort into creating a robust framework that can be applied across different game scenarios.\n\nIn terms of clarity, the paper is generally well-structured, guiding the reader through the complexities of Feint behavior formalization in a logical manner. The use of illustrative examples, particularly in the boxing game scenario, helps to ground the abstract concepts in concrete applications.\n\nThe significance of this work lies in its potential to enhance the realism and strategic depth of AI agents in multi-player games. By incorporating Feint behaviors, the authors have shown significant improvements in game rewards and diversity, which could lead to more engaging and challenging game AI. Furthermore, the unified implementation scheme for common MARL frameworks suggests broad applicability of this approach.", "weaknesses": "While the paper presents a novel approach, there are several areas where it could be strengthened:\n\n1. The primary experimental validation is conducted on a custom boxing game scenario. While this provides a good test case, it may not fully demonstrate the generalizability of the approach. The authors should consider including experiments from a wider range of game types (e.g., strategy games, team sports) to show the broad applicability of their formalization.\n\n2. While the paper provides a detailed formalization, it lacks rigorous theoretical analysis or proofs of the properties of the proposed approach. For instance, the authors could provide theoretical bounds on the performance improvements or convergence guarantees for their method.\n\n3. The paper primarily compares the performance of agents with and without Feint behaviors. However, it would be valuable to see comparisons against other existing methods for modeling deceptive or strategic behaviors in games.\n\n4. The formalization and implementation details are quite complex, which could make it challenging for others to replicate or build upon this work. The authors could consider providing a simplified version or pseudocode of key algorithms to improve accessibility.\n\n5. While the paper focuses on game simulations, it doesn't adequately address how this approach might be applied to or impact real-world game design or AI systems beyond simulations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "np3K4LttpS", "reviewer": "Reviewer_CkiB", "summary": "The paper presents a comprehensive approach to formalizing and implementing feint behaviors in multiplayer games. It introduces a new method for the automatic generation of feint behaviors using Palindrome-directed templates and combines these with high-reward actions in a Dual-Behavior Model. The paper further explores the implications of feint behaviors on game strategies, focusing on temporal and spatial impacts, and provides a unified implementation scheme. Experimental results demonstrate significant improvements in game reward gains, diversity, and minimal time overheads when incorporating feint behaviors.", "questions": "* How does the performance of the proposed feint behavior generation method compare with other state-of-the-art techniques in terms of computational efficiency and scalability?\n* Can the proposed methods be extended or adapted to non-gaming domains, such as robotics or real-world strategic planning? If so, what modifications would be necessary?\n* How does the complexity of the Palindrome-directed templates affect the learning curve and training time of the MARL models?", "limitations": "The author mentioned \"Limitation discussed in the Discussions,\" but there is no section called \"Discussion\" in the manuscript. I also couldn't find any discussion about limitations in other parts of the document.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The introduction of Palindrome-directed templates for generating feint behaviors is a novel concept that adds value to the field of game strategy formalization.\n* The paper offers the first detailed formalization of feint behaviors at both the action and strategy levels, addressing a gap in existing literature.\n* The integration of feint behaviors into common MARL frameworks and the use of diverse experimental scenarios enhance the practical applicability of the proposed methods.\n* The implementation scheme is designed to be adaptable across various MARL models, which increases its versatility and potential for widespread adoption.", "weaknesses": "* While the paper evaluates several MARL models, it would benefit from a broader comparison with more diverse baseline strategies and models to establish the relative performance gains more comprehensively, such as [1-2]. \n* The paper does not sufficiently address potential scalability issues when applying the proposed methods to larger and more complex game environments beyond the tested scenarios.\n* The focus is primarily on multiplayer games, and the applicability of the proposed methods to other domains with different characteristics is not explored in depth.\n* It is recommended to apply appropriate smoothing to the curves to enhance the readability of the figures. This is particularly important for Fig 4, where almost all the curves without feint are nearly overlapping.\n\n[1] Yu, Chao, et al. \"The surprising effectiveness of ppo in cooperative multi-agent games.\" Advances in Neural Information Processing Systems 35 (2022): 24611-24624.\n[2] Yang, Tianpei, et al. \"ASN: action semantics network for multiagent reinforcement learning.\" Autonomous Agents and Multi-Agent Systems 37.2 (2023): 45.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## [Reviewer CkiB]\n### - Apply appropriate smoothing to the curves for Figure 4\nWe thank the reviewer for this suggestion. We will add appropriate smoothing to the curves for Figure 4 for better visualization in our revision.\n\n### - How does the complexity of the Palindrome-directed templates affect the learning curve and training time of the MARL models?\nWe discussed the computational overheads in Appendix F.2. During training/gaming iterations, the Palindrome-directed Feint behaviors are dynamically generated, thus the overhead induced by them is already measured in the plots Figure.13. We want to note that the available templates (i.e., to satisfy the physical constraints in Palindrome-directed templates) can be precomputed for game simulation once having a set of reference action sequences (discussed in Appendix D.1). Thus in training/gaming iterations, the choice of Feint behaviors given an intended high-reward behavior are well constraint to a reasonable number and can be directly retrieved from pre-computed templates. There is no need to do tedious searching for all possible action sequence combinations (which is not possible in complex game scenarios.)  \n\n### - What are the computational efficiency and scalability to more complex games?\nWe respectfully identify that: examining the computational efficiency and scalability to more complex games is NOT our purpose. Our work aims to provide a low-cost approach to inject Feint into existing games (from actions to strategies). And our work demonstrates that our approach only adds negligible overheads to existing games in a variety of contexts.\n\nWe have reported our evaluation of the overhead incurred by Feint behaviors in Appendix F.2, by measuring the overhead increment at each training epoch. Our Feint implementation introduces less than 5% overhead increment in all MARL modes under all game scenarios. Our key design for this is to infer only one MARL model at each training step (explained in Appendix-E).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "luH944Jnnn", "reviewer": "Reviewer_R6PG", "summary": "The authors present a comprehensive formalization of feint behaviors in competitive multi-player games. The paper introduces a method for the automatic generation of feint behaviors using Palindrome-directed templates and combines them with high-reward actions in a Dual-Behavior Model. This formalization is incorporated into Multi-Agent Reinforcement Learning (MARL) frameworks. The authors conducted extensive evaluations using various MARL models in both two-player and six-player scenarios, demonstrating that their formalization significantly improves game rewards.", "questions": "1. The current framework of Palindrome-directed Generation of Feint Templates may not capture certain Feint behaviors. For example, while linear movements can be reversed to return to the same position, planar movements often involve feints that result in a change of location, which may not be adequately represented by this method. Additionally, there may be other types of Feint behaviors that this framework cannot express. It would be beneficial to clarify these limitations and discuss any potential Feints that fall outside the scope of this approach.\n\n2. In Section 4.2.1, the paper introduces two reward functions, R^i, in Eqs (1) and (2). However, it is unclear how these reward functions are determined. If the R^i in both equations are the same, it seems this reward is simply proposing a weighted combination of short-term and long-term rewards. If they are not the same, the authors should use distinct notations to clarify the difference between the two. Additionally, the definition of R^i is not provided, making it difficult to understand the practical implementation of these reward functions. To improve clarity, the authors can provide concrete examples illustrating how R^i is defined and calculated in both the short-term and long-term contexts.  \n\n3. In Section 4.2.2, it is unclear how the Policy Occupancy Measure is calculated for model-free policies, such as those used in deep RL. The paper may not provide a detailed explanation of how to derive this measure without a model of the environment. \n\n4. (Section 4.2.3) In the discussion of the payoff matrix, it is stated as P_i ​ ×P_i, but then it is described as an M×N matrix. This seems inconsistent. Shouldn't it be P_i ​ ×P_j ​ instead, representing the policy spaces of different agents? Please clarify how M and N correspond to the policy spaces of agents i and j respectively, and ensure that the definition and usage of the payoff matrix align with the interaction between different agents' policies.\n\n5. (Section 4.2.3) To accurately calculate the divergence between the payoff matrix and the vector a_{M+1}, it is necessary that a_{M+1} also be represented as an M×N matrix. The paper should clarify how a_{M+1}, representing the new policy occupancy measure after introducing the Feint policy, is structured as an M×N matrix, and provide detailed steps on how these values are computed to maintain coherence with the payoff matrix.\n\n6. In Section 5.1, the authors refer to Appendix D for Testbed Implementations. It is crucial to specify whether these implementations are original or derived from prior research in the main text.\n\n7. In Section 5.2, it appears that only gaming rewards are evaluated. To provide a comprehensive assessment, it is essential to detail the underlying reasons for this focus. Additionally, the results should include a thorough analysis of the types and distributions of observed Feint behaviors. Such detailed results would offer a clearer understanding of how Feint behaviors impact the overall performance and diversity in gaming scenarios. \n\n8. The citation format used in the manuscript does not adhere to the NeurIPS guidelines, making it difficult to read. For instance, in lines 20-21, \"and explore strategies based on them Wampler et al. [2010], Won et al. [2021a]\" should be revised to either [1, 2] or (Wampler et al. 2010; Won et al. 2021a) for better readability and compliance with the NeurIPS format.", "limitations": "The authors have not adequately addressed the limitations and potential negative societal impact of their work. While the paper presents a novel approach to generating and integrating Feint behaviors in MARL frameworks, it lacks a detailed discussion on the limitations of the proposed methods, such as the scope of applicability and potential challenges in real-world implementations. To improve, the authors should include a section discussing these aspects, providing a balanced view of the strengths and weaknesses of their approach, and consider potential negative outcomes and mitigation strategies.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The paper introduces a new formalization of feint behaviors, in which automatic generation of Feint behaviors via Palindrome-directed templates, and combine them with intended high-reward actions in a Dual-Behavior Model\n* The methodology addresses Feint implications on game strategies in terms of the temporal, spatial, and their collective impacts via the implementation in existing MARL frameworks.\n* In the experiments using various MARL models in both two-player and six-player scenarios, the authors demonstrate that their formalization significantly improves game rewards.", "weaknesses": "* The paper may lack clarity and consistency in the definitions and notations of key concepts, such as the reward functions in Section 4.2.1 and the Policy Occupancy Measure for model-free policies in Section 4.2.2. Additionally, the inconsistency in describing the payoff matrix dimensions in Section 4.2.3 creates confusion about the interaction between different agents' policies.\n\n* The paper also does not provide sufficient methodological details for calculating crucial measures, such as the Policy Occupancy Measure for model-free policies and the structure of the new policy occupancy measure.  \n\n* The evaluation results focus primarily on gaming rewards without a thorough analysis of the types and distributions of observed Feint behaviors. A more comprehensive assessment, including the underlying reasons for focusing on gaming rewards and the detailed impact of Feint behaviors, would enhance the understanding of the study's contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## [Reviewer  R6PG]\n\n### - In Section 4.2.1, the two reward functions, $R^i$, in Eqs (1) and (2). Are they the same?\nYes, the $R^i$ in Eqs(1) and Eqs(2) are the same, which are the game environment reward given a state $s_t$, the agent action $a_t^i$, and opponents actions $a_t^{-i}$. We do not intend to modify such a reward because this is an objective reward. Our key design here is to use a dynamic short-long-term way to calculate the accumulated rewards for the combination Dual-Behavior model and long-term actions. We highlight the effect of short-term Feint behaviors due to their characteristics and also incorporate long-term consideration, aiming to solve the “short-sight” and “far-sight” issue of previous reward calculations (discussed in Section 4.1).\n\n### - In Section 4.2.2, how is the Policy Occupancy Measure calculated for model-free policies, such as those used in deep RL?\nDirectly computing the exact occupancy measure for general Markov games is indeed generally intractable. However, since our goal is to reward Feint behaviors in terms of effective influence range, we focus on the discrepancy of occupancy measures introduced by Feint behaviors. We use the method introduced in [1] to approximate the maximization of occupancy measure difference between regular policy and Feint policy. Their key observation is that although tracking the policy in the parameter space of models is intractable, there is a one-to-one correspondence between the policy and occupancy measure. Using the inspiration from expert distillation, which treats a neural network $f_{\\hat{\\theta}}(s, a)$ as being trained to fit a randomly initialized fixed one on state-action pair $(s, a)$ dataset. By assigning an intrinsic reward $r_i^{int}(s, a) = \\|f_{\\hat{\\theta}}(s, a) - f_{\\theta}(s, a)\\|$ to the player, players are encouraged to aggressively explore state-action with large prediction errors. Thus, the occupancy measure of new policy is pushed to be different from the old ones.\n\n### - In Section 4.2.3, the discussion of the payoff matrix causes confusion. And how is a $a_{M+1}$ structured as an $M*N$ matrix after introducing the Feint policy?\nWe thank the reviewer for pointing out the confusing notation and we sincerely apologize for our oversight here. We provide the following corrections for this part:\nAn agent maintains a pool of policy \\$P_i = \\{\\pi_i^1, …, \\pi_i^M\\}\\$ while an opponent maintains a pool of policy $P_{-i} = \\{\\pi_-i^1, …, \\pi_-i^N\\}$:\n$$\nRew_{collective-diversity}(\\pi_i^{M+1}) = D(a_{M+1} \\mid\\mid A_{P_{i} \\times P_{-i}})\n$$\n\n$$\na_{M+1}^T:=(Rew_{collective}(\\pi_i^{M+1}, \\pi_{-i}^j))_{j=1}^N.\n$$\n\nIn our actual implementation, the payoff matrix (the policy pool $P_i = \\{\\pi_i^1, …, \\pi_i^M\\}$) is the regular policy sampled from the regular policy model while the $\\pi_i^{M+1}$ is the Feint policy sampled from the Feint policy model. We use the approximation method introduced in [1] to compute the divergence between vector $a_{M+1}$ and the payoff matrix, which calculates the lower bound of occupancy divergence:\n$$\nRew_{collective-diversity}(\\pi_i) \\geq \\frac{\\sigma^2_{\\min}(A)(1 - 1^\\top (A^\\top)^+ a_{M+1})^2}{M} + \\left\\| (I - A^\\top (A^\\top)^+) a_{M+1} \\right\\|^2\n$$\nwhere $(A^\\top)^+$ is the Moore-Penrose pseudoinverse of $A^\\top$, and $\\sigma_{\\min}(A)$ is the minimum singular value of $A$.\n\n### - Need to specify whether the implementations in Appendix D are original or derived from prior research in the main text.\nWe thank the reviewer for this suggestion. We will add such clarification in the main text in our revision. We clarify that the MARL framework is derived from existing research while all the scenario crafting and our Feint formalization are implemented by ourselves.\n\n### - What are the types and distributions of Feint behaviors\nWe respectfully clarify that: the authors are not clear about the questions. Would you mind clarifying more on this?\n\n### - Citation Style\nWe thank the reviewer for pointing out the citation style. We will fix them in our revision.\n\n### Reference\n[1] Xiangyu Liu, Hangtian Jia, Ying Wen, Yaodong Yang, Yujing Hu, Yingfeng Chen, Changjie Fan, and Zhipeng Hu. Unifying behavioral and response diversity for open-ended learning in zero-sum games. CoRR, abs/2106.04958, 2021. URL https://arxiv.org/abs/2106.04958.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uRlM0cjfZO", "reviewer": "Reviewer_8F3g", "summary": "This paper proposes a method to generate feint behaviors and strategies so that the agent can obtain temporal and spatial advantages when competing with opponents. Specifically, this paper first describes the characteristics of feint behaviors at action-level and proposes a Feint behavior template generator called Palindrome-directed Generation that extracts subsets of semi-symmetrical actions from an offensive behavior and synthesizes them as a Feint behavior. Then, the paper proposes a dual-behavior model, which considers the physical constraint and effectiveness when constructing effective combinations of feint behaviors and follow-up actions. Moreover, the paper formalizes the feint implications at strategy-level and proposes rewards reflecting the temporal, spatial, and collective impacts. Finally, the paper provides an implementation scheme of feint behaviors that can be integrated into the existing MARL frameworks.", "questions": "1. When combining feint behaviors with follow-up actions, this paper proposes to first select an intended follow-up high-reward behavior, and the end physical state of the feint behavior is constrained to be close to the starting physical state of the follow-up behavior. How to implement this process for MARL framework and what is the time complexity when using the palindrome-directed feint behavior template?\n\n2. In section 4.2.1, the paper proposes that \"the Dynamic Short-Long-Term temporal impacts of Feint shall be (1)...(2)...(3)...\". How do the following design of $Rew_{temporal}$ achieve this?\n\n3. In Equation (1) and (2), should $\\alpha_t$ and $\\beta_t$ be inside $\\sum_t$?", "limitations": "Please see the weakness section.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper provides the formalization of feint behaviors and the corresponding palindrome-directed feint behavior template generator. The analysis of feint behaviors and their combination with follow-up actions makes sense.\n\n2. The proposed feint behavior implementation scheme can be easily integrated into existing MARL frameworks.\n\n3. The paper is organized well and the presentation is easy to follow.", "weaknesses": "1. This paper claims that most offensive behaviors can be decomposed into three action sequences, which are Stretch-out Sequence (Sequence 1), Reward Sequence (Sequence 2), and Retract Sequence (Sequence 3) and it proposes a feint behavior template generator based on palindrome structure. This design should be effective for many feint behaviors composed at the action level. However, more complex feint behaviors belonging to the strategy level may not consist of the above three parts and thus the palindrome structure will not work. This paper may specify more clearly the scope of feint behaviors to which the proposed scheme can be applied.\n\n2. The discussion in the experiment section is relatively weak. For example, according to Appendix E, when applying the feint behavior implementation to MARL frameworks, the implementation trains a feint policy model to generate feint behaviors. The experiment results show that the feint behaviors can increase task returns. However, it is unclear whether the actions generated by the feint policy model are really feint behaviors. It is also possible that the feint policy model trained based on $Rew_{collective}$ manages to find certain good actions to beat opponents. This paper may have more discussion about this based on a clear definition of feint behaviors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## [Reviewer 8F3g]\n### - Whether the actions generated by the Feint policy model are really Feint behaviors?\nThe actions generated by Feint policy model are indeed Feint behaviors. As discussed in Appendix E, the Feint policy model is constraint by intended high-reward behaviors and corresponding available Feint templates to choose actions. The collective rewards are effective under such constraints, and thus this reward contributes to selecting good actions of Feint behaviors in a Dual-Behavior model.\n\n[### - How to implement the process of “Palindrome-directed Feint behavior generation” in MARL framework?\nWe discussed this implementation in Appendix E, which detailed explain how the Palindrome templates and Dual-Bahavior models are fused into our MARL frameworks. \n\n### - How does the design of Rew_temporal achieve the 3 points of Dynamic Short-Long-Term temporal impacts discussed in Section 4.2.1?\nThe Design of Rew_temporal achieves the 3 points discussed in Section 4.2.1 as follows:\nWe use large weighted accumulation of short-term rewards for Feint behaviors and the follow-up high-reward behaviors (the Dual-Behavior model) to address that strong correlation of Feint behaviors and follow-up high-reward behaviors.\nWe incorporate long-term reward calculation to avoid the short-sighted issue of simply focusing on short-term rewards, which address the explicit or implicit long-term effects of Feint cycles. This setting helps to stabilize the agents and prevent them from being too aggressive.\nThe design of dynamic short-term and long-term thresholds naturally reflects our action-level design for different lengths of Feint templates. When choosing different templates and composing different Dual-Behavior models in gaming iterations, the evaluation of short-term and long-term thresholds is dynamically adjusted according to the length of Dual-Behavior models.\n\n### - In Equation (1) and (2), should $\\alpha_𝑡$ and $\\beta_𝑡$ be inside $\\sum t$?\nWe thank the reviewer for pointing out our oversight on this and we sincerely apologize. The $\\alpha_𝑡$ and $\\beta_𝑡$ should indeed be inside $\\sum t$. We will fix this notation in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their constructive feedback, and address all concerns.

We address the following general questions here, and we address individual questions separately under corresponding reviews.

## General Questions

### [Reviewer: Wu2K, CkiB, R6PG] Lack of Discussion (and Identifications of) Limitations 
We hereby clarify that: the successful formation of Feint actions and strategies can be considered as a concrete example for indicating the potential unsafety of Machine Learning models. Therefore, we preserve the discussion (and identification of limitations) to ensure the knowledge sharing would NOT cause any harm. We will add a section for the discussion of negative impacts, and identifications of limitations in terms of our approach on the revision. 

### [Reviewer: Wu2K, R6PG, 8F3g] Whether Palindrome-directed Feint Formalization can cover all types of Feint behaviors? How to adapt such formalization to other Feint behaviors?
Currently, we only consider the usage of Palindrome for the purpose of Feint formation only. As for its generality, we believe it can be applied to other contexts (one can consider Feint's behaviors as a three-dimensional example from the high-order perspective). One potential approach to generalize our method is to form similar symmetries of actions under a formalization of topological manner, and then these shrunk actions can be inserted using the temporal variations. Note that we only consider this work as a proof-of-concept of other concrete examples, therefore we do not take credit for other concrete examples.

### [Reviewer: Wu2K, CkiB] How to have broader comparisons with other deceptive strategies and models?
We have addressed the generality of our approach above. As for the comparisons, we assume this is a case-by-case comparison - therefore, based on the current knowledge, we are unable to provide concrete elaborations. One hypothesis from the authors is that: given any actions may be formed in the explained manner, the tolerance of temporal variations can be an interesting metric for further investigations (in the context of the questions, asked by these two reviewers).

### [Reviewer Wu2K, CkiB] What is the broad applicability/scalability of the Feint formalization in a wide range of games, and possibly, to non-gaming domains, such as robotics or real-world strategic planning?
We consider Feint as a set of ACTIVE actions from agents. The generalization of our approach to non-gaming domains shall also take the differentiations of active and reactive actions into account. Therefore, this shall vary across different contexts. As we explained above (in terms of the generalization): though actions can be formed under our approach, we believe an outstanding point would be temporal variations.

Meta-review: This paper presents a comprehensive study on feint behaviors in multi-player games. Its key contributions lie in being the first to provide an in-depth formalization of feint behaviors at both the action-level and strategy-level, whereas past works have only introduced these concepts as a proof-of-concept. The paper first proposes a method for the automatic generation of feint behaviors, which are then combined with follow-up actions to form complete strategies. The experiments, conducted in a custom boxing game scenario using MARL, demonstrate that the proposed method improves scores when incorporating feint behaviors.

The major concern raised by the reviewers was the generalizability and scalability of the proposed method. Since all experiments were conducted in a custom boxing environment, reviewers questioned whether the approach would be effective in other environments. Although the authors were unable to provide new experiments in other environments during the rebuttal, they adequately addressed these concerns for the reviewers. Given that this is the first work to provide a comprehensive study on feint behaviors, I believe that experimenting only on the boxing environment is acceptable for this initial exploration.

In addition, while reading the paper, I found two additional issues that I believe can be easily resolved by the authors and should be carefully addressed in the final version. First, regarding reproducibility, the authors did not provide the code for their implementation. In the checklist "Open access to data and code," the authors answered No, stating that their main contribution is a unified formalization of Feint behaviors which could be widely adapted to most commonly used MARL frameworks. While they provided a detailed description of the codebase in Section 5.1, I find that this description is still insufficient for reproducing their work. To enhance reproducibility, the authors should consider either releasing their code (with the used boxing environment in the experiment) or providing pseudocode in the final version. Second, in the checklist "Experiment Statistical Significance," the authors also answered No, explaining that their experiments aimed to show that feint behaviors could significantly increase agents' rewards and game diversity by comparing these values to those of other agents in their plots. Consequently, they did not include error bars, claiming that the exact reward values are not their main concern. This claim is contradictory. Since the experiments compare agents' rewards to those of other agents, it is essential to include error bars to demonstrate that the results are statistically significant. The authors should include error bars in the final version.

Overall, I agree with the reviewers regarding the novelty and potential impact of this work, particularly its innovative approach to formalizing and implementing feint behaviors. Although feint behaviors have not been widely explored at NeurIPS, this work represents a step forward in the field.
Decision: Accept (poster)