Improving Adversarial Training for Two-player Competitive Games via Episodic Reward Engineering
Authors: Siyuan Chen
Fuyuan Zhang
Zhuo Li
Xiongfei Wu
Jianlang Chen
Pengzhan Zhao
Lei Ma
Jianjun Zhao
Venue: Accepted by TMLR ()

Abstract: In recent years, training adversarial agents has become an effective and practical approach for attacking neural network policies. However, we observe that existing methods can be further enhanced by distinguishing between states leading to win or lose and encouraging the policy training by reward engineering to prioritize winning states. In this paper, we introduce a novel adversarial training method with reward engineering for two-player competitive games. Our method extracts the historical evaluations for states from historical experiences with an episodic memory, and then incorporating these evaluations into the rewards with our proposed reward revision method to improve the adversarial policy optimization. We evaluate our approach using two-player competitive games in MuJoCo simulation environments, demonstrating that our method establishes the most promising attack performance and defense difficulty against the victims among the existing adversarial policy training techniques.

Published in Transactions on Machine Learning Research (11/2025)
Improving Adversarial Training for Two-player Competitive Games via Episodic Reward Engineering
Siyuan Chen The University of Tokyo
alachaa@@g.ec..utobp..oo
Fuyuan Zhang Zhejiang University
fuguaenshang@bzjhcccd..r
Zhuo Li Kyushu University
shuoli. Choli..codemie@@maal.coom
Xiongfei Wu University of Luzeebbourg
siongfei.wu@unite
Jianlang Chen Kyushu University
Pengzhan Zhao Hebei Normal University
Shaopengghannhhebuuuu..co
Lei Ma
ma.leiGacm.ory
The University of Tokyo, University of Alberta
Jianjun Zhao
zhoo www.shtttttkuuauaaaaaa ac.jp
Kyushu University
Reviewed on OpenReview: https: openreview. net/ forum? id= 24XtJWJCCK
Abstract
In recent years, training adversarial agents has become an effective and practical ap- proach for attacking neural network policies. However, we observe that existing meth- ods can be further enhanced by distinguishing between states leading win or lose and encouraging the policy training by reward engineering prioritize winning states. In this paper, introduce novel adversarial training method with reward engineer- ing for two-player competitive games. Our method extracts the historical evaluations for states from historical experiences with an episodic memory, and then incorporating these evaluations into the rewards with our proposed reward revision met to improve the adversarial policy optimization. We evaluate our approach using two-player compet- itive games in MuJoCo simulation environments, demonstrating that our method estab- lishes the most promising attack performance and defense difficulty against the victims among the existing adversarial policy training techniques. The source code is available at com http://hhbbb.cmmlllaaccaaa///ppe www.rrewwwwwedeeeeeeeeeeeeeeeeee
1
Introduction
Prior work shows that deep reinforcement learning (DRL) policies can be vulnerable to adversarial at- tacks Huang et. al., 2017 Kos & Song 2017). Most existing attacks on DRL policies are executed by
FFuyuan Zhang the corresponding author and is with the State Key Laboratory Blockchain and Data Security, Zhejiang University, Hangahou, China.

---

Published in Transactions on Machine Learning Research (11/2025)
scarching the adversarial examples and manipulating the environment (Huang al 2017 Kos & Song.
2017
Nguyen & Reddi 2019 However, such adversarial examples may not be applicable in the real
world (Gleave et al. 2020 Recently, training adversarial agents as attackers to DRL policies in two-player
games has been proven effective and practical |Gleave et al.] 2020 Wu et al. 2021 Guo et al.) 2021 Bui
jet al. 2022) These kinds of attacks first reduce the two-player environments to single-player environments
by fixing the victim agents, and then train the other agent to be an adversarial agent which can be trained by conventional single-agent policy training methods. Known as adversarial policy training, these attacks generate natural observations that adversarial to the victim agents, achieving significant results.
While the aforementioned adversarial policy training methods have proven effective, there is still room to improve adversarial training by providing the training process with higher quality environmental rewards to learn. Our insight is simple: utilizing the adversarial agent's historical experiences to distinguish between states that lead to wins or loses, thereby assigning higher rewards to the winning ones and lower rewards to the losing oncs. Based on this insight, we introduce novel adversarial policy training approach that leverages the analysis of information from past episodes to assess game states and improve the environmental rewards with those assessments to assist the adversarial agent in achieving better performance.
In this paper, we propose an adversarial policy training method with reward engineering mechanism for two- player competitive games. Inspired by previous works on improving the rewards for policy learning using episodic control (Blundell et al. 2016 Pritzel et al. 2017 Li et al. 2023), our method develops a neural network-based episodic memory to store historical experiences and generate state evaluations corresponding to the game outcomes. We then design a conditional reward revision method to improve the environmental rewards based on those evaluations. In our experiments, we evaluate our method on two-player competitive games in MuJoCo domains (Todorov et al.) 2012) and compare it with state-of-the-art adversarial policy training approaches (Gleave et al.) 2020 Guo et al. 2021 Wu et al.) 2021). Our experimental results show that our method establishes the most promising attack performance and defense difficulty.
In summary, this paper makes three contributions. First, we propose a novel adversarial policy learning framework with reward enhancement for two-player competitive games. Second, we introduce a neural network- based episodic memory that leverages historical experiences to evaluate states, along with a reward revision approach that incorporates these historical evaluations into environmental rewards. Third, our work demonstrrats that by identifying and highlighting the winning states based on historical experiences with reward revision, adversarial agents can achieve higher win rates against fixed victim agents and may have the potential to possess the capability to explore a more effective strategy by integrating actions from past successful strategies to defeat the victim, which is demonstrated in one Mujoco game environment in our experiments.
2
Related Work
2.1 Adversarial Attacks on DRL Policies
Previous attacks against DRL policies mainly focus on manipulating the environment to fail the victim agents. One type of attack focuses on perturbing the victim's observations, forcing its policy network to
output sub-optimal actions, and thus failed the victim agent (Russo & Proutiere 2019 Sun et al. 2020 Zhang ett al. 2021 Madry et al.] 2018 Pattanaik et al.] 2018 Pan et al.] 2022 Zhao et al. 2020).
Another kind
of attack directly perturbs the trajectory of the victim, specifically the actions the victim agent takes (Lee
ct al. 2020 Pan et al.) (022) or the rewards it receives (Ma et al. 2019: Yang ct al. 2019 Lykouris et al., 2021
effectively attack the victim. However, the above atlacks are argued to be unrealistic since the
environment cannot be manipulated (Gleave et al. 2020 Guo et al. 2021 Wu et al. 2021).
Unlike the above attacks, to simulate the real-world sccnarios, Gleave et al. successfully trained adversarial agents by PPO algorithm (Schulman et al. (017) in two-player competitive games under a striet zero- sum assumption and demonstrated the effectiveness of training adversarial agents against fixed black box victims (Glcave et al. (020). Wu al. further improved the attack performance by exploring the minimal observation differences of the shared environment to maximize deviations of the victim actions (Wu et al., 2021). Guo el al. relaxed the zero-sum assumptions of previous works and demonstrated that such attack
2

---

Published in Transactions on Machine Learning Research (11/2025)
could be achieved by maximizing the gap between the adversary and victim rewards which are approximated by observations of the adversarial agent (Guo et al. 2021). On the other hand, Bui et al. adopted imitators
of the victim policies learned by imitation learning algorithms (e.g., GAIL (Ho & Ermon 2016)) ) to roll out
the victim actions for the attacker and reached better performances (Bui et al. 2022). However, this attack is based on the specification that the victim's 's actions are visible and accessible to the imitators.
2021 wherein we have control solely over the adversarial agent and treat the victim agent black box, rendering its observations, actions, and rewards inaccessible. Meanwhile, unlkke (Gleave et al. we concentrate on leveraging the historical experiences from adversarial agents to emphasize the winning states to improve the adversarial policy training. Our experiments in Section such improvement.
This paper adopts the same setting as (Gleave et al., 2020 Guo et al.) 2021
Wu et al.
2020
Guo et al.
2021 Wu et al.) 2021
4.2 show the effectiveness of
2.2 Reward Engineering
Reward design plays a pivotal role in reinforcement learning, where poorly structured rewards can stall policy improvement. Consequently, various methods (Andrychowicz et al. 2017 Schaul et al. 2015 have been proposed to improve the reward signal. For example, Hindsight experience replay (Andrychowicz et al. 2017) sets an alternative goal for the state transitions of an episode to create a learning signal from failed attempts, and then modifies the rewards based whether the alternative goal is reached.
In recent research, episodic control (Lengyel & Dayan 2007 shows its effectiveness modilying environ- mental rewards to address sample inefficiency in various tasks, such as multi-agent tasks (Zheng et al. 2021) model-based reinforcement learning (Le et al.) 2021), and continuous control (Zhang et al. 2019 Kuznetsov & Filchenkov 2021). Previous works primarily adopt a tabular episodic memory to save experiences of past scenes, leveraging the information gained during exploration and retrieving past experiences of similar scenes to expedite policy optimization (Blundell et al.) 2016). This memory uses the state a key and a measurement of that state (which can vary by method, c.g., Q-value) as the value, storing these key-value pairs. Then, listance-based analysis (e.g., KNN) is adopted to retrieve summary statistic of similar states from the episodic memory, and the retrieved statistic can be used to guide the reward engineering to improve the training process (Hansen et al. 2018).
This paper adopts episodic control as the base since it shares a similar empirical insight of using historical experience to guide learning. Unlike methods that dircctly add statistics from cpisodic memory to the reward, we apply a conditional revision aligned with outcome-aligned learning signals and rely on neural episodic memory for efficiency. Compared with state-of-the-art episodic control method NECSA (Li et al. 2023), our approach performs better with less time cost (See Section 4.3
3 Methodology
In this work, we propose an adversarial training approach with reward engineering for training adversarial agents. Our method utilizes the conventional adversarial policy training framework and improves the rewards used for training. The workflow of our approach is shown in Figure] First, the adversarial agent interacts with the environment and gathers information (states, actions, rewards) from the episodes. This information is adjusted using our proposed reward revision method, and then saved in the experience storage (e.g., replay buffer). We calculate the objective function based on the revised rewards sampled from the experience storage and update the agent's 's policy with the objective function. In the following, we mainly elaborate on our reward engineering method based on historical experiences analysis.
3.1 Adversary in Two-player Markov Game Environment
A two-player Markov game environment can be modeled as = (S, Ap), (Ra, R2)). Here, we use a and v to represent the adversary and victim respectively. S represents a state set, and both Aa and A, are
action sets. T denotes joint state transition function T : S x A₂ x A, A(S), where A(S) is a probability
3

---

Published in Transactions on Machine Learning Research (11/2025)
Single-agent Policy Training (PPO)
Reward Engineering
Adv-Agent
Environment
Abstract Vector
Episodic Memory
Rewards
States & &cttion
LST M
States & Actions
Update
MLP
BUF
Sample
Historical Score
Episodic
Feedback
Objective
Experience
Function
Storage
Figure 1: The workflow of our adversarial policy training. The process consists of four main steps: (1) The adversarial agent interacts with the environment and gathers episode information (states, actions, rewards). (2) The episodic memory collects the information to generate historical evaluation for states. (3) The original environment rewards are revised using this evaluation. (4) The agent's policy updated based on the revised rewards.
distribution on The reward function R; 55 Aa A. X - depends on the current state, actions (aken by both agents and the next state.
Gleave et al. discovered that by fixing the victim agents, the two-player competitive games can be reduced to singlo-player games and one agent can be trained as an adversarial agent with conventional single-agent policy training methods (e.g., PPO) to attack the victim (Gleave el al. 2020). The training for the adversarial agent to defeat the fixed victim agent called adversarial policy training. Under this setting, the two- wwo-player game can be regarded as singlo-player MDP: E, (S, Aq, Ta, R²,) as the victim policy can be treated part of the environment. S becomes the state set of the adversary and the state transition function and reward function change to T. 5 x A, 5 - A(S) and R. S x Aa S RR.
Our threat model We use the same setting Gleave et al. and assume that our threat model has control over the adversarial agent and black-box access to the information of the victim agent. The adversarial agent can only interact with the environment, which the common practice in adversarial policy training. We also assume that the environment can only terminate with a terminal signal indicating three possible outcomes:
adversary win, adversary loss, or tie.
3.2 Reward Revision Based on Historical Experiences
The environmental reward R2 obtained by the adversarial agent only includes the evaluations from a single game and does not contain evaluations from the historical experiences. To enhance the adversarial policy training, our key insight is integrate state evaluations from historical experiences into the rewards, thereby providing the adversarial policy with more comprehensive rewards for learning.
As two-player Markov game is not deterministic, past episodes starting from the same state may include different state sequences. Therefore, we use state sequences, referred to as patterns, as the basis for perfor- mance evaluation in past episodes. Our choice of utilizing state sequence for performance evaluation aligns with existing research (Sutton Barto 2018 Li al. (003),, which has proved that failures in adversarial games are often the result of series of poor decisions rather than isolated states. Based on the qualification of these patterns from historical experiences, we assign higher rewards to states that lead to good pat-
4

---

Published in Transactions on Machine Learning Research (11/2025)
terns and lower rewards to states that lead to bad patterns, thereby integrating evaluations from historical experiences into the rewards.
3.2.1 Evaluation of Pattern Performance
As mentioned above, a pattern refers to a sequence of states within an episode. For a given episode e, a k-step pattern starting from time step i, denoted as Pt, refers to consecutive states in e, i.e., Pt = St S-+k-1 For example, pi , exemplifies a k-step pattern from the first state 81. Notice that a state can be considered special case of a pattern, specifically 1-step pattern.
To evaluate the past performance of the patterns, we propose that a pattern can be considered high- performing if the past episodes containing this patlern result in more wins than losses. Conversely, a pattern can be considered poor-performing if the past episodes containing it result in more losses than wins. Based on this idea, we introducc a historical score for each pattern to quantify its past performance, defined as the average cumulative reward received in past episodes that include that pattern. The cumulative reward for episode, which quantifies the adversarial agent's performance, tends to be higher in episodes where the agent wins than in those it loses. As the performance of adversarial agents improves, these cumulative rewards increase. Therefore, taking the average cumulative reward across all episodes that contain a particular pattern captures the pattern's historical effectiveness. During training, we store each pattern alongside the cumulative reward of its corresponding episodes an episodic memory. The memory then returns the average cumulative reward as the historical score for that pattern. The historical score h spor can be calculated as:
h score(pt) ( Rp.
where Rp, the average cumulative reward of past episodes containing Pt and M is an episodic memory we proposed to collect and analyze patterns in past. episodes. In Section 3.3.1 1 will further explain how we implement the episodic memory.
3.2.2 Conditional Reward Revision
Based on the historical scores of patterns, we incorporate the historical cvaluations into rewards by reward revision. As revising rewards may lead different optimal policies, inspired by polential-based shaping functions (Ng et al. [999), we propose episodic feedback to keep the reward signal mean-zero to preserve the policy-optimality. It is defined as the difference between the historical score of a pattern and the average cunulative reward of all past episodes:
8(p)) h h__score(pt) R.
(2)
where h score(p:) is the historical score of pattern and R is the average cumulative reward of all past episodes. The proof of the policy-optimality guarantee is provided in Appendix A..I On the other hand, episodic feedback also contributes to the improvement, of training. Semantically, score(pt) represents the historical evaluation of one pattern while R represents the agent's global average performance. Therefore, episodic feedback transforms the reward shaping signal from an absolute measure of quality into relative one. Instead of just rewarding 'good' patterns, it specifically rewards patterns that are better than the agent's 's current average performance. This provides more focused and informative signal for credit assignment, guiding the policy to prioritize patterns that lead to more exceptional outcomes.
With the episodic feedback, we conditionally add the episodic feedback a pattern to the reward of its initial state. Specifically, for an episode e, depending on the outcome of (whether the adversarial agent wins), we revise the rewards of the initial states of patterns e in two cases. Assume that s, is the initial state of pattern pt in episode e, whose reward is rı, and 8(pt) is the episodic feedback for Pt, the conditional
5

---

Published in Transactions on Machine Learning Research (11/2025)
reward revision can be formulated as:
+ ó(pt) x 6,
if the adversarial agent wins and ó(p:) > O,
ft + 5(pt) x, if the adversarial agent loses and 8(pz) < 0, otherwise,
(3)
Tt.
where i is the revised reward and 6 is a coefficient used to regulate the magnitude of encouragement and punishment. After the revision, we update Tt with fr.
The reason we only revise the reward in the two cases mentioned above is that the revision must adhere the win-loss rules of the two-player competitive game environment, even though this environment reduces to single-player game environment during the training process. In competitive game, we believe only states from a winning episode of the adversarial agent should be rewarded (o(ps) > 0), while states from the losing episode should be penalized (ō(p.) < 0). We will further analyze other cases in Section 4.5.3
3.3 Reward Revision Improving Adversarial Policy Training
3.3.1 Neural Network-based Episodic Memory
As discussed in Section 3.2.1 we propose using an episodic memory to generate historical scores for patterns basod on past experiences. Since reward revision with episodic memory adds to the total training time, we reduce ils overhead by adopting neural nelwork-based architecture shown in Figure 1 where episodic memory consists of LSTM followed by multi-layer perceptron (MLP). The LSTM encodes patterns into abstract vectors, and the MLP aggregates these encodings to produce historical scores. As reported in Section 4.2 the effectiveness shown in our experiments indicates that this simple architecture is sufficient for advessrrial training in two-player competitive games.
During the training, new episodes are used to update the episodic memory. Assume a newly produced episode has a cumulative reward of R and contains the set of patterns P. The episodic memory M is then trained to map all the patterns in P to the same cumulative reward R. Note that for pattern in P. the learning target is the historical score of the pattern, which is not a fixed value and will change as new episodes occur. Then, during the reward revision, M predicts historical scores for the patterns as Equation to generate historical evaluations for reward revision. Details of our implementation of the episodic memory including the architecture, the forward process and the update process can be found in Appendix A.4
3.3.2 Group-based Episodic Feedback
In practice, find that later-generated episodes exhibit higher win rates than earlier generated episodes during the training process. To achieve better performance in finding the optimal policy, we compare the historical score of pattern with the average cumulative reward from recently generated episodes when computing the episodic feedback. To achieve this, we divide the episodes into groups of size n, where n is a parameeer,,, and calculate the average reward of past episodes in the group. The average reward of the first episodes in group m can be calculatod by
Rm
R
i 1sisn.
(4)
Thus, we compute the episodic feodback of pattern pe in the ith episode of group m by
8(pr) score(pt) R." m
(5)
which is implemented in our experiments.
3.3.3 Adversarial Policy Training with Reward Revision

---

Published in Transactions on Machine Learning Research (11/2025)
Algorithm Adversarial policy training with reward revision.
Input: A: Adversarial Agent, E: Environment, M: Our episodic memory, B: Experience Storage, O:
Objective Function from the Fundamental Training Method
Parameter: h: Pattern Length, Group Size, e: Revision Coeflicient
Output: A: A Well-trained Adversarial Agent
while Training does not reach the maximum step do A interacts with E and generale state 8, action a, reward r.
A.ad(((a)) R.add(r). 4: if an episode ends then
5: P Pattern(k,
6:
Ream < Cumulative (ecpllatteeeeeer((( rewardd
7:
M. M.update(P Reum)
H Score(P) < M(P)
m GroupIndex, 4- Fgisode/ndeewWlhhnnGn
10:
R2 + Average Reward(i, m)
((P) + Episodic Feedback( H Score(P), R.))
11:
12:
R' + Reward Revision (P)),c)
13:
B - S, R'
14:
Clear S, A,
15: end if
16: if Check Update() is true then
17:
Experiences t Sample(B)
18: O(Experiences)
19: A.update(O)
20: end if
21: end while
22: return A
We implement our policy training as Algorithm First, we have the adversarial agent interact with the environment and generate states, actions and rewards (Tine 2-3). When an episode ends, we extract patterns from the episode (In practice, we use sliding window) and calculate the cumulative reward the episode (Line 5-6). We then update the episodic memory with the patterns and the cumulattiv reward (Line 7), and predict the historical score for each pattern with the episodic memory(Line 8). Subsequently, we calculate the average cumulative reward of the group with Equation 4 (line 9-11), and then utilize the average cumulative reward to calculate the episodic feedback for each pattern following Equation 5 (Line 12). With the episodic feedbacks, the rewards of the states could be revised by Equation 3 (Tine 13) under the condition stated in Section 3.2.2 After the reward revision, we store the states, actions and the revised rewards into the experience storage (Line 14). When the update condition satisfied (e.g., storage is full), we will sample some data from the storage and calculate the objective function from our selected policy training method(e.g, PPO) (Line 17-19). The adversarial agent will be updated with the objective function (Line 20). Iteration will end when the maxinum training step is reached.
4
Evaluation
4.1 Main Experiment Setup
In our main experiments, we evaluate our approach along two dimensions, attack effectiveness and defense robustness, by asking two questions: (Q1) Does it secure higher win and non-loss (win plus tie) rates? (Q2) How difficult is it for opponents to defend against the adversarial strategiess it learns?
To answer Q1, we employ PPO (Schulman et al.] 2017) as the fundamental singlo-agent policy training method train adversarial agents against well-trained Zoo agents (Bansal al.) 2018). We take three state- of-the-art approaches (Gleave et al. 2020 Wu et al. 2021 Guo et. al. 2021 baselines. Notably, Gleave

---

Published in Transactions on Machine Learning Research (11/2025)
et ol. 's method incorporates PPO for adversarial policy training without modifying PPO's fundamental mechanisms, thus effectively serving as PPO baseline. To ensure fair comparisons, we evaluate the met using the same two-player competitive games in the MuJoCo robotics simulator: VouShallNoodPsssllmooo KickAndDefend, SumoAnts and SumoHumans, running 5 seeds per environment. In the following, we briefly introduce these four environments.
ManShallNotPasHHmmaan places two humanoid robots in a corridor. One agent, the runner, must Traverse the hallway from the start line to a finish plane at the far end; the other, the blocker, aims to stop that traversal for the entire episode. All forms of physical contact are legally allowed. The game ends immediately when the runner crosses the finish lane (the runner wins) or when the fixed time limit is reached without crossing (the blocker wins).
KickAndDesend sets up humanoid kicker, humanoid defender, and single soccer ball on flat rectangular pitch. The kicker's sole objective is to propel the ball through goal mouth, while the defender tries to block clear it before that happens. Game concludes as soon as the ball fully crosses the goal line (the kicker wins) or when the time limit expires with no goal scored (the defender wins).
SumoAnts drops two four-legged "Ant" robots onto a raised circular platform. Each agent tries to remain inside the ring while forcing the opponent off the edge. A match ends when (1) the timer expires with both ants still inside the circle (tie); (2) a robot leaves the ring without having been touched by its rival at that moment (a self-out, tic); (3) both robots exit in the same simulation step so the engine cannot determine who went out first (tic); and (4) one robot leaves the platform (the other one wins).
Sumollumons follows the same ring-out rules as SumoAnts but swaps the quadrupeds for full 27- degree-offfreedoom humanoids. The objective also remains the same SumoAnts: stay on the platform and make the opponent step or fall outside.
For the hyper-parameters, we set the 0.1, pattern length k as 3 and Group Size 100. Further ppprr-paraetter analysis is detailed in Appendix A.3
To investigate Q2, we conduct two different defense robustness tests. (1) Victim retraining: we retrain the victim agents against the fixed adversaries and track whether the adversaries' win and non-loss rates decline as the victim adapts. (2) Masked observation defense: we employ masked victim agents proposed by Gleave et.al. (Gleave et al. 2020) as a simple defense method against adversarial training in two-player competitive games. In this experiment, the victim agent's observations of the adversarial agent are fixed, simulating limited perceptual capabilities to avoid adversarial actions. Together, these experiments allow us to evaluate the robustness and effectiveness of the adversarial strategies developed by our approach.
All experiments were conducted on a system rumning Ubuntu 20.04.4 with Intel Xoon E5-1650 v4 CPU, NVIDIA GeForce RTX 3090, and 128 GB of memory.
4.2
Main Results
The comparison of the win rates and non-loss rates between our approach and the baseline approaches are sunmarized in Figure 2 We can observe that our proposed method reaches 88% and 89% win rates and outperforms the baseline methods significantly in YouShallVodPassHumaa and KickAndDefend. In Sumolfumans, method also surpasses all baselines. These results indicate that by leveraging historical experiences to highlight the high-performing states, our agents demonstrate higher sample efficiency and have more potential to discover effective adversarial policies to defeat victim agents. In Sumo since the win rates of all agents against the victim are far below 50% we use the non-loss rates to measure the effectiveness of our method. From Figure [() we observe that in Auto the non-loss rate of our agent is still able to surpass that of the agent trained by Guo al., which exhibits the second highest non-loss rate.
8

---

Published in Transactions on Machine Learning Research (11/2025)
Our
Gleave ct.al.
Gui et.al.
Wo ct.al.
100
Yo./ShallKorhhassHuaaass
100
KirkAndDefand
Average Winning Rote (%)
100
SumoAnts
Average Winning Rate (%)
100
Su SumoHumans
Average Winning Rate
75
75
75
75
Average Winning
50
50
50
25
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Time Steps (207)
Time Steps (2e7)
Time Steps (2c7)
Time Steps (3e7)
(a) The comparison of win rates.
100
YouJhhallHerrbssHuuaas
100
KiekåndDeffnd
Average Non-loss Rate 1%2
100
SumoAnts
Average Non loss Rate 196)
100
Su SumoHumans
Average Non rloss Rate
Avorage Non Rate
75
75
75-
75
50
50
50
50
25
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0
Time Stops (207)
Time Steps (207)
Time Steps (267)
Time Stops (3o7)
(b) The comparison of non-loss rates.
Figure 2: The performance of our adversarial agents and baseline adversarial agents in each environment. Dashed lines represent the highest rates of each agent and non-loss rate is win rate plus tie rate. More details are shown in Table and Table 6 in Appendix A.7
We share the vidcos of agents trained by our approach and baseline approaches in Appendix(A.66 A.6 and compare their behaviors. In KickAndDefend and SumoHumans, all the adversarial agents perform similar adversarial actions to trick the victim into performing abnormal behaviors. These results align with the conclusion in Gleave et al. that adversarial agents win by confusing the victim, instead of becoming a strong opponent. However, in ground, our agent simultaneously performs the adversarial action and obstructs the victim with kicking. Kicking can be viewed non-adversarial action, illustrating that, by incorporating our improved rewards, the agent is not limited to a single winning strategy. Instead, it can draw on past successes, like non- adversarial kicking action, to explore a different but more effective strategy. Additionally, in SumoAnts, agents trained by our approach and Guo et al. both jump out of the arena at the beginning since falling out of the arena without touching the opponent is considered a draw in this game, while agents trained by Gleave et al. and Wu et al. still fight. with the victim. This suggests that same as Guo et al., our agent is also capable of discovering and exploiting game imbalances.
aasSSaallootPPsHHaaaaa while the three baseline agents attack the victim by convulsing on the
To validate whether our approach is difficult to defend against, we conduct retraining experiments the victim agents using the vanilla PPO algorithm. We report the results of our adversarial agent and baseline agents against the victim agents during the retraining in Figure 3 In YouShal Possa//n, our agent maintains a relatively high win rate during the retraining of victim agents, unlike bascline agents whose win rate quickly drops to a low level. In KichAndDefend, the win rates of our agent also decreases at a slower rate compared to the bascline agents. This indicates that our approach is more difficult defend than bascline approaches.
In order to gain better understanding of the effectiveness of our method, following the approach in Gleave et al., we have our adversarial agents play games against masked victim agents, whose obscrvation of the adversary's 's position is set to a static value corresponding to a typical initial position so that the adversarial actions may not be effective. We show the performances of our adversarial agents and baseline agents against masked victims Table We observe a significant docline in the non-loss rates of the three bascline agents, while our agent maintains a high non-loss rate against the masked victim in CouShallNotPassHummaa could be attributed to the fact that our agent not only relies adversarial actions to attack the victim,
This
9

---

Published in Transactions on Machine Learning Research (11/2025)
Our
Gleave ct.al.
et.al.
Wo ct.al.
Averege Winning Ratte (96
100
KickAndDefend
Aver Non-loss Rate (%)
Average Winning Rate (96
Sumotuumrss
75
Avarage Winning Rate
73
75
50
50
0.00
0.50
0.75
1.00
800
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Time Steps (1e7)
Time Steps (167)
Time Steps (1e7)
Time Steps (1e7)
Figure The comparison between our adversarial agent and bascline adversarial agents in each environ- ment (Gleave al., 2020 Guo et al. 2021 Wu et al.) 2021) during the retraining. Specifically, we show win rates of the agents in YouShallNotPass Pastuaaann,,,, KickAndDeffnd, SunoHumans and non-loss rate in SumoAnts. The lowest rate of each agent is depicted with dashed line. More details are shown in Table 7 in Appendix A.7
Table 1: The non-loss rates of our adversarial agent and baseline adversarial agents against masked victim agents in 100 games. Each experiment has been done times. 'Before' and 'After' indicate before and after masking the victim agent. All data reported mean rounded to the nearest integer.
Ours (%)
Gleave et al. (%)
Guo et al. (%)
Wu et al. (%)
Environment
Before
After
Before
After
Before
After
Before
After
YouShalINor MonnShaalNoPPPss//unnaaa Pussans
96 + 2
73 + 4
66 + 2
0 + 0
72 + 3
0 0 0
50 + 2
0 + 0
KickAnd Defenddd
93 +
7 1
65 +
3 + 1
63 + 2
5 5 1
69 +
5 +
SumoAnts
83 + 2
81 + 2
76 + 2
69 + 4
81 # 3
78 + 1
57 + 5
53 + 4
SumoHHmaan
92 +
90 + 1
92 +
91 + 1
92 + 0
91 +
94 + 2
92 +
but also incorporates non-adversarial actions like obstructing the victim with its body to win the game. Based on this finding, we demonstrate that in YouShalllooFPust/haaaa. our agent can defeat the victim by performing adversarial and non-adversarial actions simultancously.
4.3 Comparison with Existing Episodic Control Method
Since episodic control shares a similar insight with our approach and serves as the basis for our method, to show the effectiveness of our method, we manually implement the slate-of-the-art episodic control method NECSA (Li et al.) 2023), integrating it with PPO to train the agents attack the victims. To facilitate a fair comparison,
specifically adjust NECSA also investigate how states contribute to winning losing outcomes. NECSA uses a grid-based tabular episodic memory and groups similar states into the same grid, and then adds the average evaluation of that grid to the state's reward.
Our
PPO
NECSA
Averoge Winning Rate (96)
100
YouShallNooPessttuaans
100
KicknndDeferd
Average Non-loss Rate (%c
1000
SumoAnts
100
Su noHumans
Aueroge Winning Rate
75
Auerage Winning
50
25
25
25
0.0
0.5
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
3.
Time Steps (267)
Time Steps (267)
Time Steps (267)
Time Staps (207)
Figure 4: The comparison of performances between our method and NECSA. We show win rates of the agents in ConShallNottPssFHaamma KickAndDefend, SumoHuman and non-loss rate in SumoAnts. The highest rate of each agent depicted with a dashed line. More details are shown in Table in Appendix A.7
10

---

Published in Transactions on Machine Learning Research (11/2025)
Figure ! presents a performance comparison between agents and the NECSA agents across four vironments. The experimental results indicate that NECSA is less effective in all environments and even hinders the training process in the KickAndDefend environment. During NECSA's 's training, we observe that it repeatedly rewards states from losing episodes, which contradicts the fundamental principle of reward design in two-player competitive games. We believe this suboptima imal reward engineering strategy diminishes NECSA's 's effectiveness, further underscoring the advantage of our conditional reward revision approach.
Table 2: The mean training time (rounded to the nearest hour) for a single run with PPO, our method, and NECSA.
Methods
Environment
PPO
Ours
NECSA
YouShal MouShallNotto Pass Pass
10
12
13
KifnAddDefend
22
24
26
SumoAnts
30
33
35
SumoHumans
24
29
Both our method and NECSA require additional time for state evaluation and reward engineering, but our approach leverages a neural-network--baee memory to reduce this time cost. Table compares training times for PPO, our method, and NECSA. We obsorve that our mothod's overhead is approximately two hours which is generally acceptable in practice, and shorter than NECSA's 's training time across all environments.
4.4 Experiments in StarCraft II
To further evaluate whether our method effective in a more complex two-player competitive enviromment, we conducted experiments in StarCraft IL. We begin with two agents sharing the same initial policy: one serves as the fixed victim agent, and the other an adversarial agent trained using our method and the vanilla PPO. As shown in Figure 5 the agent trained with our method achieves 10% higher win rate than the agent trained with PPO, demonstrating that approach is still effective in more sophisticated environments.
However, the performance gain is smaller than what we observed in traditional MuJoCo game environments, We suspect this is because StarCraft II has much larger state vector, making difficult for the LSTM network to encode states into suitable abstract representations. Replacing the LSTM with a more powerful architecture such as Transformer may further enhance the effectiveness our method in such complex environments.
Our PPO
PPO
100
StarCraft
75
Average Winning
50
25
0.00
0.25
0.50
0.75
1.00
Time Stegs (1e5)
Figure 5: The comparison of the win rates between our method and PPO in StarCraft IT.
4.5
Ablation Study
4.5.1
Pattern
As stated in Section 3.2.1 we evaluate patterns instead of states generate the historical scores. The length of these patterns, however, introduces critical trade-off. While longer patterns can provide richer
11

---

Published in Transactions on Machine Learning Research (11/2025)
pallern-3
paltern-5
pallern-7
pallern-
palterm-20
Airerage Wimming Rete (16)
100
YousShillloorhhsssumaans
Aureragn Winning Robo 1%
Average Non-loss Rate (%)
100
SumoAnts
Aurerage Wimming Hate (16)
100
SumoHHumaas
75
75
75
50
50
25
25
0.0
2.0
Steps (2e7)
Time Steps
(267)
Steps (3e7)
Figure 6: The comparison of performance between agents guided by historical evaluations of patterns with varying lengths. We show win rates of the agents in MusShalNNoPPssHHaamn KickAndDefend, SumoHumans and non-loss rate in SumoAnts. The highest rate of each agent depicted with a dashed line. More details are shown in Table in Appendix A.7 A.7
temporal context, they also risk suffering from the "curse of the pattern length', where an exponentially growing pattern space causes any specific long pattern to be observed so rarely that its historical score is based on too few related samples to be a reliable average, resulting in noisy, high-variance learning signal.
To investigate this trade-off, we compare the performances of agents with different input pattern lengths. In Figure G we can see that performance initially improves when moving from single states (k=1) to short patterns (k=3 and k=5), but starts to degrade for longer patterns (k=7, 10, 20), especially in YouShallNot- PossHumans and KickAndDefend. This finding confirms both the benefit of using patterns and the critical importance of selecting an appropriate length (e.g.; 3) to balance the trade-off.
Episodic Feedhack
Historical Score
Average Winning Rate (10
100
You touihhaltiorrbsssHumanss
100
KiekAndDefend
Average Non-loss Rate 1%2
100
SumoAnts
Average Winming Rate (1)
100
Su AumoHumns
75
75
75
75
Average Winning
50
25
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Time Steps (207)
Time Steps (207)
Time Stops (267)
Time Stops (307)
Figure 7: The comparison of performances between proposed training approach guided by episodic feed- backs and historical scores. We show win rates of the agents in SumoHumans and non-loss rate in SumoAnds. The highest rate of each agent is depicted with a dashed line. More details are shown in Table 10 in Appendix A.7
KickAndDefend.
4.5.2 Episodic Feedback
In Section 3.2.2 we mention that the episodic feedback of pattern is computed by the difference between the historical score and the average cumulative reward of the group containing the episode. If episodic feedback is greater than 0, the pattern can be considered better pattern than patterns in recently generated episodes, thereby enabling the adversarial agent to scarch for an optimal strategy. To show the effectiveness of episodic leedback, we use both episodic feedback and historical score to revise the reward. Based on the results shown in Figure 7 agents trained with episodic feedback achieve approximately 10% higher win rates than those trained with historical scores across indicates that, compared historical scores, using episodic feodback for reward revision enables the agent with the ability of discovering better policy.
KickAndDefend, and SumoHumans, This
12

---

Published in Transactions on Machine Learning Research (11/2025)
Our_[wo_carr twoo_
Our_ail Bur_all_case
Aurerage Winming Rate (10)
100
You.hhilllooPhrsssuaaass
KifaAndDefand
Average Non-loss Rate 1%6
1100
SumoAnts
Aurerage Winning Rate (96)
100
Sumoouumans
Avreraee Winning Rate
50
50
50
50
25
25
25
25
0.0
2.0
1.0
3.
Steps (2e7)
Time Steps
(267)
Time Steps (2e7)
Figure 8: The comparison of performances between proposed training approach with different revision con- ditions. We show win rates of the agents in Ww.ShallNotPsssHummaa KickAndDefend, SumoHumans and non-loss rate in SumoAnts. The highest rate of each agent is depicted with a dashed line. More details are shown in Table 11 in Appendix(A.7
4.5.3 Revision Condition
As noted in Section 3.2.2 do not revise rewards in certain scenarios, such as when state from winning episode for the adversarial agent receives negative episodic feedback. If we were to extend reward revision to these scenarios, some states leading to losses would be rewarded, while some states leading to wins would be penalized, potentially causing the adversarial agent to learn a losing strategy. To demonstrate this effect, we also applied reward revisions in these additional cases and present the results in Figure g We observe that while the peak performance of the "our seur__to____ two_case" agent only about 3% higher than that of the "our all case" agents, the average performance degrades substantially under full reward revision, especially in KickAndDeffend. Similar to the results of NECSA stated in Section 4.3 this drop likely ariscs because the suboptimal reward engineering hinders the agent's 's ability to learn a winning strategy, resulting in unstable learning outcomes. In fact, we also find that states in losing episodes received positive rewards and states in winning episodes received negative rewards in all four environments when we do full reward revision. This finding further shows that reward engineering must adhere to the rules of two-player competitive games.
5 Discussion
As stated in Section 4.1 our method utilizes the PPO algorithm as the fundamental training method to train the adversarial agents. Additionally, our method can also be applied to various DRL algorithms. In the Appendix, we demonstrate the performances of our approach applicd to the bascline algorithms (Wul eet al. 2021; Guo et al.) 2021) and compare the results with those of the original baseline algorithms and KiekAndDefeend. The results show that by leveraging historical evaluations to revise the rewards, the performances of all baseline approaches get improved.
WouShalNNodPuss/Huuee
In Section 3 primarily explain episodic feedback reward-shaping function. It is also important to note that incorporating episodic feedback as a shaping term is functionally similar to the use of a state-dependent baseline (Williams [192] in policy-gradient methods, given the nature of the historical score. This score, calculated as the average cumulative return over past episodes that include a specific state (pattern), can be viewed as a Monte Carlo value estimate with a discount factor of 1 under a sparse-reward setting in which the reward equals the total cpisode return. This motivates an analogy with a statc-dependent bascline. However, the application mechanism differs: a standard baseline only changes the gradient calculation term, whereas our historical score serves reward shaping term that directly modifies the step rewards. While modifying rewards can bias the policy gradient, our episodic foedback ensures stability. As proven in Appendix A.2 the episodic feedback, with zero mean given the state, preserves the expected policy gradient, thereby leaving the optimal policy unchanged.
While the functional similarity to potential-based reward shaping and state-dependent basclines provides theoretical context, it does not fully explain the substantial performance improvements observed in our experiments. The challenge for a complete theoretical analysis centers on the issue of reward variance:
13

---

Published in Transactions on Machine Learning Research (11/2025)
specifically, how changes in step rewards using episodic feedback affect the learning process, even when all other aspects (e.g., states, actions, optimal policy, expected policy gradient) of the training algorithm remain the same. This issue is not solved in our paper, and we think a further study of it would be both difficult and highly valuable for future research.
6 Conclusion
In this paper, we propose an adversarial policy training method with reward engineering to train an adversar- ial agent more effectively and efficiently. Our method introduces an episodic memory to utilize the historical experiences to generate historical evaluations for the states and consequently revise the rewards of the states based on the evaluation thereby integrating the historical evaluations into the rewards used for adversarial training to emphasize the high-performing states. In our experiments, we demonstrate that agents trained with our approach achieve the most promising attack performance and defense difficulty. Additionally, by comparing the behaviors of adversarial agents, we discover that our attack method can explore different but optimal strategies by integrating successful actions from historical experiences. We believe our exploration of game stales and use of historical experiences improve the rewards advance adversarial policy training methods.
References
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. In International Conference on Learning Representations, 2018. URL https: //openreview.o wttp://wwww..eeeeeoooe/omd////oa
Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Leibo, Jack Rac, Daan Wierstra, and Demis Hassabis. Model-free episodic control. arXiv preprint ar Xiv:16606 04460, 2016.
The Viet Bui, Tien Mai, and Thanh H Nguyen. Imilating opponent to win: Adversarial policy imitation learning in two-player competitive games. arXiv preprint wxXi:2220016915 2022.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning. In International Conference on Learning Representations, 2020. URL h http://www...e.ooommmm/////e////
Wenbo Guo, Xian Wu, Sui Huang, and Xinyu Xing. Adversarial policy learning in two-player competitive games. International Conference on Machine Learning, pp. 3910 3919. PMLR, 2021.
Steven Hansen, Alexander Pritzel, Pablo Sprechmann, André Barreto, and Charles Blundell. Fast. deep reinforcement learning using online adjustments from the past. Advances in Neural Information Processing Systems, 31, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Sandy H. Huang, Nicolas Papernot, lan J. Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. 5th International Conference on Leurning Representations, ICLR 2017,
Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.ng net, 2017. URL https:
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. arXiv:1705.0 76450645., 2017.
orXiv preprint
14

---

Published in Transactions on Machine Learning Research (11/2025)
Igor Kuznetsov and Andrey Filchenkov. Solving Continuous Control with Episodic Memory. In Proceedings of the Thirlieth International Joint Conference Artificial Intelligence, pp. August 2021. International Joint Conferences on Artificial Intelligence Organization. ISBN -78-0-9992411- 9-6. doi: 10.24963/ 222443///caai.22. 201c//2001/3655.
Montreal, Canada,
URL https://www..j0t http://ree//oo.....aaaa.oorrrrre http://pocceeenggg/22/////////ee
Hung Le, Thommen Karimpanal George, Majid Abdolshah, Truyen Tran, and Svetha Venkatesh. Model- based episodic memory induces dynamic hybrid controls. Advances in Neural Information Processing Systems, 44:0313-30325, 2021.
Xian Yeow Lee, Sambit Ghadai, Kai Liang Tan, Chinmay Hegde, and Soumik Sarkar. Spatiotemporally constrained action space attacks on deep reinforcement learning agents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4577-4584, 2020.
Máté Lengyel and Peter Dayan. Hippocampal contributions to control: the third way. Advonces in neural information processing systems, 20, 2007.
Zhuo Li, Derui Zhu, Yujing Hu, Xiaofei Xie, Lei Ma, Yan Zheng, Yan Song, Yingleng Chen, and Jianjun Zhao. Neural episodic control with state abstraction. arXev preprint WrXiv2230.111490 2023.
Thodoris Lykouris, Max Simchowitz, Alex Slivkins, and Wen Sun. Corruption-robust exploration in episodic reinforcement learning. In Conference on Learning Theory, pp.
PMLR, 2021.
Yuzhe Ma, Xuezhou Zhang, Wen Sun, and Jerry Zhu. Policy poisoning in batch reinforcement learning and control. Advances in Neural Information Processing Systems, 32, 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial allacks. In International Conference on Learning Representations, 2018. URL htttps://openrevier
Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Ieml, volume 99, pp. 278 287, 1999.
Thanh Thi Nguyen and Vijay Janapa Reddi Deep reinforcement learning for cyber security. IEEE Trans- actions on Neural Networks and Learning Systems, 2019.
Xinlei Pan, Chaowei Xiao, Warren He, Shuang Yang, Jian Peng, Mingjie Sun, Mingyan Liu, Bo Li, and Dawn Song. Characterizing attacks on deep reinforcement learning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 222 pp. 1010-1018, Richland, SC, 2022. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450392136.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. In Proceedings of the 17th International Conference Autonomous Agents and MultiAgent Systems, AAMAS '18, pp. 2040 2042, Richland, SC, 2018. Interna- tional Foundation for Autonomous Agents and Multiagent Systems.
Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Has- sabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In International Conference on Machine Learning, pp. 2827-2836. PMLR, 2017.
Alessio Russo and Alexandre Proutiere. Optimal attacks on reinforcement learning policies. arXiv preprinl arXiv:1907.13548 2019.
Tom Schaul, John Quan, Toannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimiza- tion algorithms. ar Xiv preprint Xiv 1rXii:1777.000 707.06347, 2017.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Jianwen Sun, Tianwei Zhang, Xiaofei Xie, Lei Ma, Yan Zheng, Kangjie Chen, and Yang Liu. Stealthy and efficient adversarial attacks against deep reinforcement learning. In Proceedings of the A AAAT Conference on Artificial Intelligence, volume 34, pp. 5883 5891, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Emanuel Todorov, Tom Ercz, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robols and Systems, pp. 5026 5033. IEEE, 2012.
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229 256, 1992.
Xian Wu, Wenbo Guo, Hua Wei, and Xinyu Xing. Adversarial policy training against deep reinforcement learning. In 30th USENIX Security Symposium (USENIX Security 21), pp. 1883-1900, 2021.
Zhaoyuan Yang, Naresh Iyer, Johan Reimann, and Nurali Virani. Design of intentional backdoors in sequen- tial models. arXiv preprint Xiv: 1902. 09972, 2019.
Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsich. Robust reinforcement learning on state observations with learned optimal adversary. arXie preprint orXiv:2101.08452 2021.
Zhizheng Zhang, Jiale Chen, Zhibo Chen, and Weiping Li. Asynchronous episodic deep deterministic policy gradient: Toward continuous control in computationally complex environments. IEEE transactions on cybernetics, 514)):604-613, 2019.
Yiren Zhao, Ilia Shumailov, Han Cui, Xitong Gao, Robert Mullins, and Ross Anderson. Blackbox attacks on reinforcement learning agents using approximated temporal information. In 2020 50th Annual IEFF/IFI International Conference Dependable Systems and Networks Workshops (DSN- W), pp. 16 24. LEEE, 2020.
Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang. Episodic multi-agent reinforcement learning with curiosity-driven exploration. Advances Neural Information Processing Systems, 34, 2021.
A Appendix
A.1 Policy optimaliy guarantee of episodic feedback
Consider MDP M (S, A,T, R) and the shaped MDP M' (S, A, T.7,,, 7,,77' R2) with
R'(s,a,8') = R(s,a,8)) + 5(s),
where 8: S- is the fpisodic-feedback
For any policy x and initial state-actt action pair (s,
7 R'(, 84t1)
(s = E
1=0
E
7 (RS 81,1) 6(s)))
= Qx(saa) + E,
Σ 7 {d(s))
(6)
t==
16

---

Published in Transactions on Machine Learning Research (11/2025)
As stated in Equation 2 episodic feedback is defined
(()) = h score(s) - R.
where h (c)re(s) is the empirical mear of the episode return, and R is the running mean of the average cumulative reward over all past episodes. Both quantities unbiased estimates of the same random variable, the episode cumulative reward Reum- Hence
[[/__score(s)]] = [ = E [Bou]]
Equation immediately gives Ex (xx(((())) = substituting into (*):
00
T
Σ (5s))
= Σ ET'E)(((s)) = 0.
Because the correction term is zero, Equation 6 reduces to Q7 Qn(s, for every (s. and every policy æ. The optimal-action sets therefore coincide:
arg max Q7, = arg max Q+ (s, vs.
Hence every optimal policy in M' is also optimal in M. In other words, the episodic feedback 8(s) acts like potential-based shaping function (Ng et al. 1999), which shifts all (s, values by the same constant and thus leaves the policy optimality unchanged.
A.2
Preservation
Let the shaped reward be ri = rı + 8r Define discounted returns
G₁
G =
Aruu + 8641) = G. + Be, B. =
Σ
7
/
(8)
1=0
1=0
1=0
t=0
For policy gradient (o) and revised policy gradient (g), subtracting yields
VoJ - VoJ(O) =
Vo log Ta(a: 8t) G
Vo log mo 8t) Gt
t=
oc
V log (a st) B,
(9)
According to Equation in Appendix
A.1
E B, st] = E[8(( (s)) = 0,
V log (a 8t) B,} = 0,
therefore, VoJ((0)= (0) and the policy gradient is unbiased.
A.3 Hyper- parpeeer Analysis
For PPO typer-parameete selections, we the same parameters from Gleave et al. (2020). The hyper- parameters for our episodic memory are listed in Table 3. As mentioned in Li et al. (2023), 0.1 for e works well for episodic control methods in MuJuCo games, so we follow this setting in our experiments. We further analyze the rest two hyper-paranne parameters.
A.3.1 Pattern Length
As stated in Section 3.2 we use state sequences, referred to as patterns, as the basis for performance evaluation in past episodes. To find out the best parameter for the length patterns, we conduct experiments with different pattern lengths and show the results in Figure g We have selected different pattern lengths in our experiments. We can from Figure g that the agents reach the best performances when the pattern length is 3.
In practice h s_soor((s is the running average of the undiscounted cumulative reward over all past episodes that contained
17

---

Published in Transactions on Machine Learning Research (11/2025)
Table 3: papem-earsmeters of episodic memory used in our experiments.
Hyper-paraneter
Pattern Length k
3
Group Size n
100
Epsilon
0.1
pattern1
pattern-3
patteen-5
pattem-77
pattem-100
pattemn-20
Average Winningg (10)
100
touShallloo/assttummms
Average: Winning Rate (%)
100
Kick
Average Non-toss Rate (%c
100
SumoAnts
Averagg Winniing Rate (%6
100
Summoouumass
75
75
75
75
50
.50
25
25
0.0
0.5
2.0
0.0
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Steps (2e7)
Steps (2c7]
12c7)
Steps (3e7)
Figure 9: The performance comparison between our adversarial agent with different input pattern lengths.
A.3.2 Group Size
In Section 3.2.2 we compare the historical score of a pattern with an average cumulative reward of recently generated past episodes to calculate episodic feedback. Group size n is introduced to control the number of past episodos. If the group size is too large, some states that lead to bad patterns may be erroncously rewarded, and the magnitude of rewards and penalties reduced, thereby weakening the ability to find the optimal policy. On the other hand, if the group size is too small, it is more likely to wrongly penalize good states and reward bad states. Therefore, and 300. With the result shown in Figure [0] agents generally achieve better performance with a group size of 100. However, the unstable results in SumoHumans environment highlight the importance of carefully selecting this parameter.
conduct analysis of 5 group sizes which are 50, 100, 150, 200
A.4 Episodic Memory
In this section we share more details about the implementation of the episodic memory introduced in Sec-
tion 3.3.1
Table The architecture of the episodic memory
Module
shape
LSTM (64, 256, 1)
linearl (256, 512)
Tanh
linear2
(512, 1)
In Table 4, we give the architecture the episodic memory. The episodic memory consists of LSTM and MLP (linearl, Tanh, linear2). 64 in the shape of LSTM refers the length of the state from environment and 256 refers to the hidden length of LSTM. The LSTM is used to encode pattern into abstract vector so that the MIP can process. The MLP is used to output the historical score of the pattern, which evaluates the average performance of the pattern in past episodes.
In the Algorithm 2 we show the forward process of the episodic memory. The LSTM receives a pattern p as input and outputs an output sequence, the last hidden state vector and the last cell state vector of LSTM. The last hidden state vector will be processed with MLP, under the order of linearl, Tanh, linear2 and the ML. will output the historical score h. score of the input pattern.
18

---

Published in Transactions on Machine Learning Research (11/2025)
group-50
group-
group-150
group-200
group-300 300
Avrerage Wimming Rate (10
200
YoudShalllkorrrssssumaaa
Average Non-loss Rate 1%)
1000
SumoAnts
Aueerag Wimming Rabe (96)
200
SumoHumaans
75
Avrrrag Winning Rozo
75
75
75
50
25
25
25
Steps
Steps
Steps (2e7)
Figure 10: The comparison of win rate between our adversarial agent with different group size.
Algorithm 2 Forward process of the episodic memory.
Input: pattern (shape:[3, 64])
Output: historical score score (shape:1)
1: output, hidden, cell = LSTM(p)
2: h sccore linearl (bidden(-1.))
3: h score = Tanh(h (cchoec
4: h score lincar2(h_score
we show the update process of the episodic memory. After one episode ended, we extract patterns from the episode and calculate the cumulative reward of the episode. Then, we predict the historical score for each pattern and calculate the MSELoss between the historical score and the cumulative reward. The loss will be backpropagated to update the network.
In the Algorithum 3
A.5 Generalizability Analysis
In Section we state that our approach can be applied to various DRL algorithms. Since Gleave el al. be seen PPO, adopt our approach on the other two baseline attacks (Guo et al 2021 Wu et al.,
2021 and compare the performances with them. The results are shown in Figure 11 We can see the basclines adopting our episodic memory outperform the original basclines in YouShallNotPosHHumaa and KickAndDefend.
A.6
Videos Of Experiments
Due to the limited maximum file size for the supplementary materials, we have uploaded the videos mentioned in Section 4.2 at https://drive.googll. http:///woo..aoooooooo/o//oo///a 100__KKk222211Mra322222222 uspve link _ink
A.7 Main Results Supplementary
Supplementary tables of the figures in the main text are provided on the subsequent pages.
Table 5: The highest win rates of our agents and agents attacks against zoo victim agents are shown in Figure (aa
Environment
Our (%)
Gleave et (%)
Guo et (%)
et al. (%)
YouShallNot Husmaanns
87.62 + 7.38
60.08 = 6.22
65.77 7.60
48 48.60 = 8.99
Defeend
89.06 1: 7.98
64.37 +:: 8.61
65.32 6.92
64.76 + 8.55
AumoAnts
5.18 1 1.27
5.19 1 2.10
4.70 1.43
8.14 2.87
SumoHumans
76.35 + 8.29
69.24 + 12.16
64.49 + 7.15
62.22 + 16.86
19

---

Published in Transactions on Machine Learning Research (11/2025)
Algorithm Update process of the episodic memory.
Input: episode e
P Pattern(e)
R = Cumulative Reward(e)
3: for in P do
4: score = (pmory(p)
5: loss = MSELoss(h (h_ score, R)
(essspackpropaaationn))
end for
Guo ctall
Guo _p
Average Winning Rate (95
YouthollloottsssHuuaas
Averoge Winning Rate (96)
100
KckkndDeeend
75
755
50
25
0.0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
Time Steps (2e7)
Time Steps (2e7)
(a) The performances of adversarial agents trained with and without our episodic memory.
(Guo al.
2021
implementing
Wu elal.
Wa
100
YouShallNotPPssttuaass
Average Winning Hate (%6
1000
KckAndDefend
Average Winning Rate
50
25
25
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Time Steps (2e7)
Time Stops (2c7)
(b) The performances of adversarial agents trained by [Wu et al. with and without our episodic memory.
2021
implementing
Figure 11: The performance comparison of win rate between agents trained by Wu et al. 2021 Guo el al. 2021) implementing with and without our episodic memory in PaSSShallNoasssHfummma and KickAndDefend.
Table 6:
The highest non-loss rates of our agents and baseline agents against zoo victim agents are shown
in Figure 2(b)
Environment
Our (%)
Gleave et al. (%)
Guo et al. (%)
Wu et al. (%)
YouShalINol PassHunn Humans
87.62 1 7.38
60.08 +: 6.22
65.77 7.60
48.60 1: 8.99
Kick Defed
90.01 + 7.56
65.17 8.87
66.56 7.14
65.27 = 8.45
Anto
84.66 + 3.92
74.94 + 16.24
82.98 + 3.73
40.97 = 6.65
SumoHumans
91.68 1 7.52
91.88 + 12.18
90.49 5.48
92.55 =: 14.40
20

---

Published in Transactions on Machine Learning Research (11/2025)
Table 7:
The performances of our agents and baseline agents against retrained victim agents shown in Figure 3 We show win rates of the agents in BashhaalNooPPss//mmmmrr KickAndDefend, SumoHumans and
non-loss rate in SumoAnts.
Environment
Our (%)
Gleave et al. (%)
Guo et al. (%)
Wu et al. (%)
YouShalINooPassIHiiee
50.27 1 13.03
5.00 2.50
6.22 2.82
5.99 1 2.99
eefend
51.82 1 6.84
28.02 7.33
29.33 +: 9.84
32.38 = 9.54
SumoAnts
83.15 + 2.98
79.78 + 2.28
82.49 + 2.77
90.13 = 3.42
SumoHumans
6.17 1: 7.60
6.03 5.35
7.61 5.88
10.71 6.95
Table 8: The comparison of performances between our method and NECSA. The performances of our agents and NECSA against victim agents are shown in Figure We show win rates of the agents in YouShallNot- PossHumans, KickAndDefend, SumoHumans and non-loss rate in SumoAnta
Environment
PPО (%)
Our (%)
NECSA (%)
JouShallNot.PsssIllle PassHumans
60.08 + 6.22
87.62 7.38
71.71 +:: 9.18
Kieennd
74.94 1 16.24
89.06 1 7.98
76.52 1 16.31
Ants
83.15 + 2.98
84.66
3.92
82.49 + 2.77
SumoHumans
69.24 12.15
76.34 8.29
71.34 7.38
Table 9: The performances of our agents guided by historical evaluation of patterns with different lengths
against Zoo victim agents is shown in Figure 0
We show win rates of the agents in BashaallooPPs///ammmaa
KeceAndee Sumofhmmans and non-loss rate in Auls.
Environment
state (%)
pattern-3 (%)
pattern-5 (%)
pattern-7 (%)
pattern-10 10 (%)
pattern-22 20 (%)
82.06 + 6.77
87.62 + 7.38
77.44 11.81
76.22 + 7.26
76.02 + 5.42
73.66 + 9.61
KeeeAndDefend
83.15 + 11.29
89.06 + 7.98
87.78 + 4.07
83.24 + 9.10
81.12 + 8.66
76.84 9.12
SumoAnts
83.73 + 4.15
84.66 = 3.92
83.83 = 4.15
82.71 4.23
81.47 : 3.18
81.01 2.49
SunoHunaus
74.35 1 6.71
76.35 1 8.29
74.47 1 99666
71.56 1 8.45
71.07 1: 9.38
69.60 1: 7.14
Table 10: The performances of our agents trained with episodic feedback and historical score against zoo vic- tim agents shown in Figure 7 We show win rates of the agents in YouShallNooPass KickAndDefend, SumoHumans and non-loss rate in SumoAnts.
Environment
Episodic Feedback (%)
historical score (%)
YouShallNottPssttumass Defend
87.62 7.38
75.28 + 8,25
89.06 = 7.98
72.85 t 11.48
SumoAnts
84.66
83.66 2.30
SunoHumans
76.35 8.29
72.15 1 6.90
Table 11:
The performances of our agents with and without revision conditions against zoo victim agents is shown in Figure S We show win rates of the agents in ManShhallNotPsssFumaae KickAndDefend, SumoHu- mans and non-loss rate in AntoAnts.
Environment
Our two case (%)
Our all case (%)
YouShall PassottPPssiiinns PassHuuann
87.62 7.38
78.39 10.74
Kick Andffeend
89.06 1. 7.98
60.45 1 25.15
SumoAnts
84.66 + 3.92
80.44 + 3.73
SumoHumans
76.35 8.29
68.40 + 4.14
21


Reviews:
[{"review_id": "rmCYOxdmZQ", "reviewer": "Reviewer_HGpX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new approach for training black-box adversarial agents, which involves using historical state information to alter rewards to improve the adversarial policy. This generally resulted in beating the baseline adversarial policies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims regarding improvements over existing baselines for adversarial policies are indeed confirmed by convincing and clear evidence. Figures 2-6 demonstrate strong results, particularly Figures 2 and 3. The proposed algorithm generally outperforms existing baselines more often, although on certain select environments, this is not the case.", "audience_yn": "No", "audience_explanation": "I believe individuals in TMLR's audience may not be interested in this paper for two main reasons:\n* The proposed approach, while getting strong results, was relatively straightforward and intuitive, and doesn't really bring many new learnings to the table. The proposed approach may simply outperform due to more compute/complexity.\n* The paper was extremely difficult to follow, due to the reasons described in the requested changes below. If the paper had been more digestible, I would feel the audience may be more interested in the paper.", "requested_changes": "- The writing style and explanation of concepts/ideas is generally very difficult to understand. This made it challenging to understand the method, introduction, related works, etc (with the methodology section being extremely difficult to understand). This is critical. There are numerous mistakes in flow even in the abstract, which makes the paper difficult to understand even at a high level. Although I generally don't believe grammar/flow should affect the review of a paper, I found it to really impact the readability of this paper.\n- There are many occasions where details are inserted in seemingly random places in the paper that do not make logical sense. I highly recommend the authors rework the entire flow of the paper to be more story-like and follow more logically. This is critical. E.g. I did not find the discussion to contain relevant information to the discussion (i.e. it discussed experimental details).\n- A lot of the claims made in the paper are incorrect or misleading, these should be adjusted and are critical:\n\t- \"It has been **proved** that deep reinforcement learning (DRL) policies are vulnerable to adversarial attacks\"\n\t\t- There is no hard proof just strong evidence, it has been shown not proved\n\t- \"It is also important to note that our method is scalable and can be applied to various DRL algorithms.\"\n\t\t- There needs to be more evidence to make such claims\n- Should \"Object Function\" in Figure 1 be objective function? This Figure is unclear to me as a consequence and as a whole. A step by step explanation of this Figure in the caption would be helpful.\n- The baselines compared to, from 2019-2021, are not modern baselines, the authors should include more modern baselines.\n- There are minor formatting issues with many of the citations.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WGpMZLu8lq", "reviewer": "Reviewer_FGt1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper aims to address issues with training an agent via deep reinforcement learning to attack another agent. The authors claim that fixing the agent being attacked and casting this as a standard RL problem can be enhanced. This is done via fitting zero-mean state-specific baselines via scoring trajectories with their average cumulative reward and demeaning with the average reward of similar episodes, and then injecting this into the environment reward to shape it. \n\n## Strengths\n1. The method is practical, and seems to perform well empirically.\n2. The empirical evaluation is thorough and appears sound. \n\n## Weaknesses\nThe weaknesses mostly pertain to what Action Editor ED9N pointed out. It does not seem like these were effectively addressed. \n1. The authors do not thoroughly explore what their method means, how it is derived, and its relation to existing literature.\n- The relation to a baseline is not explored, and the relation to a critic is hand-waved away.\n- The relation to episodic control amounts to grouping by trajectories v.s. grouping by states, and so is a non-Markovian approach. Why exactly this produces any benefit in what is presumably an MDP is not explored.\n2. The theory is shallow.", "claims_and_evidence_yn": "Yes", "claims_explanation": "## Strengths\n\nRegarding the main contributions stated the authors:\n1. A novel policy learning framework with reward enhancement. This is true, the approach is evaluating similar trajectories instead of states is highlighted and explained, though the explanation could be cleared.\n2. The second claim amounts to what their method is, which is correct.\n3. The experiments are reasonably convincing, satisfying the third. Importantly, the method is viewed and developed from a practical lens, and so is reasonably easy to implement and has reasonable training time. An ablation study, as well as experiments on different environments against suitable baselines, are  provided. \n\n## Weaknesses\n1. **Relation to a critic, baseline, episodic control, etc.**\n- **On baselines and critics.**\n    - It is all well and good to cast this as a reward-shaping term, but this is quite different from vanilla reward-shaping. As Action Editor ED9N quite rightly points out, this is suspiciously similar to a critic. I would clarify that this looks closer to a baseline as in REINFORCE with baseline, instead of a TD(0) actor-critic. \n    - This differs from a baseline in three ways. Firstly, the baseline is trajectory-specific, not state-specific. Secondly, the baseline is injected into each reward of the future trajectory, and not just the reward-to-go of the current state. Thirdly, issues with the policy gradient that could result from the above modifications are resolved by making the baseline zero-mean. \n    - It seems like the authors do not realize this. The approach is cast as a practical method for improving the stability of training, but exactly what this method does and what it amounts to is not addressed by the authors.  The concerns from Action Editor ED9N are handwaved away.\n- **Against episodic control.**  The approach of grouping similar trajectories together to adjust the reward is inherently a non-Markovian approach. This is in contrast to evaluating similar states to adjust the reward, as a baseline would, as well as episodic control, which employs states and not trajectories. \n    - Within an MDP, where the state is a sufficient statistic for the expected reward-to-go, one should not expect this to yield any benefit.\n    - Why do we see improvements in Section 4.4.1? In practice, the state might not be a sufficient statistic, leading to some unobserved confounding that yields non-Markovian behavior. This can be more robust to defending against that in practice. \n    - However, employing a non-Markovian approach likely suffers from the curse of horizon, as is widely understood.\n    - The benefit of richer information v.s. the drawbacks in horizon are not explored within the ablation study in section 4.4.1. \n- Overall, this reflects on the quality of the exposition within the paper. Other issues related to this remain. For instance, which objective function was chosen for the policy in Algorithm 1 for the experiments? It is likely the PPO objective, but this has to be inferred from the writing within the experiments, as it is ambiguous whether \"we conduct retraining experiments on the victim agents using the PPO algorithm\" means retraining with the PPO objective or with the common implementation of the algorithm proper. Again, it is likely the former, but it is never clearly stated. \n2. **On theory.** \n- A proof of policy optimality is provided in the appendix. However, the proof is rather verbose, and adds complexity . The proof amounts to showing that the original reward is perturbed by a stochastic term with mean zero, so the Q-function remains the same, and the optimal policy must also remain the same. This can be expressed in a single line. I do not think that this is sufficient theoretical background per-se.\n- Regarding the relation with a baseline, there is some opportunity for a derivation of how it differs from it here. \n    - The episodic memory is not a TD(0) (or GAE(0)) critic, but there is an relation between this and REINFORCE with baseline, or GAE(1). To see this, consider the following:\n        - Let $r'(s,a) := r(s) + \\delta(s)$. It has been shown by the authors that $\\mathbb{E} \\delta(s) = 0$. Then, the policy gradient advantage estimate is $\\sum_{l=0}^{\\infty} \\gamma^l r(s_{t+l}, a_{t+l}) - V(s_t)$. \n        - The reward-shaped estimate is  $\\sum_{l=0}^{\\infty} \\gamma^l (r(s_{t+l}, a_{t+l}) + \\delta(s_{t+l})) = \\sum_{l=0}^{\\infty} \\gamma^l r(s_{t+l}, a_{t+l}) - (\\sum_{l=0}^{\\infty} - \\gamma^l \\delta(s_{t+l})) =: \\sum_{l=0}^{\\infty} \\gamma^l r(s_{t+l}, a_{t+l}) - B(s_{t:})$, where $ B(s_{t:}) := \\sum_{l=0}^{\\infty} - \\gamma^l \\delta(s_{t+l})$ is a function of the future trajectory. \n        - This at first seems to invalidate the proof of an unbiased policy gradient, but given that $B(s_{t:})$ is mean-zero, this is resolved, and it follows that the policy gradient remains the same in expectation. \n            - $\\mathbb{E}[\\sum_{l=0}^{\\infty} \\gamma^l (r(s_{t+l}, a_{t+l}) + \\delta(s_{t+l})) ]= \\mathbb{E}[\\sum_{l=0}^{\\infty} \\gamma^l r(s_{t+l}, a_{t+l}) - (\\sum_{l=0}^{\\infty} - \\gamma^l \\delta(s_{t+l}))] = \\mathbb{E}[\\sum_{l=0}^{\\infty} \\gamma^l r(s_{t+l}, a_{t+l})] -  \\sum_{l=0}^{\\infty}\\gamma^l \\mathbb{E}[\\delta(s_{t+l})] = \\mathbb{E}[\\sum_{l=0}^{\\infty} \\gamma^l r(s_{t+l}, a_{t+l})] - 0.$", "audience_yn": "Yes", "audience_explanation": "This relates to training deep RL agents to attack other agents when the other agent is fixed, which is an important enough application that there is enough literature on. The tweak appears to help empirically, for the above reasons stated in the last cell.", "requested_changes": "Explore the relation to a baseline, the benefits of the non-Markovian trajectory-based episodic control v.s. Markovian state-based episodic control, and either beef up the theory (perhaps with the derivation provided) or be more upfront about the lack of contribution there. These are critical to securing my recommendation for acceptance.", "broader_impact_concerns": "None.", "additional_comments": "The claims made are largely supported by clear and convincing evidence, and some people would be interested in this work. Still, the weaknesses raised for rejection at the last submission were not effectively addressed, and so still need to be addressed now.", "extra_scores": {}, "extra_text": {}}, {"review_id": "z3dwsSpSYO", "reviewer": "Reviewer_AeY8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a novel approach in the area of on adversarial policy training in two-player scenarios that enhances the existing PPO algorithm with a neural episodic memory to provide more fine-grained feedback on sequences within observed episodes. The introduced memory acts as an additional component that allows to detect desired positive or punish undesired negative 'patterns' (subsequences of an episode of fixed length that are indicators to a win or loss) that have been encountered during previous runs. The episodic memory is trained alongside the adversarial agent and the authors claim a better performance is comparison to existing baseline methods in the area is achieved.  The authors introduce and test a 'conditional reward revision', which limits memory guidance  patterns that rank above (or below) average particular groups of episodes (to account for later episodes consistently achieving higher rewards earlier patterns). The method and ablations are tested on four Mujoco two-player game environments, as well as on a more complex StarCraft II environment, showing good performance across all evaluations and against ablations.\n\n\n\n**Strengths**\n\n1) The idea of leveraging a neural episodic memory is intuitive. The paper is overall well written and the approach is presented in simple successive steps. Figures and plots are easy to read and interpret.\n2) The presented formalism follows common notation and is formalized to a sufficient degree. The presented proof of policy-optimality under episodic feedback seems to hold.\n3) The overall selection of environments and experimental setup seems reasonable and follows existing baseline experiments. Experiments reveal strong improvements in two of the four Mujoco environments and the authors are able to present a 10% score increase for a more complex StarCraft II scenario. The authors compare to several baselines in the field of adversarial RL and a closely related NECSA baseline with episodic control. They present several ablations with regard to the individual components of their approach and also inspect countermeasures of 'victim retraining' and 'masked observation defense' against their approach.\n\n\n\n**Weaknesses**\n\n1) While PPO is a still widely considered methods and given the strong claims on performance improvements, the paper focuses strongly on methods within the subfield, but it (or possibly the general field of adversarial attacks in RL) seems to lack behind general advances in RL (e.g., [1-3]). It is unclear to me, whether similar advantages could be achieved in the presence of possibly stronger algorithms.\n2) Variation experiments on pattern length (A.2.1) and group size (A.2.2) only consider two of the four environments in the main paper (admittedly one of the remaining four environments shows low winning performance) and evaluate only two alternative values for pattern length and group size. While the alternative variants demonstrate a parameter choice above and below the optimal chosen value, it is hard to assess the fidelity of parameters from only two alternative choices on two environments.\n3) The general setup described of Sec. 4.4.1 is unclear to me: How is the mentioned state guidance implemented? Does this imply a 1-step pattern - or is the episodic feedback omitted in this case? The authors furthermore mention \"historical scores of both states\" and to \"[...] revise the rewards with two episodic feedbacks\". I do not understand which \"both states\" are meant here and how (and why) two feedbacks are computed.\n4) Algorithm 1, line 9, states \"if The episode is the ith episode in Group m then ...\". I am wondering under which condition this statement will evaluate to false. To my understanding an episode should included in some group (with some index i assigned to it), such that it could be written as a normal assignment. Maybe I misunderstood something here and would like to ask the authors to clarify.\n\n\n\n[1] Badia, Adrià Puigdomènech, et al. \"Agent57: Outperforming the atari human benchmark.\" *International conference on machine learning*. PMLR, 2020.  \n[2] Hessel, Matteo, et al. \"Muesli: Combining improvements in policy optimization.\" *International conference on machine learning*. PMLR, 2021.  \n[3] Kapturowski, Steven, et al. \"Human-level Atari 200x faster.\" *arXiv preprint arXiv:2209.07550* (2022).", "claims_and_evidence_yn": "Yes", "claims_explanation": "While I am not intimately familiar with existing literature in this particular subfield, the presented experiments seem to clearly demonstrate improvements over existing algorithms in the field of adversarial RL and show improvements under the more complex StarCraft II setting. My main concern are the lack and comparisons to more recent RL methods under which the presented advantages might vanish (Weakness 1).\n\nSimilarly, the presented variations and ablations seem reasonable and support the necessity of individual components and choice of parameters. The sparse variations of parameters in combination with evaluation on only two environments limit insights with regard to the choice of pattern length and group size (Weakness 2).", "audience_yn": "Yes", "audience_explanation": "The introduction of a neural episodic memory in the context of adversarial learning is an interesting idea that seems to show clear improvement over existing methods and might spark further interest in the general application of adversarial methods in RL. The presented experiments and comparison to a recent baseline with episodic memory confirm a robust performance of the approach and might be suited to be considered as an additional component to more recent algorithms.", "requested_changes": "The critical requested changes mainly regard to mentioned weaknesses:\n\n1) Given the strong claims on the performance improvements of the methods, I consider weakness 1 on comparisons/integration of the idea with more recent methods to be most relevant and critical. Can the authors confirm, that the observed performance improvements still persist under more recent, possibly stronger, algorithms? Since I'm not well versed in literature of the particular subfield, I might have missed a key point that prevents more recent methods from being applied. In this case I would like to ask the authors to elaborate on why the application of such algorithms might not make a difference or possibly argue why they are out of scope for this paper.\n2) Similarly, I would like to ask the authors to clarify weaknesses 3 and 4 mentioned above. While I consider these to be critical, I believe that these points arose mainly due to a lack of explanation or confusion about the exact setup of the method.\n\n\n\n**Non-critical**\n\n1) I believe that the paper might be strengthened by probing further values of pattern length and group size, (possibly also testing on the remaining environments) to get a better perspective on the importance of parameter choice. However, since the method seems to be working without strong fine-tuning of the parameters, I consider this point to be non-critical.\n2) [Minor] While Figure 1 showcases the whole workflow of the method, it is, at first, rather unclear which parts constitute the novel contributions of the paper in contrast to existing frameworks. The authors might want to highlighting the respective components in the figure and/or state the novel parts more explicitly in the the caption.\n\n\n\n**Typos**\n\n* Sec. 3, \"Our threat model: [...] and assume[s] that\"\n* Sec. 4.1, \"For the hyper-parameters, we set the [epsilon?] as 0.1\".\n* Sec. 4.4.3, last line p. 11: \"In fact, [w]e ...\"\n* Sec. 3.3.2, \"the first i episode\" -> \"the first i episode[s]\"", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)