Ruling StarCraft Game Spitefully -Exploiting the Blind Spot of AIPowered Game Bots
Xinyu Xing, Wenbo Guo, Xian Wu, Jimmy Su#BHUSA@BLACKHATEVENTSDeep Reinforcement Learning• Deep learning has dominated many fields:• Computer vision [Krizhevsky, et al, 2012], natural language process [Socher, et•al, 2011].
Malware detection [Wang, et al, 2017], intrusion detection [Javaid, et al, 2016].

• Integrating DL into reinforcement learning – DRL:
• Extraordinary performance on many decision-making tasks.
• Robotic control [Kober, et al, 2009].
• Autonomous vehicles [O’Kelly, et al, 2019].
• Finance and business management [Cai, et al, 2018].
#BHUSA

1@BLACKHATEVENTSDRL in Games• Board games:• Go: DeepMind – AlphaGo [1], Facebook – OpenGo [2].
• Poker games: Texas hold’em [3].
• DeepMind – OpenSpiel [4].
••An open source collection of game environments.
Single- and multi- players.

Credit: https://www.irishtimes.com/business/technology/podcastdives-deep-into-science-and-ai-behind-deepmind-1.3997759

Credit: https://www.nature.com/articles/d41586-019-02156-9
Credit: https://github.com/deepmind/open_spiel/#BHUSA

2@BLACKHATEVENTSDRL in Games• Simulation games:• OpenAI – Gym [1].
••Open source toolkit for developing DRL method to control robots, play games, etc.
Atari [2], Roboschool [3], and MuJoCo [4].

Atari breakout. Credit: Wikipedia• Real-time strategy games:• StarCraft II [5].
• Dota 2 – OpenAI Five [6].
• Single- and multi- players.

Roboschool Pong. Credit: [3]Dota 2 -- defending base. Credit [6]

MuJoCo You-Shall-Not-Pass.
StarCraft II#BHUSA

3@BLACKHATEVENTSAttacks on DRL• Adversarial attacks on deep neural networks:• Training phase – data poisoning attacks.
•• Injecting a backdoor into a DNN [Liu, et al, 2017].
Testing phase – adversarial samples.
• An imperceptible perturbation on the input cause a dramatic change to the output [Goodfellow, et al, 2015].

• Deep reinforcement learning is also vulnerable to adversarial attacks.
• Perturbing an agent’s observation, action, or reward and force it to fail the task.
•Involving hijacking a game system – not practical.

Enabling a practical adversarial attack against a master agent in a two-agent gameenvironment.
#BHUSA

4@BLACKHATEVENTSAgenda• DRL basics.
••

Modeling an RL problem.
Solving an RL problem – training an DRL agent.

••Two-agent games: MuJoCo, StarCraft II.
Code structure of training an DRL bot.

• DRL-powered games.

• Existing attacks on DRL.
••

Perturbation attacks (Extension of attacks on DNN)
Practical adversarial agent attack.

• Our attack methodology.
• Evaluation.
• Conclusion.
#BHUSA

5@BLACKHATEVENTSAgenda• DRL basics.
••

Modeling an RL problem.
Training an DRL agent.

••Two-agent games: MuJoCo, StarCraft II.
Code structure of training an DRL bot.

• DRL-powered games.
• Existing attacks on DRL.
••

Perturbation attacks (Extension of attacks on DNN)
Practical adversarial agent attack.

• Our attack methodology.
• Evaluation.
• Conclusion.
#BHUSA

6@BLACKHATEVENTSModeling an RL Problem• An RL problem – a sequential decision making problem.
• An agent observes and interacts with an environment through a series of actions.
• The agent receives a reward each time taking an action.
• At each time step, system is at a certain state.

Environment

•

Agent

•
•
•
RewardObservation

ActionAgent

Receive an observation.
Executes an action.
Environment• Receive this action.
• Transit to the next state based on the transition dynamics.
• Emit an reward.
• Emit the next observation.

#BHUSA

7@BLACKHATEVENTS

Solving an RL Problem• An RL problem – a sequential decision making problem.
• The goal of an agent is to maximize its total amount of rewards.
• The goal of an RL algorithm is to learn an optimal policy, following which the agent
could receive a maximum amount of rewards over time.
GameValue functionPolicy• In RL, the total reward of an agent is formulated asvalue functions.
• State-value function: the expected total reward for an agent starting•

Demonstrations of resolving an optimal policy. Credit: David Silver [1]•from a state and taking actions by following its policy.
Action-value function: the expected total reward for an agent startingfrom a state and taking a specific action by following its policy.
An optimal policy can be obtained by maximizing the value functions.

#BHUSA

8@BLACKHATEVENTSTraining an DRL Agent• In DRL, an agent is usually modeled as an DNN.
• Policy network.
• Taking as input the observation, and output the corresponding action.
• Learning a policy is to solving the parameters of this neural network.

Observation(state)of the agent
Agent
Policy networkof the agent…
0.5 1.3 0.2 0.9 …

Updating the value network by minimizing the approximation errors.
Updating the policy network by maximizing the value function.
They usually share parameters.

…

Sharedparameter2µ⇡ N(µ, σ )⇡✓ (a|s)

…

•••

⇡

…

• Goal: optimizing the value-function.
• Using another network to approximate the value-function.
• In each iteration:

Policy network…

• Policy gradient methods - solving the network parameters.

Vv (s)

Value network#BHUSA

9@BLACKHATEVENTSAgenda• DRL basics.
••

Modeling an RL problem.
Training an DRL agent.

••Two-agent games: MuJoCo, StarCraft II.
Code structure of training an DRL bot.

• DRL-powered games.
• Existing attacks on DRL.
••

Perturbation attacks (Extension of attacks on DNN)
Practical adversarial agent attack.

• Our attack methodology.
• Evaluation.
• Conclusion.
#BHUSA

10@BLACKHATEVENTSDRL-powered Games• Two-party MuJoCo games.
• Observation: the current status of the environment: the agent’s and its opponent’s status.
• Action: the agent’s movement (moving direction and velocity).
• Reward: the agent’s status and win/lose.

• StarCraft II games.
• Observation: spatial condition of the map and the amount of resources.
• Action: building, producing, harvesting, attacking.
• Reward: game statistics and win/lose.

ntiocA#BHUSA

11@BLACKHATEVENTSTraining an DRL Bot• Overall workflow.
• Taking proximal policy optimization (PPO) as an example.
Initialize thenetwork parameters
Collecting training
trajectories based on
the current policy

Number of iterations

Updating the policy
networkUpdating the valuefunction network
Credit: [1]#BHUSA

12@BLACKHATEVENTSTraining an DRL Bot• Take StarCraft II as an example.
• Pysc2 [1]• Pysc2Extension [2]• TStarBot1[3]• Code structure• agents: defining and constructing two networks.
••• Taking as input observation and output action.
envs: environment wrapper.
• Taking action as input and output observation and reward.
bin: running the agent in the environment and training the agent.

#BHUSA

13@BLACKHATEVENTSTraining an DRL Bot• Training an DRL bot for a game.
• Usually adopting the self-play mechanism.

• The structure of the main file.
• Defining an environment using the environment wrapper.
• Defining an actor to collect the trajectories.
•• Running the current agent in the environment.
Defining a learner to train the agent.
• Receiving the collected trajectories and updating the networks.

#BHUSA

14@BLACKHATEVENTSAgenda• DRL basics.
••

Modeling an RL problem.
Training an DRL agent.

••Two-agent games: MuJoCo, StarCraft II.
Code of training an DRL bot.

• DRL-powered games.
• Existing attacks on DRL.
••

Perturbation-based attacks (Extension of attacks on DNN).
Practical adversarial agent attack.

• Our attack methodology.
• Evaluation.
• Conclusion.
#BHUSA

15@BLACKHATEVENTSPerturbation-based Attacks• Threat model.
• Perturbating the observations and force the policy•

network to output a series of sub-optimal actions.
Perturbating the output actions of the policy network.

• Example.

• Generating perturbations by using the existing attacks on DNN.
• Adding it to the observation (snapshot of the environment)

Credit: [1]Credit: Huang, et al, 2017.
#BHUSA

16@BLACKHATEVENTSPerturbation-based Attacks• In the game setup, requiring the attacker to hijack the game server.
• Identifying and exploiting the vulnerabilities of the server.
• Bypassing the defense mechanism in the server.
• Requiring professional hackers tremendous effort and time.
• Not a practical setup for beating an master agent of a two-party game.

#BHUSA

17@BLACKHATEVENTSAdversarial Agent Attack• Threat model.
• Attacker is not allowed to hijack the information flow of the victim agent.
• Manipulating the observation, action, and reward.
• Attacker could train an adversarial agent by playing with the victim agent.

• More practical in games.
••

No need to hack the game system.
Any player could play with a master agent freely.

Credit: [1]#BHUSA

18@BLACKHATEVENTSAdversarial Agent Attack• Existing technique [Gleave, et al, 2020].
••••

Treating the victim agent as a part of the environment.
Training an agent to collect maximum rewards in the environment embedded with the victim.
Maximizing the training agent’s value function by using the PPO algorithm.
Expecting to obtain a policy that could beat the victim.

• Limitations• Do not explicitly disturbing the victim agent.
• The training algorithm has less guidance for identifying the weakness of the victim.
• Cannot establishing a high game winning rate.

#BHUSA

19@BLACKHATEVENTSAgenda• DRL basics.
••

Modeling an RL problem.
Training an DRL agent.

••Two-agent games: MuJoCo, StarCraft II.
Code of training an DRL bot.

• DRL-powered games.
• Existing attacks on DRL.
••

Perturbation attacks (Extension of attacks on DNN)
Practical adversarial agent attack.

• Our attack methodology.
• Evaluation.
• Conclusion.
#BHUSA

20@BLACKHATEVENTSOur attack• Threat model.
• Practical threat model.
• Not allow to manipulate environment, opponent network.

Credit: [1]• High-level idea.
• Adversarial agent learns to disturb its opponent.
• Training an adversarial agent to not only maximizing its reward but also minimizing its•

opponent's reward.
Letting the adversarial agent take an action that deviates the victim’s next action.

#BHUSA

21@BLACKHATEVENTSOur attack• Achieving the first goal.
• Approximating the victim value function.
• Augmenting the objective function with a term that minimizes the victim reward.

• Example – Collecting resources.
• Without the added term.
•• The adversarial agent focuses only on optimizing its strategy to correct more resources.
With the added term.
• The adversarial agent learns to block the victim from collecting resources.

#BHUSA

22@BLACKHATEVENTSOur attack• Achieving the second goal.
• Explaining the actions of the victim and find out the time steps when victim takes action•

based on the adversarial agents.
The adversarial agent takes an action that introduce maximum deviation to the victim’s action
at these critical time steps.
Adversarial agent changes its action andintroduce an change to the environment.

• Example.

1.3
0.0
2.1
0.0
3.0
0.0
0.0
0.0



o1
 .. 
 .   o5  
 o6  
 o7  
 o8 
 . 
 .. o13 ô1
 .. 
 .   ô5  
 ô6  
 ô7   ô8  . 
 .. ô13

Optimal action

Optimal actions(t)Suboptimal action

WIN…

LOSS…

tk

Suboptimal action

ŝ(t)
t1…

t2

The part of the victim agent observation that represents the adversary.
23

This change forces thevictim to choose a suboptimal action.

#BHUSA@BLACKHATEVENTSAgenda• DRL basics.
••

Modeling an RL problem.
Training an DRL agent.

••Two-agent games: MuJoCo, StarCraft II.
Code of training an DRL bot.

• DRL-powered games.
• Existing attacks on DRL.
••

Perturbation attacks (Extension of attacks on DNN)
Practical adversarial agent attacks.

• Our attack methodology.
• Evaluation.
• Conclusion.
#BHUSA

24@BLACKHATEVENTSEvaluation setup• Selected games.
• MuJoCo – victim (blue), adversary (red).

••• Kick-And-Defend• You-Shall-Not-Pass• Sumo-Ants• Sumo-Humans
StarCraft II – Zerg vs. Zerg.

Measuring and Reporting the winning rate of the adversarial agent each time its policy is updated during the training
process.

#BHUSA

25@BLACKHATEVENTSQuantitively Evaluation• Comparison of winning rates.
• Red: our attack; blue: existing adversarial agent attack [Gleave, et al, 2020].
• Our attack outperforms the existing attack on most games.
• Sumo-Ants: Improve the non-lose rate.
Almost cannot win.
Low observation dimensions
Hard to disturb the victim via
the adversarial actions.
Winning + tie•••

#BHUSA

26@BLACKHATEVENTSDemo Examples

Kick-And-Defend and You-Shall-Not-Pass•

Establishing weird behaviors that fail the victim.

Sumo-Humans•

Learn a better strategy – initializing itself near theboundary and luring the victim to fall from the arena.
#BHUSA

27@BLACKHATEVENTSDemo Examples
Sumo-ants•••

Exploring the weakness of the game rule.
If one player falls from the arena without touching its opponent,
the game ends up with a draw.
The adversarial agent (red one) intentional falls from the arena
after the game begins.

Video of the StarCraft II: https://tinyurl.com/ugun2m3#BHUSA

28@BLACKHATEVENTSA Potential Defense• Retraining the victim agent against the adversarial agent with our proposed attack.
:LQQLQJ5DWH 

.LFN$QG'HIHQG

<RX6KDOO1RW3DVV

6XPR+XPDQV

6WDUFUDIW,,

6XPR$QWV        ,WHUDWLRQ H ,WHUDWLRQ H ,WHUDWLRQ H ,WHUDWLRQ H ,WHUDWLRQ H

• Improving the performance of the victim.
•••

Winning the You-Shall-Not-Pass.
Achieving a draw on three games.
Kick-And-Defend – adversarial retraining does not work.
• The unfairness of the game design – Its hard for thekicker to win.
#BHUSA

29@BLACKHATEVENTSA Potential Defense
The victim learns to ignore
the adversary and directly
go for the finish line.

The victim recognizes
the trick of the adversary
and stay where it is.
Tie game!

Victim cannot change theintentional behaviors of theadversary.
Staying Tie games!

Victim acts ever worse.
Fall into the ground!

#BHUSA30@BLACKHATEVENTSAgenda• DRL basics.
••

Modeling an RL problem.
Training an DRL agent.

••Two-agent games: MuJoCo, StarCraft II.
Code of training an DRL bot.

• DRL-powered games.
• Existing attacks on DRL.
••

Perturbation attacks (Extension of attacks on DNN)
Practical adversarial agent attacks.

• Our attack methodology.
• Evaluation.
• Conclusion.
#BHUSA

31@BLACKHATEVENTSConclusion• Attacker could train an adversarial agent to defect a bot of an AIpowered game.

• By disturbing the victim actions, the adversarial agent could exploit thevulnerabilities of the victim/game rules and thus fail the victim agent.

• Adversarial retraining does not always succeed; more advanced
techniques are needed to protect the game bots (master agent).

#BHUSA

32@BLACKHATEVENTSThank You !
Wenbo Guo
wzg13@ist.psu.edu@Henrygwb@WenboGuo4

http://www.personal.psu.edu/wzg13/#BHUSA@BLACKHATEVENTS