Learning Movement Strategies for Moving Target Defense

1 INTRODUCTION . The complexity of modern-day software technology has made the goal of deploying fully secure cyber-systems impossible . Furthermore , an attacker often has ample time to explore a deployed system before exploiting it . To level the playing field , researchers have introduced the idea of proactive cyber defenses such as Moving Target Defense . In Moving Target Defense ( MTD ) , the defender shifts between various configurations of the cyber-system ( 1 ) . This makes the attacker ’ s knowledge , gathered during the reconnaissance phase , useless at attack time as the system may have shifted to a new configuration in the window between reconnaissance and attack . To ensure that an MTD system is effective at maximizing security and minimizing the impact on the system ’ s performance , the consideration of an optimal movement strategy is important ( 2 ; 3 ) . MTD systems render themselves naturally to a game-theoretic formulation– modeling the cybersystem as a two-player game between the defender and an attacker is commonplace . The expectation is that the equilibrium of these games yields an optimal ( mixed ) strategy that guides the defender on how to move their dynamic cyber-system in the presence of a strategic and rational adversary . The notion of Strong Stackelberg Equilibrium predominantly underlies the definition of optimal strategies in these settings ( 4 ; 5 ) as the defender deploys a system first ( acting as a leader ) while the attacker , who seeks to attack the deployed system , assumes the role of the follower . In many real-world scenarios , single-stage normal-form games do not provide sufficient expressiveness to capture the switching costs of actions ( 4 ; 6 ) or reason about the adversary ’ s sequential behavior ( 7 ; 8 ) . On the other hand , works that consider modeling the MTD as a multi-stage stochastic game ( 9 ; 10 ; 11 ; 8 ) , do not model incomplete information about adversaries , a key aspect of the single-stage normal-form formalism ( known as Bayesian Stackelberg Games ( BSG ) ( 12 ; 6 ) ) . To address these concerns about expressiveness , while remaining scalable for use in cyber-security settings , we propose the unifying framework of Bayesian Stackelberg Markov Games ( BSMG ) . We show that BSMGs can be used to model various Moving Target Defense scenarios , capturing the uncertainty over attacker types and sequential impacts of attacks and switching defenses . We characterize the notion of optimal strategy as the Strong Stackelberg Equilibrium of BSMGs and show that the robust ( movement ) strategy improves the state-of-the-art found by previous game-theoretic modeling . While multi-stage game models are ubiquitous in security settings , expecting experts to provide detailed models about rewards and system transitions is considered unrealistic . Thus , researchers have considered techniques in reinforcement learning to learn optimal movement policies over time ( 13 ; 14 ; 15 ; 16 ) . Unfortunately , these works ignore ( 1 ) the strategic nature and the rational behavior of an adversary and ( 2 ) the incomplete knowledge a defender may possess about their opponent . This , as we show in our experiments , results in a new attack surface where the defender ’ s movement policy can be exploited by an adversary . To mitigate this , we bridge the knowledge gap between existing work , and techniques in multi-agent reinforcement learning by proposing a Bayesian Strong Stackelberg Q-learning ( BSS-Q ) approach ( graphically shown in Figure 1 ) . First , we can show that BSS-Q converges to the Strong Stackelberg Equilibrium of BSMGs . Second , we design an Open-AI gym ( 17 ) style multi-agent environment for two Moving Target Defenses ( one for web-application and the other for cloud-network security ) and compare the effectiveness of policies learned by BSS-Q against existing state-of-the-art static policies and other reinforcement learning agents . In the next section , we motivate the need for a unifying framework and formally describe the proposed game-theoretic model of BSMGs . We briefly discuss how two Moving Target Defenses are modeled as BSGMs . We then introduce the Bayesian Strong Stackelberg Q-learning approach and show that it converges to the SSE of BSMGs , followed by a section showcasing experimental results . Finally , before concluding , we discuss related work . 2 BAYESIAN STACKELBERG MARKOV GAMES ( BSMGS ) . Markov Games ( MGs ) ( 18 ) are used to model multi-agent interactions in sequential planning problems . Under this framework , a player can reason about the behavior of other agents ( co-operative or adversarial ) and come up with policies that adhere to some notion of equilibrium ( where no agent can gain by deviating away from the action or strategy profile ) . While MGs have been widely used to model adversarial scenarios , they suffer from two major shortcomings– ( 1 ) they do not consider incomplete information about the adversary ( 19 ; 7 ; 20 ; 13 ) and/or ( 2 ) they consider weak threat models where the attacker has no information about the defender ’ s policy ( 21 ; 14 ) . On the other hand , Bayesian Stackelberg Games ( 22 ; 6 ) are a single-stage game-theoretic formalism that addresses both of these concerns but can not be trivially generalized to sequential settings . To overcome these challenges of expressiveness required for Moving Target Defenses ( MTDs ) while ensuring scalability , we introduce the formalism of Bayesian Stackelberg Markov Games ( BSMGs ) . BSMGs extend Bayesian Stackelberg Games ( BSGs ) to multi-stage sequential games . While one can consider using existing formalism in Markov Games that capture incomplete information , they face severe scalability issues and have thus been unpopular in cyber-security domains ( we discuss how BSMG is situated in this landscape of models in section 5 ) . In the context of Moving Target Defense ( MTD ) , BSMG acts as a unifying framework helping us characterize optimal movement policies against strategic adversaries , capture transition dynamics and costs of the underlying cyber-system , aid in reasoning about stronger threat models , and consider incomplete information about strategic adversaries . Formally , a BSMG can be represented by the tuple ( P , S , Θ , A , τ , U , γD , γA ) where , • P = { D , A = { A1 , A2 , . . .At } } where D denotes the leader ( defender ) and A denotes the follower ( attacker ) . In our model , only the second player has t types . • S = { s1 , s2 , . . . , sk } are k ( finite ) states of the game , • Θ = { θ1 , θ2 , . . . θk } denotes k probability distributions ( for k states ) over the t attackers and θi ( s ) denotes the probability of i-th attacker type in state s • A = { AD , AA1 , . . . AAt } denotes the action set of the player and Ai ( s ) represents the set of actions/pure strategies available to player i in state s. • τ i ( s , aD , aAi , s′ ) represents the probability of reaching a state s′ ∈ S from the state s ∈ S when the D chooses aD and attacker type i choose the action aAi , • U = { UD , UA1 , . . . , UAt } where UD ( s , aD , aAi ) and U i ( s , aD , aAi ) represents the re- ward/utility of D and an attacker type Ai respectively if , in state s , actions aD and aAi are chosen by the players , • γi 7→ [ 0 , 1 ) is the discount factor for player i . We will assume that γD = γA〉 = γ . In BSMGs the individual stage games constitute normal-form Bayesian games with a distribution over attacker types ; this is in contrast to the unit probability over a single adversary type in MGs . Both in physical-security ( 22 ) and cyber-security ( 6 ) domains , defenders are known to have knowledge about follower types , a classic case of known-unknowns . BSMGs provide the expressive power to represent this information ; precisely θs represents the probability estimate with which a defender believes a certain kind of adversary is encountered in a particular state s of the game . Note that a defender D is expected to deploy a system first . Thus , a strong threat model assumes that all the attacker types Ai know the defender ’ s policy , making the Bayesian notion of Stackelberg Equilibrium an appropriate solution concept for such games . For a normal-form game , let a defender ’ s mixed policy be denoted as x and let us denote an attacker type Ai ’ s response set ( i.e . a set of best responses to x ) as Ri ( x ) . If the response set for all adversary types is singleton , then the action profile ( x , R1 ( x ) , . . . Rt ( x ) ) constitutes a Stackelberg Equilibrium of the normal-form game ( 23 ) . When the response set contains more than one action , the final response chosen can yield different rewards for D. In such cases , a popular assumption made in general-sum games is to consider the response that results in the optimal rewards for D ; this is termed as the Strong Stackelberg Equilibrium ( SSE ) ( 24 ; 22 ; 4 ; 6 ) . In contrast to the notion of Weak Stackelberg Equilibrium , which considers the pessimistic case , an SSE is guaranteed to exist and yields a unique game value to the defender regardless of the particular SSE chosen ( 25 ; 26 ) . Thus , we consider SSEs as the solution concept in BSMGs and highlight a few properties about player strategies at equilibrium for BSMGs ( the proofs are deferred to Appendix A ) . Lemma 1 . For a given policy of the leader/defender in BSMG , every follower/attacker type will have a deterministic policy in all states s ∈ S that is an optimal response . Corollary 1 . For an SSE policy of the defender , denoted as x , each attacker typeAi has a deterministic best policy qi . The action profile ( x , q1 , . . . qt ) denotes the SSE of the BSMG . Lemma 2 . If an action profile ( x , q1 , . . . , qt ) yields the equilibrium values V Dx , q and V Aix , q to the players and is an SSE of BSMG , then ∀s ∈ S ( x ( s ) , q1 ( s ) , . . . , qt ( s ) ) is an SSE of the bi-matrix Bayesian game represented by the Q-values QD , ix , qi ( s ) , Q Ai x , qi ( s ) ∀i ∈ { 1 , . . . , t } . When the parameters of a game are provided up-front , an approach similar to calculating Strong Stackelberg Equilibrium in Bayesian Games ( 12 ) alongside Mixed-Integer Non-Linear Programming approaches ( 7 ) or Bellman-style approaches for Markov Games ( 20 ) can be leveraged to find the defender ’ s policy . In contrast , when game-parameters are difficult to provide upfront but interaction with an environment is considered possible , we can resort to reinforcement learning techniques . Before proposing our model-free multi-agent reinforcement learning method in the next section , we briefly discuss how the various MTDs , used later in the experiments , are modeled as BSMGs .

Summary: This paper introduces a Bayesian Stackelberg Markov Game (BSMG) model that considers a defender’s uncertainty over attackers’ types when implementing defensive strategies. It also proposes to use a Bayesian Strong Stackelberg Q-learning method to learn defense policies by first simulating an adversary to obtain feedback of an attack and then computing the Bayesian Strong Stackelberg Equilibrium for the BSMG with a solver. In this way, this work relaxes the assumption that the defender knows attackers’ types in existing game-theoretic models for moving target defense.