Learning Movement Strategies for Moving Target Defense

1 INTRODUCTION . The complexity of modern-day software technology has made the goal of deploying fully secure cyber-systems impossible . Furthermore , an attacker often has ample time to explore a deployed system before exploiting it . To level the playing field , researchers have introduced the idea of proactive cyber defenses such as Moving Target Defense . In Moving Target Defense ( MTD ) , the defender shifts between various configurations of the cyber-system ( 1 ) . This makes the attacker ’ s knowledge , gathered during the reconnaissance phase , useless at attack time as the system may have shifted to a new configuration in the window between reconnaissance and attack . To ensure that an MTD system is effective at maximizing security and minimizing the impact on the system ’ s performance , the consideration of an optimal movement strategy is important ( 2 ; 3 ) . MTD systems render themselves naturally to a game-theoretic formulation– modeling the cybersystem as a two-player game between the defender and an attacker is commonplace . The expectation is that the equilibrium of these games yields an optimal ( mixed ) strategy that guides the defender on how to move their dynamic cyber-system in the presence of a strategic and rational adversary . The notion of Strong Stackelberg Equilibrium predominantly underlies the definition of optimal strategies in these settings ( 4 ; 5 ) as the defender deploys a system first ( acting as a leader ) while the attacker , who seeks to attack the deployed system , assumes the role of the follower . In many real-world scenarios , single-stage normal-form games do not provide sufficient expressiveness to capture the switching costs of actions ( 4 ; 6 ) or reason about the adversary ’ s sequential behavior ( 7 ; 8 ) . On the other hand , works that consider modeling the MTD as a multi-stage stochastic game ( 9 ; 10 ; 11 ; 8 ) , do not model incomplete information about adversaries , a key aspect of the single-stage normal-form formalism ( known as Bayesian Stackelberg Games ( BSG ) ( 12 ; 6 ) ) . To address these concerns about expressiveness , while remaining scalable for use in cyber-security settings , we propose the unifying framework of Bayesian Stackelberg Markov Games ( BSMG ) . We show that BSMGs can be used to model various Moving Target Defense scenarios , capturing the uncertainty over attacker types and sequential impacts of attacks and switching defenses . We characterize the notion of optimal strategy as the Strong Stackelberg Equilibrium of BSMGs and show that the robust ( movement ) strategy improves the state-of-the-art found by previous game-theoretic modeling . While multi-stage game models are ubiquitous in security settings , expecting experts to provide detailed models about rewards and system transitions is considered unrealistic . Thus , researchers have considered techniques in reinforcement learning to learn optimal movement policies over time ( 13 ; 14 ; 15 ; 16 ) . Unfortunately , these works ignore ( 1 ) the strategic nature and the rational behavior of an adversary and ( 2 ) the incomplete knowledge a defender may possess about their opponent . This , as we show in our experiments , results in a new attack surface where the defender ’ s movement policy can be exploited by an adversary . To mitigate this , we bridge the knowledge gap between existing work , and techniques in multi-agent reinforcement learning by proposing a Bayesian Strong Stackelberg Q-learning ( BSS-Q ) approach ( graphically shown in Figure 1 ) . First , we can show that BSS-Q converges to the Strong Stackelberg Equilibrium of BSMGs . Second , we design an Open-AI gym ( 17 ) style multi-agent environment for two Moving Target Defenses ( one for web-application and the other for cloud-network security ) and compare the effectiveness of policies learned by BSS-Q against existing state-of-the-art static policies and other reinforcement learning agents . In the next section , we motivate the need for a unifying framework and formally describe the proposed game-theoretic model of BSMGs . We briefly discuss how two Moving Target Defenses are modeled as BSGMs . We then introduce the Bayesian Strong Stackelberg Q-learning approach and show that it converges to the SSE of BSMGs , followed by a section showcasing experimental results . Finally , before concluding , we discuss related work . 2 BAYESIAN STACKELBERG MARKOV GAMES ( BSMGS ) . Markov Games ( MGs ) ( 18 ) are used to model multi-agent interactions in sequential planning problems . Under this framework , a player can reason about the behavior of other agents ( co-operative or adversarial ) and come up with policies that adhere to some notion of equilibrium ( where no agent can gain by deviating away from the action or strategy profile ) . While MGs have been widely used to model adversarial scenarios , they suffer from two major shortcomings– ( 1 ) they do not consider incomplete information about the adversary ( 19 ; 7 ; 20 ; 13 ) and/or ( 2 ) they consider weak threat models where the attacker has no information about the defender ’ s policy ( 21 ; 14 ) . On the other hand , Bayesian Stackelberg Games ( 22 ; 6 ) are a single-stage game-theoretic formalism that addresses both of these concerns but can not be trivially generalized to sequential settings . To overcome these challenges of expressiveness required for Moving Target Defenses ( MTDs ) while ensuring scalability , we introduce the formalism of Bayesian Stackelberg Markov Games ( BSMGs ) . BSMGs extend Bayesian Stackelberg Games ( BSGs ) to multi-stage sequential games . While one can consider using existing formalism in Markov Games that capture incomplete information , they face severe scalability issues and have thus been unpopular in cyber-security domains ( we discuss how BSMG is situated in this landscape of models in section 5 ) . In the context of Moving Target Defense ( MTD ) , BSMG acts as a unifying framework helping us characterize optimal movement policies against strategic adversaries , capture transition dynamics and costs of the underlying cyber-system , aid in reasoning about stronger threat models , and consider incomplete information about strategic adversaries . Formally , a BSMG can be represented by the tuple ( P , S , Θ , A , τ , U , γD , γA ) where , • P = { D , A = { A1 , A2 , . . .At } } where D denotes the leader ( defender ) and A denotes the follower ( attacker ) . In our model , only the second player has t types . • S = { s1 , s2 , . . . , sk } are k ( finite ) states of the game , • Θ = { θ1 , θ2 , . . . θk } denotes k probability distributions ( for k states ) over the t attackers and θi ( s ) denotes the probability of i-th attacker type in state s • A = { AD , AA1 , . . . AAt } denotes the action set of the player and Ai ( s ) represents the set of actions/pure strategies available to player i in state s. • τ i ( s , aD , aAi , s′ ) represents the probability of reaching a state s′ ∈ S from the state s ∈ S when the D chooses aD and attacker type i choose the action aAi , • U = { UD , UA1 , . . . , UAt } where UD ( s , aD , aAi ) and U i ( s , aD , aAi ) represents the re- ward/utility of D and an attacker type Ai respectively if , in state s , actions aD and aAi are chosen by the players , • γi 7→ [ 0 , 1 ) is the discount factor for player i . We will assume that γD = γA〉 = γ . In BSMGs the individual stage games constitute normal-form Bayesian games with a distribution over attacker types ; this is in contrast to the unit probability over a single adversary type in MGs . Both in physical-security ( 22 ) and cyber-security ( 6 ) domains , defenders are known to have knowledge about follower types , a classic case of known-unknowns . BSMGs provide the expressive power to represent this information ; precisely θs represents the probability estimate with which a defender believes a certain kind of adversary is encountered in a particular state s of the game . Note that a defender D is expected to deploy a system first . Thus , a strong threat model assumes that all the attacker types Ai know the defender ’ s policy , making the Bayesian notion of Stackelberg Equilibrium an appropriate solution concept for such games . For a normal-form game , let a defender ’ s mixed policy be denoted as x and let us denote an attacker type Ai ’ s response set ( i.e . a set of best responses to x ) as Ri ( x ) . If the response set for all adversary types is singleton , then the action profile ( x , R1 ( x ) , . . . Rt ( x ) ) constitutes a Stackelberg Equilibrium of the normal-form game ( 23 ) . When the response set contains more than one action , the final response chosen can yield different rewards for D. In such cases , a popular assumption made in general-sum games is to consider the response that results in the optimal rewards for D ; this is termed as the Strong Stackelberg Equilibrium ( SSE ) ( 24 ; 22 ; 4 ; 6 ) . In contrast to the notion of Weak Stackelberg Equilibrium , which considers the pessimistic case , an SSE is guaranteed to exist and yields a unique game value to the defender regardless of the particular SSE chosen ( 25 ; 26 ) . Thus , we consider SSEs as the solution concept in BSMGs and highlight a few properties about player strategies at equilibrium for BSMGs ( the proofs are deferred to Appendix A ) . Lemma 1 . For a given policy of the leader/defender in BSMG , every follower/attacker type will have a deterministic policy in all states s ∈ S that is an optimal response . Corollary 1 . For an SSE policy of the defender , denoted as x , each attacker typeAi has a deterministic best policy qi . The action profile ( x , q1 , . . . qt ) denotes the SSE of the BSMG . Lemma 2 . If an action profile ( x , q1 , . . . , qt ) yields the equilibrium values V Dx , q and V Aix , q to the players and is an SSE of BSMG , then ∀s ∈ S ( x ( s ) , q1 ( s ) , . . . , qt ( s ) ) is an SSE of the bi-matrix Bayesian game represented by the Q-values QD , ix , qi ( s ) , Q Ai x , qi ( s ) ∀i ∈ { 1 , . . . , t } . When the parameters of a game are provided up-front , an approach similar to calculating Strong Stackelberg Equilibrium in Bayesian Games ( 12 ) alongside Mixed-Integer Non-Linear Programming approaches ( 7 ) or Bellman-style approaches for Markov Games ( 20 ) can be leveraged to find the defender ’ s policy . In contrast , when game-parameters are difficult to provide upfront but interaction with an environment is considered possible , we can resort to reinforcement learning techniques . Before proposing our model-free multi-agent reinforcement learning method in the next section , we briefly discuss how the various MTDs , used later in the experiments , are modeled as BSMGs .

Summary: This paper proposes the game-theoretic model of Bayesian Stackelberg Markov Games (BSMGs), a generalization of Markov games, as a formalism for studying Moving Target Defense (MTD) systems, a type of defender-attacker game with applications to cybersecurity. An algorithm for finding the Stackelberg equilibrium in BSMGs, called Bayesian Strong Stackelberg Q-Learning (BSS-Q) is proposed, and an OpenAI Gym-style environment for testing the derived policies in particular MTD settings is introduced, which allows for empirical evaluation of the policies' effectiveness. The paper then shows experimental results supporting the BSS-Q algorithm's success at finding the Strong Stackelberg Equilibrium of BSMGs.