Year: 2019

Abstract:
We re-think the Two-Player Reinforcement Learning (RL) as an instance of a distribution sampling problem in infinite dimensions. Using the powerful Stochastic Gradient Langevin Dynamics, we propose a new two-player RL algorithm, which is a sampling variant of the two-player policy gradient method. Our new algorithm consistently outperforms existing baselines, in terms of generalization across differing training and testing conditions, on several MuJoCo environments.

heading: 1 INTRODUCTION
text: Reinforcement learning (RL) promise automated solutions to many real-world tasks with beyondhuman performance. Indeed, recent advances in policy gradient methods (Sutton et al., 2000; Silver et al., 2014; Schulman et al., 2015; 2017) and deep reinforcement learning have demonstrated impressive performance in games (Mnih et al., 2015; Silver et al., 2017), continuous control (Lillicrap et al., 2015), and robotics (Levine et al., 2016) towards this grand challenge.
Despite the success of deep RL, the progress is still upset by the fragility in real-life deployments. In particular, majority of these methods fail to perform well when there is some difference between training and testing scenarios, thereby posting serious safety and security concerns. To this end, learning policies that are robust to environmental shifts, mismatched configurations, and even mismatched control actions is becoming increasingly more important.
A powerful framework to learning robust policies is to interpret the changing of the environment as an adversarial perturbation. This notion naturally lends itself to a two-player minimax problem involving a pair of agents, a protagonist and an adversary, where the protagonist learns to fulfill the original task goals while being robust to the disruptions generated by its adversary.
Two prominent examples along this research vein, differing in how they model the adversary, are the Robust Adversarial Reinforcement Learning (RARL) (Pinto et al., 2017) and Noisy Robust Markov Decision Process (NR-MDP) (Tessler et al., 2019). Despite achieving impressive performance in practice, these existing frameworks heavily rely on heuristic algorithms, and hence, suffer from lack of theoretical guarantees, even in idealistic cases with infinite data as well as computational power.
One critical challenge in robust RL setting is that while maximizing rewards is a well-studied subject in classical/deep RL, the needed two-player minimax version is significantly more complicated to solve both in theory and practice. For instance, Tessler et al. (2019) prove that it is in fact strictly suboptimal to directly apply (deterministic) policy gradient steps to their NR-MDP max-min objectives. Owing to the lack of a better algorithm, the policy gradient is nonetheless still employed in their experiments; similar comments also apply to (Pinto et al., 2017).
Our paper precisely bridges this gap between theory and practice in previous works, by proposing the first theoretically convergent algorithm for robust RL. Our key idea is to switch from optimizing the max-min reward to sampling from the optimal randomized policies, which corresponds to finding a mixed Nash Equilibrium (NE) (Nash et al., 1950) in the max-min objective.
It is a classical fact in game theory that, while deterministic minimax objective is often ill-posed, the mixed NE is well-behaved under very mild assumptions. Furthermore, algorithmic approaches to finding mixed NE with finite strategies have been studied extensively (Freund & Schapire, 1999; Nemirovski, 2004) and is recently extended to the case of infinite strategies (Hsieh et al., 2019). In particular, (Hsieh et al., 2019) show that, by using the Stochastic Gradient Langevin Dynamics
(SGLD) (Welling & Teh, 2011) to take samples from randomized strategies, one can find a mixed NE of Generative Adversarial Networks (Goodfellow et al., 2014).
Our work introduces the same mixed NE perspective to the max-min objectives in robust RL, and substantiate the ensuing theoretical framework with extensive experimental evidence. We apply the new sampling framework to the well-known Deep Deterministic Policy Gradient (DDPG) method in the scope of NR-MDP. We demonstrate that the new algorithm achieves clearly superior performance in its generalization capabilities.
Intriguingly, we also observe that the idea of mixed strategy in single-player RL (i.e., the non-robust or one-player formulation) can lead to substantially more robust policies over the standard DDPG algorithm. More precisely, we represent the agent’s policy as a distribution over deterministic policies, and aim to sample from the distribution µ(π) ∝ exp(− 1σJ(π)) where π denotes a deterministic policy, J the associated expected reward, and σ a temperature parameter going to 0 during training.
Our numerical evidence demonstrates that DDPG combined with this sampling approach for the actor update leads to learned policies that generalize better to unseen MDPs, when compared to the state-of-the-art DDPG variant of Tessler et al. (2019) while using similar computational resources.

heading: 2 BACKGROUND

heading: 2.1 STOCHASTIC GRADIENT LANGEVIN DYNAMICS (SGLD)
text: For any probability distribution p (z) ∝ exp (−g (z)), the Stochastic Gradient Langevin Dynamics (SGLD) Welling & Teh (2011) iterates as
zk+1 ← zk − γ [ ∇̂zg (z) ] z=zk + √ 2γ ξk, (1)
where γ is the step-size, ∇̂zg (z) is an unbiased estimator of ∇zg (z), > 0 is a temperature parameter, and ξk ∼ N (0, I) is a standard normal vector, independently drawn across different iterations. In some cases, the convergence rate of SGLD can be improved by scaling the noise using a positive-definite symmetric matrix C. We thus define a preconditioned variant of the above update equation 1 as follows:
zk+1 ← zk − γC−1 [ ∇̂zg (z) ] z=zk + √ 2γ C− 1 2 ξk. (2)
In the experiments, we use a RMSProp-preconditioned version of the SGLD (Li et al., 2016).

heading: 2.2 INFINITE-DIMENSIONAL BI-LINEAR GAMES
text: In this section, we review some of the key results from (Hsieh et al., 2019). We denote the set of all probability measures on Z by P (Z), and the set of all functions on Z by F (Z). Given a (sufficiently regular) function h : Θ×Ω→ R, consider the following objective (a two-player game with infinitely many strategies):
max p∈P(Θ) min q∈P(Ω) f (p, q) := E θ∼p [ E ω∼q [h (θ, ω)] ] . (3)
A pair (p∗, q∗) achieving the max-min value in equation 3 is called a mixed Nash Equilibrium (NE). Define the operator G : P (Ω)→ F (Θ), and its adjoint operator G† : P (Θ)→ F (Ω) as follows:
Gq (θ) := E ω∼q [h (θ, ω)] ∈ F (Θ)
G†p (ω) := E θ∼p [h (θ, ω)] ∈ F (Ω) .
Denoting 〈p, g〉 := E z∼p [g (z)] for any probability measure p and function g on Z , we can write f as f (p, q) = 〈p,Gq〉 = 〈 G†p, q 〉 . Furthermore, the derivative (the analogue of gradient in infinite
dimension) of f (p, q) with respect to p is simply Gq, and the derivative of f (p, q) with respect to q is G†p; i.e., ∇pf (p, q) = Gq, and ∇qf (p, q) = G†p.
Algorithm 1 Infinite-Dimensional Entropic Mirror Descent-v2 Input: Initial distributions p1, q1, and learning rate η for t = 1, 2, . . . , T − 1 do pt+1 (θ) ∝ exp ( +η ∑ s≤tGqs (θ)
) qt+1 (ω) ∝ exp ( −η ∑ s≤tG †ps (ω) )
end for Output: p̄T = 1T ∑T t=1 pt and q̄T = 1 T ∑T t=1 qt
Algorithm 2 Approximate Infinite-Dimensional Entropic Mirror Descent
Input: ω1, θ1 ← random initialization, SGLD step-size {γt}Tt=1, thermal noise of SGLD { t} T t=1, warmup steps for SGLD {Kt}Tt=1, exponential damping factor β, standard normal noise ξk, ξ′k. for t = 1, 2, . . . , T − 1 do ω̄t, ω (1) t ← ωt ; θ̄t, θ (1) t ← θt
for k = 1, 2, . . . ,Kt do θ
(k+1) t ← θ (k) t + γt [ ̂∇θh (θ, ωt) ] θ=θ
(k) t
+ √
2γt tξk
ω (k+1) t ← ω (k) t − γt [ ̂∇ωh (θt, ω) ] ω=ω
(k) t
+ √
2γt tξ ′ k
ω̄t ← (1− β) ω̄t + βω(k+1)t ; θ̄t ← (1− β) θ̄t + βθ (k+1) t
end for ωt+1 ← (1− β)ωt + βω̄t ; θt+1 ← (1− β) θt + βθ̄t
end for Output: ωT , θT .
Conceptually, problem (3) can be solved via the so-called infinite-dimensional entropic mirror descent; see Algorithm 1. Hsieh et al. (2019) have proved the convergence (to the mixed NE) rate of Algorithm 1. But this algorithm is infinite-dimensional and requires infinite computational power to implement. For practical interest, by leveraging the SGLD sampling techniques and using some practical relaxations, Hsieh et al. (2019) have proposed a simplified variant of Algorithm 1. The pseudocode for their resulting algorithm can be found in Algorithm 2.

heading: 3 TWO-PLAYER MARKOV GAMES
text: Markov Decision Process: We consider a Markov Decision Process (MDP) represented by M1 := (S,A, T1, γ, P0, R1), where the state and action spaces are denoted by S and A respectively. We focus on continuous control tasks, where the actions are real-valued, i.e., A = Rd. T1 : S × S × A → [0, 1] captures the state transition dynamics, i.e., T1 (s′ | s, a) denotes the probability of landing in state s′ by taking action a from state s. Here γ is the discounting factor, P0 : S → [0, 1] is the initial distribution over states S, and R1 : S ×A → R is the reward.
Two-Player Zero-Sum Markov Games: Consider a two-player zero-sum Markov game Littman (1994); Perolat et al. (2015), where at each step of the game, both players simultaneously choose an action. The reward each player gets after one step depends on the state and the joint action of both players. Furthermore, the transition kernel of the game is controlled jointly by both the players. In this work, we only consider simultaneous games, not the turn-based games.
This game can be described by an MDPM2 = (S,A,A′, T2, γ, R2, P0), where A and A′ are the continuous set of actions the players can take, T2 : S × A × A′ × S → R is the state transition probability, and R2 : S × A×A′ → R is the reward for both players. Consider an agent executing a policy µ : S → A, and an adversary executing a policy ν : S → A′ in the environmentM. At each timestep t, both players observe the state st and take actions at = µ (st) and a′t = ν (st). In the zero-sum game, the agent gets a reward rt = R2 (st, at, a′t) while the adversary gets a negative reward −rt.
This two-player zero-sum Markov game formulation has been used to model the following robust RL settings:
• Robust Adversarial Reinforcement Learning (RARL) (Pinto et al., 2017), where the power of the adversary is limited by the action space A′ of the adversary.
• Noisy Robust Markov Decision Process (NR-MDP) (Tessler et al., 2019), where A′ = A, T2 (st+1 | st, at, a′t) = T1 (st+1 | st, āt), and R2 (st, at, a′t) = R1 (st, āt), with āt = (1− δ)at + δa′t, for δ ∈ (0, 1). The power of the adversary is limited by δ.
In our adversarial game, we consider the following performance objective:
J (µ, ν) = E [ ∞∑ t=1 γt−1rt ∣∣∣∣ µ, ν,M ] ,
where ∑∞ t=1 γ
t−1rt be the random cumulative return. In particular, we consider the parameterized policies {µθ : θ ∈ Θ}, and {νω : ω ∈ Ω}. By an abuse of notation, we denote J (θ, ω) = J (µθ, νω). We consider the following objective:
max θ∈Θ min ω∈Ω J (θ, ω) . (4)
Note that J is non-convex/concave in both θ and ω. Instead of solving equation 4 directly, we focus on the mixed strategy formulation of equation 4. In other words, we consider the set of all probability distributions over Θ and Ω, and we search for the optimal distribution that solves the following program:
max p∈P(Θ) min q∈P(Ω) f (p, q) := E θ∼p [ E ω∼q [J (θ, ω)] ] . (5)
Then, we can use the techniques from Section 2.2 to solve the above problem.

heading: 4 EXPERIMENTS
text: In this section, we demonstrate the effectiveness of using infinite-dimensional sampling techniques to solve the robust RL problem.
Two-Player DDPG: As a case study, we consider NR-MDP setting with δ = 0.1 (as recommended in Section 6.3 of (Tessler et al., 2019)). We design a two-player variant of DDPG (Lillicrap et al., 2015) algorithm by adapting the Algorithm 2. As opposed to standard DDPG, in two-player DDPG two actor networks output two deterministic policies, the protagonist and adversary policies, denoted by µθ and νω . The critic is trained to estimate the Q-function of the joint-policy. The gradients of the protagonist and adversary parameters are given in Proposition 5 of (Tessler et al., 2019). The resulting algorithm is given in Algorithm 3.
We compare the performance of our algorithm against the baseline algorithm proposed in (Tessler et al., 2019) (see Algorithm 4). (Tessler et al., 2019) have suggested a training ratio of 1 : 1 for actors and critic updates. Note that the action noise is injected while collecting transitions for the replay buffer. In Fujimoto et al. (2018), authors noted that the action noise drawn from the OrnsteinUhlenbeck Uhlenbeck & Ornstein (1930) process offered no performance benefits. Thus we also consider uncorrelated Gaussian noise.
Setup: We evaluate the performance of Algorithm 3 and Algorithm 4 on standard continuous control benchmarks available on OpenAI Gym Brockman et al. (2016) utilizing the MuJoCo environment Todorov et al. (2012). Specifically, we benchmark on eight tasks: Walker, Hopper, HalfCheetah, Ant, Swimmer, Reacher, Humanoid, and InvertedPendulum. Details of these environments can be found in Brockman et al. (2016) and on the GitHub website.
The Algorithm 3 implementation is based on the codebase from (Tessler et al., 2019). For all the algorithms, we use a two-layer feedforward neural network structure of (64, 64, tanh) for both actors (agent and adversary) and critic. The optimizer we use to update the critic is Adam Kingma & Ba (2015) with a learning rate of 10−3. The target networks are soft-updated with τ = 0.999.
For the baseline, the actors are trained with RMSProp optimizer. For our algorithm, the actors are updated according to Algorithm 2 with warmup steps Kt = min { 15, b(1 + 10−5)tc } , and thermal noise σt = σ0 × (1 − 5 × 10−5)t. The hyperparameters that are not related to exploration (see Table 1) are identical to both algorithms that are compared.
And we tuned only the exploration-related hyper-parameters (for both algorithms) by grid search: (a) Algorithm 3 with (σ0, σ) ∈ { 10−2, 10−3, 10−4, 10−5 } × {0, 0.01, 0.1, 0.2, 0.3, 0.4} ; (b) Algorithm 4 with σ ∈ {0, 0.01, 0.1, 0.2, 0.3, 0.4}. For each algorithm-environment pair, we identified the best performing exploration hyperparameter configuration (see Tables 2 and 3).
Each algorithm is trained on 0.5M samples (i.e., 0.5M time steps in the environment). We run our experiments, for each environment, with 5 different seeds. The exploration noise is turned off for evaluation.
Evaluation: We evaluate the robustness of both algorithms under different testing conditions, and in the presence of adversarial disturbances in the testing environment. We train both algorithms with the standard mass variables in OpenAI Gym. At test time, we evaluate the learned policies by changing the mass values (without adversarial perturbations) and estimating the cumulative rewards. As shown in Figure 1, our Algorithm 3 outperforms the baseline Algorithm 4 in terms of robustness. We also evaluate the robustness of the learned policies under both test condition changes, and adversarial disturbances (see Figure 2).
One-Player DDPG: We evaluate the robustness of one-player variants of Algorithm 3, and Algorithm 4, i.e., we consider the NR-MDP setting with δ = 0. In this case, we set Kt = 1 for
Algorithm 3 (this choice of Kt makes the computational complexity of both algorithms equal). The results are presented in Figures 3 and 4.
Here, we remark that Algorithm 3 with δ = 0, and Kt = 1 is simply the standard DDPG with actor being updated by preconditioned version of SGLD. Thus we achieve robustness under dif-
ferent testing conditions with just a simple change in the DDPG algorithm and without additional computational cost.

heading: 5 CONCLUSION
text: In this work, we studied the robust reinforcement learning problem. By adapting the approximate infinite-dimensional entropic mirror descent from (Hsieh et al., 2019), we design a robust variant of DDPG algorithm, under the NR-MDP setting. In our experiments, we evaluated the robustness of our algorithm on several continuous control tasks, and found that our algorithm clearly outperformed the baseline algorithm from (Tessler et al., 2019).

heading: A ALGORITHMS AND HYPERPARAMETER DETAILS
text: Most of the values for the hyperparameters (in Table 1) are chosen from Dhariwal et al. (2017). In addition, for Algorithm 3, we set
1. thermal noise σt = σ0 × (1− 5× 10−5)t, where σ0 ∈ { 10−2, 10−3, 10−4, 10−5 } .
2. warmup steps Kt = min { 15, b(1 + 10−5)tc }
The best performing values for every environment are presented in Tables 2 and 3.

heading: B TOY EXAMPLE
text: Setting. We consider the following simplified/modified hide and seek (running and chasing) game from Baker et al. (2019):
• Two agents (a hider and a seeker) are placed in 2d-square environment without any obstacles. The hider wants to be away from the seeker as much as possible.
• The state (at time t) is represented by st = ( `hidet , ` seek t , v hide t , v seek t ) ∈ S, where `hidet is
the location of the hider, and vseekt is the velocity of the seeker. We assume that both agents observe the full state.
• The action space is same for both agents, i.e., ahidet , aseekt ∈ A = {left, right,up,down}. • The reward function is given by Rt = −Rhide (st) = Rseek (st) = 1d(`hidet ,`seekt ) , i.e.,
inverse of the current distance between the hider and seeker. • The additional parameters fhide, and f seek denote the force of the hider and the seeker
respectively. We fix fhide = 4, and let f seek ∈ {2, 3, 4}. • The transition dynamics T ( st+1 | st, ahidet , aseekt ; fhide, f seek ) depends on the parameters
fhide, and f seek. • We have pre-trained (three) competitive seeker policies π̄fseek for all f seek ∈ {2, 3, 4}.
Training. For training, we fix setting with fhide = 4, and f seek = 4. We train the parametric policies πhideθ , and π seek w using the following two approaches (generate an episode . . . , St, A hide t , A seek t , Rt, St+1, . . . using current parameters, and define Gt = ∑T τ=0 γ τRt+τ ):
• Gradient descent ascent (GDA) approach: θt+1 ← θt − ηtGt [ ∇θ lnπhideθ ( Ahidet | St )] θ=θt
wt+1 ← wt + ηtGt [ ∇w lnπseekw ( Aseekt | St )] w=wt
• Stochastic Gradient Langevin Dynamics (SGLD) approach: θt+1 ← θt − ηtGt [ ∇θ lnπhideθ ( Ahidet | St )] θ=θt + √ 2ηtN (0, I)
wt+1 ← wt + ηtGt [ ∇w lnπseekw ( Aseekt | St )] w=wt + √ 2ηtN (0, I)
We denote the final hider policy parameters resulting from the above methods as θGDA, and θSGLD.
Evaluation. We compare the performance of the policies πhideθGDA , and π hide θSGLD under three different settings: T ( st+1 | st, ahidet , aseekt ; fhide, f seek ) , aseekt ∼ π̄vseek (· | st), fhide = 4, and f seek ∈ {2, 3, 4}. The results (averaged over 10 initial seeds, and 100 episodes per seed) are shown in Figure 5. In this simplified setting, on average, πhideθSGLD performs better than π hide θGDA
, but the difference in performance is not that significant as in the MuJoCo environments.

heading: C ADDITIONAL RESULTS
text: The results for the best performing seeds are presented in Figures 6 and 7.
Algorithm 3 Two-Player DDPG with SGLD Actors (SGLD) Hyperparameters: see Table 1 Initialize (randomly) policy parameters ω1, θ1, and Q-function parameter φ. Initialize the target network parameters ωtarg ← ω1, θtarg ← θ1, and φtarg ← φ. Initialize replay buffer D. Initialize m← 0 ; m′ ← 0. t← 1. repeat
Observe state s, and select actions a = µθt(s) + ξ ; a ′ = νωt(s) + ξ ′, where ξ, ξ′ ∼ N (0, σI) Execute the action ā = (1− δ)a+ δa′ in the environment. Observe reward r, next state s′, and done signal d to indicate whether s′ is terminal. Store (s, ā, r, s′, d) in replay buffer D. If s′ is terminal, reset the environment state. if it’s time to update then
for however many updates do ω̄t, ω (1) t ← ωt ; θ̄t, θ (1) t ← θt
for k = 1, 2, . . . ,Kt do Sample a random minibatch of N transitions B = {(s, ā, r, s′, d)} from D. Compute targets y (r, s′, d) = r+γ (1− d)Qφtarg ( s′, (1− δ)µθtarg (s′) + δνωtarg (s′) ) .
Update critic by one step of (preconditioned) gradient descent using∇φL (φ), where
L (φ) = 1
N ∑ (s,ā,r,s′,d)∈B (y (r, s′, d)−Qφ (s, ā)) 2 .
Compute the (agent and adversary) policy gradient estimates:
̂∇θJ (θ, ωt) = 1− δ N ∑ s∈D ∇θµθ (s)∇āQφ (s, ā) |ā=(1−δ)µθ(s)+δνωt (s)
̂∇ωJ (θt, ω) = δ
N ∑ s∈D ∇ωνω (s)∇āQφ (s, ā) |ā=(1−δ)µθt (s)+δνω(s).
g ← [ ̂∇θJ (θ, ωt) ] θ=θ
(k) t
; m← αm+ (1− α) g g ; C ← diag (√ m+ ) θ
(k+1) t ← θ (k) t + ηC
−1g + √ 2ησtC − 12 ξ, where ξ ∼ N (0, I) g′ ← [ ̂∇ωJ (θt, ω) ] ω=ω
(k) t
; m′ ← αm′ + (1− α) g′ g′ ; D ← diag (√ m′ + ) ω
(k+1) t ← ω (k) t − ηD−1g′ + √ 2ησtD
− 12 ξ′, where ξ′ ∼ N (0, I) ω̄t ← (1− β) ω̄t + βω(k+1)t ; θ̄t ← (1− β) θ̄t + βθ (k+1) t Update the target networks:
φtarg ← τφtarg + (1− τ)φ
θtarg ← τθtarg + (1− τ)θ(k+1)t ωtarg ← τωtarg + (1− τ)ω(k+1)t
end for ωt+1 ← (1− β)ωt + βω̄t ; θt+1 ← (1− β) θt + βθ̄t t← t+ 1.
end for end if
until convergence Output: ωT , θT .
Algorithm 4 Two-Player DDPG with RMSProp Actors (Baseline) Hyperparameters: see Table 1 Initialize (randomly) policy parameters ω1, θ1, and Q-function parameter φ. Initialize the target network parameters ωtarg ← ω1, θtarg ← θ1, and φtarg ← φ. Initialize replay buffer D. Initialize m← 0 ; m′ ← 0. t← 1. repeat
Observe state s, and select actions a = µθt(s) + ξ ; a ′ = νωt(s) + ξ ′, where ξ, ξ′ ∼ N (0, σI) Execute the action ā = (1− δ)a+ δa′ in the environment. Observe reward r, next state s′, and done signal d to indicate whether s′ is terminal. Store (s, ā, r, s′, d) in replay buffer D. If s′ is terminal, reset the environment state. if it’s time to update then
for however many updates do Sample a random minibatch of N transitions B = {(s, ā, r, s′, d)} from D. Compute targets y (r, s′, d) = r + γ (1− d)Qφtarg ( s′, (1− δ)µθtarg (s′) + δνωtarg (s′) ) .
Update critic by one step of (preconditioned) gradient descent using∇φL (φ), where
L (φ) = 1
N ∑ (s,ā,r,s′,d)∈B (y (r, s′, d)−Qφ (s, ā)) 2 .
Compute the (agent and adversary) policy gradient estimates:
̂∇θJ (θ, ωt) = 1− δ N ∑ s∈D ∇θµθ (s)∇āQφ (s, ā) |ā=(1−δ)µθ(s)+δνωt (s)
̂∇ωJ (θt, ω) = δ
N ∑ s∈D ∇ωνω (s)∇āQφ (s, ā) |ā=(1−δ)µθt (s)+δνω(s).
g ← [ ̂∇θJ (θ, ωt) ] θ=θt ; m← αm+ (1− α) g g ; C ← diag (√ m+ ) θt+1 ← θt + ηC−1g g′ ← [ ̂∇ωJ (θt, ω) ] ω=ωt ; m′ ← αm′ + (1− α) g′ g′ ; D ← diag (√ m′ +
) ωt+1 ← ωt − ηD−1g′ Update the target networks:
φtarg ← τφtarg + (1− τ)φ θtarg ← τθtarg + (1− τ)θt+1 ωtarg ← τωtarg + (1− τ)ωt+1
t← t+ 1. end for end if until convergence Output: ωT , θT .

Review summaries:
The paper considers the task of learning robust policies. Specifically, they focus on the noisy-robust (NR) variant of the action robust framework proposed in [1]. As noted in [1], as strong duality does not hold in the NR variant it can not be solved using deterministic policies (pure strategies).