Title: Robust Reinforcement Learning with General Utility
Venue: NeurIPS 2024
Accepted: yes
Keywords: robust reinforcement Learning

general utility

minimax optimization
Abstract: Reinforcement Learning (RL) problem with general utility is a powerful decision making framework that covers standard RL with cumulative cost, exploration problems, and demonstration learning. Existing works on RL with general utility do not consider the robustness under environmental perturbation, which is important to adapt RL system in the real-world environment that differs from the training environment. To train a robust policy, we propose a robust RL framework with general utility,  which subsumes many existing RL frameworks including RL, robust RL, RL with general utility, constrained RL, robust constrained RL, pure exploration, robust entropy regularized RL, etc. Then we focus on popular convex utility functions, with which our proposed learning framework is a challenging nonconvex-nonconcave minimax optimization problem, and design a two-phase stochastic policy gradient type algorithm and obtain its sample complexity result for gradient convergence. Furthermore, for convex utility on a widely used polyhedral ambiguity set, we design an algorithm and obtain its convergence rate to a global optimal solution.

Robust Reinforcement Learning with General Utility

Abstract

Reinforcement Learning (RL) problem with general utility is a powerful decision making framework that covers standard RL with cumulative cost, exploration problems, and demonstration learning. Existing works on RL with general utility do not consider the robustness under environmental perturbation, which is important to adapt RL system in the real-world environment that differs from the training environment. To train a robust policy, we propose a robust RL framework with general utility, which subsumes many existing RL frameworks including RL, robust RL, RL with general utility, constrained RL, robust constrained RL, pure exploration, robust entropy regularized RL, etc. Then we focus on popular convex utility functions, with which our proposed learning framework is a challenging nonconvex-nonconcave minimax optimization problem, and design a two-phase stochastic policy gradient type algorithm and obtain its sample complexity result for gradient convergence. Furthermore, for convex utility on a widely used polyhedral ambiguity set, we design an algorithm and obtain its convergence rate to a global optimal solution.

1 Introduction

Reinforcement learning (RL) is an important decision-making framework [41] aiming to find the optimal policy that minimizes accumulative cost, which is also a linear utility function of occupancy measure. Recent works have extended standard RL to more general utility functions to account for a variety of practical needs, including risk-sensitive applications [22, 8, 52], exploration maximization [18, 54, 51, 13, 6], and safety constraints [54, 51, 13]. There are provably convergent algorithms to solve RL with general utility [54, 55, 6]. However, these works study RL with general utility in a fixed environment, which may fail in many applications where the policy is trained in a simulation environment but implemented in a different real-world environment [37, 56].

To make the policy robust to such environmental change, robust RL has been proposed to find the optimal robust policy under the worst possible environment [36, 20, 48, 45, 14]. However, all the existing robust RL works restrict to linear utility function to our knowledge. Therefore, we aim to answer the following research question:

Q: Can we train a robust policy for RL with general utility and obtain convergence results?

1.1 Our Contributions

We affirmatively answer this question by proposing robust RL with general utility, the first learning framework that obtains a robust policy for general utility in the worst possible environment. It is formulated as a minimax optimization problem min θ ∈ Θ max ξ ∈ Ξ f ( λ θ,ξ ) where f is the utility function and λ θ,ξ is the occupancy measure under the policy parameter θ ∈ Θ and the environmental

transition kernel parameter ξ ∈ Ξ . Our robust RL with general utility is a combination of its two important special cases, namely, RL with general utility [54] (formulated as min θ ∈ Θ f ( λ θ,ξ ) where the environmental parameter ξ is fixed) and robust RL [36] where f is restricted to linear utility function. This new learning framework also covers many other existing RL frameworks including constrained RL [2] and robust constrained RL [43] with safety critical applications such as healthcare and unmanned drones, entropy regularized RL [10] and its robust extension [32] which help agents learn from human demonstration, pure exploration [18] with application to explore an environment with sparse reward signals and its robust extension, etc. These examples use convex utility functions f , which is the focus of this paper. See Section 2.1 for details of these examples.

Then, we focus on designing provably convergent algorithms for our new proposed learning framework with the widely used convex utility function f . In this case, our objective min θ max ξ f ( λ θ,ξ ) is still a highly challenging nonconvex-nonconcave minimax optimization problem. Hence, we have to utilize the structure and properties of λ θ,ξ to design algorithms and obtain convergence results. To elaborate, we design a projected stochastic gradient descent ascent algorithm with two phases. Interestingly, the first phase targeted at the objective function f obtains a stationary point of a different envelope function. Hence, we add a second phase targeted at a corrected objective ˜ f to converge to a near-stationary solution of the original objective f . The convergence analysis is non-trivial with two novel techniques. First, we have proved a projected gradient dominance property (Proposition 4) that is much stronger than the existing one on convex utility, with less assumptions, no bias term and applicability to more general parameterized policy. Second, in the convergence analysis of the second phase, we obtain convergence to a global Nash equilibrium (thus a stationary point) of ˜ f by Proposition 4, which is close to a stationary point of f by proving that ∇ ξ ˜ f ( λ θ,ξ ) ≈∇ ξ f ( λ θ,ξ ) .

Furthermore, with convex utility function f and the widely used s -rectangular polyhedral ambiguity set Ξ (including the popular L 1 and L ∞ ambiguity sets), we design an alternative algorithm which converges to a global optimal solution of this nonconvex-nonconcave optimization problem at a sublinear rate O (1 /K ) (Theorem 3). This is much more challenging than global convergence for convex RL (that is, RL with convex utility function and fixed ξ ) [54, 51, 6] and for robust RL with linear utility satisfying Bellman equation [36, 20, 45, 15, 25], so we need novel algorithm design and novel techniques. First, we prove that arg max ξ f ( λ θ,ξ ) can be obtained in the finite set of vertices V (Ξ) (Proposition 6). This is intuitive if f ( λ θ, · ) is a convex function but in many applications, only f ( λ ) is convex. To solve this challenge, we prove a novel local invertibility property of λ θ, · (Proposition 5) by checking Bellman equation of λ θ,ξ state by state in two cases. Then we prove Proposition 6 using a novel state-by-state extension from an optimal non-vertex ξ to an optimal vertex ξ . Second, the major difficulty to design our algorithm is to find a descent direction of Γ( θ ) := max ξ f ( λ θ,ξ ) . We select the near-optimal vertices ξ ∈ Ξ k ⊂ V (Ξ) that may affect the optimization progress Γ( θ k +1 ) -Γ( θ k ) , and find the descent direction with provably large descent for all the corresponding functions { f ( λ θ k ,ξ ) } ξ ∈ Ξ k (Proposition 7) via convex optimization. Third, by Proposition 6, the global convergence measure ∆ k := Γ( θ k ) -min θ Γ( θ ) at each iteration k either is O (1 /k ) -close to optimal ( Γ( θ k ) ≤ O (1 /k ) ) or enjoys large descent (Eq. (26)), so we prove the convergence in 3 cases: O (1 /K ) -optimal final θ K , iterate Eq. (26) from θ 0 or from a O (1 /k ) -optimal intermediate θ k .

1.2 Related Works

RL with General Utility. Standard RL aims to optimize over the accumulated reward/cost [21, 41]. Some early operation research works focus on other non-linear objectives such as variance-penalized MDPs [12], risk-sensitive objectives [22, 8, 52], entropy exploration [18], constrained RL [2, 1, 35] and learning from demonstration [39, 3].

[54] proposes RL with general utilities to cover the above applications and applies variational policy gradient method that provably converges to the global optimal solution for convex utility. [55] proposes a variance reduced policy gradient algorithm which requires ˜ O ( ϵ -3 ) samples to achieve an ϵ -stationary policy for general utility and ˜ O ( ϵ -2 ) samples to achieve an ϵ -global optimal policy for convex utility and overparameterized policy. [51] provides a meta-algorithm to solve the convex MDP problem as a min-max game between a policy player and a cost player who produces rewards that the policy player must maximize. They further show that any method-solving problems under the standard RL settings can be used to solve the more general convex MDP problem. [27] obtains policy

gradient theorem for RL with general utilities. [6] proposes a simpler single-loop parameter-free normalized policy gradient algorithm with recursive momentum variance reduction. This algorithm requires ˜ O ( ϵ -3 ) samples to achieve ϵ -stationary policy in general and ˜ O ( ϵ -2 ) samples to achieve ϵ -global optimal policy for convex utility. For large finite state action spaces, it requires ˜ O ( ϵ -4 ) samples to achieve ϵ -stationary policy via linear function approximation of the occupancy measure. [53] proposes decentralized multi-agent RL with general utilities. [13] shows that convex RL is a subclass of multi-agent mean-field games.

Robust RL. Robust RL is designed to learn a policy that is robust to perturbation of environmental factors. Usually robust RL is NP-hard [45], but becomes tractable for ambiguity sets that is ( s, a ) -rectangular [36, 20, 45, 44, 29, 56] or s -rectangular [45, 42, 23, 26]. Methods to solve robust RL include value iteration [36, 20, 45, 15, 25], policy iteration [20, 4, 24] and policy gradient [29, 44, 56, 42, 26, 17, 28].

2 Robust Reinforcement Learning with General Utility

Notations. The space of probability distribution on a space X is denoted as ∆ X . If X is finite, we denote its cardinality as |X| . ∥ · ∥ p denotes p -norm of vectors and ∥ · ∥ = ∥ · ∥ 2 by default.

Reinforcement Learning with General Utility. Reinforcement Learning (RL) with general utility is an emerging learning framework [54, 55, 6], specified by a tuple ⟨S , A , p ξ , f, ρ, γ ⟩ , with finite state space S , finite action space A , transition kernel p ξ ∈ (∆ S ) S×A parameterized by ξ ∈ Ξ ( Ξ ⊂ R d Ξ is convex and compact), discount factor γ ∈ (0 , 1) , general utility function f : ∆ S×A → R and the distribution ρ ∈ ∆ S of the initial state s 0 . At time t , given the environmental state s t , the agent takes action a t ∼ π θ ( ·| s t ) based on a policy π θ ∈ (∆ A ) S parameterized by θ ∈ Θ ( Θ ⊂ R d Θ is convex). Then the environment transitions to state s t +1 ∼ p ξ ( ·| s t , a t ) . The occupancy measure λ θ,ξ ∈ ∆ S×A at ( s, a ) ∈S×A is defined below.

where P π θ ,p ξ denotes the probability measure of the Markov chain { s t , a t } t ≥ 0 induced by policy π θ , transition kernel p ξ and the initial distribution ρ . The aim of the agent is to find the optimal policy π θ that solves min θ ∈ Θ f ( λ θ,ξ ) given fixed transition kernel p ξ . Here, f ( λ θ,ξ ) can be seen as the overall cost of selecting policy π θ in the environment p ξ , and there are many examples of the utility function f covering a variety of applications (See Section 2.1). However, existing works on RL with general utility assume a fixed environmental transition kernel p ξ , which may fail in many applications where the policy is deployed in a real-world environment different from the simulation environment for training. To obtain a policy that is robust to such environmental change, we propose a new learning framework called robust RL with General Utility as follows.

Our Proposed Robust RL with General Utility. The goal of our proposed robust RL with general utility is to find an optimal robust policy under the worst possible environmental parameter ξ from an ambiguity set Ξ , as formulated by the following minimax optimization problem with general utility function f .

In practice, the distance between the real-world environment (for deployment) and simulation environment (for training) is assumed to be bounded. Therefore, Ξ is usually set as a neighborhood around the nominal kernel ̂ ξ estimated from the simulation environment, i.e. Ξ = { ξ ∈ R d Θ : d ( ξ, ̂ ξ ) ≤ d 0 } with distance measure d and the distance upper bound d 0 ≥ 0 .

2.1 Examples of Our Robust RL with General Utility

Example 1: RL with General Utility.

When Ξ = { ̂ ξ } for a fixed environmental parameter ̂ ξ , our proposed robust RL with general utility (2) reduces to (non-robust) RL with general utility min θ ∈ Θ f ( λ θ, ̂ ξ ) , as introduced above.

Example 2: Robust Constrained RL and Its Special Cases.

Robust constrained RL [38, 43, 40] is an emerging learning framework where an agent should obey safety conditions in all possible real-world environments, which is important in safety critical applications such as healthcare and unmanned aerial vehicle [43]. For math formulation, denote c (0) , c (1) , . . . , c ( K ) as cost functions S × A → R . At time t , the agent receives performance-related cost c (0) ( s t , a t ) and safety-related costs { c ( k ) ( s t , a t ) } K k =1 . Define value functions V ( k ) θ,ξ and robust value functions V ( k ) θ as follows.

Then robust constrained RL is formulated as the following constrained policy optimization problem.

where τ k ∈ R is the safety threshold, and V ( k ) θ ≤ τ k means that the safety constraints V ( k ) θ,ξ ≤ τ k holds for any environmental parameter ξ ∈ Ξ .

Proposition 1. The robust constrained RL problem (4) is a special case of our proposed robust RL with general utility (2) using the following convex utility function f .

After removing the safety constraints, robust constrained RL reduces to an important special case called robust RL (formulated as min θ ∈ Θ max ξ ∈ Ξ ⟨ c (0) , λ θ,ξ ⟩ with linear utility function f ( λ ) = ⟨ c (0) , λ ⟩ ) [36]. Furthermore, when Ξ = { ̂ ξ } for fixed ̂ ξ , robust constrained RL and robust RL reduce to constrained RL [2] and RL [41] respectively. All these examples are important special cases of our proposed robust RL with general utility based on Proposition 1.

Example 3: Robust Entropy Regularized RL and Its Special Cases.

Robust entropy regularized RL is also an important RL framework with application to imitation learning and inverse reinforcement learning which help agents learn from human experts' demonstration [32, 33], and is formulated as the following minimax optimization problem.

where c is a cost function, λ θ,ξ ( s ) = ∑ a λ θ,ξ ( s, a ) is the state occupancy measure, and H [ π θ ( ·| s )] = -∑ a π θ ( a | s ) log π θ ( a | s ) is the entropy regularizer (with coefficient µ ≥ 0 ) which encourages the agent to explore more states and actions and helps to prevent early convergence to sub-optimal policies.

Proposition 2. The robust entropy regularized RL problem (6) is a special case of our proposed robust RL with general utility (2) using the following convex utility function f .

When µ = 0 , robust entropy regularized RL (6) reduces to robust RL [36]. When c ≡ 0 but µ > 0 , robust entropy regularized RL reduces to robust pure exploration . Furthermore, when Ξ = { ̂ ξ } , robust entropy regularized RL , robust RL and robust pure exploration reduce to entropy regularized RL [10], RL [41] and pure exploration [18] respectively. All these examples are important special cases of our proposed robust RL with general utility based on Proposition 2.

2.2 Gradients for Our Robust RL with General Utility

Theorem 1. The gradients of the objective function (2) for our proposed robust RL with general utility can be computed as follows.

We make the following standard assumptions which are also used in RL with general utility [55, 6]. Assumption 1. There exist constants l π θ , L π θ , l p ξ , L p ξ > 0 such that for all s, s ′ ∈ S , a ∈ A θ, θ ′ ∈ Θ and ξ, ξ ′ ∈ Ξ , we have

,

Assumption 2. There exist constants l λ , L λ > 0 such that for all λ, λ ′ ∈ ∆ S×A , ∥∇ λ f ( λ ) ∥ ≤ l λ and ∥∇ λ f ( λ ′ ) -∇ λ f ( λ ) ∥ ≤ L λ ∥ λ ′ -λ ∥ .

Proposition 3. Under Assumptions 1 and 2, the gradients (8) and (9) satisfy the following bounds for any θ, θ ′ ∈ Θ and ξ, ξ ′ ∈ Ξ .

In practice, the exact gradients (8) and (9) are unavailable and can only be estimated via stochastic samples. We refer the details to Appendix C as those largely follow [6].

Define the following projected gradients with stepsizes b, a > 0 , which have been used to measure convergence of algorithms to stationary points of optimization [30, 5, 47] and RL problems [49, 46, 34].

3 Gradient Convergence for Convex Utility

Assumption 3. The utility function f ( λ ) is convex.

Robust RL with convex utility functions f subsumes many important special cases, including robust constrained RL, robust entropy regularized RL, constrained RL, robust RL, RL, pure exploration, etc., as shown in Examples 2 and 3 in Section 2.1.

Partially inspired by the gradient descent ascent (GDA) algorithm [31] for nonconvex-concave minimax optimization, we design the projected stochastic GDA algorithm (Algorithm 1) with two phases to solve robust RL with convex utility. The first phase (called original phase ) can be seen as projected stochastic GDA algorithm on the original objective function f . Specifically, in the k -th the outer loop with fixed ξ k , the inner loop applies T projected stochastic gradient descent steps (14) to obtain θ which converges to the global solution of Φ( ξ ) := min f ( λ ) as f is convex. Then, k k θ ∈ Θ θ,ξ k

we update ξ k using the projected stochastic gradient ascent step (15). However, the output ˜ ξ of the first phase only converges to a stationary point of the following the envelope function ˜ Φ 1 .

To converge to a stationary point of f , we add the second phase (called corrected phase ) which applies projected stochastic GDA to the following corrected objective.

Algorithm 1 Projected Stochastic Gradient Descent Ascent Algorithm For Convex Utility

1: Hyperparameters: K , T , K ′ , T ′ , α , β , a , b , L ξ,ξ , { m ( k ) λ , H ( k ) λ , m ( k ) θ , H ( k ) θ } 4 k =1 .

2: Initialize: ξ 0 ∈ Ξ , θ 0 , θ K ∈ Θ .

# Begin original phase to solve the original optimization problem (2)

3: for Iterations k = 0 , 1 , . . . , K -1 do

4: Initialize θ k, 0 ← θ 0 .

5: for Inner steps t = 0 , 1 , . . . , T -1 do

6: Obtain g ( θ ) k,t ≈∇ θ f ( λ θ k,t ,ξ k ) by Algorithm 3 with hyperparameters m (1) λ , H (1) λ , m (1) θ , H (1) θ .

7: Apply the following projected stochastic policy gradient descent step.

8: end for

9: Assign θ k ← θ k,T .

10: Obtain g ( ξ ) k ≈∇ ξ f ( λ θ k ,ξ k ) by Algorithm 3 with hyperparameters m (2) λ , H (2) λ , m (2) ξ , H (2) ξ .

11: Apply the following projected stochastic gradient descent step.

13: Obtain ˜ ξ from { ξ k } K -1 k =0 uniformly at random. # Begin corrected phase to solve the corrected optimization problem

12: end for

14: for Iterations k = K,K +1 , . . . , K + K ′ -1 do

15: Initialize ξ k, 0 ← ˜ ξ .

16: for Inner steps t = 0 , 1 , . . . , T ′ -1 do

17:

Obtain g ( ξ ) k,t ≈∇ ξ f ( λ θ k ,ξ k,t ) by Algorithm 3 with hyperparameters m (3) λ , H (3) λ , m (3) ξ , H (3) ξ .

18: Apply the following projected stochastic gradient ascent step.

(19)

19: end for

20: Assign ξ k ← ξ k,T ′ .

21: Obtain g ( θ ) k ≈ ∇ θ f ( λ θ k ,ξ k ) by Algorithm 3 with hyperparameters m (4) λ , H (4) λ , m (4) θ , H (4) θ .

22: Apply the following projected stochastic gradient descent step.

23: end for

24: Output: ( θ ˜ k , ξ ˜ k ) where ˜ k is obtained from { K,K +1 , . . . , K + K ′ -1 } uniformly at random.

The convergence analysis of Algorithm 1 is challenging largely because f ( λ θ,ξ ) is only a convex function of λ θ,ξ not of θ . To tackle this challenge for non-robust convex RL with fixed ξ , [54] assumed that a global Lipschitz continuous inverse mapping from λ θ,ξ to θ exists. [55, 6] relaxed this assumption to the following assumption of local inverse mapping, which covers the popular direct policy parameterization π θ ( a | s ) = θ s,a [6] and softmax policy parameterization π θ ( a | s ) = exp( θ s,a ) ∑ a ′ exp( θ s,a ′ ) (see Proposition 8 for the proof).

Assumption 4 (Local Invertibility of λ · ,ξ ) . There exists constants ℓ λ -1 > 0 and δ ∈ (0 , 1) such that for any fixed θ ∈ Θ and ξ ∈ Ξ , the occupancy measure (1) satisfies:

There exists sets U θ,ξ ⊂ Θ and V λ θ,ξ ⊂ ∆ S×A that contain θ and λ θ,ξ respectively, such that λ θ,ξ : U θ,ξ →V λ θ,ξ is a bijection. Its inverse denoted as λ -1 θ,ξ is ℓ λ -1 -Lipscthiz.

There exists at least one optimal policy θ ∗ ( ξ ) ∈ arg min θ ′ ∈ Θ f ( λ θ ′ ,ξ ) such that for any δ ∈ [0 , δ ] , (1 -δ ) λ θ,ξ + δλ θ ∗ ( ξ ) ,ξ ∈ V λ θ,ξ .

Proposition 4 (Projected Gradient Dominance for Convex Utility) . Under Assumptions 1-4, the utility function f satisfies the following gradient dominance property for any θ ∈ Θ and ξ ∈ Ξ .

Remark: Proposition 4 indicates that the function f ( λ · ,ξ ) is projected gradient dominant for convex utility function f , which is important in the convergence analysis of Algorithm 1. Our Proposition 4 is stronger than Lemma F.7 of [6], a similar gradient dominance property for convex RL which requires assumption of positive definite Fisher information matrix, involves bias in the error term and focuses on unconstrained optimization with softmax parameterized policy (a special of our general parameterized policy with constrained variable θ ∈ Θ ).

Technical Novelty. In our proof, to tackle the constraint θ ∈ Θ which is more challenging than the unconstrained case Θ = R |S||A| in [55, 6], we apply Assumption 4 to θ ′ := θ -βG ( θ ) β ( θ, ξ ) not to the obvious choice θ , which yields θ δ ∈ Θ for any δ ∈ [0 , δ ] such that λ θ δ ,ξ = (1 -δ ) λ θ ′ ,ξ + δλ θ ∗ ( ξ ) ,ξ ∈ V λ θ,ξ . Then

where (i) uses the projection property ( θ δ -θ ′ ) ⊤ [ G ( θ ) β ( θ, ξ ) -∇ θ f ( λ θ,ξ )] ≤ 0 and (ii) uses ∥ θ δ -θ ′ ∥ ≤ O ( δ ) . The above bound implies Eq. (20) since f is convex and ℓ θ -Lipscthiz.

Assumption 5. Ξ is convex and compact with diameter D Ξ := max ξ,ξ ′ ∈ Ξ ∥ ξ ′ -ξ ∥ > 0 .

Assumption 5 holds for the commonly used direct kernel parameterization p ξ ( s ′ | s, a ) = ξ ( s, a, s ′ ) (for all s, s ′ ∈ S and a ∈ A )[42, 28, 26, 17] and Ξ defined a compact neighborhood around a nominal transition kernel parameter ̂ ξ .

We show the gradient convergence result of Algorithm 1 by the following theorem and demonstrate the gradient convergence by the experiments in Appendix A.

Theorem 2 (Gradient Convergence for Convex Utility) . Suppose Assumptions 1-5 hold. For any precision 0 < ϵ ≤ 48 L ξ,ξ ˜ L [ √ 2 ℓ λ -1 ( L θ,θ +4 ˜ L ) + ℓ θ ] , we can always find proper hyperparameter values of Algorithm 1 (see Eqs. (127) -(150) in Appendix N.6 for these hyperparamter values) such that the algorithm output ( θ ˜ k , ξ ˜ k ) is an ϵ -close to a stationary point, that is, E [ ∥ G ( θ ) b ( θ ˜ k , ξ ˜ k ) ∥ 2 ] ≤ ϵ 2 and E [ ∥ G ( ξ ) a ( θ ˜ k , ξ ˜ k ) ∥ 2 ] ≤ ϵ 2 with projected gradients G ( θ ) b and G ( ξ ) a defined in Eq. (13) . The number of required stochastic samples is O [ log 2 [(1 -γ ) -1 ϵ -1 ] (1 -γ ) 25 ϵ 10 ] .

Proof Sketch of Theorem 2 and Technical Novelty. Inspired by Appendix D of [31], ˜ ξ from the first phase satisfies E ∥∇ ˜ Φ( ˜ ξ ) ∥ 2 → 0 (see Appendix N.2). Then, ξ k := ξ k,T ′ from the inner update (16) of the second phase converges to the unique maximizer (denoted as ξ ∗ k ) of the L ξ,ξ -concave function ˜ f ( θ k , · ) as T ′ → + ∞ (see Appendix N.3). This means the update step (17) is approximately projected gradient descent for min θ ∈ Θ ˜ Ψ( θ ) , which yields the convergence rate of E [ ∥ G ( θ ) b ( θ ˜ k , ξ ˜ k ) ∥ 2 ] (see Appendix N.4).

However, the biggest challenge is to obtain the convergence rate of E [ ∥ G ( ξ ) a ( θ ˜ k , ξ ˜ k ) ∥ 2 ] (see Appendix N.5), which corresponds to ∇ ξ f while the second corrected phase aims at the corrected objective ˜ f . To show that ∇ ξ ˜ f ( θ k , ξ k ) ≈ ∇ ξ f ( θ k , ξ k ) , note that ∇ ξ ˜ f ( θ k , ξ k ) -∇ ξ f ( θ k , ξ k ) = -2 L ξ,ξ ( ξ k -˜ ξ ) and that ∇ ˜ Φ( ˜ ξ ) = 2 L ξ,ξ [ ξ ∗ ( ˜ ξ ) -˜ ξ ] ≈ 0 (already proved) where ξ ∗ ( ˜ ξ ) is the unique maximizer of Φ( ξ ′ ) -L ξ,ξ ∥ ξ ′ -˜ ξ ∥ 2 , a strongly concave function of ξ ′ in Eq. (18). Hence, it suffices to show ξ k ≈ ξ ∗ ( ˜ ξ ) . Note that ( θ k , ξ k ) is an approximate Nash equilibrium of ˜ f , i.e., ξ k ≈ ξ ∗ k := arg max ξ ∈ Ξ ˜ f ( θ k , ξ ) (proved above) and θ k ≈ arg min θ ∈ Θ ˜ f ( θ, ξ k ) (derived below).

where (i) uses Proposition 4. Hence, based on the property of Nash equilibrium , we have ξ k ≈ arg max ξ ψ ( ξ ) = ξ ∗ ( ˜ ξ ) where ψ ( ξ ) := min θ ∈ Θ ˜ f ( θ, ξ ) = Φ( ξ ) -L ξ,ξ ∥ ξ -˜ ξ ∥ 2 .

4 Global Convergence on Polyhedral Ambiguity Set

This section aims to obtain a global optimal policy θ ∗ that minimizes the robust utility Γ( θ ) def = max ξ ∈ Ξ f ( λ θ,ξ ) . This maximization is challenging for convex utility f . In contrast, global conver-

gence results have been obtained without such challenge in some important special cases, including convex RL with fixed ξ [54, 51, 6] and robust RL where linear utility f is amenable to both min θ ∈ Θ and max ξ ∈ Ξ [36, 45, 42, 23, 26]. Fortunately, we will show that by using the popular s -rectangular polyhedral ambiguity set Ξ , arg max ξ ∈ Ξ f ( λ θ,ξ ) always exists among the finitely many vertices of Ξ .

4.1 S-rectangular Polyhedral Ambiguity Set

In this subsection, we will introduce the popular s -rectangular polyhedral ambiguity set, and derive its important propositions for designing globally converged algorithm.

Fhe global convergence is generally NP-hard, even for the important special case called robust RL with linear utility , [45]. A common practice to make the problem tractable is to use direct kernel parameterization p ξ ( s ′ | s, a ) = ξ ( s, a, s ′ ) [42, 28, 26, 17] and assume the ambiguity set Ξ to satisfy some certain rectangularity conditions, such as s -rectangularity defined below [45, 42, 23, 26].

Assumption 6. We use direct kernel parameterization and assume that Ξ is s -rectangular, i.e., Ξ = × s ∈S Ξ s := { ξ ∈ (∆ S ) S×A : ξ ( s, · , · ) ∈ Ξ s , ∀ s ∈ S} , a Cartesian product of Ξ s ⊂ (∆ S ) A .

Proposition 5 (Local Invertibility of λ θ, · ) . Suppose Assumption 6 holds and Ξ is a convex set. For any θ ∈ Θ , ξ 0 , ξ 1 ∈ Ξ and δ ∈ [0 , 1] , define the following kernel parameters ξ δ ∈ (∆ S ) S×A .

where λ θ,ξ ( s ) := ∑ a ∈A λ θ,ξ ( s, a ) for any s ∈ S , θ ∈ Θ and ξ ∈ Ξ . Then ξ δ ∈ Ξ and its corresponding occupancy measure is λ θ,ξ δ = δλ θ,ξ 1 +(1 -δ ) λ θ,ξ 0 .

Remark: Proposition 5 indicates that the mapping from ξ to λ θ,ξ is locally invertible for s -rectangular set Ξ , which is important to solve the aforementioned challenge that convex utility is not amenable for max ξ ∈ Ξ f ( λ θ,ξ ) . This role is similar to that played by the local invertibility assumption (Assumption 4) for policy θ . To our knowledge, Proposition 5 has never been obtained in the existing literature.

Assumption 7. Under Assumption 6, for every s ∈ S , Ξ s is a polyhedron spanned by a finite set of vertices V (Ξ s ):= { ξ ( s ) m } M s m =1 ⊂ Ξ s , i.e., Ξ s = {∑ M s m =1 ν m ξ ( s ) m : ν m ≥ 0 , ∑ M s m =1 ν m = 1 } .

Polyhedral ambiguity set defined by Assumption 7 includes the widely used s -rectangular L 1 and L ∞ ambiguity sets, defined as Ξ = { ξ ∈ (∆ S ) S×A : ∥ ξ ( s, : , :) -̂ ξ ( s, : , :) ∥ p ≤ α s , ∀ s ∈ S} for p ∈ { 1 , ∞} respectively [7, 19, 16], where ̂ ξ ∈ Ξ is the nominal transition kernel usually obtained via empirical estimation. On polyhedral ambiguity set, the optimal kernels arg max ξ ∈ Ξ f ( λ θ,ξ ) can always be obtained at the vertices of Ξ , as shown below.

Proposition 6. Under Assumptions 3, 6 and 7, for any θ ∈ Θ , we have max ξ ∈ Ξ f ( λ θ,ξ ) = max ξ ∈ V (Ξ) f ( λ θ,ξ ) , where V (Ξ) = × s ∈S V (Ξ s ) is the vertex set.

̸

Technical Novelty. Suppose a non-vertex kernel ξ ∗ ∈ arg max ξ ∈ Ξ f ( λ θ,ξ ) /V (Ξ) is optimal. Since f is convex, if λ θ,ξ ∗ is a convex combination of λ θ,ξ ∗ 1 and λ θ,ξ ( ϵ ) for some ξ 1 , ξ 0 ∈ Ξ (corresponding to ξ ∗ 1 , ξ ( ϵ ) respectively in the proof in Appendix I), then ξ 1 , ξ 0 are also optimal. Ideally, if ξ 1 ∈ V (Ξ) or ξ 0 ∈ V (Ξ) , the proof is done. However, this is not guaranteed since in Proposition 5 and Assumption 6, the convex combination coefficients differ among the states s ∈ S . To solve this challenge, it suffices to find such optimal ξ 1 that differs from ξ ∗ at only one state s such that the non-vertex ξ ∗ ( s ) / ∈ V (Ξ s ) is replaced with vertex ξ 1 ( s ) ∈ V (Ξ s ) . Then we can conduct such change from non-vertex to vertex for only one state s at a time until the kernel becomes vertex at every state, while keeping the optimality all the way. To find such ξ 1 ( s ) , note that on polyhedral set Ξ s , there always exist ξ 1 ( s ) ∈ V (Ξ s ) and ξ 0 ( s ) ∈ Ξ s such that the non-vertex point ξ ∗ ( s ) is a convex combination of ξ 1 ( s ) and ξ 0 ( s ) , while ξ ∗ ( s ′ ) = ξ 1 ( s ′ ) = ξ 0 ( s ′ ) for any s ′ = s . Hence, there exists δ ∈ [0 , 1] such that ξ ∗ = ξ δ defined by Proposition 5, which implies that λ θ,ξ ∗ is a convex combination of λ θ,ξ ∗ 1 and λ θ,ξ ( ϵ ) .

4.2 Globally Converged Algorithm

The original objective (2) is equivalent to the minimization problem min θ ∈ Θ Γ( θ ) , where Γ( θ ) := max ξ ∈ V (Ξ) f ( λ θ,ξ ) with finite vertex set V (Ξ) based on Proposition 6. A natural choice to solve this minimization problem is the following policy update rule (for simplicity we consider the unconstrained policy space Θ = R d Θ as in [55, 6]).

where β k > 0 is the stepsize and d k is a unit descent direction of Γ( θ k ) . Subgradient descent method seems an obvi-

Algorithm 2 Globally Converged Algorithm for Convex Utility on Polyhedral Ambiguity Set

1: Hyperparameters: K , { σ k , ϵ k , β k } K -1 k =0 .

2: Initialize: θ 0 ∈ Θ .

3: for Iterations k = 0 , 1 , . . . , K -1 do

4: Calculate λ θ k ,ξ , f ( λ θ k ,ξ ) and ∇ θ f ( λ θ k ,ξ ) for all ξ ∈ V (Ξ) .

5: Select near-optimal vertices Ξ k := { ξ ∈ V (Ξ) : f ( λ θ k ,ξ ) ≥ max ξ ′ ∈ V (Ξ) f ( λ θ k ,ξ ′ ) -σ k } .

6: Find d ′ k ∈ B 1 := { d ∈ R d Θ : ∥ d ∥ ≤ 1 } such that A ( d ′ ) ≤ min ∈ A ( d ) + ϵ .

k k d B 1 k k ( A k is defined in Eq. (23) . One way to solve min d ∈ B 1 A k ( d ) is to apply projected subgradient method (28) and obtain d ′ k ∈ arg max d ∈{ d k,t :0 ≤ t ≤ T } A k ( d ) .)

7: Let d k := d ′ k / ∥ d ′ k ∥ and obtain θ k +1 by Eq. (22).

8: end for

9: Output: ( θ K , ξ K ) where ξ K ∈ arg max ξ ∈ V (Ξ) f ( λ θ K ,ξ ) .

ous choice for d k which aligns with the direction of a subgradient ∇ θ f ( λ θ k ,ξ k ) where ξ k ∈ arg max ξ ∈ V (Ξ) f ( λ θ k ,ξ ) . However, the convergence analysis of subgradient descent method [11] requires the convexity of f ( λ · ,ξ k ) which does not hold in our setting, and the function value is not monotonically decreasing. To solve these challenges, we design Algorithm 2 which selects near-optimal vertices Ξ k := { ξ ∈ V (Ξ) : f ( λ θ k ,ξ ) ≥ max ξ ′ ∈ V (Ξ) f ( λ θ k ,ξ ′ ) -σ k } with a certain threshold σ k > 0 and obtains d k by solving the convex optimization problem min d ∈ B 1 A k ( d ) up to precision ϵ k > 0 , where A k ( d ) below denotes effective descent of Γ( θ k ) along the direction d .

Here we only care about the near-optimal vertices in Ξ k ⊂ V (Ξ) because for any worse vertices ξ ∈ V (Ξ) / Ξ k , f ( θ k , ξ ) < max ξ ′ ∈ V (Ξ) f ( λ θ k ,ξ ′ ) -σ k implies f ( θ k +1 , ξ ) < max ξ ′ ∈ V (Ξ) f ( λ θ k +1 ,ξ ′ ) for appropriate σ k > 0 . This means such worse ξ can not affect the optimization progress Γ( θ k ) -Γ( θ k +1 ) . Hence, by solving min d ∈ B 1 A k ( d ) , we can obtain a direction d k in which all the potentially effective function values { f ( λ θ k ,ξ ) } ξ ∈ Ξ k have uniformly large amount of descent -∇ f ( λ θ k ,ξ ) ⊤ d k .

To analyze the global convergence of Algorithm 2, we want to guarantee sufficient descent Γ( θ k ) -Γ( θ k +1 ) whenever θ k is not close to optimal. It suffices to slightly alter Assumption 4 as follows.

Assumption 8. A variant of Assumption 4 holds which replaces the non-robust optimal policy θ ∗ ( ξ ) with a robust optimal policy θ ∗ ∈ arg min θ ∈ Θ Γ( θ ) and shrinks the range from ξ ∈ Ξ to ξ ∈ V (Ξ) .

Remark: Assumption 8 is no stronger than Assumption 4 and also covers the popular direct policy parameterization. Also, Assumption 8 guarantees that from any policy θ ∈ Θ , there exists a partial curve { θ δ : δ ∈ [0 , δ ] } towards a robust optimal policy θ ∗ such that λ θ δ ,ξ = (1 -δ ) λ θ,ξ + δλ θ ∗ ,ξ , so we can utilize convexity of f and obtain the following important sufficient descent property.

Proposition 7 (Sufficient Descent on Polyhedral Ambiguity Set) . Under Assumptions 1-3 and 8, at any θ ∈ Θ := R d Θ , there exists a unit descent direction d ( ∥ d ∥ = 1 ) such that

where θ ∗ ∈ arg min θ ′ ∈ Θ Γ( θ ) is given by Assumption 8 and x + := max( x, 0) for any x ∈ R .

Remark: d in Proposition 7 is a good descent direction since whenever the function value gap f ( λ θ,ξ ) -f ( λ θ ∗ ,ξ ) > 0 , it is dominated by the gradient descent amount -∇ θ f ( λ θ,ξ ) ⊤ d > 0 . Unlike existing gradient dominance properties for robust RL [42, 26, 17], f ( λ θ,ξ ) -f ( λ θ ∗ ,ξ ) ≤ 0 is possible so we use [ · ] + to cover all cases. This brings challenge and thus novel techniques to obtain the first global convergence result of our robust RL with general convex utility as follows.

Theorem 3 (Global Convergence for Convex Utility on Polyhedral Ambiguity Set) . Implement Algorithm 2 with β k = 2 √ 2 ℓ λ k +2 , σ k = 4 √ 2 ℓ θ ℓ λ k +2 and any ϵ k > 0 . Then under Assumptions 1-3, 6-8, the

algorithm output θ K has the following global convergence rate.

Remark: The convergence rate O (1 /K ) matches the state-of-the-art of policy gradient type methods for robust RL [26], while the error term ϵ k results from solving the convex optimization problem min d ∈ B 1 A k ( d ) in line 6 of Algorithm 2.

Technical Novelty. Applying Proposition 7 to Algorithm 2 with σ k = 2 β k ℓ θ , we have

To overcome the main difficulty caused by [ · ] + above, we analyze each k -th iteration in 2 cases A k ( d ′ k ) ≥ 0 and A k ( d ′ k ) < 0 . If A k ( d ′ k ) ≥ 0 , then ∆ k ≤ √ 2 ℓ λ -1 ϵ k + 2 β k ℓ θ and thus ∆ k +1 ≤ √ 2 ℓ λ -1 ϵ k +3 β k ℓ θ ; If A k ( d ′ k ) < 0 , then in Eq. (26) we replace A k ( d ′ k ) with A k ( d k ) ≤ A k ( d ′ k ) < 0 and remove [ · ] + . This along with Γ( θ k +1 ) -Γ( θ k ) ≤ β k A k ( d k ) + L θ,θ 2 β 2 k (by smoothness) implies

Then we obtain the rate (25) in 3 cases : If A k ( d ′ k ) < 0 for all k = 0 , 1 , . . . , K -1 , iterate Eq. (27) from ∆ 0 ; If A K -1 ( d K -1 ) ≥ 0 , ∆ K ≤ √ 2 ℓ λ -1 ϵ K + σ k ; If A K ′ -1 ( d K ′ -1 ) ≥ 0 while A k ( d k ) < 0 for all k = K ′ , . . . , K -1 , iterate Eq. (27) from ∆ K ′ ≤ √ 2 ℓ λ -1 ϵ K ′ -1 +3 β K ′ -1 ℓ θ .

Algorithm 2 involves convex optimization problems min d ∈ B 1 A k ( d ) , which can be solved via the following projected subgradient method for t = 0 , 1 , . . . , T -1 .

The best direction d ′ k ∈ arg max d ∈{ d k,t :0 ≤ t ≤ T } A k ( d ) from the above subgradient method achieves ϵ k accuracy within T = O ( ϵ -2 k ) steps [11], which yields the following complexity result.

Corollary 1. Under the conditions of Theorem 3, for any ϵ > 0 , implement Algorithm 2 with K = 8 ℓ λ -1 ϵ -1 ( ℓ λ -1 L θ,θ +2 √ 2 ℓ θ ) iterations and T = 36 ℓ 2 λ -1 ℓ 2 θ ϵ -2 subgradient descent updates (28) with stepsize α = ϵ 3 ℓ λ -1 ℓ 2 θ to obtain d ′ k . Then the output θ K achieves Γ( θ K ) -min θ ′ ∈ Θ Γ( θ ′ ) ≤ ϵ .

Finally, we can prove that all these Assumptions 1-8 required by our convergence results (Theorems 2 and 3) can be satisfied by the following examples.

Proposition 8. Assumptions 1-8 are all satisfied if we use the following choices:

Softmax policy parameterization π θ ( a | s ) = exp( θ s,a ) ∑ a ′ exp( θ s,a ′ ) , where θ ∈ Θ = [ -R,R ] |S|×|A| for some constant R > 0 to prevent π θ ( a | s ) from approaching 0.

Direct kernel parameterization p ξ ( s ′ | s, a ) = ξ s,a,s ′ with s -rectangular L 1 or L ∞ ambiguity sets defined as Ξ = { ξ ∈ (∆ S ) S×A : ∥ ξ ( s, : , :) -̂ ξ ( s, : , :) ∥ p ≤ α s , ∀ s ∈ S} for p ∈ { 1 , ∞} respectively, where the fixed nominal kernel ̂ ξ satisfies ̂ ξ ( s, a, s ′ ) > α s , ∀ s, a, s ′ to prevent p ξ ( s ′ | s, a ) from approaching 0.

The utility function f ( λ ) defined in Eq. (7) for robust entropy regularized RL and its special cases, within the range λ ∈ Λ = { λ θ,ξ : θ ∈ Θ , ξ ∈ Ξ } for the domains Θ and Ξ selected above.

5 Conclusion

In this work, we propose robust RL with general utility, the first learning framework that obtains a robust policy for RL with general utility. We propose a stochastic policy gradient type algorithm for convex utilities and obtains its sample complexity result for gradient convergence. Furthermore, for convex utility on polyhedral ambiguity set, we propose an alternative policy gradient type algorithm and obtain its global convergence rate. Note that this globally converged algorithm requires enumeration among many vertices, and thus it is an important future direction to reduce enumeration by utilizing structural properties. In addition, to extend the results to large or continuous state-action space is also an interesting direction.

Acknowledgments

This work was partially supported by NSF IIS 2347592, 2347604, 2348159, 2348169, DBI 2405416, CCF 2348306, CNS 2347617.

References

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning , pages 22-31, 2017.

Eitan Altman. Constrained Markov decision processes . Routledge, 2021.

Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems , 57(5):469-483, 2009.

Kishan Panaganti Badrinath and Dileep Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference on Machine Learning , pages 511-520, 2021.

Maxim Viktorovich Balashov. The gradient projection algorithm for a proximally smooth set and a function with lipschitz continuous gradient. Sbornik: Mathematics , 211(4):481, 2020.

Anas Barakat, Ilyas Fatkhullin, and Niao He. Reinforcement learning with general utilities: Simpler variance reduction and large state-action space. In International Conference on Machine Learning , 2023.

Bahram Behzadian, Marek Petrik, and Chin Pang Ho. Fast algorithms for l ∞ -constrained srectangular robust mdps. In Proceedings of the International Conference on Neural Information Processing Systems (Neurips) , 2021.

Vivek S Borkar and Sean P Meyn. Risk-sensitive optimal control for markov decision processes with monotone cost. Mathematics of Operations Research , 27(1):192-209, 2002.

Winfried Bruns and Joseph Gubeladze. Polytopes, rings, and K-theory . Springer Science & Business Media, 2009.

Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research , 70(4):25632578, 2022.

Yuxin Chen. Subgradient methods. https://yuxinchen2020.github.io/ele522 _optimization/lectures/subgradient_methods.pdf , 2020.

Jerzy A Filar, Lodewijk CM Kallenberg, and Huey-Miin Lee. Variance-penalized markov decision processes. Mathematics of Operations Research , 14(1):147-161, 1989.

Matthieu Geist, Julien Pérolat, Mathieu Laurière, Romuald Elie, Sarah Perrin, Oliver Bachem, Rémi Munos, and Olivier Pietquin. Concave utility reinforcement learning: The mean-field game viewpoint. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems , pages 489-497, 2022.

Mohammad Ghavamzadeh, Marek Petrik, and Yinlam Chow. Safe policy improvement by minimizing robust baseline regret. In Proceedings of the International Conference on Neural Information Processing Systems (Neurips) , volume 29, 2016.

Julien Grand-Clément and Christian Kroer. Scalable first-order methods for robust mdps. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 12086-12094, 2021.

Julien Grand-Clément and Marek Petrik. Reducing blackwell and average optimality to discounted mdps via the blackwell discount factor. In Proceedings of the International Conference on Neural Information Processing Systems (Neurips) , 2023.

Etash Kumar Guha and Jason D Lee. Solving robust mdps through no-regret dynamics. ArXiv:2305.19035 , 2023.

Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning , pages 2681-2691, 2019.

Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Partial policy iteration for l1-robust markov decision processes. Journal of Machine Learning Research , 22(275):1-46, 2021.

Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research , 30(2):257-280, 2005.

Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research , 4:237-285, 1996.

Lodewijk CM Kallenberg. Survey of linear programming for standard and nonstandard markovian control problems. part i: Theory. Zeitschrift für Operations Research , 40:1-42, 1994.

Navdeep Kumar, Esther Derman, Matthieu Geist, Kfir Levy, and Shie Mannor. Policy gradient for rectangular robust markov decision processes. In Proceedings of the International Conference on Neural Information Processing Systems (Neurips) , 2023.

Navdeep Kumar, Kfir Levy, Kaixin Wang, and Shie Mannor. Efficient policy iteration for robust markov decision processes via regularization. ArXiv:2205.14327 , 2022.

Navdeep Kumar, Kfir Levy, Kaixin Wang, and Shie Mannor. An efficient solution to srectangular robust markov decision processes. ArXiv:2301.13642 , 2023.

Navdeep Kumar, Ilnura Usmanova, Kfir Yehuda Levy, and Shie Mannor. Towards faster global convergence of robust policy gradient methods. In Sixteenth European Workshop on Reinforcement Learning , 2023.

Navdeep Kumar, Kaixin Wang, Kfir Levy, and Shie Mannor. Policy gradient for reinforcement learning with general utilities. ArXiv:2210.00991 , 2023.

Mengmeng Li, Tobias Sutter, and Daniel Kuhn. Policy gradient algorithms for robust mdps with non-rectangular uncertainty sets. ArXiv:2305.19004 , 2023.

Yan Li, Guanghui Lan, and Tuo Zhao. First-order policy optimization for robust markov decision process. ArXiv:2209.10579 , 2023.

Zhize Li and Jian Li. A simple proximal stochastic gradient method for nonsmooth nonconvex optimization. In Proceedings of the International Conference on Neural Information Processing Systems (Neurips) , pages 5569-5579, 2018.

Tianyi Lin, Chi Jin, and Michael Jordan. On gradient descent ascent for nonconvex-concave minimax problems. In International Conference on Machine Learning , pages 6083-6093, 2020.

Tien Mai and Patrick Jaillet. Robust entropy-regularized markov decision processes. ArXiv:2112.15364 , 2021.

Daniel J Mankowitz, Nir Levine, Rae Jeong, Abbas Abdolmaleki, Jost Tobias Springenberg, Yuanyuan Shi, Jackie Kay, Todd Hester, Timothy Mann, and Martin Riedmiller. Robust reinforcement learning for continuous control with model misspecification. In Proceedings of the International Conference on Learning Representations (ICLR) , 2020.

Weichao Mao, Lin Yang, Kaiqing Zhang, and Tamer Basar. On improving model-free algorithms for decentralized multi-agent reinforcement learning. In International Conference on Machine Learning , pages 15007-15049, 2022.

Sobhan Miryoosefi, Kianté Brantley, Hal Daumé, Miroslav Dudík, and Robert E Schapire. Reinforcement learning with convex constraints. In Proceedings of the International Conference on Neural Information Processing Systems (Neurips) , pages 14093-14102, 2019.

Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research , 53(5):780-798, 2005.

Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 3803-3810. IEEE, 2018.

Reazul Hasan Russel, Mouhacine Benosman, and Jeroen Van Baar. Robust constrained-mdps: Soft-constrained robust policy optimization under model uncertainty. ArXiv:2010.04870 , 2020.

Stefan Schaal. Learning from demonstration. In Proceedings of the International Conference on Neural Information Processing Systems (Neurips) , volume 9, 1996.

Zhongchang Sun, Sihong He, Fei Miao, and Shaofeng Zou. Constrained reinforcement learning under model mismatch. ArXiv:2405.01327 , 2024.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Qiuhao Wang, Chin Pang Ho, and Marek Petrik. Policy gradient in robust mdps with global convergence guarantee. In Proceedings of the International Conference on Machine Learning (ICML) , volume 202, pages 35763-35797, 23-29 Jul 2023.

Yue Wang, Fei Miao, and Shaofeng Zou. Robust constrained reinforcement learning. ArXiv:2209.06866 , 2022.

Yue Wang and Shaofeng Zou. Policy gradient method for robust reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML) , pages 23484-23526, 2022.

Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Mathematics of Operations Research , 38(1):153-183, 2013.

Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research , 23(282):1-36, 2022.

Tesi Xiao, Krishna Balasubramanian, and Saeed Ghadimi. A projection-free algorithm for constrained stochastic multi-level composition optimization. In Proceedings of the International Conference on Neural Information Processing Systems (Neurips) , 2022.

Huan Xu and Shie Mannor. Parametric regret in uncertain markov decision processes. In Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference , pages 3606-3613. IEEE, 2009.

Pan Xu, Felicia Gao, and Quanquan Gu. Sample efficient policy gradient methods with recursive variance reduction. In Proceedings of the International Conference on Learning Representations (ICLR) , 2020.

Rui Yuan, Robert M Gower, and Alessandro Lazaric. A general sample complexity analysis of vanilla policy gradient. In International Conference on Artificial Intelligence and Statistics , pages 3332-3380, 2022.

Tom Zahavy, Brendan O'Donoghue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex mdps. In Proceedings of the International Conference on Neural Information Processing Systems (Neurips) , volume 34, pages 25746-25759, 2021.

Junyu Zhang, Amrit Singh Bedi, Mengdi Wang, and Alec Koppel. Cautious reinforcement learning via distributional risk in the dual domain. ArXiv:2002.12475 , 2020.

Junyu Zhang, Amrit Singh Bedi, Mengdi Wang, and Alec Koppel. Multi-agent reinforcement learning with general utilities via decentralized shadow reward actor-critic. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 9031-9039, 2022.

Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvári, and Mengdi Wang. Variational policy gradient method for reinforcement learning with general utilities. In Proceedings of the International Conference on Neural Information Processing Systems (Neurips) , pages 4572-4583, 2020.

Junyu Zhang, Chengzhuo Ni, Zheng Yu, Csaba Szepesvari, and Mengdi Wang. On the convergence and sample efficiency of variance-reduced policy gradient method. In Proceedings of the International Conference on Neural Information Processing Systems (Neurips) , 2021.

Ruida Zhou, Tao Liu, Min Cheng, Dileep Kalathil, Panganamala Kumar, and Chao Tian. Natural actor-critic for robust reinforcement learning with function approximation. In Proceedings of the International Conference on Neural Information Processing Systems (Neurips) , 2023.

Appendix

Table of Contents

A Experiments

In this section, we present simulation results of Algorithm 1 for convex utility.

Simulation Setting. We choose S = { 1 , 2 , · · · , S } with S = 10 states and A = { 1 , 2 , · · · , A } with A = 5 actions. The discount factor is γ = 0 . 95 and we select uniform distribution as the

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Sample Complexity

1e7

0.0046

0.0048

0.0050

0.0052

0.0054

G

(

)

b

(

k

,

k

)

2 +

G

(

)

a

(

k

,

k

)

2

initial state distribution ρ . To optimize the objective function (2), we apply direct parameterization to policy parameter θ s,a = π ( a | s ) ∈ Θ = (∆ A ) S and transition kernel parameter ξ s,a,s ′ = p ( s ′ | s, a ) ∈ (∆ S ) S×A . In order to preserve ξ (: , : , s ′ ) ∈ ∆ S , We select nominal kernel ξ ( · , · , s ′ ) as | 10+ ε s ′ | ∑ s ′ | 10+ ε s ′ | , where ε s ′ i.i.d ∼ N (0 , 1) for each s ′ ∈ S . Then we select sufficiently small radius r = 0 . 01 < min s,a,s ′ ξ s,a,s ′ and use the L 2 ambiguity set Ξ := { ξ : ∥ ξ ( s, : , :) -ξ ( s, : , :) ∥ ≤ r } (for transition kernel) such that all ξ ∈ Ξ have all positive entries. As for the general utility function f , we use the following convex entropy function with application to exploration (Example 2.2 of [54]).

where λ θ,ξ ( s ) := ∑ a ∈A λ θ,ξ ( s, a ) denotes the state visitation measure for any s ∈ S , θ ∈ Θ and ξ ∈ Ξ .

Hyperparameters. For Algorithm 1, we use the following hyperparameters obtained from finetuning but not from Theorem 2: K = 200 , T = 25 , K ′ = 300 , T ′ = 25 , α = 0 . 002 , β = 0 . 001 , a = 0 . 002 , b = 0 . 002 , L ξ,ξ = 20 , m (1) λ = 15 , H (1) λ = 100 , m (1) θ = 15 , H (1) θ = 100 , m (2) λ = 15 , H (2) λ = 100 , m (2) ξ = 15 , H (2) ξ = 100 , m (3) λ = 10 , H (3) λ = 100 , m (3) ξ = 10 , H (3) ξ = 100 , m (4) λ = 10 , H (4) λ = 100 , m (4) θ = 10 , H (4) θ = 100 .

Environment. The experiment is implemented on Python 3.8 on AMD EPYC-7313 CPU with 3.00GHz, which costs about 1.5 hours in total.

Results. The numerical result of Algorithm 1 is shown in Figure 1. Here the y-axis is the norm of the true projected gradient √ ∥ G ( θ ) b ( θ k , ξ k ) ∥ 2 + ∥ G ( ξ ) a ( θ k , ξ k ) ∥ 2 at each outer iteration k of both phases of Algorithm 1 (separated by the green vertical dashed line), and the x-axis is the sample complexity (i.e., the total number of generated samples up to iteration k ). Figure 1 shows that the projected gradient decays and converges to a small value, which matches Theorem 2.

B Supporting Lemmas

Lemma 1. Under Assumption 1, for any s, s ′ ∈ S , a ∈ A , θ, θ ′ ∈ Θ , ξ, ξ ′ ∈ Ξ , we have

Proof. Based on Assumption 1, the following inequalities holds for all s, s ′ ∈ S , a ∈ A , θ ∈ Θ , ξ ∈ Ξ , which by Lagrange mean value theorem directly proves Eq. (30)

Then we prove Eq. (31) as follows.

where (i) uses Eq. (30) and Assumption 1.

Lemma 2. For any θ ∈ Θ and ξ ∈ Ξ , the occupancy measure λ θ,ξ defined by Eq. (1) is the unique solution to the following Bellman equation of λ ∈ R |S|×|A| .

Therefore, the state occupancy measure λ θ,ξ ( s ) := ∑ λ θ,ξ ( s, a )

Proof. First, we can prove that λ θ,ξ satisfies Eq. (32) as follows.

Next, we prove the uniqueness. Suppose λ 1 , λ 2 ∈ R |S|×|A| satisfies Eq. (32). Then we have

which implies that (1 -γ ) ∑ s,a | λ 2 ( s, a ) -λ 1 ( s, a ) | ≤ 0 and thus λ 2 = λ 1 , i.e., the solution to Eq. (32) is unique.

Finally, we will prove Eq. (33). Note that

where (i) uses Eq. (32). Then Eq. (33) can be proved by substituting the above equality into Eq. (32).

Lemma 3. Under Assumption 1, the occupancy measure (1) satisfies the following Lipschitz properties for any θ, θ ′ ∈ Θ and ξ, ξ ′ ∈ Ξ .

Proof. For any θ, θ ′ ∈ Θ and ξ, ξ ′ ∈ Ξ , we have

where (i) uses Eq. (32), (ii) uses triangular inequality, (iii) uses Lemma 1, ∑ a ′ π θ ′ ( a ′ | s ′ ) 2 ≤ 1 and ∑ s ′ [(1 -γ ) ρ ( s ′ ) + γ ∑ s,a λ θ,ξ ( s, a ) p ξ ( s ′ | s, a )] = 1 and (iv) uses Jensen's inequality. Then Eq. (34) can be proved by rearranging the above inequality.

Lemma 4. The distance between any pair of probability vectors x, y ∈ ∆ X on finite space X has the upper bound that ∥ x -y ∥ ≤ √ 2 .

Proof. Denote d = |X| and x j , y j as the j -th entry of x, y respectively. Then

Lemma 5. Under Assumptions 1-2, the projected gradients in (13) have the following properties.

Proof. The proof for ∥ G ( θ ) β ( θ, ξ ) ∥ in Eq. (35) simply follows from the contraction property of projection as follows.

Then, ∥∇ θ f ( λ θ,ξ ) ∥ ≤ ℓ θ by Proposition 3. The proof logic for ∥ G ( ξ ) β ( θ, ξ ) ∥ is the same.

Next, we prove Eq. (36) as follows and the proof of Eq. (37) follows the same logic.

where (i) uses Proposition 3.

Lemma 6. Suppose X ⊂ R d is a closed convex set. For any x ∈ R d and x ′ ∈ X , we have

Proof. For any δ ∈ (0 , 1] , x δ := δx ′ + (1 -δ )proj X ( x ) belongs to the convex set X since x ′ , proj X ( x ) ∈ X . Then based on the definition of projection we have

The above inequality can be rearranged as follows

which proves Eq. (39) as δ → +0 .

C Stochastic Gradients

To get the stochastic estimation of the gradients (8) and (9), we first estimate the occupancy measure (1) as follows.

where ✶ {·} is an indicator function and τ ( λ ) := { τ ( λ ) i } m λ i =1 contains m λ independent trajectories τ ( λ ) i := { s ( λ ) i,h , a ( λ ) i,h } H λ -1 h =0 ( i = 1 , . . . , m λ ) of length H λ generated from the policy π θ and transition kernel p ξ . Then the estimated cost function is ̂ c := ∇ λ f [ ̂ λ ( τ ( λ ) )] .

Algorithm 3 Obtain Stochastic Gradients at ( θ, ξ )

1: Input: z := ( θ, ξ ) ∈ Z := Θ ×Z .

2: Hyperparameters: m λ , H λ , m θ , H θ , m ξ , H ξ .

3: Generate independent trajectories τ ( λ ) i := { s ( λ ) i,h , a ( λ ) i,h } H λ -1 h =0 ( i = 1 , . . . , m λ ) from π θ , p ξ .

4: Obtain ̂ λ ( τ ( λ ) ; s, a ) for every s, a ∈ S × A by Eq. (40) with τ ( λ ) := { τ ( λ ) i } m λ i =1 .

5: Obtain ̂ c := ∇ λ f [ ̂ λ ( τ ( λ ) )] .

6: Generate independent trajectories τ ( θ ) i := { s ( θ ) i,h , a ( θ ) i,h } H θ -1 h =0 ( i = 1 , . . . , m θ ) from π θ , p ξ .

7: Obtain g ( θ ) ( τ ( θ ) , θ, ξ, ̂ c ) by Eq. (41) with τ ( θ ) := { τ ( θ ) i } m θ i =1 .

8: Generate independent trajectories τ ( ξ ) i := { s ( ξ ) i,h , a ( ξ ) i,h } H ξ -1 h =0 ( i = 1 , . . . , m ξ ) from π θ , p ξ .

9: Obtain g ( ξ ) ( τ ( ξ ) , θ, ξ, ̂ c ) by Eq. (42) with τ ( ξ ) := { τ ( ξ ) i } m ξ i =1 .

10: Output: g ( θ ) ( τ ( θ ) , θ, ξ, ̂ c ) ≈ ∇ θ f ( λ θ,ξ ) , g ( ξ ) ( τ ( ξ ) , θ, ξ, ̂ c ) ≈ ∇ ξ f ( λ θ,ξ ) .

The stochastic gradients (8) and (9) can be approximated respectively by the following stochastic sample averaged values known as GPOMDP [50].

where τ ( θ ) := { τ ( θ ) i } m θ i =1 and τ ( ξ ) := { τ ( ξ ) i } m ξ i =1 contain m θ independent trajectories τ ( θ ) i := { s ( θ ) i,h , a ( θ ) i,h } H θ -1 h =0 ( i = 1 , . . . , m θ ) and m ξ independent trajectories τ ( ξ ) i := { s ( ξ ) i,h , a ( ξ ) i,h } H ξ -1 h =0 ∪ { s ( ξ ) i,H ξ } ( i = 1 , . . . , m ξ ) respectively, both generated from the policy π θ and transition kernel p ξ .

We summarize the procedure of obtaining the stochastic gradients (8) and (9) in Algorithm 3. These stochastic gradients approximate the true gradients with the following error bounds.

Proposition 9. Under Assumptions 1 and 2, the stochastic gradients (41) and (42) have the following error bounds.

D Proof of Proposition 1

As follows, we slightly rewrite the utility function f defined in Eq. (5), by replacing λ with λ θ,ξ .

Therefore, for any θ ∈ Θ , we have

.

Recalling the definition of the robust value function (3), i.e., V ( k ) θ def = max ξ ∈ Ξ ⟨ c ( k ) , λ θ,ξ ⟩ , the equation above can be rewritten as follows.

.

Therefore, θ ∈ Θ minimizes max ξ ∈ Ξ f ( λ θ,ξ ) if and only if θ solves the constrained robust RL problem (4), as repeated below.

Finally, we will prove that f ( λ ) defined in Eq. (5) is a convex function. Note that A k = { λ ∈ ∆ S×A : ⟨ c ( k ) , λ ⟩ ≤ τ k } is a convex set, so A = ∩ K k =1 A k is also a convex set. Then for any λ 1 , λ 0 ∈ ∆ S×A and α ∈ [0 , 1] , we aim to prove that

If either λ 1 / ∈ A or λ 0 / ∈ A , then Eq. (45) obviously holds as the right side equals + ∞ . Otherwise, if λ 1 , λ 0 ∈ A , then δλ 1 +(1 -δ ) λ 0 ∈ A as A is a convex set, and thus Eq. (45) holds with equality as proved below.

Proof of Proposition 2

E

The utility function f in Eq. (7) satisfies

where (i) uses λ θ,ξ ( s ) = ∑ a λ θ,ξ ( s, a ) and Eq. (33) that λ θ,ξ ( s, a ) = λ θ,ξ ( s ) π θ ( a | s ) , and (ii) denotes the entropy function that H [ π θ ( ·| s )] = -∑ a π θ ( a | s ) log π θ ( a | s ) . The above function is exactly the minimax objective function (6) of the robust entropy regularized RL.

Finally, we will prove that f ( λ ) defined in Eq. (7) is a convex function. For any λ 0 , λ 1 ∈ ∆ S×A and δ ∈ [0 , 1] , denote λ δ = δλ 1 +(1 -δ ) λ 0 , λ δ ( s ) = ∑ a λ δ ( s, a ) and policy π δ ( a | s ) = λ δ ( s,a ) λ δ ( s ) . Then, the convexity of f can be proved as follows.

F Proof of Proposition 3

The first formula of Eq. (10) can be proved as follows and the second formula can be proved in the same way.

where (i) uses Eq. (41) and (ii) uses c ( s t , a t ) ∈ [0 , 1] and Assumption 1. Define the following V function.

For any fixed cost function c : S × A → R , the gradient ∇ θ V θ,ξ ( c ) can be rewritten as follows.

where (i) uses Eq. (5) of [6] and (ii) uses the occupancy measure (1) and defines the following Q function.

The above Q function has the following upper bound

where c max := max s,a | c ( s, a ) | and also satisfies the following Bellman equation.

Therefore, for any θ, θ ′ ∈ Θ , ξ, ξ ′ ∈ Ξ and fixed cost function c , we have

where (i) uses Eq. (49). Rearranging the above inequality yields that

where (i) uses Lemma 1. For any θ ∈ Θ , ξ ∈ Ξ and fixed cost functions c, c ′ , we have

where (i) uses Eq. (48).

Therefore, Eq. (11) can be proved as follows.

where (i) uses the gradient (47), (ii) uses Assumptions 1-2 and Eqs. (49), (51) and (52) with c max = ∥∇ λ f ( λ θ ′ ,ξ ′ ) ∥ ∞ replaced by its upper bound ℓ λ , (iii) uses Assumption 2, (iv) uses Lemma 3 and (v) uses Assumption 1.

The proof of Eq. (12) follows the same logic. To elaborate, ∇ ξ V θ,ξ ( c ) can be derived as follows in a similar way to the derivation of Eq. (47)

where (i) uses the occupancy measure (1) and defines the following V function.

The above V function has the following upper bound

where c max := max s,a | c ( s, a ) | and also satisfies the following Bellman equation.

As a result,

where (i) uses Eq. (55) and Lemma 1. Rearranging the above inequality yields that

Similar to Eq. (52), we have

Therefore, we can prove Eq. (12) as follows.

where (i) uses the gradient (53) and denotes ∇ λ f ( λ θ ′ ,ξ ′ )( s, a ) = ∂f ( λ θ ′ ,ξ ′ ) ∂λ ( s,a ) as the ( s, a ) -th element of ∇ λ f ( λ θ ′ ,ξ ′ ) , (ii) uses ∇ ξ p ξ ′ ( s ′ | s, a ) = p ξ ′ ( s ′ | s, a ) ∇ ξ log p ξ ′ ( s ′ | s, a ) , Assumptions 1-2 and Eqs. (31), (55), (57) and (58) with c max = ∥∇ λ f ( λ θ ′ ,ξ ′ ) ∥ ∞ replaced by its upper bound ℓ λ , (iii) uses Assumptions 2 and (iv) uses Eq. (34).

G Proof of Proposition 4

Implement one projected gradient step from θ and obtain θ ′ = proj Θ ( θ -β ∇ θ f ( λ θ,ξ ) ) = θ -βG ( θ ) β ( θ, ξ ) where the projected gradient G ( θ ) β ( θ, ξ ) is defined by Eq. (13). Based on Assumption 4, for any δ ∈ [0 , δ ] , there exists θ δ ∈ Θ such that λ θ δ ,ξ = (1 -δ ) λ θ ′ ,ξ + δλ θ ∗ ( ξ ) ,ξ ∈ V λ θ,ξ . Based on Assumption 4, for any δ ∈ [0 , δ ] , there exists θ δ ∈ Θ such that λ θ δ ,ξ = (1 -δ ) λ θ ′ ,ξ + δλ θ ∗ ( ξ ) ,ξ ∈ V λ θ ′ ,ξ . Then we have,

where (i) uses the L θ,θ -smoothness of f ( λ · ,ξ ) based on Proposition 3, (ii) uses Lemma 4. By applying Lemma 6 to X = Θ , x = θ -β ∇ θ f ( λ θ,ξ ) , x ′ = θ δ ∈ Θ (so proj X ( x ) = θ ′ = θ -βG ( θ ) β ( θ, ξ ) ), we obtain that

Then on one hand, f ( λ θ δ ,ξ ) has the following lower bound.

where (i) and (iii) use L θ,θ -smoothness of f ( λ · ,ξ ) based on Proposition 3, (ii) uses Eq. (60), and (iv) uses Lemma 5, ∥ θ δ -θ ′ ∥ ≤ √ 2 δℓ λ -1 (obtained in the same way as Eq. (59)) and θ ′ -θ = -βG ( θ ) β ( θ, ξ ) . On the other hand, f ( λ θ δ ,ξ ) has the following upper bound since f is convex.

The above two inequalities (61) and (62) imply that

Finally, we prove Eq. (20) as follows.

where (i) uses Eq. (10) which implies that f λ · ,ξ is ℓ θ -Lipschitz, (ii) uses θ ′ -θ = -βG ( θ ) β ( θ, ξ ) and Eq. (63).

H Proof of Proposition 5

We will first prove that ξ δ ∈ Ξ . Note that the s -rectangular ambiguity set Ξ can be expressed as a Cartesian product of Ξ s for all s ∈ S . Hence, as Ξ is convex, Ξ s is convex for all s ∈ S . Therefore, for any s ∈ S , ξ δ ( s, · , · ) ∈ Ξ s since it is a convex combination of ξ 0 ( s, · , · ) ∈ Ξ s and ξ 1 ( s, · , · ) ∈ Ξ s defined by Eq. (21), so ξ δ ∈ Ξ .

Next, we will prove λ θ,ξ δ = δλ θ,ξ 1 +(1 -δ ) λ θ,ξ 0 . Denote λ δ := δλ θ,ξ 1 +(1 -δ ) λ θ,ξ 0 , so the aim becomes to prove λ θ,ξ δ = λ δ . Based on Lemma 2, it suffices to prove the following equation.

For each s ∈ S , consider the following two cases.

(Case 1): λ θ,ξ 1 ( s ) > 0 or λ θ,ξ 0 ( s ) > 0 . Note that

(Case 2): λ θ,ξ 0 ( s ) = λ θ,ξ 1 ( s ) = 0 . In this case, λ δ ( s ) = δλ θ,ξ 1 ( s )+(1 -δ ) λ θ,ξ 0 ( s ) = 0 and thus λ δ ( s, a ) = λ θ,ξ 1 ( s, a ) = λ θ,ξ 0 ( s, a ) = 0 for any a ∈ A , so Eq. (65) also holds for any choice of ξ δ ( s, · , · ) .

Therefore, we can prove Eq. (64) as follows.

where (i) uses Eq. (65) and λ δ := δλ θ,ξ 1 +(1 -δ ) λ θ,ξ 0 and (ii) applies Lemma 2 to λ θ,ξ 1 and λ θ,ξ 0 .

I Proof of Proposition 6

Fix any θ ∈ Θ , and there exists at least one ξ ∗ ∈ arg max ξ ′ ∈ Ξ f ( λ θ,ξ ′ ) . If ξ ∗ ∈ V (Ξ) := × s ∈S V (Ξ s ) , then this proposition directly holds. Hence, we focus on the case where ξ ∗ / ∈ V (Ξ) , which means ξ ∗ ( s 0 ) / ∈ V (Ξ s 0 ) for at least one s 0 ∈ S .

Based on Assumption 6-7, there exists a probability vector ν := [ ν 1 , . . . , ν M s 0 ] such that ξ ∗ ( s 0 ) = ∑ M s 0 m =1 ν m ξ ( s 0 ) m , where we denote ξ ′ ( s ) := ξ ′ ( s, · , · ) ∈ Ξ s for all ξ ′ ∈ Ξ . Without loss of generality, we assume ν 1 = max 1 ≤ m ≤ M s ν m (Otherwise we can make this assumption hold by permutating the elements in each Ξ s .).

̸

For any ϵ > 0 , define ξ ∗ 1 , ξ ( ϵ ) ∈ (∆ S ) S×A such that ξ ∗ 1 ( s ) = ξ ( ϵ ) ( s ) = ξ ∗ ( s ) for any s = s 0 , while at s = s 0 we define ξ ∗ 1 ( s 0 ) = ξ ( s 0 ) 1 ∈ V (Ξ s 0 ) and

which implies that

It is easily seen that ξ ∗ 1 ∈ Ξ by its definition. Since lim ϵ → +0 [(1 + ϵ ) ν 1 ( s ) -ϵ ] = ν 1 ( s ) = max 1 ≤ m ≤ M s ν m ( s ) > 0 , there exists a sufficiently small ϵ > 0 such that [(1 + ϵ ) ν 1 ( s ) -ϵ, (1 + ϵ ) ν 2 ( s ) , . . . , (1+ ϵ ) ν M s ( s )] ∈ [0 , 1] | Ξ s | is a probability vector and thus ξ ( ϵ ) ∈ Ξ . Furthermore, select arbitrary δ ∈ [0 , 1] if λ θ,ξ ( ϵ ) ( s 0 ) = λ θ,ξ ∗ 1 ( s 0 ) = 0 and the following δ otherwise.

where λ θ,ξ ( s ) := ∑ a ∈A λ θ,ξ ( s, a ) is defined as the state occupancy measure for any s ∈ S , θ ∈ Θ and ξ ∈ Ξ . Then it can be directly verified that ξ ∗ satisfies the following equality.

Hence, based on Proposition 5, ξ ∗ satisfies

On one hand, f ( λ θ,ξ ∗ 1 ) ≤ f ( λ θ,ξ ∗ ) and f ( λ θ,ξ ( ϵ ) ) ≤ f ( λ θ,ξ ∗ ) since ξ ∗ ξ ∗ ∈ arg max ξ ′ ∈ Ξ f ( λ θ,ξ ′ ) . On the other hand, the above Eq. (68) along with convexity of f implies that

Therefore, f ( λ θ,ξ ∗ 1 ) = f ( λ θ,ξ ( ϵ ) ) = f ( λ θ,ξ ∗ ) = max ξ ′ ∈ Ξ f ( λ θ,ξ ′ ) . If ξ ∗ 1 ∈ V (Ξ) , then the proof is done. Otherwise, note that ξ ∗ 0 ( s 0 ) / ∈ V (Ξ s 0 ) while ξ ∗ 1 ( s 0 ) ∈ V (Ξ s 0 ) , and ξ ∗ 1 ( s ) = ξ ∗ 0 ( s ) for any s = s 0 . Hence, in the same way, we can obtain the sequence ξ ∗ 2 , ξ ∗ 3 , . . . , ξ ∗ N that satisfies the following conditions by changing non-vertex into vertex at one state each time until no non-vertex remains (i.e., until the condition 2 below holds):

̸

̸

For 1 ≤ k ≤ N -1 , ξ ∗ k ( s k ) / ∈ V (Ξ s k ) while ξ ∗ k +1 ( s k ) ∈ V (Ξ s k ) , and ξ ∗ k ( s ) = ξ ∗ k +1 ( s ) for any s = s k .

ξ ∗ N ∈ V (Ξ) .

For 1 ≤ k ≤ N , f ( λ θ,ξ ∗ k ) = max ξ ′ ∈ Ξ f ( λ θ,ξ ′ ) .

As a result, we find the optimal vertex ξ ∗ N ∈ V (Ξ) ∩ arg max ξ ′ ∈ Ξ f ( λ θ,ξ ′ ) , which concludes the proof.

J Proof of Proposition 7

Based on Assumption 8, for any θ ∈ Θ and δ ∈ [0 , δ ] , there exists θ δ ∈ Θ such that λ θ δ ,ξ = (1 -δ ) λ θ,ξ + δλ θ ∗ ,ξ .

where (i) uses the L θ,θ -smoothness of f ( λ · ,ξ ) based on Proposition 3, (ii) uses Lemma 4. Hence, on one hand, using L θ,θ -smoothness of f ( λ · ,ξ ) based on Proposition 3, f ( λ θ δ ,ξ ) has the following lower bound.

On the other hand, f ( λ θ δ ,ξ ) has the following upper bound since f is convex.

Combining the above two inequalities, we obtain that

where (i) uses Eq. (71), and (ii) uses Eq. (69) and assumes f ( λ θ,ξ ) ≥ min θ ′ ∈ Θ max ξ ′ ∈ Ξ f ( λ θ ′ ,ξ ′ ) without loss of generality (otherwise, Eq. (24) trivially holds).

Based on the Bolzano-Weierstrass theorem, there exists a sequence δ n → +0 such that θ δn -θ ∥ θ δn -θ ∥ → d ∈ R d Θ as n → + ∞ . Hence, ∥ d ∥ = 1 and we can conclude the proof by letting δ = δ n and n → + ∞ in the above inequality.

K Proof of Proposition 8

K.1 Proof of Assumptions 1, 2 and 3

Proposition 2 indicates that the utility function f defined by Eq. (7) is convex, which proves Assumption 3.

To prove Assumptions 1 and 2, it suffices to prove that the following functions have bounded first-order and second-order derivatives for any ( s, a, s ′ ) ∈ S × A × S .

For any θ ∈ Θ = [ -R,R ] |S|×|A| , we have

When p = 1 or p = ∞ , any ξ ∈ Ξ satisfies

where ξ min > 0 since it is minimum over finitely many positive numbers ̂ ξ ( s, a, s ′ ) -α s . Then for any θ ∈ Θ and ξ ∈ Ξ , we have

Finally, for any ( s, a, s ′ ) , ( s 1 , a 1 , s ′ 1 ) , ( s 2 , a 2 , s ′ 2 ) ∈ S × A × S , we obtain all the derivative bounds as follows, where {·} is an indicator function.

K.2 Proof of Assumptions 5, 6 and 7 about ambiguity set Ξ

It is straightforward to verify that the ambiguity set Ξ = { ξ ∈ (∆ S ) S×A : ∥ ξ ( s, : , :) -̂ ξ ( s, : , :) ∥ p ≤ α s , ∀ s ∈ S} ( p ∈ { 1 , ∞} ) is convex and compact, which proves Assumption 5.

Assumption 6 can be proved easily by letting Ξ s = { ξ s ∈ (∆ S ) A : ∥ ξ s -̂ ξ ( s, : , :) ∥ p ≤ α s } ( p ∈ { 1 , ∞} ).

Then we will prove Assumption 7, that is, Ξ s = { ξ s ∈ (∆ S ) A : ∥ ξ s -̂ ξ ( s, : , :) ∥ p ≤ α s } ( p ∈ { 1 , ∞} ) is a polyhedron. Based on Definition 1.1 and Theorem 1.26 of [9], it is equivalent to prove that Ξ s is bounded (already proved above) and is an intersection of finitely many closed half-planes (obvious based on the definitions of ∥ · ∥ 1 and ∥ · ∥ ∞ ).

K.3 Proof of Assumptions 4 and 8

We will only prove Assumption 8, since Assumption 4 can be proved in the same way.

Fix any ξ ∈ Ξ and θ ∈ Θ = [ -R,R ] |S|×|A| . Then we select any θ ∗ = θ ∗ ( θ ) ∈ arg min θ ′ ∈ Θ min ∥ θ ′ -θ ∥ ∞ where Θ min := arg min θ ′ ∈ Θ Γ( θ ′ ) is a compact set since Γ is a continuous function.

Define the following notations.

λ ( δ ) θ,ξ = (1 -δ ) λ θ,ξ + δλ θ ∗ ,ξ for δ ∈ [0 , 1] (we select δ = 1 ).

Policy π ( δ ) θ,ξ defined as π ( δ ) θ,ξ ( a | s ) = λ ( δ ) θ,ξ ( s,a ) λ ( δ ) θ,ξ ( s ) where λ ( δ ) θ,ξ ( s ) = ∑ a ′ λ ( δ ) θ,ξ ( s, a ′ )

(Note that λ ( δ ) θ,ξ ( s ) ≥ λ ( δ ) θ,ξ ( s, a ) ≥ λ min > 0 , so λ ( δ ) θ,ξ ( s ) can be the denominator and π ( δ ) θ,ξ ( a | s ) > 0 ).

θ ( δ ) θ,ξ ∈ R |S|×A with each entry defined as follows.

which is valid since π ( δ ) θ,ξ ( a | s ) > 0 and π θ ( a | s ) ≥ π min > 0 .

V θ,ξ = { λ ( δ ) θ,ξ : δ ∈ [0 , 1] } ⊂ ∆ S×A .

Now, it remains to prove the following two statements.

(P4): The mapping θ ( δ ) θ,ξ → λ ( δ ) θ,ξ from U θ,ξ to V θ,ξ is a bijection and is Lipschitz continuous in both directions.

(P1) obviously follows from Eq. (76).

Note that ( θ ( δ ) θ,ξ ) s,a defined by Eq. (76) is a monotone function of δ ∈ [0 , 1] , and (P1) implies that ( θ (0) θ,ξ ) s,a = θ s,a ∈ [ -R,R ] and θ (1) θ,ξ = θ ∗ s,a ∈ [ -R,R ] . Therefore, ( θ ( δ ) θ,ξ ) s,a ∈ [ -R,R ] which proves (P2).

To prove (P3), rewrite Eq. (76) as follows.

where we denote c δ ( s ) := log [ λ ( δ ) θ,ξ ( s ) ] -log [ (1 -δ ) λ θ,ξ ( s ) ∑ a ′ exp( θ s,a ′ ) + δλ θ ∗ ,ξ ( s ) ∑ a ′ exp( θ ∗ s,a ′ ) ] . Therefore,

Note that for any θ ′ ∈ R |S|×|A| , we have

where (i) uses Lemma 2. Then we have

where (i) uses Eq. (78), (ii) uses λ ( δ ) θ,ξ = (1 -δ ) λ θ,ξ + δλ θ ∗ ,ξ and (iii) uses the Eq. (79) for θ ′ ∈ { θ ∗ , θ } . Based on Lemma 2, the equality above implies (P3).

Next, we prove (P4). Note that the mapping from θ to λ ( δ ) θ,ξ = λ θ ( δ ) θ,ξ ,ξ is Lipschitz continuous based on Lemma 3. Hence, we only need to consider its reverse mapping.

If λ θ ∗ ,ξ = λ θ,ξ , then π θ ∗ = π θ . Hence, θ ∈ Θ min := arg min θ ′ ∈ Θ Γ( θ ′ ) because Γ( θ ′ ) = max ξ ′ ∈ Ξ f ( λ θ ′ ,ξ ′ ) can be seen as a function of π θ ′ . Therefore, θ ∗ = θ which means U θ,ξ = { θ } and V θ,ξ = { λ θ,ξ } are singletons. In this case, (P4) trivially holds.

̸

Therefore, we focus on the case where λ θ ∗ ,ξ = λ θ,ξ . Before proving (P4), we will prove the following statement.

(P5) There exists a constant L ′ > 0 such that ∥ θ ∗ -θ ∥ ∞ ≤ L ′ ∥ λ θ ∗ ,ξ -λ θ,ξ ∥ ∞ for any θ ∈ Θ = [ -R,R ] |S|×|A| .

Define θ ′∗ ∈ θ ∈ R |S|×|A| such that θ ′∗ s,a = θ ∗ s,a + 1 |A| ∑ a ′ ( θ s,a ′ -θ ∗ s,a ′ ) . Then it can be easily verified that

Note that π θ ( a | s ) = exp( θ s,a ) ∑ a ′ exp( θ s,a ′ ) and π θ ′ ( a | s ) = exp( θ ′ s,a ) ∑ a ′ exp( θ ′ s,a ′ ) , so θ ′ s,a -θ s,a = log π θ ′ ( a | s ) -log π θ ( a | s ) + u ( s ) where u ( s ) := log [ ∑ a ′ exp( θ ′ s,a ′ ) ] -log [ ∑ a ′ exp( θ s,a ′ ) ] . Then combining with Eq. (81), we obtain that u ( s ) = 1 |A| ∑ a ′ [log π θ ( a ′ | s ) -log π θ ′ ( a ′ | s )] . Therefore,

where (i) uses Eqs. (73) and (80) which imply that π θ ( a | s ) , π θ ∗ ( a | s ) = π θ ′ ( a | s ) ∈ [ π min , 1] for θ, θ ∗ ∈ Θ , (ii) uses λ θ ∗ ,ξ ( s ) , λ θ,ξ ( s ) ≥ |A| λ min for θ, θ ∗ ∈ Θ as a result of Eq. (75).

Based on the definition of θ ′∗ , we have max a ′ θ ′∗ s,a ′ -min a ′ θ ′∗ s,a ′ = max a ′ θ s,a ′ -min a ′ θ s,a ′ ≤ 2 R . Therefore, for each s ∈ S , there are three cases: -R ≤ min a ′ θ ′∗ s,a ′ ≤ max a ′ θ ′∗ s,a ′ ≤ R , max a ′ θ ′∗ s,a ′ > R and min a ′ θ ′∗ s,a ′ < -R , and we can define θ ′′ ∈ R |S|×|A| as follows

.

It can be easily verified that the θ ′′ defined above satisfies θ ′′ ∈ Θ = [ -R,R ] |S|×|A| (since max a ′ θ ′∗ s,a ′ -min a ′ θ ′∗ s,a ′ ≤ 2 R ) and π θ ′′ = π θ ′∗ = π θ ∗ (the second = comes from Eq. (80)). Therefore, θ ′′ ∈ Θ min and thus

where (i) uses θ ′′ ∈ Θ min and θ ∗ ∈ arg min θ ′ ∈ Θ min ∥ θ ′ -θ ∥ ∞ . To further obtain the upper bound of ∥ θ ′′ -θ ′∗ ∥ ∞ in the above inequality, we discuss the three aforementioned cases.

(Case II): When max a ′ θ ′∗ s,a ′ > R , we have

where (i) uses θ ∈ Θ = [ -R,R ] |S|×|A| .

(Case III): When min a ′ θ ′∗ s,a ′ < -R , we have

where (i) uses θ ∈ Θ = [ -R,R ] |S|×|A| .

Summarizing the above three cases, we obtain that ∥ θ ′′ -θ ′∗ ∥ ∞ ≤ ∥ θ ′∗ -θ ∥ ∞ and therefore Eq. (83) implies that

where (i) uses Eq. (82). This proves (P5).

We consider Eq. (76) as a function of δ and take its derivative as follows.

where (i) uses θ, θ ∗ ∈ Θ = [ -R,R ] |S|×|A| and Eq. (75), (ii) uses θ, θ ∗ ∈ Θ = [ -R,R ] |S|×|A| , (iii) uses Eq. (84).

Therefore, for any δ, δ ′ ∈ [0 , 1] , we have

which proves the statement (P4) and thus proves Assumption 8.

Assumption 4 can be proved in the same way simply by replacing θ ∗ with any θ ∗ ( ξ ) ∈ arg min θ ′ ∈ Θ f ( λ θ ′ ,ξ ) .

L Proof of Proposition 9

The estimated occupancy measure (40) is an unbiased estimator of the following truncated occupancy measure with truncation level H λ .

Denote ̂ λ ( τ ( λ ) ) := [ ̂ λ ( τ ( λ ) ; s, a ) ] s,a ∈S×A ∈ R |S||A| , λ ( H λ ) θ,ξ := [ λ ( H λ ) θ,ξ ( s, a ) ] s,a ∈S×A ∈ R |S||A| , λ θ,ξ := [ λ θ,ξ ( s, a ) ] s,a ∈S×A ∈ R |S||A| , Then the estimation error of occupancy measure has the following upper bound.

where (i) uses E ∥ X ∥ 2 = Var X + ∥ E X ∥ 2 for random vector X := ̂ λ ( τ ( λ ) ) -λ θ,ξ , (ii) uses Eqs. (1) and (86) and uses the fact that ̂ λ defined by Eq. (40) is average among the m λ i.i.d. individual estimators ̂ λ i ( τ ( λ ) i ; s, a ) := (1 -γ ) ∑ H λ -1 h =0 γ h ✶ { s ( λ ) i,h = s, a ( λ ) i,h = a } for i = 1 , . . . , m λ , (iii) uses Var X ≤ E ∥ X ∥ 2 for random vector X := ̂ λ 1 ( τ ( λ ) ) and P π θ ,p ξ ( s t = s, a t = a | s 0 ∼ ρ ) ∈ [0 , 1] , and (iv) uses 0 ≤∥ ̂ λ 1 ( τ ( λ ) i ∥ 2 ≤ ∑ s,a ̂ λ 1 ( τ ( λ ) i ; s, a ) = 1 and ∑ s,a P π θ ,p ξ ( s t = s, a t = a | s 0 ∼ ρ ) = 1 .

Define the cost function as c := ∇ λ f ( λ θ,ξ ) . The error of estimating c by ̂ c := ∇ λ f [ ̂ λ ( τ ( λ ) )] has the following upper bounds.

where (i) uses Assumption 2 and (ii) uses Eq. (87). Also, c and ̂ c have the following norm bound based on Assumption 2.

Note that g ( θ ) ( τ ( θ ) , θ, ξ, c ) defined by Eq. (41) (replace ̂ c with c ) is the average of the following m θ i.i.d. individual stochastic gradients.

Then it can be easily seen that g ( θ ) i ( τ ( θ ) i , θ, ξ, c ) defined above is an unbiased estimator of the following truncated policy gradient.

Also, g ( θ ) i ( τ ( θ ) i , θ, ξ, c ) can be bounded as follows by using Eq. (90) and Assumption 1.

Therefore, we can prove Eq. (43) as follows, and Eq. (44) can be proved using the same logic.

where (i) uses Eqs. (8),(41),(92) and ∇ θ f ( H θ ) ( λ θ,ξ ) = E π θ ,p ξ g ( θ ) ( τ ( θ ) , θ, ξ, c ) , (ii) uses Eq. (90), Assumption 1 and g ( θ ) ( τ ( θ ) , θ, ξ, c ) = 1 m θ ∑ m θ i =1 g ( θ ) i ( τ ( θ ) i , θ, ξ, c ) where { g ( θ ) i ( τ ( θ ) i , θ, ξ, c ) } m θ i =1 are independent, (iii) uses Eqs. (89) and (93).

M Proof of Theorem 1

The policy gradient (8) is proved by Eqs. (5) and (6) of [6]. We will next prove the transition gradient (9).

Under the transition kernel p ξ and policy π θ , the probability of obtaining a certain trajectory τ t = { s h , a h } t h =0 ∪ { s t +1 } with initial state distribution ρ can be expressed as follows.

Hence, the gradient of the log of the above probability can be computed as follows.

Denote c := ∇ λ f ( λ θ,ξ ) . Then the transition gradient (9) can be obtained as follows.

where (i) uses Eq. (94).

N Proof of Theorem 2

Based on Proposition 9, we define the following error terms E ( ξ ) j and E ( θ ) j ( i, j ∈ { 1 , 2 , 3 , 4 } ) to bound the estimation errors of the stochastic gradients in lines 6, 10, 17 and 21 of Algorithm 1 respectively.

Then we prove Theorem 2 in the following procedures.

N.1 Convergence Rate of Inner Update Step (14) of the First Original Phase

Lemma 7. Suppose Assumptions 1-4 hold. Apply the projected stochastic gradient descent step (14) in Algorithm 1 to the policy optimization problem min θ ∈ Θ f ( λ θ,ξ k ) with fixed ξ k ∈ Ξ . Select stepsize α = 1 2 L θ,θ and initialization θ k, 0 = θ 0 . Then the output θ k := θ k,T globally converges at the following rate for any δ ∈ [0 , δ ] .

where E ( θ ) 1 is defined in Eq. (95) .

Proof. For any θ k,t ∈ Θ in the update rule (14), there exists at least one optimal policy θ ∗ k,t ∈ arg min θ ′ ∈ Θ f ( λ θ ′ ,ξ k ) such that for any δ ∈ [0 , δ ] , there exists θ ( δ ) k,t ∈ Θ that satisfies λ θ ( δ ) k,t ,ξ k = (1 -δ ) λ θ k,t ,ξ k + δλ θ ∗ k,t ,ξ k . Since f is convex, we have

Then, we analyze the optimization progress of the stochastic gradient descent step (14) as follows.

where (i) and (iii) use L θ,θ -smoothness of f ( λ · ,ξ k ) based on Proposition 3, (ii) uses the update rule (14) with stepsize α = 1 2 L θ,θ which implies that θ k,t +1 ∈ arg min θ ∈ Θ [ ( g ( θ ) k,t ) ⊤ ( θ -θ k,t ) + L θ,θ ∥ θ -θ k,t ∥ 2 ] , (iv) uses Eq. (98) and the following inequality.

where (i) uses Assumption 4 and (ii) uses Lemma 4. Rearranging Eq. (99) and taking conditional expectation, we obtain that

where (i) uses Eq. (43) in Proposition 9 and E ( θ ) 1 defined in Eq. (95) with j = 1 . Then the convergence rate (97) can be proved by iterating the above inequality as follows.

N.2 Convergence Rate of E [ ∥∇ ˜ Φ( ˜ ξ ) ∥ 2 ] from the First Original Phase

Lemma 8. Implement the first original phase of Algorithm 1 with stepsizes α = 1 2 L θ,θ and β = 1 2 L ξ,ξ √ K . The inner projected stochastic gradient descent step (14) is implemented up to precision ϵ 0 > 0 as follows.

Then, the output ˜ ξ of the first original phase has the following convergence rate.

where E ( ξ ) 2 is defined in Eq. (96) with j = 2 .

Proof. For any fixed ξ ∈ Ξ , define the optimal policy parameter θ ∗ ( ξ ) and the optimal utility value Φ( ξ ) as follows.

Since f ( λ θ, · ) is L ξ,ξ -smooth for any θ ∈ Θ based on Proposition 3, for any ( θ, ξ ) ∈ Θ × Ξ , f ( λ θ,ξ ′ ) -L ξ,ξ 2 ∥ ξ ′ -ξ ∥ 2 is a concave function of ξ ′ . As a result, Φ( ξ ′ ) -L ξ,ξ ∥ ξ ′ -ξ ∥ 2 is a L ξ,ξ -strongly concave function of ξ ′ and thus it has the following unique maximizer.

Accordingly, we define the following Moreau envelope function (repeat Eq. (18)).

Based on Lemma 3.6 of [31], ˜ Φ is L ξ,ξ -smooth with

Similar to Lemma D.3 of [31], we obtain the following ascent property of the above envelope function ˜ Φ for any k = 0 , 1 , . . . , K -1 .

where (i) uses Eqs. (15) and (105), (ii) holds for any τ k > 0 whose value is to be determined later, (iii) uses contraction property of projection and ξ ∗ ( ξ k ) ∈ Ξ , (iv) uses Propositions 3-9 and the error

term E ( ξ ) 2 defined in Eq. (96), (v) uses Eqs. (100) and (103), (vi) uses Eqs. (104)-(105) which imply that ˜ Φ( ξ k ) = Φ[ ξ ∗ ( ξ k )] -L ξ,ξ ∥ ξ ∗ ( ξ k ) -ξ k ∥ 2 ≥ Φ( ξ k ) , (vii) uses Eq. (106). Taking unconditional expectation of Eq. (107) and telescoping it over k = 0 , 1 , . . . , K -1 with β = 1 2 L ξ,ξ √ K ∈ [ 0 , 1 2 L ξ,ξ ] ,

τ k ≡ βL ξ,ξ ≤ 1 , we obtain that

where (i) uses the following range of ˜ Φ( ξ ) (defined in Eq. (105)) that holds for any ξ ∈ Ξ .

As a result,

N.3 Convergence of the Inner Update Step (16) of the Second Corrected Phase

Next we focus on the second corrected phase which aims to solve the following minimax optimization problem (repeat Eq. (19)).

where ˜ ξ is obtained from { ξ k } K -1 k =0 uniformly at random in the first original phase. Based on Proposition 3, it can be easily verified that ˜ f has the following smoothness properties and ˜ f ( θ, · ) is L ξ,ξ -strongly concave.

Next, we will see the convergence rate of the projected stochastic gradient ascent steps (16) to the following optimal variable, which is unique due to strong concavity of ˜ f ( θ k , · ) .

The optimization progress of each step of Eq. (16) for k = K,K + 1 , . . . , K + K ′ -1 can be bounded as follows.

where (i) uses Eq. (16), ξ ∗ k ∈ Ξ and contraction property of projection, (ii) holds for any c k > 0 whose value will be assigned later, (iii) uses ∇ ξ ˜ f ( θ k , ξ ∗ k ) = 0 and the definition of ˜ f in Eq. (110), (iv) uses Proposition 9, the error term E ( ξ ) 3 defined in Eq. (96) as well as the 3 L ξ,ξ -smoothness and L ξ,ξ -strongly concavity of ˜ f ( θ k , · ) (see Eq. (112)), and (v) uses a = 1 9 L ξ,ξ and c k = 1 17 . Iterating the unconditional expectation of Eq. (114) over t = 0 , 1 , . . . , T ′ -1 , we obtain that

where the second ≤ denotes D Ξ := sup ξ,ξ ′ ∈ Ξ ∥ ξ ′ -ξ ∥ as the diameter of the compact set Ξ .

N.4 Convergence Rate of E [ ∥ G ( θ ) b ( θ ˜ k , ξ ˜ k ) ∥ 2 ]

Since ˜ f ( θ, · ) is strongly concave, it has unique maximizer ˜ ξ ∗ ( θ ) and the corresponding function value ˜ Ψ( θ ) defined as follows.

Furthermore, since ˜ f ( θ, · ) is L ξ,ξ -strongly concave and ˜ f has the smoothness properties (111) and (112), we can easily obtain that ˜ ξ ∗ ( θ ) is ( L ξ,θ /L ξ,ξ ) -Lipschitz and ˜ Ψ is ˜ L := L θ,θ + L θ,ξ L ξ,θ /L ξ,ξ -smooth with the following gradient, following the proof of Lemma 4.3 in [31]. 2

Note that for any k = K,... , K + K ′ -1 , the projected stochastic gradient ascent step (17) satisfies

where (i) uses θ k ∈ Θ and the definition of projection and (ii) uses the stochastic gradient descent step (17). The above inequality implies that

Then, we analyze the optimization progress of the potential function (117) along the projected stochastic gradient descent step (17) as follows.

where (i) uses the ˜ L := L θ,θ + L θ,ξ L ξ,θ /L ξ,ξ -smoothness of ˜ Ψ , (ii) uses Eq. (118), (iii) uses CauchySchwartz inequality and Eq. (119), (iv) uses b = 1 4 ˜ L , Propositions 3-9 and the error term E ( θ ) 4 defined by Eq. (95), (v) uses Eq. (115). Denote G ( θ ) b ( θ k , ξ k ) = 1 b [ θ k -proj Θ [ θ k -b ∇ θ f ( λ θ k ,ξ k )] ] for k = K,... , K + K ′ -1 . Its norm can be bounded as follows.

where (i) uses Eqs. (17) and (120) as well as G ( θ ) b ( θ k , ξ k ) = 1 b [ θ k -proj Θ [ θ k -b ∇ θ f ( λ θ k ,ξ k )] ] , (ii) uses Eq. (110), and (iii) uses Proposition 9 and the error term E ( θ ) 4 defined by Eq. (95). By rearranging the above inequality and averaging it over k = K,K +1 , . . . , K + K ′ -1 , we obtain the convergence rate of E [ ∥ G ( θ ) b ( θ ˜ k , ξ ˜ k ) ∥ 2 ] as follows.

where (i) uses Eqs. (116)-(117) and selects the stepsize b = 1 4 ˜ L , (ii) uses Eq. (110), and (iii) uses f ∗ := min θ ∈ Θ max ξ ∈ Ξ f ( λ θ,ξ ) and Γ( θ ) := max ξ ∈ Ξ f ( λ θ,ξ ) .

N.5 Convergence Rate of E [ ∥ G ( ξ ) a ( θ ˜ k , ξ ˜ k ) ∥ 2 ]

Denote ψ ( ξ ) := min θ ∈ Θ ˜ f ( θ, ξ ) = Φ( ξ ) -L ξ,ξ ∥ ξ -˜ ξ ∥ 2 . Then, on one hand,

where (i) uses Eq. (104), (ii) uses Eq. (113), (iii) uses ∇ ξ ˜ f ( θ k , ξ ∗ k ) = 0 at the optimal variable ξ ∗ k defined by Eq. (113) and 3 L ξ,ξ -smoothness of ˜ f ( θ, · ) implied by Eq. (112). On the other hand, since ˜ f ( θ, · ) is L ξ,ξ -strongly concave, ψ is L ξ,ξ -strongly concave. Hence,

where (i) uses ∇ ψ [ ξ ∗ ( ˜ ξ ) ] = 0 at the unique optimizer ξ ∗ ( ˜ ξ ) = arg max ξ ∈ Ξ ψ ( ξ ) (see Eq. (104)). Then, we have

where (i) uses proj Ξ ( ξ ∗ k + a ∇ ξ ˜ f ( θ k , ξ ∗ k ) ) -ξ ∗ k = 0 based on Eq. (113), (ii) uses the definition of ˜ f given by Eq. (110), (iii) uses 3 L ξ,ξ -smoothness of ˜ f ( θ k , · ) based on Eq. (112), (iv) uses Eqs. (106), (123) and (124), (v) uses Eq. (110). Taking expectation of the above Eq. (125) and averaging it over k = K,... , K + K ′ -1 , we obtain the convergence rate of E [ ∥ G ( ξ ) a ( θ ˜ k , ξ ˜ k ) ∥ 2 ] as follows.

where (i) uses Eqs. (20), (101) and (115), (ii) uses Eq. (122), (iii) uses ℓ θ , ℓ ξ = O [(1 -γ ) -2 ] , L θ,ξ , L ξ,ξ , ˜ L = O [(1 -γ ) -3 ] based on Proposition 3 and selects stepsizes a = 1 9 L ξ,ξ = O [(1 -γ ) 3 ] , b = 1 4 ˜ L = O [(1 -γ ) 3 ] , β = 1 2 L ξ,ξ √ K = O [ K -1 / 2 (1 -γ ) 3 ] .

N.6 Substituting Hyperparameters

Denote δ = min [ δ, ϵ 2 5760 L ξ,ξ L θ,θ ℓ 2 λ -1 , 1 2 ] = O [(1 -γ ) 6 ϵ 2 ] . Then we select the following hyperparameter values.

m

(2)

λ

=

148512

ϵ

2

L

(1

2

λ

ℓ

-

p

2

ξ

γ

|S||A|

)

4

=

O

[(1

-

γ

)

ϵ

]

,

(140)

-

4

-

2

Substituting the above hyperparameter choices into Eqs. (95) and (96), we have

where we used [ 1 + H ( k ) θ (1 -γ ) ] 2 γ H ( k ) θ ≤ 2 for H ( k ) θ ≥ 4 1 -γ log ( 2 1 -γ ) ( k = 1 , 2 , 3 , 4 ). Lemma 7 implies that for any fixed ξ ∈ Ξ , θ T ′ obtained from the update rule (14) satisfies

where (i) uses Lemma 7, (ii) uses Eqs. (128), (151) and δ ≤ ϵ 2 5760 L ξ,ξ L θ,θ ℓ 2 λ -1 , (iii) uses log(1 -δ ) ≤ -δ for δ ∈ [0 , 1 / 2] . Hence, the above inequality implies that ϵ 0 defined by Lemma 8 satisfies ϵ 0 ≤ ϵ 2 480 L ξ,ξ .

As a result, we can prove that E [ ∥ G ( ξ ) a ( θ ˜ k , ξ ˜ k ) ∥ 2 ] ≤ ϵ 2 and E [ ∥ G ( θ ) b ( θ ˜ k , ξ ˜ k ) ∥ 2 ] ≤ ϵ 2 by substituting ϵ 0 ≤ ϵ 2 480 L ξ,ξ and Eqs. (127)-(154) into the convergence rates (122) and (126). The number of samples required by Algorithm 1 is

O Proof of Theorem 3

Note that σ k = 2 β k ℓ θ , so by the definition of Ξ k := { ξ ∈ V (Ξ) : f ( λ θ k ,ξ ) ≥ max ξ ′ ∈ V (Ξ) f ( λ θ k ,ξ ′ ) -2 β k ℓ θ } we have

Hence,

where (i) uses Eq. (10) in Proposition 3 which implies that f ( λ · ,ξ ) is ℓ θ -Lipschitz continuous, (ii) uses Eq. (155) and the update rule (22) with ∥ d k ∥ = 1 . Eqs. (155) and (156) respectively imply the following two equations.

where ξ ∗ k +1 ∈ arg max ξ ′ ∈ Ξ k f ( λ θ k +1 ,ξ ′ ) .

Based on Proposition 7, there exists a unit descent direction ˜ d k ( ∥ ˜ d k ∥ = 1 ) such that

Then we have

where (i) uses Proposition 6, (ii) uses Ξ k := { ξ ∈ V (Ξ) : f ( λ θ k ,ξ ) ≥ max ξ ′ ∈ V (Ξ) f ( λ θ k ,ξ ′ ) -2 β k ℓ θ } , (iii) uses Eq. (159), (iv) uses A k ( ˜ d k ) ≥ min d ∈ B 1 A k ( d ) ≥ A k ( d ′ k ) -ϵ k based on line 6 of Algorithm 2.

O.1 Analyze the k -th Iteration

(Case 1): If A k ( d ′ k ) ≥ 0 , then Eq. (160) implies that

Hence, by ℓ θ -Lipschitz continuity of Γ( · ) := max ξ ∈ Ξ f ( λ · ,ξ ) (based on Proposition 3), we have

where (i) uses Eq. (161) and the update rule (155) with ∥ d k ∥ = 1 .

(Case 2): If A k ( d ′ k ) < 0 , then since ∥ d ′ k ∥ ≤ 1 , d k = d ′ k / ∥ d ′ k ∥ satisfies A k ( d k ) ≤ A k ( d ′ k ) < 0 . Hence, Eq. (160) implies that

As a result, we bound the one-step optimization progress of Γ( θ k ) as follows.

where (i) uses Eqs. (157) and (158), (ii) uses L θ,θ -smoothness of f ( λ · ,ξ ) based on Proposition 3, (iii) uses the update rule (22) with ∥ d k ∥ = 1 , (iv) uses ξ ∗ k +1 ∈ Ξ k and the definition of A k in Eq. (23), (v) uses Eq. (163). Rearranging the above inequality yields that

where (i) uses β k = 2 √ 2 ℓ λ -1 k +2 .

O.2 Obtain the Convergence Rate (25)

(Case 1): If A k ( d ′ k ) < 0 for all k = 0 , 1 , . . . , K -1 , then we iterate Eq. (164) over k = 0 , 1 , . . . , K -1 as follows.

(Case 2): If A K -1 ( d K -1 ) ≥ 0 , then Eq. (162) holds for k = K -1 , i.e.,

(Case 3): If A K ′ -1 ( d K ′ -1 ) ≥ 0 for some K ′ ∈ { 1 , . . . , K -1 } while A k ( d k ) < 0 for all k = K ′ , . . . , K -1 , then we iterate Eq. (164) over k = K ′ , . . . , K -1 as follows.

where (i) applies Eq. (162) to k = K ′ and (ii) uses β k = 2 √ 2 ℓ λ -1 k +2 .

In sum, the convergence rate (25) holds in all the above three cases.

P Proof of Corollary 1

Note that A k ( d ) := max ξ ∈ Ξ k [ ∇ θ f ( λ θ k ,ξ ) ⊤ d ] defined by Eq. (23) is convex and ℓ θ -Lipschitz continuous. Hence, the best direction d k ∈ arg max d ∈{ d k,t :0 ≤ t ≤ T k } A k ( d ) obtained from the subgradient method (28) converge at the following rate [11].

where (i) denotes d ∗ k ∈ arg max d ∈ B 1 A k ( d ) , (ii) uses T = 36 ℓ 2 λ -1 ℓ 2 θ ϵ 2 = O [(1 -γ ) -4 ϵ -2 ] , α = ϵ 3 ℓ λ -1 ℓ 2 θ and ∥ d k, 0 ∥ , ∥ d ∗ k ∥ ≤ 1 . Hence, the accuracy ϵ k = ϵ 3 ℓ λ -1 is achieved in Algorithm 2. Substituting ϵ k = ϵ 3 ℓ λ -1 , K = 8 ℓ λ -1 ϵ ( ℓ λ -1 L θ,θ +2 √ 2 ℓ θ ) = O [(1 -γ ) -3 ϵ -1 ] into the convergence rate (25), we obtain that

The above ϵ accuracy above requires K | V (Ξ) | = O [ | V (Ξ) | (1 -γ ) -3 ϵ -1 ] evaluations to λ θ k ,ξ , f ( λ θ k ,ξ ) and ∇ θ f ( λ θ k ,ξ ) , KT = O [(1 -γ ) -7 ϵ -3 ] subgradient updates (28), and K = O [(1 -γ ) -3 ϵ -1 ] gradient descent updates to the policy gradient descent updates (22).

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and introduction, we have accurately reflect our contributions, namely, our proposed new learning framework (robust RL with general utility), and our proposed algorithms as well as their convergence results for convex utilities, concave utilities and other utilities that satisfy weak Minty variational inequality.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We mention the limitations in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All our lemmas, propositions and theorems contain assumptions (if there are) and the corresponding proofs are shown in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In the experiment section, we have provided all details needed to reproduce our experiments, including the problem setup (e.g. utilization function, transition kernel parameterization, policy parameterization, ambiguity set, discount factor), the algorithms we implemented (as shown in our algorithm boxes) and the hyperparameter choices.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have uploaded our code which generates the simulation data for our experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have shown the experimental settings in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We report one time implementation of each algorithm.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have shown the hardware and computing resource in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We do not see any ethics violation of our theoretical study.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We do not see any societal impact of our foundational theoretical study since it is not tied to particular applications, let alone deployments.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not release data. The code is simply implementation of the algorithms in this work, which does not pose any risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We have uploaded an anonymized zip file containing our code and a readme file on how to use the code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors combine the topics of robust RL with general non-linear utility. They use policy gradient formulae from general utility RL and combine those with gradient algorithms for minimax problems due to Lin et al. The authors claim to have a convergence theory for the sample based algorithm that ensures convergence to a stationary point (convergence of gradient).\n\nSTRENGTHS:\nThe topic is interesting and it is very reasonable to combine the different setups in the way the authors propose. The algorithm is clever, the results look (mostly) reasonable, a very extensive theoretical study is carried out.\n\nWEAKNESSES:\nIt seems to me the authors did not understand the Landau O notation, at least the way it is used does not make much sense. The O notation is only reasonable (and only defined for) in a limiting sense where typically one limiting variable is chosen. For a fixed number there is no sense in the O notation, it could allow to multiply that number by any real which of course is non-sense! Looking at Theorem 2 the problem becomes visible. The statement as is does not make sense: The accuracy $\\epsilon$ is fixed. Then a number of parameters of the algorithm are fixed as K=O(...), T=O(...), K'=O(...). What does that mean, O in what variable? Please compare with your Corollary 1, that statement makes sense. I went into the proofs to see if this is only imprecisely formulated, but it got even worse. The proof is full of estimates of the type $A\\leq O(...)$ for a number $A$ and, even worse, $O(...)\\leq A$. What is that supposed to be? The proofs are way too long (and this is not the referee's job) to check if the proofs can be saved or if there are O-constants that build up and spoil the proof. I am not too skeptical, the theorems looks somewhat reasonable. On the other hand a lot can happen if all constants are swept under the carpet, I cannot confirm correctness of the results. The analysis should be completely reworked and might make the article interesting for the next top tier conference.\n\nHere are some further points:\n- There is no clear motivation given for the article. As a Mathematician I agree the question is interesting, for a leading ML conference there should be some clearer motivation. The example is extremely artificial and not convincing at all.\n- It is not particularly pleasant to read the article. I believe there are way too many repetitive (and imprecise) citations. One clear citation for the general utility policy gradient theorem should be enough, most citations given do not include such a variant. To me the article is too technical, a little bit of story telling would be very much appreciated. Perhaps the authors might want to think of splitting the article in two.\n- There should be examples for the assumptions. For instance, what standard class of policies satisfy the policy assumptions? This is certainly standard for PG methods but almost all articles discuss some examples (softmax, linear softmax, ...).\n- The complexity \"result\" of Theorem 2 is pretty extreme. For the usual $\\gamma=0.99$ and $\\epsilon=0.01$ the powers make it essentially infinite. I do not criticise the result, that's what the Maths gives. Still, the authors should point out that this is a theoretical estimate with no practical implication. Similarly, the theoretical iteration numbers and batch sizes are ridiculously large. This is a theoretical worst case estimate (perfectly fine with me) but it should be mentioned that in practice those numbers would not be used. If I am not wrong those numbers are not used in their own simulation example (which should also be mentioned).\n\nQUESTIONS:\n- It might be interesting to think about unbiased gradient estimators for general utility RL. Such have been found in the past years for linear RL, I guess something similar works in the non-linear case. \n- Assumption 1, assumption for p. That assumption is somewhat crucial. I do not have a feeling, there is no discussion, if (or not) that assumption is strong. In what situation is the assumption satisfied? Is there any interesting situation in which the assumption is satisfied?\n- Assumption 4 looks brutal to me. Is there any way to verify that assumption? Is there any example for which it can be verified? Perhaps that was addressed in past work, but some discussion is needed.\n- The example: If I am not wrong the choices of parameters do not fit to the theory. Do they? If not, that should at least be discussed. How about the assumptions, can they be checked in that example? If not, I fear the article might prove something on the empty set.\n\nLIMITATIONS:\nAs mentioned above I cannot confirm that I agree with the theoretical results. The statement of Theorem 2 seems incorrect and the proofs continue in the same fashion. Since the article is of only theoretical nature I do not think it can be published in the current form at the top conference. \n\nNonetheless, I believe the research direction is very reasonable and the authors have made very important and very non-trivial research progress! In my view the article needs a full major revision which is not in the scope of a conference revision.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this submission, the authors tackle a new problem: how to train a policy with a general utility when one needs to be robust to some uncertainty in the environment. More precisely, they are looking for the solution of min-max problem: the minimization over a set of parametric policies of the worst case over a set of possible transition probabilities of a (convex) functional of the occupancy measure. They propose a policy gradient type algorithm for which they prove that the gradient goes to 0. They also introduce a more complex algorithm for a specific classical uncertainty set (s-regular polyhedral ambiguity set) that is provably convergent toward the global optimal solution.\n\nSTRENGTHS:\n- The authors attack a novel problem (the combination of general utility and robustness) for which they construct algorithms with theoretical guarantees.\n\n- The content is very technical, but the writing makes it understandable. I really liked the way the authors stress their specific contribution.\n\n- The proof seem correct, although I did not check all of them extensively.\n\n- The results are supported by some numerical experiments).\n\nWEAKNESSES:\n- The nonconvex case is only handled in the Appendix. I liked the fact that they have much more in their pocket than the content of the main article. The nonconvex case could nevertheless have been mentioned only in the conclusion as it is not really part of the main article.\n\n- The numerical experiments should be part of the main article as they add a lot to the theoretical results.\n\n- The results are very technical and some explanations (for example for the main theorems) could be beneficial to the readers.\n\nQUESTIONS:\nCan the author explain if the control on the size of the gradient with respect to \\xi corresponds to a convergence for the functional itself or just that the algorithm is going to slow down?\n\n\nTypos and misc:\n- 134: $\\quad$ before $,$\n- 309: $\\epsilon_k$ only defined in Algorithm\n\nLIMITATIONS:\nThe complexity of their algorithms is quite high, as it is mentioned, and the convergences are local or in a specific setting, but this makes sense for a first paper in a topic.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to incorporate robustness in reinforcement learning environments by allowing the transition kernel to be within a polyhedral s-rectangular uncertainty set to handle general utility functions. It applies a stochastic gradient descent with (gradient sampling subroutines) algorithm designed for global convergence and presents numerical experiments to showcase its convergence properties.\n\nSTRENGTHS:\nThe methodology behind robustifying the transition kernel is well thought out. The theory is supported by proofs that are generally clear to follow. The convergence behavior is validated by numerical simulations.\n\nWEAKNESSES:\n- Modeling motivation: The paper lacks a comparative analysis to demonstrate superiority over existing non-robust methods in handling scenarios like constrained and risk-averse RL. To solidify its contributions, it should include empirical or theoretical evidence comparing its performance against established methods, especially in real-world scenarios that involve distributional shifts.\n\n- Computational costs. Detailed evaluations of the computational overhead comparing to non-robust approaches needs to be presented to demonstrate the price of robustness and usefulness of the modeling.\n\n- Scalability. The robustness in transition kernels is modeled through s-rectangular uncertainty with direct parametrization of the transition kernels, which appears to be less advantageous for scalability and generalization in complex or continuous environments. The paper should discuss potential adaptations or extensions of the method that can handle larger state spaces without compromising theoretical integrity.\n\n- Hyperparameters. The algorithm requires managing an extensive set of hyperparameters, complicating its practical implementation. The paper should propose methods to reduce hyperparameter complexity, such as adaptive tuning or simplified parameter settings, to enhance usability and accessibility.\n\nQUESTIONS:\n- Comparison with Barakat et al. 2023: It would be helpful if the authors could provide a more detailed comparative analysis of their approach with the work by Barakat et al., especially in terms of algorithmic differences and performance in continuous state-action spaces.\n- $\\ell_{\\lambda^{-1}}$: please provide an explicit example of this constant and its dependence on |S| and |A|\n- Line 230: Inversible -> Invertible\n\nLIMITATIONS:\nThe authors touched upon the limitations of theoretical complexity", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
The paper studies the problem of robust RL with general utility function, which is looking at maximizing a general (possibly non-convex) utility function with the worst-case possible transition kernels in an ambiguity set. The paper provides convergence analysis for a wide range of utility functions and ambiguity set.

STRENGTHS:
The technical contribution mainly comes from the following convergence analysis:

1. With convex utility, a two-phase projected stochastic gradient descent ascent algorithm is shown to converge to stationary point.
2. WIth convex utility and an s-rectangular polyhedral ambiguity set, global convergence is proven.
3. The gradient convergence is also proven for concave utility and utility that satisfies weak Minty variational inequality.

WEAKNESSES:
My main concern is about the motivation, interpretation of the results, along with potential tightness results. Please see question section.

QUESTIONS:
1. I think the authors might do a better job motivating and introducing the problem at the introduciton section. I'm confused about the exact definition of robust RL with general utility until the end of section 2. Is it possible to provide intuitive explanations at the beginning? 

2. it might be easier to understand the 4 different settings if the authors can motivate with good examples in practice (convex utility, concave utility, polyhedral ambiguity set and weak Minty variational inequality).

3. There have been a lot of assumptions in the paper before the convergence results. Can authors comment on how practical those assumptions are? And under such assumptions, can the authors comment on the tightness of the results in terms of sample complexity / iteration complexity?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13