Reinforcement Learning with Ex-Post Max-Min Fairness

We consider reinforcement learning with vectorial rewards , where the agent receives a vector of K ≥ 2 different types of rewards at each time step . The agent aims to maximize the minimum total reward among the K reward types . Different from existing works that focus on maximizing the minimum expected total reward , i.e . ex-ante max-min fairness , we maximize the expected minimum total reward , i.e . ex-post max-min fairness . Through an example and numerical experiments , we show that the optimal policy for the former objective generally does not converge to optimality under the latter , even as the number of time steps T grows . Our main contribution is a novel algorithm , Online-ReOpt , that achieves near-optimality under our objective , assuming an optimization oracle that returns a near-optimal policy given any scalar reward . The expected objective value under Online-ReOpt is shown to converge to the asymptotic optimum as T increases . Finally , we propose offline variants to ease the burden of online computation in Online-ReOpt , and we propose generalizations from the max-min objective to concave utility maximization . 1 INTRODUCTION . The prevailing paradigm in reinforcement learning ( RL ) concerns the maximization of a single scalar reward . On one hand , optimizing a single scalar reward is sufficient for modeling simple tasks . On the other hand , in many complex tasks there are often multiple , potentially competing , rewards to be maximized . Expressing the objective function as a single linear combination of the rewards can be constraining and insufficiently expressive for the nature of these complex tasks . In addition , a suitable choice of the linear combination is often not clear a priori . In this work , we consider the reinforcement learning with max-min fairness ( RL-MMF ) problem . The agent accumulates a vector of K ≥ 1 time-average rewards V̄1 : T = ( V̄1 : T , k ) Kk=1 ∈ RK in T time steps , and aims to maximize E [ mink∈ { 1 , ... , K } V̄1 : T , k ] . The maximization objective represents ex-post max-min fairness , in contrast to the objective of ex-ante max-min fairness by maximizing mink∈ { 1 , ... , K } E [ V̄1 : T , k ] . Our main contributions are the design and analysis of the Online-ReOpt algorithm , which achieves near-optimality for the ex-post max-min fairness objective . More specifically , the objective under Online-ReOpt converges to the optimum as T increases . Our algorithm design involves a novel adaptation of the multiplicative weight update method ( Arora et al. , 2012 ) , in conjunction with a judiciously designed re-optimization schedule . The schedule ensures that the agent adapts his decision to the total vectorial reward collected at a current time point , while allowing enough time for the currently adopted policy to converge before switching to another policy . En route , we highlight crucial differences between the ex-ante and ex-post max-min fairness objectives , by showing that an optimal algorithm for the former needs not converge to the optimality even when T increases . Finally , our results are extended to the case of maximizing E [ g ( V̄1 : T ) ] , where g is a Lipschitz continuous and concave reward function . 2 RELATED WORKS . The Reinforcement Learning with Max-Min Fairness ( RL-MMF ) problem described is related to an emerging body of research on RL with ex-ante concave reward maximization . The class of ex-ante concave reward maximization problems include the maximization of g ( E [ V̄1 : T ] ) , as well as its exante variants , including the long term average variant g ( E [ limT→∞ V̄1 : T ] ) and its infinite horizon discounted reward variant . The function g : RK → R is assumed to be concave . The class of ex-ante concave reward maximization problems is studied by the following research works . Chow et al . ( 2017 ) study the case where g is specialized to the Conditional Value-at-Risk objective . Hazan et al . ( 2019 ) study the case when g models the entropy function over the probability distribution over the state space , in order to construct a policy which induces a distribution over the state space that is as close to the uniform distribution as possible . Miryoosefi et al . ( 2019 ) study the case of minimizing the distance between E [ V̄1 : T ] and a target set in RK . Lee et al . ( 2019 ) study the objective of state marginal matching , which aims to make the state marginal distribution match a given target state distribution . Pareto optimality of E [ V̄1 : T ] and its ex-ante variants are studied in ( Mannor & Shimkin , 2004 ; Gábor et al. , 1998 ; Barrett & Narayanan , 2008 ; Van Moffaert & Nowé , 2014 ) . Lastly , a recent work Zahavy et al . ( 2021 ) provides a unifying framework that encompasses many of the previously mentioned works , by studying the problem of maximizing g ( E [ V̄1 : T ] ) and its ex-ante variants , where g is concave and Lipschitz continuous . Our contributions , which concern the ex-post max-min fairness E [ mink∈ { 1 , ... , K } V̄1 : T , k ] and its generalization to the ex-post concave case , are crucially different from the body of works on the ex-ante case . The difference is further highlighted in the forthcoming Section 3.2 . Additionally , a body of works Altman ( 1999 ) ; Tessler et al . ( 2019 ) ; Le et al . ( 2019 ) ; Liu et al . ( 2020 ) study the setting where g is a linear function , subject to the constraint that E [ V̄1 : T ] ( or its ex-ante variants ) is contained in a convex feasible region , such as a polytope . There is another line of research works Tarbouriech & Lazaric ( 2019 ) ; Cheung ( 2019 ) ; Brantley et al . ( 2020 ) focusing on various online settings . The works Tarbouriech & Lazaric ( 2019 ) ; Cheung ( 2019 ) focus on the ex-post setting like ours , but they crucially assume that the underlying g is smooth , which is not the case for our max-min objective nor the case of Lipschitz continuous concave functions . In addition , the optimality gap ( quantified by the notion of regret ) degrades linearly with the number of states , which makes their applications to large scale problems challenging . Brantley et al . ( 2020 ) focus on the ex-ante setting , different from our ex-post setting , and their optimality gap also degrades linearly with the number of states . 3 MODEL . Set up . An instance of the Reinforcement Learning with Max-Min Fairness ( RL-MMF ) problem is specified by the tuple ( S , s1 , A , T , O ) . The set S is a finite state space , and s1 ∈ S is the initial state . In the collection A = { As } s∈S , the set As contains the actions that the agent can take when he is at state s. Each set As is finite . The quantity T ∈ N is the number of time steps . When the agent takes action a ∈ As at state s , he receives the array of stochastic outcomes ( s′ , U ( s , a ) ) , governed by the outcome distribution O ( s , a ) . For brevity , we abbreviate the relationship as ( s′ , U ( s , a ) ) ∼ O ( s , a ) . The outcome s′ ∈ S is the subsequent state he transits to . The outcome U ( s , a ) = ( Uk ( s , a ) ) Kk=1 is a random vector lying in [ −1 , 1 ] K almost surely . The random variable Uk ( s , a ) is the amount of type-k stochastic reward the agent receives . We allow the random variables s′ , U1 ( s , a ) , . . . UK ( s , a ) to be arbitrarily correlated . Dynamics . At time t ∈ { 1 , . . . T } , the agent observes his current state st. Then , he selects an action at ∈ Ast . After that , he receives the stochastic feedback ( st+1 , Vt ( st , at ) ) ∼ O ( st , at ) . We denote Vt ( st , at ) = ( Vt , k ( st , at ) ) K k=1 , where Vt , k ( st , at ) is the type-k stochastic reward received at time t. The agent select the actions { at } Tt=1 with a policy π = { πt } Tt=1 , which is a collection of functions . For each t , the function πt inputs the history Ht−1 = ∪t−1q=1 { sq , aq , Vq ( sq , aq ) } and the current state { st } , and outputs at ∈ Ast . We use the notation aπt to highlight that the action is chosen under policy π . A policy π is stationary if for all t , Ht−1 , st it holds that πt ( Ht−1 , st ) = π̄ ( st ) for some function π̄ , where π̄ ( s ) ∈ As for all s. With a slight abuse of notation , we identify a stationary policy with the function π̄ . Objective . We denote V̄ π1 : t = 1t ∑t q=1 Vq ( sq , a π q ) as the time average vectorial reward during time 1 to t under policy π . The agent ’ s over-arching goal is to design a policy π that maximizes E [ gmin ( V̄ π1 : T ) ] , where gmin : RK → R is defined as gmin ( v ) = mink∈ { 1 , ... , K } vk . Denoting V̄ π1 : T , k as the k-th component of the vector V̄ π1 : T , the value gmin ( V̄ π 1 : T ) = mink V̄ π 1 : T , k is the minimum time average reward , among the reward types 1 , . . . , K. The function gmin is concave , and is 1-Lipschitz w.r.t . ‖ · ‖∞ over the domain RK . When K = 1 , the RL-MMF problem reduces to the conventional RL problem with scalar reward maximization . The case of K > 1 is more subtle . Generally , the optimizing agent needs to focus on different reward types in different time steps , contingent upon the amounts of the different reward types at the current time step . Since the max-min fairness objective could lead to an intractable optimization problem , we aim to design a near-optimal policy for the RL-MMF problem . 3.1 REGRET . We quantify the near-optimality of a policy π by the notion of regret , which is the difference between a benchmark opt ( P ( gmin ) ) and the expected reward E [ gmin ( V̄ π1 : T ) ] . Formally , the regret of a policy π in a T time step horizon is Reg ( π , T ) = opt ( P ( gmin ) ) − E [ gmin ( V̄ π1 : T ) ] . ( 1 ) The benchmark opt ( P ( gmin ) ) is a fluid approximation to the expected optimum . To define opt ( P ( gmin ) ) , we introduce the notation p = { p ( s′|s , a ) } s∈S , a∈As , where p ( s′|s , a ) is the probability of transiting to s′ from s , a . In addition , we introduce v = { v ( s , a ) } s∈S , a∈As , where v ( s , a ) = E [ U ( s , a ) ] is the vector of the K expected rewards . The benchmark opt ( P ( gmin ) ) is the optimal value of the maximization problem P ( gmin ) . For any g : RK → R , we define P ( g ) : max x g ∑ s∈S , a∈As v ( s , a ) x ( s , a ) s.t . ∑ a∈As x ( s , a ) = ∑ s′∈S , a′∈As′ p ( s|s′ , a′ ) x ( s′ , a′ ) ∀s ∈ S ( 2a ) ∑ s∈S , a∈As x ( s , a ) = 1 ( 2b ) x ( s , a ) ≥ 0 ∀s ∈ S , a ∈ As . ( 2c ) The concave maximization problem P ( gmin ) serves as a fluid relaxation to RL-MMF . For each s ∈ S , a ∈ As , the variable x ( s , a ) can be interpreted as the frequency of the agent visiting state s and taking action a . The set of constraints ( 2a ) stipulates that the rate of transiting out of a state s is equal to the rate of transiting into the state s for each s ∈ S , while the sets of constraints ( 2b , 2c ) require that { x ( s , a ) } s∈S , a∈As forms a probability distribution over the state-action pairs . Consequently , opt ( P ( gmin ) ) is an asymptotic ( in T ) upper bound to the expected optimum . Our goal is to design a policy π such that its regret1 Reg ( T ) satisfies Reg ( T ) = opt ( P ( gmin ) ) − E [ gmin ( V̄ π1 : T ) ] ≤ D T γ ( 3 ) holds for all initial state s1 ∈ S and all T ∈ N , with parameters D , γ > 0 independent of T . We assume the access to an optimization oracle Λ , which returns a near-optimal policy given any scalar reward . For ϑ ∈ RK , define the linear function gϑ : RK → R as gϑ ( w ) = ϑ > w = ∑K k=1 ϑkwk . The oracle Λ inputs ϑ ∈ RK , and outputs a policy π satisfying opt ( P ( gϑ ) ) − E [ gϑ ( V̄ π1 : T ) ] = opt ( P ( gϑ ) ) − E [ ϑ > V̄ π1 : T ] ≤ Dlin T β ( 4 ) for all initial state s1 ∈ S and all T ∈ N , with parameters Dlin , β > 0 independent of T . By assuming β > 0 , we are assuming that the output policy π is near-optimal , in the sense that the difference opt ( P ( gϑ ) ) − E [ ϑ > V̄ π1 : T ] converges to 0 as T tends to the infinity . A higher β signifies a 1We omit the notation with π for brevity sake faster convergence , representing a higher degree of near-optimality . We refer to ϑ as a scalarization of v , with the resulting scalarized reward being ϑ > v ( s , a ) for each s , a . Our algorithmic frameworks involve invoking Λ as a sub-routine on different ϑ ’ s . In other words , we assume an algorithmic sub-routine that solves the underlying RL problem with scalar reward ( the case of K = 1 ) , and delivers an algorithm that ensures max-min fairness ( the case of K ≥ 1 ) . Finally , while the main text focuses on gmin , our algorithm design and analysis can be generalized to the case of concave g , as detailed in Appendix C. 3.2 COMPARISON BETWEEN MAXIMIZING E [ gMIN ( V̄ π1 : T ) ] AND gMIN ( E [ V̄ π1 : T ] ) Before introducing our algorithms , we illustrate the difference between the objectives of maximizing E [ gmin ( V̄ π1 : T ) ] and gmin ( E [ V̄ π1 : T ] ) by the deterministic instance in Figure 1 , with initial state s1 = so . ) ( 00 ) ( ( ) The figure depicts an instance with K = 2 . An arc represents an action that leads to a transition from its tail to its head . For example , the arc from so to s ` represents the action ao ` , with p ( s ` | so , ao ` ) = 1 . Likewise , the loop at s ` represents the action a `` with p ( s ` | s ` , a `` ) = 1 . Each arc is labeled with its vectorial reward , which is deterministic . For example , with certainty we have V ( so , ao ` ) = ( 0 0 ) and V ( s ` , a `` ) = ( 0 1 ) . Consider two stationary policies π ` , πr , defined as π ` ( sr ) = aro , π ` ( so ) = ao ` , π ` ( s ` ) = a `` and πr ( sr ) = arr , πr ( so ) = aor , πr ( s ` ) = a ` o . The policy π ` always seeks to transit to s ` , and then loop at s ` indefinitely , likewise for πr . With certainty , V̄ π ` 1 : T = ( 0 1−1/T ) , V̄ π r 1 : T = ( 1−1/T 0 ) . The objective gmin ( E [ V̄ π1 : T ] ) is maximized by choosing πran uniformly at random from the collection { π ` , πr } . We have E [ V̄ πran1 : T ] = ( 1/2−1/ ( 2T ) 1/2−1/ ( 2T ) ) , leading to the optimal value of 1/2 − 1/ ( 2T ) . More generally , existing research focuses on maximizing g ( E [ V̄ π1 : T ] ) for certain concave g , and the related objectives of maximizing g ( limT→∞ E [ V̄ π1 : T ] ) or g ( E [ ∑∞ t=1 α tVt ( st , a π t ) ] ) , where α ∈ ( 0 , 1 ) is the discount factor . In these research works , a near-optimal policy π is constructed by first generating a collection Π of stationary policies , then sampling π uniformly at random from Π. Interestingly , πran is sub-optimal for maximizing E [ gmin ( V̄ π1 : T ) ] . Indeed , Pr ( V̄ πran 1 : T = ( 0 1−1/T ) ) = Pr ( V̄ πran1 : T = ( 1−1/T 0 ) ) = 1/2 , so we have E [ gmin ( V̄ πran1 : T ) ] = 0 for all T . Now , consider the deterministic policy πsw , which first follows π ` for the first bT/2c time steps , then follows πr in the remaining dT/2e time steps . We have V̄ πsw1 : T , k ≥ 1/2 − 2/T for each k ∈ { 1 , 2 } , meaning that gmin ( V̄ πsw 1 : T ) ≥ 1/2− 2/T . Note that gmin ( E [ V̄ πsw 1 : T ] ) ≥ gmin ( E [ V̄ πran 1 : T ] ) − 2/T , so the policy πsw is also near-optimal for maximizing gmin ( E [ V̄ π1 : T ] ) . Altogether , an optimal policy for maximizing gmin ( E [ V̄ π1 : T ] ) can be far from optimal for maximizing E [ gmin ( V̄ π1 : T ) ] . In addition , for the latter objective , it is intuitive to imitate πsw , which is to partition the horizon into episodes and run a suitable stationary policy during each episode . A weakness to πsw is that its partitioning requires the knowledge on T . While our algorithm follows the intuition to imitate πsw , we propose an alternate partitioning that allows does not require T as an input .

Summary: The paper considers RL problem with a vectorial reward of $K$ dimension. The authors consider to maximize the minimal value function $\mathbb{E}[g_{min}(\bar{V}_{1:T})]$ and achieve a regret bound of $O(\sqrt{logK}T^{\frac{2}{3}})$ given an oracle to solve the RL problem with scalar reward. The authors also discuss the offline variant of the proposed algorithm. Finally, the authors verify their algorithm with a queuing system.