Reinforcement Learning with Ex-Post Max-Min Fairness

We consider reinforcement learning with vectorial rewards , where the agent receives a vector of K ≥ 2 different types of rewards at each time step . The agent aims to maximize the minimum total reward among the K reward types . Different from existing works that focus on maximizing the minimum expected total reward , i.e . ex-ante max-min fairness , we maximize the expected minimum total reward , i.e . ex-post max-min fairness . Through an example and numerical experiments , we show that the optimal policy for the former objective generally does not converge to optimality under the latter , even as the number of time steps T grows . Our main contribution is a novel algorithm , Online-ReOpt , that achieves near-optimality under our objective , assuming an optimization oracle that returns a near-optimal policy given any scalar reward . The expected objective value under Online-ReOpt is shown to converge to the asymptotic optimum as T increases . Finally , we propose offline variants to ease the burden of online computation in Online-ReOpt , and we propose generalizations from the max-min objective to concave utility maximization . 1 INTRODUCTION . The prevailing paradigm in reinforcement learning ( RL ) concerns the maximization of a single scalar reward . On one hand , optimizing a single scalar reward is sufficient for modeling simple tasks . On the other hand , in many complex tasks there are often multiple , potentially competing , rewards to be maximized . Expressing the objective function as a single linear combination of the rewards can be constraining and insufficiently expressive for the nature of these complex tasks . In addition , a suitable choice of the linear combination is often not clear a priori . In this work , we consider the reinforcement learning with max-min fairness ( RL-MMF ) problem . The agent accumulates a vector of K ≥ 1 time-average rewards V̄1 : T = ( V̄1 : T , k ) Kk=1 ∈ RK in T time steps , and aims to maximize E [ mink∈ { 1 , ... , K } V̄1 : T , k ] . The maximization objective represents ex-post max-min fairness , in contrast to the objective of ex-ante max-min fairness by maximizing mink∈ { 1 , ... , K } E [ V̄1 : T , k ] . Our main contributions are the design and analysis of the Online-ReOpt algorithm , which achieves near-optimality for the ex-post max-min fairness objective . More specifically , the objective under Online-ReOpt converges to the optimum as T increases . Our algorithm design involves a novel adaptation of the multiplicative weight update method ( Arora et al. , 2012 ) , in conjunction with a judiciously designed re-optimization schedule . The schedule ensures that the agent adapts his decision to the total vectorial reward collected at a current time point , while allowing enough time for the currently adopted policy to converge before switching to another policy . En route , we highlight crucial differences between the ex-ante and ex-post max-min fairness objectives , by showing that an optimal algorithm for the former needs not converge to the optimality even when T increases . Finally , our results are extended to the case of maximizing E [ g ( V̄1 : T ) ] , where g is a Lipschitz continuous and concave reward function . 2 RELATED WORKS . The Reinforcement Learning with Max-Min Fairness ( RL-MMF ) problem described is related to an emerging body of research on RL with ex-ante concave reward maximization . The class of ex-ante concave reward maximization problems include the maximization of g ( E [ V̄1 : T ] ) , as well as its exante variants , including the long term average variant g ( E [ limT→∞ V̄1 : T ] ) and its infinite horizon discounted reward variant . The function g : RK → R is assumed to be concave . The class of ex-ante concave reward maximization problems is studied by the following research works . Chow et al . ( 2017 ) study the case where g is specialized to the Conditional Value-at-Risk objective . Hazan et al . ( 2019 ) study the case when g models the entropy function over the probability distribution over the state space , in order to construct a policy which induces a distribution over the state space that is as close to the uniform distribution as possible . Miryoosefi et al . ( 2019 ) study the case of minimizing the distance between E [ V̄1 : T ] and a target set in RK . Lee et al . ( 2019 ) study the objective of state marginal matching , which aims to make the state marginal distribution match a given target state distribution . Pareto optimality of E [ V̄1 : T ] and its ex-ante variants are studied in ( Mannor & Shimkin , 2004 ; Gábor et al. , 1998 ; Barrett & Narayanan , 2008 ; Van Moffaert & Nowé , 2014 ) . Lastly , a recent work Zahavy et al . ( 2021 ) provides a unifying framework that encompasses many of the previously mentioned works , by studying the problem of maximizing g ( E [ V̄1 : T ] ) and its ex-ante variants , where g is concave and Lipschitz continuous . Our contributions , which concern the ex-post max-min fairness E [ mink∈ { 1 , ... , K } V̄1 : T , k ] and its generalization to the ex-post concave case , are crucially different from the body of works on the ex-ante case . The difference is further highlighted in the forthcoming Section 3.2 . Additionally , a body of works Altman ( 1999 ) ; Tessler et al . ( 2019 ) ; Le et al . ( 2019 ) ; Liu et al . ( 2020 ) study the setting where g is a linear function , subject to the constraint that E [ V̄1 : T ] ( or its ex-ante variants ) is contained in a convex feasible region , such as a polytope . There is another line of research works Tarbouriech & Lazaric ( 2019 ) ; Cheung ( 2019 ) ; Brantley et al . ( 2020 ) focusing on various online settings . The works Tarbouriech & Lazaric ( 2019 ) ; Cheung ( 2019 ) focus on the ex-post setting like ours , but they crucially assume that the underlying g is smooth , which is not the case for our max-min objective nor the case of Lipschitz continuous concave functions . In addition , the optimality gap ( quantified by the notion of regret ) degrades linearly with the number of states , which makes their applications to large scale problems challenging . Brantley et al . ( 2020 ) focus on the ex-ante setting , different from our ex-post setting , and their optimality gap also degrades linearly with the number of states . 3 MODEL . Set up . An instance of the Reinforcement Learning with Max-Min Fairness ( RL-MMF ) problem is specified by the tuple ( S , s1 , A , T , O ) . The set S is a finite state space , and s1 ∈ S is the initial state . In the collection A = { As } s∈S , the set As contains the actions that the agent can take when he is at state s. Each set As is finite . The quantity T ∈ N is the number of time steps . When the agent takes action a ∈ As at state s , he receives the array of stochastic outcomes ( s′ , U ( s , a ) ) , governed by the outcome distribution O ( s , a ) . For brevity , we abbreviate the relationship as ( s′ , U ( s , a ) ) ∼ O ( s , a ) . The outcome s′ ∈ S is the subsequent state he transits to . The outcome U ( s , a ) = ( Uk ( s , a ) ) Kk=1 is a random vector lying in [ −1 , 1 ] K almost surely . The random variable Uk ( s , a ) is the amount of type-k stochastic reward the agent receives . We allow the random variables s′ , U1 ( s , a ) , . . . UK ( s , a ) to be arbitrarily correlated . Dynamics . At time t ∈ { 1 , . . . T } , the agent observes his current state st. Then , he selects an action at ∈ Ast . After that , he receives the stochastic feedback ( st+1 , Vt ( st , at ) ) ∼ O ( st , at ) . We denote Vt ( st , at ) = ( Vt , k ( st , at ) ) K k=1 , where Vt , k ( st , at ) is the type-k stochastic reward received at time t. The agent select the actions { at } Tt=1 with a policy π = { πt } Tt=1 , which is a collection of functions . For each t , the function πt inputs the history Ht−1 = ∪t−1q=1 { sq , aq , Vq ( sq , aq ) } and the current state { st } , and outputs at ∈ Ast . We use the notation aπt to highlight that the action is chosen under policy π . A policy π is stationary if for all t , Ht−1 , st it holds that πt ( Ht−1 , st ) = π̄ ( st ) for some function π̄ , where π̄ ( s ) ∈ As for all s. With a slight abuse of notation , we identify a stationary policy with the function π̄ . Objective . We denote V̄ π1 : t = 1t ∑t q=1 Vq ( sq , a π q ) as the time average vectorial reward during time 1 to t under policy π . The agent ’ s over-arching goal is to design a policy π that maximizes E [ gmin ( V̄ π1 : T ) ] , where gmin : RK → R is defined as gmin ( v ) = mink∈ { 1 , ... , K } vk . Denoting V̄ π1 : T , k as the k-th component of the vector V̄ π1 : T , the value gmin ( V̄ π 1 : T ) = mink V̄ π 1 : T , k is the minimum time average reward , among the reward types 1 , . . . , K. The function gmin is concave , and is 1-Lipschitz w.r.t . ‖ · ‖∞ over the domain RK . When K = 1 , the RL-MMF problem reduces to the conventional RL problem with scalar reward maximization . The case of K > 1 is more subtle . Generally , the optimizing agent needs to focus on different reward types in different time steps , contingent upon the amounts of the different reward types at the current time step . Since the max-min fairness objective could lead to an intractable optimization problem , we aim to design a near-optimal policy for the RL-MMF problem . 3.1 REGRET . We quantify the near-optimality of a policy π by the notion of regret , which is the difference between a benchmark opt ( P ( gmin ) ) and the expected reward E [ gmin ( V̄ π1 : T ) ] . Formally , the regret of a policy π in a T time step horizon is Reg ( π , T ) = opt ( P ( gmin ) ) − E [ gmin ( V̄ π1 : T ) ] . ( 1 ) The benchmark opt ( P ( gmin ) ) is a fluid approximation to the expected optimum . To define opt ( P ( gmin ) ) , we introduce the notation p = { p ( s′|s , a ) } s∈S , a∈As , where p ( s′|s , a ) is the probability of transiting to s′ from s , a . In addition , we introduce v = { v ( s , a ) } s∈S , a∈As , where v ( s , a ) = E [ U ( s , a ) ] is the vector of the K expected rewards . The benchmark opt ( P ( gmin ) ) is the optimal value of the maximization problem P ( gmin ) . For any g : RK → R , we define P ( g ) : max x g ∑ s∈S , a∈As v ( s , a ) x ( s , a ) s.t . ∑ a∈As x ( s , a ) = ∑ s′∈S , a′∈As′ p ( s|s′ , a′ ) x ( s′ , a′ ) ∀s ∈ S ( 2a ) ∑ s∈S , a∈As x ( s , a ) = 1 ( 2b ) x ( s , a ) ≥ 0 ∀s ∈ S , a ∈ As . ( 2c ) The concave maximization problem P ( gmin ) serves as a fluid relaxation to RL-MMF . For each s ∈ S , a ∈ As , the variable x ( s , a ) can be interpreted as the frequency of the agent visiting state s and taking action a . The set of constraints ( 2a ) stipulates that the rate of transiting out of a state s is equal to the rate of transiting into the state s for each s ∈ S , while the sets of constraints ( 2b , 2c ) require that { x ( s , a ) } s∈S , a∈As forms a probability distribution over the state-action pairs . Consequently , opt ( P ( gmin ) ) is an asymptotic ( in T ) upper bound to the expected optimum . Our goal is to design a policy π such that its regret1 Reg ( T ) satisfies Reg ( T ) = opt ( P ( gmin ) ) − E [ gmin ( V̄ π1 : T ) ] ≤ D T γ ( 3 ) holds for all initial state s1 ∈ S and all T ∈ N , with parameters D , γ > 0 independent of T . We assume the access to an optimization oracle Λ , which returns a near-optimal policy given any scalar reward . For ϑ ∈ RK , define the linear function gϑ : RK → R as gϑ ( w ) = ϑ > w = ∑K k=1 ϑkwk . The oracle Λ inputs ϑ ∈ RK , and outputs a policy π satisfying opt ( P ( gϑ ) ) − E [ gϑ ( V̄ π1 : T ) ] = opt ( P ( gϑ ) ) − E [ ϑ > V̄ π1 : T ] ≤ Dlin T β ( 4 ) for all initial state s1 ∈ S and all T ∈ N , with parameters Dlin , β > 0 independent of T . By assuming β > 0 , we are assuming that the output policy π is near-optimal , in the sense that the difference opt ( P ( gϑ ) ) − E [ ϑ > V̄ π1 : T ] converges to 0 as T tends to the infinity . A higher β signifies a 1We omit the notation with π for brevity sake faster convergence , representing a higher degree of near-optimality . We refer to ϑ as a scalarization of v , with the resulting scalarized reward being ϑ > v ( s , a ) for each s , a . Our algorithmic frameworks involve invoking Λ as a sub-routine on different ϑ ’ s . In other words , we assume an algorithmic sub-routine that solves the underlying RL problem with scalar reward ( the case of K = 1 ) , and delivers an algorithm that ensures max-min fairness ( the case of K ≥ 1 ) . Finally , while the main text focuses on gmin , our algorithm design and analysis can be generalized to the case of concave g , as detailed in Appendix C. 3.2 COMPARISON BETWEEN MAXIMIZING E [ gMIN ( V̄ π1 : T ) ] AND gMIN ( E [ V̄ π1 : T ] ) Before introducing our algorithms , we illustrate the difference between the objectives of maximizing E [ gmin ( V̄ π1 : T ) ] and gmin ( E [ V̄ π1 : T ] ) by the deterministic instance in Figure 1 , with initial state s1 = so . ) ( 00 ) ( ( ) The figure depicts an instance with K = 2 . An arc represents an action that leads to a transition from its tail to its head . For example , the arc from so to s ` represents the action ao ` , with p ( s ` | so , ao ` ) = 1 . Likewise , the loop at s ` represents the action a `` with p ( s ` | s ` , a `` ) = 1 . Each arc is labeled with its vectorial reward , which is deterministic . For example , with certainty we have V ( so , ao ` ) = ( 0 0 ) and V ( s ` , a `` ) = ( 0 1 ) . Consider two stationary policies π ` , πr , defined as π ` ( sr ) = aro , π ` ( so ) = ao ` , π ` ( s ` ) = a `` and πr ( sr ) = arr , πr ( so ) = aor , πr ( s ` ) = a ` o . The policy π ` always seeks to transit to s ` , and then loop at s ` indefinitely , likewise for πr . With certainty , V̄ π ` 1 : T = ( 0 1−1/T ) , V̄ π r 1 : T = ( 1−1/T 0 ) . The objective gmin ( E [ V̄ π1 : T ] ) is maximized by choosing πran uniformly at random from the collection { π ` , πr } . We have E [ V̄ πran1 : T ] = ( 1/2−1/ ( 2T ) 1/2−1/ ( 2T ) ) , leading to the optimal value of 1/2 − 1/ ( 2T ) . More generally , existing research focuses on maximizing g ( E [ V̄ π1 : T ] ) for certain concave g , and the related objectives of maximizing g ( limT→∞ E [ V̄ π1 : T ] ) or g ( E [ ∑∞ t=1 α tVt ( st , a π t ) ] ) , where α ∈ ( 0 , 1 ) is the discount factor . In these research works , a near-optimal policy π is constructed by first generating a collection Π of stationary policies , then sampling π uniformly at random from Π. Interestingly , πran is sub-optimal for maximizing E [ gmin ( V̄ π1 : T ) ] . Indeed , Pr ( V̄ πran 1 : T = ( 0 1−1/T ) ) = Pr ( V̄ πran1 : T = ( 1−1/T 0 ) ) = 1/2 , so we have E [ gmin ( V̄ πran1 : T ) ] = 0 for all T . Now , consider the deterministic policy πsw , which first follows π ` for the first bT/2c time steps , then follows πr in the remaining dT/2e time steps . We have V̄ πsw1 : T , k ≥ 1/2 − 2/T for each k ∈ { 1 , 2 } , meaning that gmin ( V̄ πsw 1 : T ) ≥ 1/2− 2/T . Note that gmin ( E [ V̄ πsw 1 : T ] ) ≥ gmin ( E [ V̄ πran 1 : T ] ) − 2/T , so the policy πsw is also near-optimal for maximizing gmin ( E [ V̄ π1 : T ] ) . Altogether , an optimal policy for maximizing gmin ( E [ V̄ π1 : T ] ) can be far from optimal for maximizing E [ gmin ( V̄ π1 : T ) ] . In addition , for the latter objective , it is intuitive to imitate πsw , which is to partition the horizon into episodes and run a suitable stationary policy during each episode . A weakness to πsw is that its partitioning requires the knowledge on T . While our algorithm follows the intuition to imitate πsw , we propose an alternate partitioning that allows does not require T as an input .

Summary: The paper studies an RL problem, where the rewards are given as a vector at each time step. The goal is to find a policy that balances the total rewards on different dimensions of the reward vector, i.e., one that maximizes the total reward of the worst dimension. More specifically, the paper takes an ex-post perspective and presents an online algorithm that achieves near optimality. An offline variant of this online algorithm is also proposed to alleviate the heavy cost of online computation in some applications. The authors also conducted experiments to evaluate the proposed algorithms.