Efficient Reinforcement Learning in Resource Allocation Problems Through Permutation Invariant Multi-task Learning

1 INTRODUCTION . Sample efficiency in reinforcement learning ( RL ) is an elusive goal . Recent attempts at increasing the sample efficiency of RL implementations have focused to a large extent on incorporating models into the training process : Xu et al . ( 2019 ) ; Clavera et al . ( 2018 ) ; Zhang et al . ( 2018 ) ; Berkenkamp et al . ( 2017 ) ; Ke et al . ( 2019 ) ; Yarats et al . ( 2019 ) ; Huang et al . ( 2019 ) ; Chua et al . ( 2018 ) ; Serban et al . ( 2018 ) . The models encapsulate knowledge explicitly , complementing the experiences that are gained by sampling from the RL environment . Another means towards increasing the availability of samples for a reinforcement learner is by tilting the training towards one that will better transfer to related tasks : if the training process is sufficiently well adapted to more than one task , then the training of a particular task should be able to benefit from samples from the other related tasks . This idea was explored a decade ago in Lazaric & Ghavamzadeh ( 2010 ) and has been gaining traction ever since , as researchers try to increase the reach of deep reinforcement learning from its comfortable footing in solving games outrageously well to solving other important problems . Yu ( 2018 ) discusses a number of methods for increasing sample efficiency in RL and includes experience transfer as one important avenue , covering the transfer of samples , as we do here , transfer of representation or skills , and jumpstarting models which are then ready to be quickly , i.e . with few samples , updated to different tasks . D ’ Eramo et al . ( 2020 ) address the same idea , noting that multi-task learning can improve the learning of each individual task , motivated by robotics-type tasks with underlying commonality , such as balancing a single vs. a double pendulum , or hopping vs. walking . We are interested in exploiting the ability of multi-task learning to solve the sample efficiency problem of RL . Our setting does not apply to all problem classes nor does it seek to exploit the kind of physical similarities found in robotics tasks that form the motivation of Lazaric & Ghavamzadeh ( 2010 ) ; D ’ Eramo et al . ( 2020 ) . Rather , we show that there are a number of reinforcement learning tasks with a particular fundamental property that makes them ideal candidates for multi-task learning with the goal of increasing the availability of samples for their training . We refer to this property as permutation invariance . It is present in very diverse tasks : we illustrate it on a financial portfolio optimization problem , whereby trades are executed sequentially over a given time horizon , and on the problem of meta-learning in a federated supervised learning setting . Permutation invariance in the financial portfolio problem exhibits itself as follows : consider the task of allocating a portion of wealth to each of a number of financial instruments using a trading policy . If the trading policy is permutation invariant , one can change the order of the instruments without changing the policy . This allows one to generate multiple portfolio optimization tasks from a given set of financial instruments . A commonality between applications that have this property is that they concern sequential resource allocation : at each time step , the resource allocation scores the quality of each available candidate entity ( for example a financial instrument in the above example ) , then based on those scores , apportions out the resource ( the total wealth to invest , in the above example ) among the entities at that time step , so that over the horizon of interest , the reward is maximized . Sequential resource allocation problems include applications such as sequential allocation of budget , sequential allocation of space , e.g . in IT systems , hotels , delivery vehicles , sequential allocation of people to work slots or appointments , etc . Many such applications possess permutation invariance in that the ordering of the entities , i.e . where the resources are allocated , can change without changing the resulting optimal allocation . We show that under this form of permutation invariance , it is possible to derive a bound on the performance of the policy . The bound is an extension of that of Lazaric & Ghavamzadeh ( 2010 ) , and while similar to , provides additional information beyond the bound of D ’ Eramo et al . ( 2020 ) . We use the bound to motivate an algorithm that allows for substantially improved results as compared with solving each task on its own . The bound and the algorithm are first analyzed on a synthetic problem that validates the bound in our theorem and confirms the multi-task gain that the theory predicts . Hessel et al . ( 2018 ) ; Bram et al . ( 2019 ) have cautioned against degrading of the performance on each task when some tasks bias the updates to the detriment of others in multi-task learning . They claim that some tasks have a greater density or magnitude of in-task rewards and hence a disproportionate impact on the learning process . In our setting , deleterious effects of some tasks on others could also arise . The algorithm we propose handles this through a form of prioritized sampling , where priorities are put on the tasks themselves , and acts like a prioritized experience replay buffer , applied to a multi-task learning problem . We show empirically that the priorities thus defined protect the overall learning problem from the deleterious effects that unrelated or unhelpful tasks could otherwise have on the policy . The contributions of this work are as follows : ( 1 ) we identify the permutation invariance property of the class of reinforcement learning problems involving sequential resource allocation , ( 2 ) we define a method to increase sample efficiency in these reinforcement learning problems by leveraging this property of permutation invariance ; ( 3 ) we provide a theoretical performance bound for the class of problems ; ( 4 ) we validate experimentally the utility of permutation variance on sample efficiency as well as the validity of the bound on a synthetic problem ; and ( 5 ) we illustrate two real-world RL resource allocation tasks for which this property holds and demonstrate the benefits of the proposed method on sample efficiency and thus also on the overall performance of the models . 2 RELATED WORK . A notable first stream of work on leveraging multi-task learning for enhancing RL performance on single tasks can be found in Wilson et al . ( 2007 ) ; Lazaric & Ghavamzadeh ( 2010 ) which consider , as we do , that there is an underlying MDP from which the multiple tasks can be thought to derive . They use however a Bayesian approach and propose a different algorithmic method than ours . Our results extend performance bounds by Lazaric et al . ( 2012 ) on single-task RL . As noted by Yu ( 2018 ) , jumpstarting , or distilling experiences and representations of relevant policies is another means to increasing sample efficiency in solving a new but related problem . Rusu et al . ( 2016 ) uses this idea in so-called progressive neural networks and Parisotto et al . ( 2015 ) leverage multiple experts to guide the derivation of a general policy . With a similar objective , Teh et al . ( 2017 ) define a policy centroid , that is , a shared distilled policy , that captures the commonalities across the behaviors in the tasks . In all of these distillation-type methods , the tasks considered are simple or complex games . Teh et al . ( 2017 ) note that their policy centroid method , distral , is likely to be affected by task interference , in that differences across tasks may degrade the performance of the resulting policy of any of the constituent tasks . This topic was studied by Hessel et al . ( 2018 ) ; Bram et al . ( 2019 ) . Hessel et al . ( 2018 ) proposed a solution to this by extending the so-called PopArt normalization van Hasselt et al . ( 2016 ) to re-scale the updates of each task so that the different characteristics of the task-specific reward do not skew the learning process . Bram et al . ( 2019 ) use a different approach that learns attention weights of the sub-networks of each task and discards those that are not relevant or helpful . Vuong et al . ( 2019 ) ; D ’ Eramo et al . ( 2020 ) are , like our work , concerned with sharing of experiences to facilitate a more sample-efficient learning process . Vuong et al . ( 2019 ) suggest identifying the shared portions of tasks to allow sharing of samples in those portions . The work of D ’ Eramo et al . ( 2020 ) is in some ways quite similar to ours : the authors ’ goal is the same and they derive a bound as we do on the performance in this setting . However , their setting is different in that their tasks have both shared and task-specific components , and their bound becomes tighter only as the number of tasks increases . In our setting , we do not require a task-specific component , and we are able to show how the distance between the MDPs of each task , in addition to the number of tasks , affects the strength of the bound . Recently , permutation invariance has been exploited in deep multi-agent reinforcement learning ( Liu et al. , 2019 ) where the invariance properties arise naturally in a homogeneous multi-agent setting . Their work employs permutation invariance in learning the critic whereas in our case the entire learned policy employs permutation invariance . 3 PRELIMINARIES . We begin by defining notation . For a measurable space with domain X , let S ( X ) denote the set of probability measures over X , and B ( X ; L ) the space of bounded measurable functions with domain X and bound 0 < L < ∞ . For a measure ρ ∈ S ( X ) and a measurable function f : X → R , the l2 ( ρ ) -norm of f is ‖f‖ρ , and for a set of n points X1 , · · · , Xn ∈ X , the empirical norm , ‖f‖n is ‖f‖2ρ = ∫ f ( x ) 2ρ ( dx ) and ‖f‖2n = 1 n n∑ t=1 f ( Xt ) 2 . Let ‖f‖∞ = supx∈X |f ( x ) | be the supremum norm of f . Consider a set of MDPs indexed by t. Each MDP is denoted by a tupleMt = 〈X , A , Rt , Pt , γ〉 , where X , a bounded closed subset of the s-dimensional Euclidean space , is a common state space ; A is a common action space , Rt : X ×A → R is a task specific reward function uniformly bounded by Rmax , Pt is a task specific transition kernel such that Pt ( ·|x , a ) is a distribution over X for all x ∈ X and a ∈ A , and γ ∈ ( 0 , 1 ) is a common discount factor . Deterministic policies are denoted by π : X → A . For a given policy π , the MDPMt is reduced to a Markov chainMπt = 〈X , Rπt , Pπt , γ〉 with reward function Rπt ( x ) = Rt ( x , π ( x ) ) , transition kernel P π t ( ·|x ) = Pt ( ·|x , π ( x ) ) , and stationary distribution ρπt . The value function V πt for MDP t is defined as the unique fixed-point of the Bellman operator T πt : B ( X ; Vmax = Rmax/ ( 1− γ ) ) → B ( X ; Vmax ) , given by ( T πt V ) ( x ) = Rπt ( x ) + γ ∫ X Pπt ( dy|x ) V ( y ) . Let π∗t denote the optimal policy forMt . The optimal value function V π∗t t forMt is defined as the unique fixed-point of its optimal Bellman operator T π ∗ t t which is defined by ( T π ∗ t t V ) ( x ) = max a∈A [ Rt ( x , a ) + γ ∫ X Pt ( dy|x , a ) V ( y ) ] . To approximate the value function V , we use a linear approximation architecture with parameters α ∈ Rd and basis functions ϕi ∈ B ( X ; L ) for i = 1 , · · · , d. Let ϕ ( · ) = ( ϕ1 ( · ) , · · · , ϕd ( · ) ) T ∈ Rd be the feature vector and F the linear function space spanned by basis functions ϕi . Thus , F = { fα | α ∈ Rd and fα ( · ) = ϕ ( · ) Tα } . Consider a learning task to dynamically allocate a common resource across entities Ut ⊆ U . Each t corresponds to a task , but for now take t to be an arbitrary fixed index . At each time step n , the decision maker observes states xn = ( xi , n ) i∈Ut of the entities , where xi , n is the state of entity i , and takes action an = ( ai , n ) i∈Ut , where ai , n is the share of the resource allocated to entity i . The total resource capacity is normalized to 1 for convenience . Therefore , allocations satisfy 0 ≤ ai , n ≤ 1 and∑ i∈Ut ai , n = 1 . We consider policy πθ ( xn ) parameterized by θ . Assume that we have access to the reward function Rt as well as a simulator that generates a trajectory of length N given any arbitrary policy πθ . The objective of the learning task is to maximize Jt ( θ ) = E [ N∑ n=1 γn−1Rt ( xn , an ) ∣∣∣∣∣ an+1 = πθ ( xn ) , xn+1 ∼ Pt ( ·|xn , an ) , x1 ∼ Pt ( · ) ] . In many settings , N is small and simulators are inaccurate ; therefore , trajectories generated by the simulator are poor representations of the actual transition dynamics . This occurs in batch RL where trajectories are rollouts from a dataset . In these cases , policies overfit and generalize poorly .

Summary: This paper addresses sequential resource allocation problem using reinforcement learning, where sample efficiency is the focus of the paper. The authors identify a key property in the targeted resource allocation problems -- the permutation invariance -- which intrinsically implies the independency of samples at different time steps. From this property the paper extends the work of D’Eramo et al. (2020) and derives a potentially tighter bound for the gap between the optimal policy and the policy learned from multiple tasks. The paper also designs a new algorithms that prioritizes sampling in multi-task learning that addresses the bias between training and target tasks. Empirical evaluations on financial portfolio allocation and meta federated learning demonstrate the effectiveness of the proposed approach.