Monotonic Robust Policy Optimization with Model Discrepancy

1 INTRODUCTION . With deep neural network approximation , deep reinforcement learning ( DRL ) has extended classical reinforcement learning ( RL ) algorithms to successfully solving complex control tasks , e.g. , playing computer games with human-level performance ( Mnih et al. , 2013 ; Silver et al. , 2018 ) and continuous robotic control ( Schulman et al. , 2017 ) . By random exploration , DRL often requires tremendous amounts of data to train a reliable policy . It is thus infeasible for many tasks , such as robotic control and autonomous driving , as training in the real world is not only time-consuming and expensive , but also dangerous . Therefore , training is often conducted on a very limited set of samples , resulting in overfitting and poor generalization capability . One alternative solution is to learn a policy in a simulator ( i.e. , source/training environment ) and then transfer it to the real world ( i.e. , target/testing environment ) . Currently , it is impossible to model the exact environment and physics of the real world . For instance , the physical effects like nonrigidity and fluid dynamics are quite difficult to be accurately modeled by simulation . How to mitigate the model discrepancy between the training and target environments remains challenging for the generalization in RL . To simulate the dynamics of the environment , domain randomization ( DR ) , a simple but effective method is proposed . It randomizes the simulator ( e.g. , by randomizing the distribution of environment parameters ) to generate a variety of environments for training the policy in the source domain . Compared with training in a single environment , recent researches have shown that policies learned through an ensemble of environment dynamics obtained by DR achieve better generalization performance with respect to the expected return . The expected return is referred to as the average per- formance across all the trajectories sampled from different environments . Since these trajectories , regardless of their performance , are uniformly sampled , the trajectories with the worst performance would severely degrade the overall performance . In contrast , another line of research on the generalization in RL is from the perspective of control theory , i.e. , learning policies that are robust to environment perturbations . Robust RL algorithms learn policies , also using model ensembles produced by perturbing the parameters of the nominal model . EPOpt ( Rajeswaran et al. , 2017 ) , a representative of them , trains policy solely on the worst performing subset , i.e. , trajectories with the worst α percentile of returns , while discarding all the higher performing trajectories . In other words , it seeks a higher worst-case performance at the cost of degradation on the average performance . In general , robust RL algorithms may sacrifice performance on many environment variants and focus only on environments with the worst performance , such that the policy learned will not behave very badly in a previously unseen environment . In this paper , we focus on the generalization issue in RL , and aim to mitigate the model discrepancy of the transition dynamics between the training and target environments . Considering that both the average and worst-case performance are equally important for evaluating the generalization capability of the policy , we propose a policy optimization approach in which the distribution of the sampled trajectories are specifically designed for concurrently improving both the average and worst-case performance . Our main contributions are summarized as follows . • For a given policy and a wide range of environments , we theoretically derive a lower bound for the worst-case expected return of that policy over all the environments , and prove that maximizing this lower bound ( equivalent to maximizing the worst-case performance ) can be achieved by solving an average performance maximization problem , subject to constraints that bound the update step in policy optimization and statistical distance between the worst and average case environments . To the best of our knowledge , this theoretical analysis of the relationship between the worst-case and average performance is reported for the first time , which provides a practical guidance for updating policies towards both the worst-case and average performance maximization . • Trajectories obtained from diverse environments may contribute differently to the generalization capacity of the policy . Therefore , in face of a huge amount of trajectories , the problem that which types of trajectories are likely to mostly affect the generalization performance should be considered . Unlike traditional uniform sampling without the worst-case performance guarantee , and different from the worst α percentile sampling in which the parameter α is empirically preset , we propose a criterion for the sampling trajectory selection based on the proposed worst-case and average performance maximization , with which both the environment diversity and the worst-case environments are taken into account . • Based on the proposed theorem , we develop a monotonic robust policy optimization ( MRPO ) algorithm to learn the optimal policy with both the maximum worst-case and average performance . Specifically , MRPO carries out a two-step optimization to update the policy and the distribution of the sampled trajectories , respectively . We further prove that the policy optimization problem can be transformed to trust region policy optimization ( TRPO ) ( Schulman et al. , 2015 ) on all possible environments , such that the policy update can be implemented by the commonly used proximal policy optimization ( PPO ) algorithm ( Schulman et al. , 2017 ) . Finally , we prove that by updating the policy with the MRPO , the worst-case expected return can be monotonically increased . • To greatly reduce the computational complexity , we impose Lipschitz continuity assumptions on the transition dynamics and propose a practical implementation of MRPO . We then conduct experiments on five robot control tasks with variable transition dynamics of environments , and show that MRPO can improve both the average and worst-case performance in the training environments compared to DR and Robust RL baselines , and significantly facilitate the learned policy with a better generalization capability in unseen testing environments . 2 BACKGROUND . Under the standard RL setting , the environment is modeled as a Markov decision process ( MDP ) defined by a tuple < S , A , T , R > . S is the state space and A is the action space . For the convenience of derivation , we assume they are finite . T : S × A × S → [ 0 , 1 ] is the transition dynamics determined by the environment parameter p ∈ P , where P denotes the environment pa- rameter space . For example in robot control , environment parameter could be physical coefficient that directly affect the control like friction of joints and torso mass . Throughout this paper , by environment p , we mean that an environment has the transition dynamics determined by parameter p. R : S × A → R is the reward function . At each time step t , the agent observes the state st ∈ S and takes an action at ∈ A guided by policy π ( at|st ) . Then , the agent will receive a reward rt = R ( st , at ) and the environment shifts from current state st to the next state st+1 with probability T ( st+1|st , at , p ) . The goal of RL is to search for a policy π that maximizes the expected cumulative discounted reward η ( π|p ) = Eτ [ G ( τ |p ) ] , G ( τ |p ) = ∑∞ t=0 γ trt . τ = { st , at , rt , st+1 } ∞t=0 denotes the trajectory generated by policy π in environment p and γ ∈ [ 0 , 1 ] is the discount factor . We can then define the state value function as Vπ ( s ) = E [ ∑∞ k=0 γ krt+k|st = s ] , the ac- tion value function as Qπ ( s , a ) = E [ ∑∞ k=0 γ krt+k|st = s , at = a ] , and the advantage function as Aπ ( s , a ) = Qπ ( s , a ) − Vπ ( s ) . We denote the state distribution under environment p and policy π as Pπ ( s|p ) and that at time step t as P tπ ( s|p ) . During the policy optimization in RL , by updating the current policy π to a new policy π̃ , Schulman et al . ( 2015 ) prove that η ( π̃|p ) ≥ Lπ ( π̃|p ) − 2λγ ( 1− γ ) 2 β2 , Lπ ( π̃|p ) = η ( π|p ) + Es∼Pπ ( ·|p ) , a∼π ( ·|s ) [ π̃ ( a|s ) π ( a|s ) Aπ ( s , a ) ] ( 1 ) where λ = maxs |Ea∼π ( a|s ) [ Aπ ( s , a ) ] | is the maximum mean advantage following current policy π and β = maxsDTV ( π ( ·|s ) ‖π̃ ( ·|s ) ) is the maximum total variation ( TV ) distance between π and π̃ . The policy ’ s expected return after updating can be monotonically improved by maximizing the lower bound in ( 1 ) w.r.t . π̃ . Based on this and with certain approximation , Schulman et al . ( 2015 ) then propose a algorithm named trust region policy optimization ( TRPO ) that optimizes π̃ towards the direction of maximizing Lπ ( π̃ ) , subject to the trust region constraint β ≤ δ . In standard RL , environment parameter p is fixed without any model discrepancy . While under the domain randomization ( DR ) settings , because of the existence of model discrepancy , environment parameter should actually be a random variable p following a probability distribution P over P . By introducing DR , the goal of policy optimization is to maximize the mean expected cumulative discounted reward over all possible environment parameters , i.e. , maxπ Ep∼P [ η ( π|p ) ] . In face of model discrepancy , our goal is to provide a performance improvement guarantee for the worst-case environments , and meanwhile to improve the average performance over all environments . Lemma 1 . Guided by a certain policy π , there exists a non-negative constant C ≥ 0 , such that the expected cumulative discounted reward in environment with the worst-case performance satisfies : η ( π|pw ) − Ep∼P [ η ( π|p ) ] ≥ −C , ( 2 ) where environment pw corresponds to the worst-case performance , and C is related to pw and π . Proof . See Appendix A.1 for details . Theorem 1 . In MDPs where reward function is bounded , for any distribution P overP , by updating the current policy π to a new policy π̃ , the following bound holds : η ( π̃|pw ) ≥ Ep∼P [ η ( π̃|p ) ] − 2|r|max γEp∼P [ ( pw‖p ) ] ( 1− γ ) 2 − 4|r|maxα ( 1− γ ) 2 , ( 3 ) where ( pw‖p ) , maxt Es′∼P t ( ·|pw ) Ea∼π ( ·|s′ ) DTV ( T ( s|s′ , a , pw ) ‖T ( s|s′ , a , p ) ) , environment pw corresponds to the worst-case performance under the current policy π , and α , maxt Es′∼P t ( ·|pw ) DTV ( π ( a|s′ ) ‖π̃ ( a|s′ ) ) . Proof . See Appendix A.2 for details , and Appendix A.7 for bounded reward function condition . In ( 3 ) , ( pw‖p ) specifies the model discrepancy between two environments pw and p in terms of the maximum expected TV distance of their transition dynamics of all time steps in trajectory sampled in environment pw using policy π , and α denotes the maximum expected TV distance of two policies along trajectory sampled in environment pw using policy π . In general , the RHS of ( 3 ) provides a lower-bound for the expected return achieved in the worst-case environment pw , where the first term denotes the mean expected cumulative discounted reward over all environments following the Algorithm 1 Monotonic Robust Policy Optimization 1 : Initialize policy π0 , uniform distribution of environment parameters U , number of environment parameters sampled per iteration M , maximum number of iterations N and maximum episode length T . 2 : for k = 0 to N − 1 do 3 : Sample a set of environment parameters { pi } M−1i=0 according to U . 4 : for i = 0 to M − 1 do 5 : Sample L trajectories { τi , j } L−1j=0 in environment pi using πk . 6 : Determine pkw = argminpi∈ { pi } M−1i=0 ∑L−1 j=0 G ( τi , j |pi ) /L . 7 : Compute Ê ( pi , πk ) = ∑L−1 j=0 G ( τi , j |pi ) L − 2|r|maxγ ( pi‖pkw ) ( 1−γ ) 2 for environment pi . 8 : end for 9 : Select the trajectory set T = { τi : Ê ( pi , πk ) ≥ Ê ( pkw , πk ) } . 10 : Use PPO for policy optimization on T to get the updated policy πk+1 . 11 : end for sampling distribution P , while the other two terms can be considered as penalization on a large TV distance between the worst-case environment pw and the average case , and a large update step from the current policy π to the new policy π̃ , respectively . Therefore , by maximizing this lower bound , we can improve the worst-case performance , which in practice is equivalent to the following constrained optimization problem with two constraints : max π̃ , P Ep∼P [ η ( π̃|p ) ] s.t . α ≤ δ1 , Ep∼P [ ( pw‖p ) ] ≤ δ2 . ( 4 ) The optimization objective is to maximize the mean expected cumulative discounted reward over all possible environments , by updating not only the policy π̃ , but the environment parameter ’ s sampling distribution P . The first constraint imposes a similar trust region to TRPO ( Schulman et al. , 2017 ) that constrains the update step in policy optimization . In addition , we further propose a new trust region constraint on the sampling distribution P that the TV distance between the worst-case environment pw and average case over P is bounded , such that by achieving the optimization objective in ( 4 ) , the worst-case performance is also improved . To solve the constrained optimization problem in ( 4 ) , we need to seek for the optimal policy π̃ and the distribution P of the sampled trajectories . In practice , we carry out a two-step optimization procedure to simplify the computational complexity . First , we fix the policy by letting π̃ = π , and optimize the objective in ( 4 ) w.r.t . the distribution P . In this case , we no longer need to consider the first constraint on the policy update , and thus can convert the second constraint on the sampling distribution into the objective with the guidance of Theorem 1 , formulating the following unconstrained optimization problem : max P Ep∼P [ E ( p , π̃ ) ] , ( 5 ) where we denote E ( p , π̃ ) , η ( π̃|p ) − 2|r|maxγ ( p‖pw ) ( 1−γ ) 2 . The first term in E ( p , π̃ ) indicates policy π̃ ’ s performance in environment p , while the second term measures the model discrepancy between environment p and pw . Since the objective function in ( 5 ) is linear to P , we can update P by assigning a higher probability to environment p with higher E ( p , π̃ ) . As a consequence , sampling according to E ( p , π̃ ) would increase the sampling probability of environments with both poor and good-enough performance , and avoid being trapped in the worst-case environment . Specifically , we propose to select samples from environment p that meets E ( p , π̃ ) ≥ E ( pw , π̃ ) for the training of policy π̃ , which is equivalent to assigning a zero probability to the other samples . In the second step , we target at optimizing the policy π with the updated distribution P fixed , i.e. , the following optimization problem : max π̃ Ep∼P [ η ( π̃|p ) ] s.t . α ≤ δ1 ( 6 ) Optimization in ( 6 ) can be transformed to a trust region robust policy optimization similar to TRPO and solve it practically with PPO ( refer to Appendix A.3 and Schulman et al . ( 2017 ) for more information ) . To summarize , we propose a monotonic robust policy optimization ( MRPO ) in Algorithm1 . At each iteration k , we uniformly sample M environments and run a trajectory for each sampled environment . For each environment pi , we sampleL trajectories { τi , j } Lj=1 , approximate η ( πk|pi ) with∑L−1 j=0 G ( τi , j |pi ) /L , and determine the worst-case environment pw based on ∑L−1 j=0 G ( τi , j |pi ) /L of a given set of environments piM−1i=0 , We then optimize the policy with PPO on the selected trajectory subset T according to E ( pi , πk ) and E ( pkw , πk ) . We now formally show that by maximizing the lower bound provided in Theorem1 , the worst-case performance within all the environments can be monotonically improved by MRPO . Theorem 2 . The sequence of policy { π1 , π2 , . . . , πN } generated by Algorithm1 is guaranteed with the monotonic worst-case performance improvement , i.e. , η ( π1|p1w ) ≤ η ( π2|p2w ) ≤ · · · ≤ η ( πN |pNw ) , ( 7 ) where pkw denotes the parameter of environment with the worst-case performance guided by the current policy πk at iteration k. Proof . See Appendix A.4 for details .

Summary: This paper introduces Monotonic Robust Policy Optimization (MRPO), an RL algorithm that aims to jointly optimize policy and domain sampling distribution, with the goal of improving policy performance for both average and worst-case scenarios and addressing the model discrepancy between the training and target environments. They derive a lower bound for the worst-case performance, which comprises the average performance, policy change, and the statistical distance between the worst and average case environments. A TRPO-like monotonic performance improvement guarantee is provided for the worst-case expected return. Finally, a practical approximation to MRPO is proposed, which imposes the assumption on Lipschitz continuity with respect to the environment parameters and circumvents the estimation of total variation distance between the worst-case environment and the sampled environment. Experiments are conducted on three control tasks with diverse transition dynamics parameters, where MRPO could improve both average and worst-case performance in the training environments, and it shows better generalization to the unseen test environments than baseline algorithms.