A General Theory of Relativity in Reinforcement Learning

1 INTRODUCTION . Deep reinforcement learning ( RL ) has demonstrated its great successes in recent years , including breakthrough of solving a number of challenging problems like Atari ( Mnih et al. , 2015 ) , GO ( Silver et al. , 2016 ; 2017 ) , DOTA2 ( Berner et al. , 2019 ) and StarCraft II ( Vinyals et al. , 2019 ) , with humanlevel performance or even beyond . These successes demonstrate that current deep RL methods are capable to explore and exploit sufficiently in huge observation and action spaces , as long as sufficient and effective data samples can be generated for training , such as the cases in games . For example , AlphaGo Zero ( Silver et al. , 2017 ) costs 3 days of training over 4.9 millions self-play games , and OpenAI Five ( Berner et al. , 2019 ) and AlphaStar ( Vinyals et al. , 2019 ) spend months of training using thousands of GPUs/TPUs over billions of generated matches . However , for environments that prohibit infinite interactions , e.g. , robotics , real life traffic control and autopilot , etc. , applying general RL is difficult because generating data is extremely expensive and slow . Even if parallel data collection is possible , for example , by deploying multiple robots or vehicles running simultaneously , the scale of collected data is still far below that in virtual games . Worse still , exploration in these environments is considerably limited for safety reasons , which further reduces the effectiveness of the generated data . Due to the above challenges , similar significant advances like in solving virtual games have not been witnessed in these applications yet . Generally , there are three tracks of approaches targeting to alleviate the aforementioned situation to promote the widespread application of RL . They are improving the data efficiency , transfer learning and simulator engineering . To improve data efficiency , many recent efforts have been paid on investigating offline RL algorithms ( Siegel et al. , 2020 ; Fujimoto et al. , 2019 ; Kumar et al. , 2019 ; Wu et al. , 2019 ; Wang et al. , 2018 ) . Compared to standard on-policy or off-policy RL , offline RL ( also known as batch RL ) aims to effectively use previously collected experiences stored in a given dataset , like supervised learning , without online interactions with the environment . The stored experiences may not be generated by a fixed or known policy , so offline RL algorithms can leverage any previously collected data and learn a provably better policy than those who generated the experiences in the dataset . Although offline RL can effectively take advantage of finite data samples , solving a complex real-world task still requires huge amount of high quality offline experiences . Another way to increase data efficiency is to adopt model-based RL . Compared to model-free methods , model-based RL ( Kaiser et al. , 2019 ; Janner et al. , 2019 ; Moerland et al. , 2020 ) learns a dynamics model that mimics the transitions in the true environment , and then the policy can feel free to interact with the learned dynamics instead of the true environment . It has been proved that the true return can be improved by interacting with the learned dynamics model when the model error is bounded ( Janner et al. , 2019 ) . However , learning an accurate dynamics model still requires sufficient transition data by interacting with the true environment , specifically for complex dynamics with noisy transitions . Transfer learning in RL ( Zhu et al. , 2020 ) is practically useful to adapt a policy learned in a source environment to solve another task in the target environment . In the context of this paper , we consider the case where the policy can feel free to explore in the source environment , while the amount of collected data in the target environment should be as small as possible . When the source environment is a simulated one while the target environment takes place in reality , the transfer problem is also known as the simulation to reality ( sim2real ) problem . The simplest way to do transfer is to train the policy in the source environment and then use the converged parameters as warm start for a new policy or part of its parameters in the target environment , so that the amount of interactions with the target is expected to be largely reduced , as long as the tasks and dynamics in the two environments are closely related . Training a shared or partially shared policy in both the source and target environments is an alternative method which also belongs to the multi-task reinforcement learning scope ( Hessel et al. , 2019 ) . Domain adaptation has been demonstrated to be another useful technique ( Ibarz et al. , 2021 ) . Such methods try to bridge the gap between the source and target environments using some adaptation networks . For example , adapter networks were introduced to convert the input in simulation to be close to the real-world observation , by utilizing the generative adversarial model ( James et al. , 2017 ; Shrivastava et al. , 2017 ; Bousmalis et al. , 2017 ; 2018 ; Rao et al. , 2020 ) , or on the contrary an inverse network was trained to convert real-world observation to that in simulation ( James et al. , 2019 ) . Using such adapter networks , the policy only needs to be trained in the source environment , and then it can directly be applied in the target environment . An important concept in transfer learning is that instead of directly deploying RL in the target environment , a source environment is considered as a proxy . Sharing this spirit , the last track of approaches tries to build a proxy simulator that is as close as possible to the target environment , and hence we refer to such methods as simulator engineering . For example , in robotics control problems , there are many mature toolboxes can offer simulation engineering , including MuJoCo , PyBullet , Gazebo , etc . Model-based RL can also be viewed as a specific form of simulator engineering that the simulator is composed by a pure neural network , which is trained to approach the target environment as with lower model error as possible , while this might require a large amount of dynamics data in the target environment as mentioned above . Actually , to achieve more efficient and accurate simulator engineering , one recent rising direction is to integrate differentiable programming and physical systems to build a trainable simulator , which follows the physical laws as in reality and also whose key factors , such as the mass , length and friction of some objects , are trainable like the parameters in neural networks . Representative examples include the DiffTaichi ( Hu et al. , 2020 ) , Brax ( Freeman et al. , 2021 ) and Nimble ( Werling et al. , 2021 ) . Overall , the existing methods focus on either directly improving the data efficiency in the target environment or bridging/reducing the gap between a proxy environment and the target environment , and there lacks a principled theory that can incorporate the learning in the two environments through an unified framework , and explain the intrinsic relationship between the expected returns in the two environments from the perspective of RL . In this paper , we inherit the spirit in transfer learning and consider two environments , where one is free to interact and another is the goal to solve , and the number of interactions in the goal environment should be as small as possible . We believe that there exist some explicit connections between the expected returns in the two environments , given two different policies , from the very fundamental perspective of RL . To verify this , we formally define two Markov Decision Processes ( MDPs ) and then explicitly derive the difference between the expected returns in the two MDPs . In the following context , with the RL convention , an environment is equivalent to an MDP . Specifically , let P ( s′|s , a ) and P ′ ( s′|s , a ) denote two dynamics transition functions in any two arbitrary MDPs sharing the same state and action spaces , where ( s , a , s′ ) is the tuple of the state , action and next state . Let π′ ( a|s ) and π ( a|s ) denote two arbitrary policies , and denote J ( P , π ) as the cumulative expected return given P and π . Then , we aim to investigate the difference J ( P ′ , π ) − J ( P , π′ ) , which is referred to as the relativity gap between the two MDPs . It turns out that the relativity gap has a very interesting and compact form that integrates the interactions in both environments . Now , suppose P and P ′ are the dynamics functions in the source and target MDPs respectively , and J ( P , π′ ) has been maximized by optimizing π′ . Then , with fixed P , P ′ and π′ , maximizing the relativity gap over π by constraining π to be close to π′ will also improve the return J ( P ′ , π ) in the target MDP ; on the other hand , for trainable P , minimizing the relativity gap by optimizing P given a fixed policy π = π′ will reduce the dynamics gap , similar to what is done by conventional model-based RL methods . Based on the above two principles , our theoretical results suggest two general algorithms referred to as Relative Policy Optimization ( RPO ) and Relative Transition Optimization ( RTO ) , respectively . RPO updates the policy using the relative policy gradient to transfer the policy evaluated in the source environment to maximize the return in the target environment , while RTO updates a dynamics model using the relative transition gradient to reduce the value gap in the two environments . Then , applying RPO and RTO simultaneously offers a complete algorithm named Relative Policy-Transition Optimization ( RPTO ) , which can transfer the policy from the source to the target smoothly . RPO , RTO and RPTO interact with the two environments simultaneously , so that data collections from two environments , policy and/or transition updates are completed in a closed loop to form a principled learning framework . In the experimental section , we show how to practically apply RPO , RTO and RPTO algorithms . We demonstrate the effectiveness of these methods in the classic control problems in OpenAI gym with both discrete and continuous actions , by varying the physical variables like mass , length and gravity of the objects to create policy transfer problems . At the last section , we discuss a few new directions based on the proposed relativity theory , which are worthy future investigations . 2 PRELIMINARIES . 2.1 REINFORCEMENT LEARNING . A standard RL problem can be described by a tuple 〈E , A , S , P , r , γ , π〉 , where E indicates the environment that is an MDP with dynamics transition probability P ; at each time step t , st ∈ S is the global state in the state space S , and at ∈ A is the action executed by the agent at time step t from the action space A ; the dynamics transition function P ( st+1|st , at ) is the probability of the state transition ( st , at ) −→ st+1 ; for the most general case , the reward r ( st , at , st+1 ) can be written as a function of st , at and st+1 , while in many tasks it only relies on one or two of them , or it is even a constant in sparse rewards problem . For notation simplicity , we usually write r ( st , at , st+1 ) as rt ; γ ∈ [ 0 , 1 ] is a discount factor and π ( at|st ) denotes a stochastic policy . The following equations define some important quantities in reinforcement learning . The objective of RL is to maximize the expected discounted return J ( P , π ) = Es0 , a0 , ···∼P , π [ ∞∑ t=0 γtrt ] , where s0 ∼ P ( s0 ) , at ∼ π ( at|st ) , st+1 ∼ P ( st+1|st , at ) . At time step t , the state-action value QP , π , value function V P , π , and advantage AP , π are defined as QP , π ( st , at ) = Est+1 , at+1 , ···∼P , π [ ∑∞ l=0 γ lrt+l ] , V P , π ( st ) = Eat , st+1 , ···∼P , π [ ∑∞ l=0 γ lrt+l ] , AP , π ( s , a ) = QP , π ( s , a ) − V P , π ( s ) . In the above standard definitions , we explicitly show their dependence on both the dynamics P and policy π , since we will analyze these functions defined on variant dynamics and policies . This convention will be kept throughout the paper .

Summary: This paper proposed a way to decompose the difference between values of two policies in two MDPs respectively. Such a decomposition results in two parts, the first one is the difference between values of one policy under two MDPs; the second part is the difference between values of two policies under the same MDP. Using this decomposition, the paper then proposed three algorithms.   The first algorithm, called RPO, is used when there are two MDPs that the agent can interact with. The agent uses data from both two MDPs to get a good policy for one of the two MDPs. Such an algorithm is expected to be useful when gathering data from one MDP is costly while it from the other MDP is much cheaper.   The second algorithm is a model learning algorithm. In this setting, the agent only interacts with one MDP and learns a model to approximate the MDP. The RTO algorithm is different from the classic model learning algorithm (regression) in that the model is learned to achieve some consistency between the predicted values from the model and from the MDP.  The third algorithm combines the first two algorithms and is a full model-based algorithm. Specifically, now the agent only interacts with one MDP and learns a model of that MDP using RTO. Meanwhile, it also maintains and updates a policy that is expected to perform well in the MDP, using data from both the MDP and the model.