Understanding Domain Randomization for Sim-to-real Transfer

1 INTRODUCTION . Reinforcement Learning ( RL ) is concerned with sequential decision making , in which the agent interacts with the environment to maximize its cumulative rewards . This framework has achieved tremendous empirical successes in various fields such as Atari games , Go and StarCraft ( Mnih et al. , 2013 ; Silver et al. , 2017 ; Vinyals et al. , 2019 ) . However , state-of-the-art algorithms often require a large amount of training samples to achieve such a good performance . While feasible in applications that have a good simulator such as the examples above , these methods are limited in applications where interactions with the real environment are costly and risky , such as healthcare and robotics . One solution to this challenge is sim-to-real transfer ( Floreano et al. , 2008 ; Kober et al. , 2013 ) . The basic idea is to train an RL agent in a simulator that approximates the real world and then transfer the trained agent to the real environment . This paradigm has been widely applied , especially in robotics ( Rusu et al. , 2017 ; Peng et al. , 2018 ; Chebotar et al. , 2019 ) and autonomous driving ( Pouyanfar et al. , 2019 ; Niu et al. , 2021 ) . Sim-to-real transfer is appealing as it provides an essentially unlimited amount of data to the agent , and reduces the costs and risks in training . However , sim-to-real transfer faces the fundamental challenge that the policy trained in the simulated environment may have degenerated performance in the real world due to the sim-to-real gap—the mismatch between simulated and real environments . In addition to building higher-fidelity simulators to alleviate this gap , domain randomization is another popular method ( Sadeghi & Levine , 2016 ; Tobin et al. , 2017 ; Peng et al. , 2018 ; OpenAI et al. , 2018 ) . Instead of training the agent in a single simulated environment , domain randomization randomizes the dynamics of the environment , thus exposes the agent to a diverse set of environments in the training phase . Policies learned entirely in the simulated environment with domain randomization can be directly transferred to the physical world with good performance ( Sadeghi & Levine , 2016 ; Matas et al. , 2018 ; OpenAI et al. , 2018 ) . In this paper , we focus on understanding sim-to-real transfer and domain randomization from a theoretical perspective . The empirical successes raise the question : can we provide guarantees for the sub-optimality gap of the policy that is trained in a simulator with domain randomization and directly transferred to the physical world ? To do so , we formulate the simulator as a set of MDPs with tunable latent variables , which corresponds to unknown parameters such as friction coefficient or wind velocity in the real physical world . We model the training process with domain randomization as finding an optimal history-dependent policy for a latent MDP , in which an MDP is randomly drawn from a set of MDPs in the simulator at the beginning of each episode . Our contributions can be summarized as follows : • We propose a novel formulation of sim-to-real transfer and establish the connection between domain randomization and the latent MDP model ( Kwon et al. , 2021 ) . The latent MDP model illustrates the uniform sampling nature of domain randomization , and helps to analyze the sim-to-real gap for the policy obtained from domain randomization . • We study the optimality of domain randomization in three different settings . Our results indicate that the sim-to-real gap of the policy trained in the simulation can be o ( H ) when the randomized simulator class is finite or satisfies certain smoothness condition , where H is the horizon of the real-world interaction . We also provide a lower bound showing that such benign conditions are necessary for efficient learning . Our theory highlights the importance of using memory ( i.e. , history-dependent policies ) in domain randomization . • To analyze the optimality of domain randomization , we propose a novel proof framework which reduces the problem of bounding the sim-to-real gap of domain randomization to the problem of designing efficient learning algorithms for infinite-horizon MDPs , which we believe are of independent interest . • As a byproduct of our proof , we provide the first provably efficient model-based algorithm for learning infinite-horizon average-reward MDPs with general function approximation ( Algorithm 4 in Appendix C.3 ) . Our algorithm achieves a regret bound of Õ ( D √ deT ) , where T is the total timesteps and de is a complexity measure of a certain function class F that depends on the eluder dimension ( Russo & Van Roy , 2013 ; Osband & Van Roy , 2014 ) . 2 RELATED WORK . Sim-to-Real and Domain Randomization The basic idea of sim-to-real is to first train an RL agent in simulation , and then transfer it to the real environment . This idea has been widely applied to problems such as robotics ( e.g. , Ng et al. , 2006 ; Bousmalis et al. , 2018 ; Tan et al. , 2018 ; OpenAI et al. , 2018 ) and autonomous driving ( e.g. , Pouyanfar et al. , 2019 ; Niu et al. , 2021 ) . To alleviate the influence of reality gap , previous works have proposed different methods to help with sim-to-real transfer , including progressive networks ( Rusu et al. , 2017 ) , inverse dynamics models ( Christiano et al. , 2016 ) and Bayesian methods ( Cutler & How , 2015 ; Pautrat et al. , 2018 ) . Domain randomization is an alternative approach to making the learned policy to be more adaptive to different environments ( Sadeghi & Levine , 2016 ; Tobin et al. , 2017 ; Peng et al. , 2018 ; OpenAI et al. , 2018 ) , thus greatly reducing the number of real-world interactions . There are also theoretical works related to sim-to-real transfer . Jiang ( 2018 ) uses the number of different state-action pairs as a measure of the gap between the simulator and the real environment . Under the assumption that the number of different pairs is constant , they prove the hardness of sim-to-real transfer and propose efficient adaptation algorithms with further conditions . Feng et al . ( 2019 ) prove that an approximate simulator model can effectively reduce the sample complexity in the real environment by eliminating sub-optimal actions from the policy search space . Zhong et al . ( 2019 ) formulate a theoretical sim-to-real framework using the rich observation Markov decision processes ( ROMDPs ) , and show that the transfer can result in a smaller real-world sample complexity . None of these results study benefits of domain randomization in sim-to-real transfer . Furthermore , all above works require real-world samples to fine-tune their policy during training , while our work and the domain randomization algorithm do not . POMDPs and Latent MDPs Partially observable Markov decision processes ( POMDPs ) are a general framework for sequential decision-making problems when the state is not fully observable ( Smallwood & Sondik , 1973 ; Kaelbling et al. , 1998 ; Vlassis et al. , 2012 ; Jin et al. , 2020a ; Xiong et al. , 2021 ) . Latent MDPs ( Kwon et al. , 2021 ) , or LMDPs , are a special type of POMDPs , in which the real environment is randomly sampled from a set of MDPs at the beginning of each episode . This model has been widely investigated with different names such as hidden-model MDPs and multi-model MDPs . There are also results studying the planning problem in LMDPs , when the true parameters of the model is given ( Chades et al. , 2012 ; Buchholz & Scheftelowitsch , 2019 ; Steimle et al. , 2021 ) . Kwon et al . ( 2021 ) consider the regret minimization problem for LMDPs , and provide efficient learning algorithms under different conditions . We remark that all works mentioned above focus on the problems of finding the optimal policies for POMDPs or latent MDPs , which is perpendicular to the central problem of this paper—bounding the performance gap of transferring the optimal policies of latent MDPs from simulation to the real environment . Infinite-horizon Average-Reward MDPs Recent theoretical progress has produced many provably sample-efficient algorithms for RL in infinite-horizon average-reward setting . Nearly matching upper bounds and lower bounds are known for the tabular setting ( Jaksch et al. , 2010 ; Fruit et al. , 2018 ; Zhang & Ji , 2019 ; Wei et al. , 2020 ) . Beyond the tabular case , Wei et al . ( 2021 ) propose efficient algorithms for infinite-horizon MDPs with linear function approximation . To the best of our knowledge , our result ( Algorithm 4 ) is the first efficient algorithm with near-optimal regret for infinite-horizon average-reward MDPs with general function approximation . 3 PRELIMINARIES . 3.1 EPISODIC MDPS . We consider episodic RL problems where each MDP is specified byM = ( S , A , P , R , H , s1 ) . S and A are the state and the action space with cardinality S and A respectively . We assume that S and A are finite but can be extremely large . P : S × A → ∆ ( S ) is the transition probability matrix so that P ( ·|s , a ) gives the distribution over states if action a is taken on state s , R : S × A → [ 0 , 1 ] is the reward function . H is the number of steps in one episode . For simplicity , we assume the agent always starts from the same state in each episode , and use s1 to denote the initial state at step h = 1 . It is straight-forward to extend our results to the case with random initialization . At step h ∈ [ H ] , the agent observes the current state sh ∈ S , takes action ah ∈ A , receives reward R ( sh , ah ) , and transits to state sh+1 with probability P ( sh+1|sh , ah ) . The episode ends when sH+1 is reached . We consider the history-dependent policy class Π , where π ∈ Π is a collection of mappings from the history observations to the distributions over actions . Specifically , we use trajh = { ( s1 , a1 , s2 , a2 , · · · , sh ) | si ∈ S , ai ∈ A , i ∈ [ h ] } to denote the set of all possible trajectories of history till step h. We define a policy π ∈ Π to be a collection of H policy functions { πh : trajh → ∆ ( A ) } h∈ [ H ] . We define V πM , h : S → R to be the value function at step h under policy π on MDP M , i.e. , V πM , h ( s ) = EM , π [ ∑H t=hR ( st , at ) | sh = s ] . Accordingly , we define QπM , h : S × A → R to be the Q-value function at step h : QπM , h ( s , a ) = EM , π [ R ( sh , ah ) + ∑H t=h+1R ( st , at ) | sh = s , ah = a ] . We use π∗M to denote the optimal policy for a single MDPM . It can be shown that there exists π∗M such that the policy at step h depends on only the state at step h but not any other prior history . That is , π∗M can be expressed as a collection of H policy functions mapping from S to ∆ ( A ) . We use V ∗M , h and Q ∗ M , h to denote the optimal value and Q-functions under the optimal policy π ∗ M at step h .

Summary: The paper is a theoretical analysis of domain randomization in the context of latent MDPs. The paper provides bounds on a definition of the sim-to-real gap, defined as the difference between the optimal value function and a "domain randomization oracle" value function. In this context the oracle is defined as a history dependent policy which uses the first steps in the environment to improve its belief on the latent variable and then optimally behaves according to this belief.   Provided bounds can be applied in three different contexts: finite domain randomization (i.e. domain randomization on a finite number of domains) satisfying a separation conditions (i.e. a requirement on the existence of a state-action pair which leads to sufficiently different next states), finite domain randomization without the separation condition and infinite domain randomization.   Bounds are expressed in terms of D (i.e. a bound on number of time steps required to reach any other state in the state space) and of H (i.e. the number of steps in an episode). In the case of finite simulators with separation the domain gap is O(D log^3(H)). In the case of finite simulators without separation the domain gap is O(D H^(1/2) log^(1/2)(H)). In the case of infinite domain randomization the gap is approximatively O(D H^(1/2) log^(1/2)(H)).   Bounds are provided with assumptions and most of these assumptions are justified by proving that removing these assumptions opens the possibility to counterexamples which do not meet the provided bounds.