Sim and Real: Better Together

1 Introduction . Reinforcement learning ( RL ) is a framework where an agent interacts with an unknown environment , receives a feedback from it , and optimizes its performance accordingly [ 44 , 3 ] . There have been attempts of learning a control policy directly from real world samples [ 28 , 49 , 36 , 21 ] . However , in many cases , learning from the actual environment may be slow , costly , or dangerous , while learning from a simulated system can be fast , cheap , and safe . The advantages of learning from simulation are counterbalanced by the reality-gap [ 18 ] : the loss of fidelity due to modeling limitations , parameter errors , and lack of variety in physical properties . The quality of the simulation may vary : when the simulation mimics the reality well , we can train the agent on the simulation and then transfer the policy to the real environment , in a one shot manner ( e.g. , [ 2 ] ) . However in many cases , simulation demonstrates low fidelity which leads to the following question : Can we mitigate the differences between real environments ( `` real '' ) and simulations ( `` sim '' ) thereof , so as to train an agent that learns from both , and performs well in the real one ? In this work , we propose to learn simultaneously on real and sim , while controlling the rate in which we collect samples from each environment and controlling the rate in which we use these samples in the policy optimization . This synergy offers a speed-fidelity trade-off and harnesses the advantage of each domain . Moreover , the simulation speed encourages exploration that helps to accelerate the learning process . The real system in turn can improve exploitation in the sense that it mitigates the challenges of sim-to-real policy transfer , and encourages the learner to converge to relevant solutions . ∗This research was conducted during an internship in Bosch Center of AI . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A general scheme describing our proposed setup is depicted in Figure 1 . In a nutshell , there is a single agent interacting with K environments ( on the left ) . Each sample provided by an environment is pushed into a corresponding replay buffer ( RB ) . On the right , the agent pulls samples from the RBs and is trained on them . In the sim-to-real scheme , K = 2 . In the specific scheme for mixing real and sim samples in the learning process , separate probability measures for collecting samples and for optimizing parameters policies are used . The off-policy nature of our scheme enables separation between real and sim samples which in turn helps controlling the rate of real samples used in the optimization process . In this work we discuss two RL algorithms that can be used with this scheme : ( 1 ) off-policy linear actor critic with mixing sim and real samples and ( 2 ) Deep Deterministic Policy Gradient ( DDPG ; [ 29 ] ) mixing scheme variant based on neural networks . We analyze the asymptotic convergence of the linear algorithm and demonstrate the mixing samples variant of DDPG in a sim-to-real environment . The naive approach in which one pushes the state-action-reward-next-state tuples into a single shared replay buffer is prone to failures due to the imbalance between simulation and real roll-outs . To overcome this , we maintain separate replay buffers for each of the environments ( e.g. , in the case of a single robot and a simulator we would have two replay buffers ) . This allows us to extract the maximum valuable information from reality by distinguishing its tuples from those generated by other environments , while continuously improving the agent using data from all input streams . Importantly , although the rate of samples is skewed in favor of the simulation , the learning may be carried out using a different rate . In a sense , the mechanism we suggest is a version of the importance sampling technique [ 10 ] . Our main contributions in this work are as follows : 1 . We present a method for incorporating real system samples and simulation samples in a policy optimization process while distinguishing between the rate of collecting samples and the rate of using them . 2 . We analyze the asymptotic convergence of our proposed mixing real and sim scheme . 3 . To the best of our knowledge , we provide for the first time theoretical analysis of the dynamics and properties of replay buffer such as its Markovity and the explicit probability measure induces by the replay buffer . 4 . We demonstrate our findings in a simulation of sim-to-real , with two simulations where one is a distorted version of the other and analyze it empirically . 2 Related Work . Sim-to-Real : Sim-to-Real is a long investigated topic in robotics where one aims to reduce the reality gap between the real system and its digital twin implementation . A general framework where we transfer results from one domain to another is domain adaptation . In vision , this approach have helped to gain state-of-the-art results [ 15 , 43 , 32 , 9 , 23 , 42 ] . In our work , we focus on the physical aspects of the sim-to-real gap . Related to domain adaptation , is the approach of domain randomization , where the randomization is done in simulation in order to robustify and enhance the detection and object recognition capability [ 45 , 40 , 19 , 47 ] . Recently , James et al . [ 20 ] proposed a method where both simulation and reality are adapted to a common domain . Andrychowicz et al . [ 2 ] extensively randomize the task of reaching a cube pose where one-shot transfer is achieved but with large sample complexity . Randomization may also be applied to dynamics , e.g. , [ 35 ] , where robustness to inaccuracy in real world parameters is achieved . Another approach in Sim-to-Real is how to change the simulation in the light of real samples . In [ 11 ] the agent learns mainly from simulation but its parameters are updated to match the behavior in reality by reducing the difference between simulation and reality roll-outs . Our method is a direct approach that incorporates phenomena that is difficult to simulate accurately . In Bayesian context , [ 39 ] provide a principled framework to reason about the uncertainty in simulation parameters . Kang et al . [ 22 ] investigated how real system and simulation data can be combined in training deep RL algorithms . They separate between the data types by using real data to learn about the dynamics of the system , and simulated data to learn a generalizing perception system . Our method mix real and simulation data by controlling the rate of streaming each data type into the learning agent . Replay Buffer analysis : Large portion of RL algorithms use replay buffers [ 30 , 33 ] but here we review only works that provide some analysis . Several works study the effect of replay buffer size on the agent performance [ 51 , 31 ] . Our focus is the effect of controlling the rate of collecting samples and the rate of using them in the optimization process . Fedus et al . [ 14 ] investigated the effect of the ratio between these rates on the learning process through simulated experiments , while our focus is on the theoretical aspects . Other works studied the criteria for prioritizing transitions to enhance learning [ 41 , 34 , 50 ] . In case of multiple agents that share their policy , Horgan et al . [ 16 ] argue in favor of a shared replay buffer for all agents and a prioritizing mechanism . We , on the other hand , emphasize the advantage of separating replay buffers when collecting samples from different environments to enable a mixing management in the learning process . Stochastic Approximation : Our proposed algorithm is based on the Stochastic Approximation method [ 27 ] . Konda and Tsitsiklis [ 25 ] proposed the actor-critic algorithm , and established the asymptotic convergence for the two time-scale actor-critic , with TD ( λ ) learning-based critic . Bhatnagar et al . [ 6 ] proved the convergence result for the original actor-critic and natural actor-critic methods . Di Castro and Meir [ 13 ] proposed a single time-scale actor-critic algorithm and proved its convergence . Recently , several finite sample analyses were applied by [ 48 , 52 , 12 ] and more but these works have not analyzed the replay buffer asymptotic behavior while we do . 3 Setup . We model the problem using a Markov Decision Process ( MDP ; [ 38 ] ) , where S and A are the state space and action space , respectively . We let P ( s′|s , a ) denote the probability of transitioning from state s ∈ S to state s′ ∈ S when applying action a ∈ A . The MDP measure P ( s′|s , a ) and the policy measure πθ ( a|s ) induce together a Markov Chain ( MC ) measure Pθ ( s′|s ) ( Pθ is matrix form ) . We consider a probabilistic policy πθ ( a|s ) , parameterized by θ ∈ Θ ⊂ Rd which expresses the probability of the agent to choose an action a given that it is in state s. We let µθ denote the stationary distribution induced by the policy πθ . The reward function is denoted by r ( s , a ) . Throughout the paper we assume the following . Assumption 1 . 1 . The set Θ is compact . 2 . The reward |r ( · , · ) | ≤ 1 for all s ∈ S , a ∈ A . Assumption 2 . For any policy πθ , the induced Markov chain of the MDP process { sτ } τ≥0 is irreducible and aperiodic . The goal of the agent is to find a policy that maximizes the average reward that the agent receives during its interaction with the environment [ 38 ] . Under an ergodicity assumption , the average reward over time eventually converges to the expected reward under the stationary distribution [ 3 ] : ηθ , lim T→∞ ∑T τ=0 r ( sτ , aτ ) T = Es∼µθ , a∼πθ [ r ( s , a ) ] . ( 1 ) The state-value function evaluates the overall expected accumulated rewards given a starting state s and a policy πθ V πθ ( s ) , E [ ∞∑ τ=0 ( r ( sτ , aτ ) − ηθ ) ∣∣∣∣∣ s0 = s , πθ ] , ( 2 ) where the actions follow the policy aτ ∼ πθ ( ·|sτ ) and the next state follows the transition probability sτ+1 ∼ P ( ·|sτ , aτ ) . Denote vθ to be the vector value function defined in ( 2 ) . Therefore , the vectorial Bellman Equation ( BE ) for a fixed policy πθ ( ·|· ) is vθ = r − ηθ + Pθvθ , where r is a vector of rewards for each state [ 38 ] . We recall that the solution to the BE is unique up to an additive constant . In order to have a unique solution , we choose a state s∗ to be of value 0 , i.e. , V πθ ( s∗ ) = 0 ( due to Assumption 2 , s∗ can be any of s ∈ S ) . In our specific setup , we consider a model where there are K MDPs , denoted by Mk , all share the same state space S , action space A , and reward function r ( s , a ) . The environment dynamics , though , are different , and are denoted by a transition function Pk ( ·|· , · ) . Together with a shared policy πθ ( ·|· ) , each Mk is induced by a state transition measure Pθ , k ( s′|s ) and a stationary distribution µθ , k . Let ηθ , k = Es∼µθ , k , a∼πθ [ r ( s , a ) ] and define the average reward over K environments , η̄θ = Ek∼β , s∼µθ , k , a∼πθ [ r ( s , a ) ] = K∑ k=1 βkηθ , k , ( 3 ) where β is a distribution which will be defined in Section 4 . The following assumption resembles Assumption 2 for K environments . Assumption 3 . For any policy πθ , the induced Markov chain of MDPMk is irreducible and aperiodic for all k = 1 . . .K . We define νk to be the throughput of Mk and it is defined as the number of samples MDP Mk provides for a unit time . In sim-to-real context , this setup can practically handle several robots and several simulation instances . We assume for the sim-to-real scenario that νs > νr . Since the samples from real arrive at a lower throughput than the sim , if we push the samples into two separate Replay Buffers ( RB ; [ 30 , 33 ] ) based on their sources , we can leverage the relatively scarce , but valuable samples that originated in the real system . This observation is the main motivation for our `` Mixing Sim and Real '' scheme , presented in the next section .

Summary: The paper presents a reply-buffer based algorithm to learn simultaneously from multiple environments. The target application is learning in a mixture of simulated and real environments but the theoretical analysis is conducted in the context of multiple environments. The proposed approach is sim-real-mix which maintains a reply buffer for each of the available environments; actor and critic updates are computed with a TD error which results from a random sampling from the available reply buffers according to a static categorial distribution.    The proposed sim-real-mix algorithm (theoretical analysis limited to linear actor-critic) is proved to converge to fixed points for both the actor and the critic. Additionally, bounds on the fixed points are given in case the case of two environments (i.e. sim and real) for which a distance between the transition probabilities is upper bounded.  Additionally, the authors propose some experimental evaluations of their sim-real-mix algorithm, testing it on two slightly different instances of the same simulated environments. The idea is to simulate the situation which occurs when learning from two environments (e.g. one sim environment and one real environment) with the aim of solving a task in one environment (i.e. the real environment) while taking advantage of data produced by both environments, assumed to share some similarities. The experimental evaluation aims at understanding how the proposed sim-real-mix algorithm behaves when varying the categorical distributions used for sampling (Figure 2 and 3). Additionally, authors compare the proposed strategy with a standard sim2real fine tuning (i.e. learn in sim and fine-tuning in real) and with a modified sim2real (i.e. learn and sim and fine-tune with the proposed sim-real-mix strategy). Interestingly it is observed that fine-tuning with sim-real-mix data is better than fine-tuning with real data only.