Sim and Real: Better Together
Year: 2021

Abstract:
Simulation is used extensively in autonomous systems, particularly in robotic manipulation. By far, the most common approach is to train a controller in simulation, and then use it as an initial starting point for the real system. We demonstrate how to learn simultaneously from both simulation and interaction with the real environment. We propose an algorithm for balancing the large number of samples from the high throughput but less accurate simulation and the low-throughput, high-fidelity and costly samples from the real environment. We achieve that by maintaining a replay buffer for each environment the agent interacts with. We analyze such multi-environment interaction theoretically, and provide convergence properties, through a novel theoretical replay buffer analysis. We demonstrate the efficacy of our method on a sim-to-real environment.

heading: 1 Introduction
text: Reinforcement learning (RL) is a framework where an agent interacts with an unknown environment, receives a feedback from it, and optimizes its performance accordingly [44, 3]. There have been attempts of learning a control policy directly from real world samples [28, 49, 36, 21]. However, in many cases, learning from the actual environment may be slow, costly, or dangerous, while learning from a simulated system can be fast, cheap, and safe. The advantages of learning from simulation are counterbalanced by the reality-gap [18]: the loss of fidelity due to modeling limitations, parameter errors, and lack of variety in physical properties. The quality of the simulation may vary: when the simulation mimics the reality well, we can train the agent on the simulation and then transfer the policy to the real environment, in a one shot manner (e.g., [2]). However in many cases, simulation demonstrates low fidelity which leads to the following question: Can we mitigate the differences between real environments ("real") and simulations ("sim") thereof, so as to train an agent that learns from both, and performs well in the real one?
In this work, we propose to learn simultaneously on real and sim, while controlling the rate in which we collect samples from each environment and controlling the rate in which we use these samples in the policy optimization. This synergy offers a speed-fidelity trade-off and harnesses the advantage of each domain. Moreover, the simulation speed encourages exploration that helps to accelerate the learning process. The real system in turn can improve exploitation in the sense that it mitigates the challenges of sim-to-real policy transfer, and encourages the learner to converge to relevant solutions.
∗This research was conducted during an internship in Bosch Center of AI.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
A general scheme describing our proposed setup is depicted in Figure 1. In a nutshell, there is a single agent interacting with K environments (on the left). Each sample provided by an environment is pushed into a corresponding replay buffer (RB). On the right, the agent pulls samples from the RBs and is trained on them. In the sim-to-real scheme, K = 2.
In the specific scheme for mixing real and sim samples in the learning process, separate probability measures for collecting samples and for optimizing parameters policies are used. The off-policy nature of our scheme enables separation between real and sim samples which in turn helps controlling the rate of real samples used in the optimization process. In this work we discuss two RL algorithms that can be used with this scheme: (1) off-policy linear actor critic with mixing sim and real samples and (2) Deep Deterministic Policy Gradient (DDPG; [29]) mixing scheme variant based on neural networks. We analyze the asymptotic convergence of the linear algorithm and demonstrate the mixing samples variant of DDPG in a sim-to-real environment.
The naive approach in which one pushes the state-action-reward-next-state tuples into a single shared replay buffer is prone to failures due to the imbalance between simulation and real roll-outs. To overcome this, we maintain separate replay buffers for each of the environments (e.g., in the case of a single robot and a simulator we would have two replay buffers). This allows us to extract the maximum valuable information from reality by distinguishing its tuples from those generated by other environments, while continuously improving the agent using data from all input streams. Importantly, although the rate of samples is skewed in favor of the simulation, the learning may be carried out using a different rate. In a sense, the mechanism we suggest is a version of the importance sampling technique [10].
Our main contributions in this work are as follows:
1. We present a method for incorporating real system samples and simulation samples in a policy optimization process while distinguishing between the rate of collecting samples and the rate of using them.
2. We analyze the asymptotic convergence of our proposed mixing real and sim scheme.
3. To the best of our knowledge, we provide for the first time theoretical analysis of the dynamics and properties of replay buffer such as its Markovity and the explicit probability measure induces by the replay buffer.
4. We demonstrate our findings in a simulation of sim-to-real, with two simulations where one is a distorted version of the other and analyze it empirically.

heading: 2 Related Work
text: Sim-to-Real: Sim-to-Real is a long investigated topic in robotics where one aims to reduce the reality gap between the real system and its digital twin implementation. A general framework where we transfer results from one domain to another is domain adaptation. In vision, this approach have helped to gain state-of-the-art results [15, 43, 32, 9, 23, 42]. In our work, we focus on the physical aspects of the sim-to-real gap. Related to domain adaptation, is the approach of domain randomization, where the randomization is done in simulation in order to robustify and enhance the detection and object recognition capability [45, 40, 19, 47]. Recently, James et al. [20] proposed a method where both simulation and reality are adapted to a common domain. Andrychowicz et al. [2] extensively randomize the task of reaching a cube pose where one-shot transfer is achieved but with large sample complexity. Randomization may also be applied to dynamics, e.g., [35], where robustness to inaccuracy in real world parameters is achieved.
Another approach in Sim-to-Real is how to change the simulation in the light of real samples. In [11] the agent learns mainly from simulation but its parameters are updated to match the behavior in reality by reducing the difference between simulation and reality roll-outs. Our method is a direct approach that incorporates phenomena that is difficult to simulate accurately. In Bayesian context, [39] provide a principled framework to reason about the uncertainty in simulation parameters. Kang et al. [22] investigated how real system and simulation data can be combined in training deep RL algorithms. They separate between the data types by using real data to learn about the dynamics of the system, and simulated data to learn a generalizing perception system. Our method mix real and simulation data by controlling the rate of streaming each data type into the learning agent.
Replay Buffer analysis: Large portion of RL algorithms use replay buffers [30, 33] but here we review only works that provide some analysis. Several works study the effect of replay buffer size on the agent performance [51, 31]. Our focus is the effect of controlling the rate of collecting samples and the rate of using them in the optimization process. Fedus et al. [14] investigated the effect of the ratio between these rates on the learning process through simulated experiments, while our focus is on the theoretical aspects. Other works studied the criteria for prioritizing transitions to enhance learning [41, 34, 50]. In case of multiple agents that share their policy, Horgan et al. [16] argue in favor of a shared replay buffer for all agents and a prioritizing mechanism. We, on the other hand, emphasize the advantage of separating replay buffers when collecting samples from different environments to enable a mixing management in the learning process.
Stochastic Approximation: Our proposed algorithm is based on the Stochastic Approximation method [27]. Konda and Tsitsiklis [25] proposed the actor-critic algorithm, and established the asymptotic convergence for the two time-scale actor-critic, with TD(λ) learning-based critic. Bhatnagar et al. [6] proved the convergence result for the original actor-critic and natural actor-critic methods. Di Castro and Meir [13] proposed a single time-scale actor-critic algorithm and proved its convergence. Recently, several finite sample analyses were applied by [48, 52, 12] and more but these works have not analyzed the replay buffer asymptotic behavior while we do.

heading: 3 Setup
text: We model the problem using a Markov Decision Process (MDP; [38]), where S and A are the state space and action space, respectively. We let P (s′|s, a) denote the probability of transitioning from state s ∈ S to state s′ ∈ S when applying action a ∈ A. The MDP measure P (s′|s, a) and the policy measure πθ(a|s) induce together a Markov Chain (MC) measure Pθ(s′|s) (Pθ is matrix form). We consider a probabilistic policy πθ(a|s), parameterized by θ ∈ Θ ⊂ Rd which expresses the probability of the agent to choose an action a given that it is in state s. We let µθ denote the stationary distribution induced by the policy πθ. The reward function is denoted by r(s, a). Throughout the paper we assume the following.
Assumption 1. 1. The set Θ is compact. 2. The reward |r(·, ·)| ≤ 1 for all s ∈ S, a ∈ A. Assumption 2. For any policy πθ, the induced Markov chain of the MDP process {sτ}τ≥0 is irreducible and aperiodic.
The goal of the agent is to find a policy that maximizes the average reward that the agent receives during its interaction with the environment [38]. Under an ergodicity assumption, the average reward
over time eventually converges to the expected reward under the stationary distribution [3]:
ηθ , lim T→∞
∑T τ=0 r(sτ , aτ )
T = Es∼µθ,a∼πθ [r(s, a)]. (1)
The state-value function evaluates the overall expected accumulated rewards given a starting state s and a policy πθ
V πθ (s) , E
[ ∞∑
τ=0
(r(sτ , aτ )− ηθ) ∣∣∣∣∣ s0 = s, πθ ] , (2)
where the actions follow the policy aτ ∼ πθ(·|sτ ) and the next state follows the transition probability sτ+1 ∼ P (·|sτ , aτ ). Denote vθ to be the vector value function defined in (2). Therefore, the vectorial Bellman Equation (BE) for a fixed policy πθ(·|·) is vθ = r − ηθ + Pθvθ, where r is a vector of rewards for each state [38]. We recall that the solution to the BE is unique up to an additive constant. In order to have a unique solution, we choose a state s∗ to be of value 0, i.e., V πθ (s∗) = 0 (due to Assumption 2, s∗ can be any of s ∈ S). In our specific setup, we consider a model where there are K MDPs, denoted by Mk, all share the same state space S , action space A, and reward function r(s, a). The environment dynamics, though, are different, and are denoted by a transition function Pk(·|·, ·). Together with a shared policy πθ(·|·), each Mk is induced by a state transition measure Pθ,k(s′|s) and a stationary distribution µθ,k. Let ηθ,k = Es∼µθ,k,a∼πθ [r(s, a)] and define the average reward over K environments,
η̄θ = Ek∼β,s∼µθ,k,a∼πθ [r(s, a)] = K∑
k=1
βkηθ,k, (3)
where β is a distribution which will be defined in Section 4. The following assumption resembles Assumption 2 for K environments. Assumption 3. For any policy πθ, the induced Markov chain of MDPMk is irreducible and aperiodic for all k = 1 . . .K.
We define νk to be the throughput of Mk and it is defined as the number of samples MDP Mk provides for a unit time. In sim-to-real context, this setup can practically handle several robots and several simulation instances. We assume for the sim-to-real scenario that νs > νr.
Since the samples from real arrive at a lower throughput than the sim, if we push the samples into two separate Replay Buffers (RB; [30, 33]) based on their sources, we can leverage the relatively scarce, but valuable samples that originated in the real system. This observation is the main motivation for our "Mixing Sim and Real" scheme, presented in the next section.

heading: 4 Mixing Sim and Real Algorithm
text: In order to reconcile the dynamics disparity, we propose our Mixing Sim and Real Algorithm with Linear Actor Critic, presented in Algorithm 1 and described in Figure 1. We considerK environments, modeled as MDPs, M1, . . . ,MK , where the agent maintain a replay buffer RB(k) for each MDP, respectively. For the sake of analysis simplicity, we replace {νk} with the following random variable. The agent chooses an environment to communicate with according to I ∼ Categorial(q1, . . . , qK) where q , [q1, . . . , qK ], qi ≥ 0, and ∑ i qi = 1. The agent collects transitions {si, ai, ri, s′i} from the chosen environment and stores them in the corresponding RB(i). In order to approximate the rates {νk}Kk=1 correctly, we choose qi = νi/ ∑ k νk for the agent to interact according to the rates.
We train the agent in an off-policy manner. The agent selects RB(j) for sampling the next batch for training according to J ∼ Categorial(β1, . . . , βK) where β , [β1, . . . , βK ], βj ≥ 0, and∑ j βj = 1. This distribution remains static, and hence the selections in time are i.i.d
2. In addition, the β distribution that selects which samples to train over should be different than the q distribution that controls the throughput each environments pushes samples to the RB. In that way, scarce samples from the real environment can get higher influence on the training.
2We note that one could remove this restriction and think of other schemes in which the replay buffer selection distribution changes over time based on some prescribed optimization goal, cost, etc.
Algorithm 1 Mixing Sim and Real with Linear Actor Critic 1: Initialize Replay Buffers RB(k) with size N and initialize tk = 0 for k = 1, . . .K. 2: Initialize actor parameters θ0, critic parameters v0 and average reward estimator η0. 3: for τ = 0, . . . do 4: Sample i ∼ q, interact with Mi according to policy πθτ and add the transition
{si,ti , ai,ti , ri,ti , si,ti+1} to RB(i). Increment ti ← ti + 1. 5: Sample j ∼ β and choose Nbatch transitions from RB(j) denoted as {Õzj,n(τ)}Nbatchz=1 . 6: δ(Õz) = r̃z − ητ + φ(s̃′z)>vτ − φ(s̃z)>vτ 7: Update average reward
ητ+1 = ητ + α η τ ( 1 Nbatch ∑ z r̃
z − ητ ) 8: Update critic vτ+1 = vτ + αvτ 1 Nbatch ∑ z δ(Õz)φ(s̃z)
9: Update actor θτ+1 = Γ ( θτ − αθτ 1Nbatch ∑ z δ(Õz)∇θ log πθ(ãz|s̃z) )
10: end for
Once a RB is selected, the sampled batch is used for optimizing the actor and the critic parameters. In this work, we propose a two time scale linear actor critic optimization scheme [25], which is an RB-based version of [6] Algorithm. We analyze its convergence properties in Section 5. We note, however, that other optimization schemes can be provided, such as DDPG [29], which we use in our experiments.
We define a tuple of indices (k, n) where k corresponds to RB(k) and n corresponds to the n-th sample in this RB(k). In addition, it corresponds to time t(k, n) where this is the time when the agent interacted with the k-th MDP and the n-sample was added to RB(k). Let Õk,n(τ) , {s̃k,n, ãk,n, r̃k,n, s̃′k,n} be a transition sampled at time τ from RB(k). Whenever it is clear from the context, we simply use Õ. The temporal difference (TD) error δ(Õ) is a random quantity based on a single sampled transition from RB(k), δ(Õ) = r(s̃, ã) − η + φ(s̃′)>v − φ(s̃)>v, where V̂ πθv (s) = φ(s)>v is a linear approximation for V πθ (s), φ(s) ∈ Rd is a feature vector for state s and v ∈ Rd is a parameter vector. In Algorithm 1, average reward, critic and actor parameters are updated based on the TD error (see lines 7 - 9). Note that for the actor updates, we use a projection Γ(·) that projects any θ ∈ Rd to a compact set Θ whenever θ /∈ Θ. In order to gain understanding of our proposed setup, in the next section we characterize the behaviour of the iterations in Algorithm 1.

heading: 5 Convergence Analysis for Mixing Sim and Real with Linear Approximation
text: The standard tool in the literature for analyzing iterations of processes such as two time scale ActorCritic in the context of RL is SA; Stochastic Approximation [26, 7, 4]. This analysis technique includes two parts: proving the existence of a fixed point, and bounding the rate of convergence to this fixed point. By far, the most popular methods for proving convergence is the Ordinary Differential Equation (ODE) method. Usually, the iteration should demonstrate either some monotonicity property, or a contraction feature in order for the iteration to converge.
Although in practice such algorithms (after some tuning) usually converge to an objective value, it is not always guaranteed. To achieve that in a stochastic approximation setup, the main known result shows that the iteration can be decomposed into a deterministic function, which depends only on the problem parameters, and a martingale difference noise, which is bounded in some way.
In this section we show that the iterations of Algorithm 1 converge to a stable point of a corresponding ODE. We begin with showing that the process of sampling transitions from RBs is a Markov process. Afterward, we show that if the original Markov chain is irreducible and aperiodic, then also the RBs Markov process is irreducible and aperiodic. This property is required for proving the convergence of the iterations in Algorithm 1 using SA tools. We conclude this section with showing that if in some
sense sim is close to real, then the properties of the mixed process are close to the properties of both sim and real.

heading: 5.1 Asymptotic Convergence of Algorithm 1
text: Let RB(k) be a replay buffer storing the last N transitions from MDP k. Let RBτ (k) be the state of RB(k) at time τ , i.e., RBτ (k) , {Ok,1, . . . ,Ok,N}, where Ok,n = {sk,n, ak,n, rk,n, s′k,n} is a transition tuple pushed at some time t(k, n) < τ . We denote the collection of all RBτ (k) as K⋃ k=1 RBτ (k). We define Iτ and Jτ be i.i.d random processes based on I and J , respectively. We define Yτ to be the process induced by Algorithm 1, i.e.,
Yτ =
[ K⋃
k=1
RBτ (k), Iτ , Jτ
] . (4)
The next lemma states the Yτ is Markovian. The proof is deferred to the Supplementary material A.1. Lemma 1 (Yτ induced by Algorithm 1 is Markovian). 1. The random process Yτ is a Markovian. 2. Under Assumption 3, there exists some τ ′ > 0 such that Yτ is irreducible and aperiodic for τ ≥ τ ′. Next, we present several assumptions that are necessary for proving the convergence of Algorithm 1. The first assumption is a standard requirement for policy gradient methods. Assumption 4. For any state–action pair (s, a), πθ(a|s) is continuously differentiable in the parameter θ.
Proving convergence for a general function approximation is hard. In our case we demonstrate the convergence for a linear function approximation (LFA; [4]). In matrix form, it can be expressed as V = Φv where Φ ∈ R|S|×d. The following assumption is needed for the uniqueness of the convergence point of the critic. Assumption 5. 1. The matrix Φ has full rank. 2. The functions φ(s) are Liphschitz in s and bounded. 3. For every v ∈ Rd, Φv 6= e where e is a vector of ones. In order to get a with probability 1 using the SA convergence, the following standard assumption is needed. Note that in the actor-critic setup we need two time-scales convergence, thus, in this assumption the critic is a ‘faster’ recursion than the actor. Assumption 6. The step-sizes {αητ}, {αvτ}, {αθτ}, τ ≥ 0 satisfy ∑∞ τ α η τ = ∑∞ τ α v τ = ∑∞ τ α
θ τ =∞,∑∞
τ (α η τ ) 2, ∑∞ τ (α v τ ) 2, ∑∞ τ (α θ τ )
2 <∞ and αθτ = o(αvτ ). We define the induced MC for the time t(k, n) with a corresponding parameter θt(k,n). For this parameter, we denote with Pt(k,n) the transition matrix at that time and the corresponding state distribution vector ρt(k,n) (both induced by the policy πθt(k,n)). Finally, we define the following diagonal matrix St(k,n) , diag(ρt(k,n)) and the reward vector rt(k,n) with elements rt(k,n)(s) =∑ a πθt(k,n)(a|s)r(s, a). Based on these definitions we define
Aτ , K∑
k=1
N∑
n=1
βk N St(k,n)
( Pt(k,n) − I ) , bτ , K∑
k=1
N∑
n=1
βk N St(k,n)
( rt(k,n) − ηθ,ke ) . (5)
where I is the identity matrix and e is a vector of ones. The intuition behind Aτ and bτ is the following. For an online TD(0)-learning under a stationary policy we have a fixed point at the solution to the equation Φ>D(P − I)Φv + Φ>D(r − η) = 0 ([4]; Lemma 6.5). In our case, since we have K RBs where each one with N samples entered at different times, we have a superposition of all these samples. When τ →∞, ρt(k,n) → µθ,k for all index n. We let Sθ,k , diag(µθ,k) and define
Aθ , K∑
k=1
βkSθ,k (Pθ,k − I) , bθ , K∑
k=1
βkSθ,k (rθ,k − ηθ,ke) . (6)
For proving the convergence of the critic, we assume the policy is fixed. Thus, for each RB the induced MC is one for all the samples in this RB, so the sum over N disappear for Aθ and bθ. Now we are ready to prove the following theorems, regarding Algorithm 1. We note that Theorems 2 and 3 state the critic and actor convergence.
Theorem 2. (Convergence of the Critic to a fixed point) Under Assumptions 1-6, for any given π and {ητ}, {vτ} as in the updates in Algorithm 1, we have ητ → η̄θ and vτ → vπ with probability 1, where vπ is obtained as a unique solution to Φ>AθΦv + Φ >bθ = 0.
The proof for Theorem 2 follows the proof for Lemma 5 in [6], see more details in the supplementary material A.2. For establishing the convergence of the actor updates, we define additional terms. Let Z denote the set of asymptotically stable equilibria of the ODE θ̇ = Γ̂(−∇θη̄θ) and let Z be the -neighborhood of Z . Let V̄ πθk (s̃) = ∑ ã πθ(ã|s̃) ( r(s̃, ã)− ηθ,k + ∑ s̃′ Pk(s̃ ′|s̃, ã)φ(s̃′)>vπθ ) , and define
ξπθ =
K∑
k=1
βk ∑
s̃
µθ,k(s̃) ( φ(s̃)>∇θvπθ −∇θV̄ πθk (s̃) ) .
Theorem 3. (Convergence of the actor) Under Assumptions 1-6, given > 0, ∃δ > 0 such that for θτ , τ ≥ 0 obtained using Algorithm 1, if supθτ ‖ξπθτ ‖ < δ, then θτ → Z as τ →∞ with probability one. The proof for Theorem 3 follows the proof for Theorem 2 in [6] and is given in the supplementary material A.3.

heading: 5.2 Sim2Real Asymptotic Convergence Properties
text: In this section we analyze the convergence properties of the Mixing Sim and Real algorithm we use. The main idea is that if sim and real are close in their dynamics through the MDP transition matrix many properties of their MDPs under the same policy are close as well. Moreover, we show that under the assumption of sim close to real, any process derived from both processes is close to both sim and real. Assumption 7. (Closeness of sim and real). For all s, s′ ∈ S, a ∈ A, we have |Ps(s′|s, a) − Pr(s
′|s, a)| ≤ s2r. The following theorem states that if Assumption 7 holds then the convergence points of sim, real, and the mixed process (as defined in Algorithm 1) converge to close points. Theorem 4. Consider a policy πθ(a|s) and Assumptions 1, 2, and 7. Then, for each s, s′ ∈ S, a ∈ A, and ∀θ ∈ Θ we have: 1. The induced MC of sim and real, MCs and MCr, satisfy |P θs (s′|s)− P θr (s′|s) ≤ BP , |A| s2r. 2. Let P̃ θs ∈ R(|S|−1)×(|S|−1) where its elements are identical to the first (|S| − 1) × (|S| − 1) elements of P θs . The corresponding stationary distributions satisfy |µθs(s) − µθr(s)| ≤ Bµ , BP |S|3 minθ∈Θ √ SR2m, where Rm is the largest eigenvalue of the matrix P̃ θ s . 3. The convergence points for the average reward and value functions under the policy for sim and real satisfy ‖ηθs − ηθr‖ ≤ Bη , Bµ|S| and ‖vθs − vθr‖ ≤ Bµ. The proof for Theorem 4 is in the supplementary material B. Based on this Theorem, it follows immediately that any convex combination of "close" enough sim and real share the same properties as both sim and real. We defer to supplementary material the precise statement.

heading: 6 Experimental Evaluation
text: In this section we evaluate the performance of our proposed algorithm on two Fetch Push environments [37], one acts as the real environment and the other is the simulation environment 3. Although our theoretical results are on the proposed mixing scheme with linear function approximation, in this section we focus on non-linear methodologies, i.e., using neural networks. We set K = 2 meaning there is only one real and one simulation environments. We denote by qr the probability of collecting samples from the real environment and by βr the probability of choosing samples from the real environment for the optimization process. We are interested in demonstrating the effect of different qr and βr values on the learning process. In addition, we investigate different mixing strategies for combining real and sim samples:
3The code for the experiments is available at: https://github.com/sdicastro/ SimAndRealBetterTogether.
1. "Mixed": real and sim episodes are collected according to Algorithm 1.
2. "Real only": The agent collects and optimize only real samples (i.e., qr = 1 and βr = 1).
3. "Sim only": The agent collects and optimize only sim samples (i.e., qr = 0 and βr = 0).
4. "Sim first": At the beginning, the agent collects and optimizes only sim samples. When the success rate in the sim environment reaches 0.7, we switch to sampling and optimizing only using real samples.
5. "Sim-dependent": At the beginning, the agent collects and optimizes only sim samples. When the success rate in the sim environment reaches 0.7, we switch to the "Mixed" strategy.
In the Fetch Push task, a robot arm needs to push an object on a table to a certain goal point. The state is represented by the gripper, object and target position and pose, as well as their velocities and angular velocities4. The action specifies the desired gripper position at the next time-step. The agent gets a reward of -1, if the desired goal was not yet achieved and 0 if it was achieved within some tolerance. To solve the task we used our mixing sim and real algorithm and replaced the linear actor-critic optimization scheme (lines 6-9 in Algorithm 1) with DDPG [29] together with Hindsight Experience Replay (HER; [1]) optimization scheme. We created the real and sim environments using the Mujoco simulator [46]. The difference between the environments is the friction between the object and the table. We preceded the following experiments with an experiment to depict a region of friction parameters where training the task using only sim samples and using the trained policy in the real environment does not solve the task (see supplementary material Section C.3).
We emphasize that we evaluate the performance in each experiment according to the success rate in the real environment, as this is the environment of final interest. In addition, we seek for mixing strategies that achieve the lowest number of real samples since usually they are costly and harder to get than sim samples.
Different qr values: We fix optimization parameter βr = 0.5 and test different collection parameter qr = 0, 0.1, 0.3, 0.5, 0.7, 0.9, 1. Results are presented in Figure 2. We notice that when the agent is trained using "Sim only" strategy (qr = 0), it fails to solve the task in real (Figure 2a). Next, when the agent is trained using "Real only" strategy (qr = 1), the task is solved. However, for achieving 0.9 success rate, "Real only" requires approximately 20K real episodes and to increase it to success rate of 1, it requires approximately 40K real episodes (Figures 2b and 2c). Observing the qr values in-between, we see that qr = 0.1 achieves the best performance – it uses fewer (≈ 10K) real episodes to achieve high success rates compared to the "Real only" strategy. Notice that as qr increases the
4The final dimension is 28 after removing non-informative dimensions.
performance deteriorates. This phenomenon can be explained due to the mixed samples distribution. When qr is low, most of the data distribution is based on sim, and real samples do not change it much, but only "fine tune" the learning. When qr increases, the data distribution is composed of two different environments which may confuse the agent.
Different βr values: In this experiment, we fix qr = 0.1 and test for βr = 0.1, 0.15, 0.3, . . . 0.9. Results are presented in Figure 3. When βr is low and equals qr, the agent fails to solve the task (Figure 3a). However, when βr is higher than qr, the performance improves where no significant differences are observed for βr = 0.3, 0.5, 0.7. For βr = 0.7, the algorithm achieves the best performance: high success rate of 0.9 while using fewer real episodes and fewer sim episodes compared to other βr values (Figures 3b and 3c). Interestingly, when βr is too high (with respect to qr, i.e., βr = 0.9) the performance deteriorates.
Different Mixing Strategies: We tested different mixing strategies. "Mixed", "Sim first" and "Simdependent" as described above. Results are presented in Figure 4. Using the "Sim-dependent" strategy reduced the required real and sim episodes to achieve 0.9 success rate comparing to the "Mixed" strategy with the same qr and βr values (Figure 4c). When using "Sim first" strategy, we observe that although in the beginning of the learning it uses only sim samples, once it switches to use only real samples, the agent requires many more real episodes to achieve success rate of 0.9 (compared to the "Mixed" and "Sim-dependent" strategies; Figures 4b and 4c). Although the most common approach is to train a policy in simulation and then use it as an initial starting point for the real system, we see that applying the mixing strategy after transferring the policy to real can reduce further the required real episodes while maintaining high success rate.

heading: 7 Conclusions and Future Work
text: In this work we analyzed a mixing strategy between simulation and real system samples. By separating the rate of collecting samples from each environment and the rate of choosing samples for the optimization process, we were able to achieve a significant reduction in the amount of real environment samples, comparing to the common strategy of using the same rate for both collection and optimization phases. This reduction is of special interest since usually the real samples are costly and harder to achieve. We believe this work can lead to a new line of research. First, finite sample analysis for our proposed algorithm can reveal its exact sample complexity. Comparing it to the sample complexity of learning only on real environment can emphasis the advantage of using the mixing strategy. Second, other replay buffer prioritization schemes can now be theoretically analyzed using the dynamics and properties of replay buffers we have developed. Third, our approach is limited to the online case, where new samples are collected during training. Adapting our approach to the offline case can discover new venues in the offline RL research. Fourth, learning the real samples collection rate and adapting it during training can further improve our approach.

heading: 8 Acknowledgements
text: Research was conducted under ISF grant number 2199/20.

Review summaries:
The paper addresses the challenge of sim-to-real transfer, i.e., how to use simulations for robot learning in the real world. It proposes the use of information from rollouts in both the simulated and the real world. The underlying technique is reminiscent of importance sampling. While the approach is explained for transfer between two environments (sim and real) it is applicable to any situation that involves K environments. At a basic level, the approach keeps track of K individual Replay Buffers which are used for RL. An agent chooses with some probability an environment and then with yet another probability samples a replay buffer to update the policy. In effect, the data from both the simulated and the real environment are used for training albeit with different mixing probabilities. The core contribution (in my opinion) of this paper is a theoretical proof that guarantees convergence of this mixing scheme. The proof shows that sampling transitions is a Markov process and that under some (reasonably benign) conditions it converges to a fix point. The approach is rigorous and the appendix includes all the individual steps of the proof. Two critical assumptions that are made in the proof is that the real and simulated transitions are close to each other (defined as difference of probabilities is below a threshold). In addition, the current proof assumes a linear approximation of the value function. The approach is evaluated on the Fetch Push environment in OpenAI Gym. In the experiments, however, a nonlinear function approximation is used which is not in line with the theoretical results. In addition, only simulation is used -- no real-robot experiments have been performed. To this end, the difference between the simulated and the "real" environment was the friction coefficient.       

The paper presents a reply-buffer based algorithm to learn simultaneously from multiple environments. The target application is learning in a mixture of simulated and real environments but the theoretical analysis is conducted in the context of multiple environments. The proposed approach is sim-real-mix which maintains a reply buffer for each of the available environments; actor and critic updates are computed with a TD error which results from a random sampling from the available reply buffers according to a static categorial distribution.    The proposed sim-real-mix algorithm (theoretical analysis limited to linear actor-critic) is proved to converge to fixed points for both the actor and the critic. Additionally, bounds on the fixed points are given in case the case of two environments (i.e. sim and real) for which a distance between the transition probabilities is upper bounded.  Additionally, the authors propose some experimental evaluations of their sim-real-mix algorithm, testing it on two slightly different instances of the same simulated environments. The idea is to simulate the situation which occurs when learning from two environments (e.g. one sim environment and one real environment) with the aim of solving a task in one environment (i.e. the real environment) while taking advantage of data produced by both environments, assumed to share some similarities. The experimental evaluation aims at understanding how the proposed sim-real-mix algorithm behaves when varying the categorical distributions used for sampling (Figure 2 and 3). Additionally, authors compare the proposed strategy with a standard sim2real fine tuning (i.e. learn in sim and fine-tuning in real) and with a modified sim2real (i.e. learn and sim and fine-tune with the proposed sim-real-mix strategy). Interestingly it is observed that fine-tuning with sim-real-mix data is better than fine-tuning with real data only. 

This is a paper on sim-to-real transfer, proposing a new method for combining data from both the sim and real domains in order to obtain a better tradeoff between sample efficiency and quality of learning. This is a natural and good idea, although it appears novel in this proposed form.   The technical contribution is to devise a scheme to store data in multiple replay buffers, corresponding to the different sim and real sources, then using this data in a mixed linear actor critic scheme by suitably sampling the source and data points online. This paper attempts to present an analysis of the mixing scheme to argue convergence despite the difference in environments, as long as the sim and real domains are not too different in a certain sense. Finally, there is an empirical evaluation involving multiple simulations of a simple robotic pushing task.  

The paper combines simulation and the real world data by maintaining a replay buffer for both. The agent selects both the environment as well as the replay buffer with respective probabilities and uses that to estimate the TD error and update the policy parameters. They test this on a simple and well known `FetchPush` task and provide a theoretical convergence analysis.   On a high level, the paper is about how to mix and balance simulation and real world data to increase the probability of the transfer of policies from simulation to the real world.