Sim and Real: Better Together
Authors: Shirli Di-Castro Shashua
Dotan Di Castro
Shie Mannor
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement Learning
Robotics
Replay buffer

Abstract: Simulation is used extensively in autonomous systems, particularly in robotic manipulation. By far, the most common approach is to train a controller in simulation, and then use it as an initial starting point for the real system. We demonstrate how to learn simultaneously from both simulation and interaction with the real environment. We propose an algorithm for balancing the large number of samples from the high throughput but less accurate simulation and the low-throughput, high-fidelity and costly samples from the real environment. We achieve that by maintaining a replay buffer for each environment the agent interacts with. We analyze such multi-environment interaction theoretically, and provide convergence properties, through a novel theoretical replay buffer analysis.  We demonstrate the efficacy of our method on a sim-to-real environment.

Sim and Real: Better Together
Shirli Di Castro Shashua Technion Institute Technology Haifa, Israel shirlidi@techmion
Shie Mannor
Dotan Di Castro
Technion and NVIDIA Research Israel shie@technion.
Bosch Center of AI Haifa, Israel
dotan. ietanddicasto@il. htt.:iii..osc..ccoooooo bosch.
smannor@nvidia. com
Abstract
Simulation is used extensively in autonomous systems, particularly in robotic ma- nipulation. By far, the most common approach is to train controller in simulation, and then use it as an initial starting point for the real system. We demonstrate how to learn simultaneously from both simulation and interaction with the real environment. We propose an algorithm for balancing the large number samples from the high throughput but less accurate simulation and the tttthrrugh high-fidelity and costly samples from the real environment. We achieve that by maintaining replay buffer for each environment the agent interacts with. We analyze such multi-environment interaction theoretically. and provide convergence properties, through a novel theoretical replay buffer analysis. We demonstrate the efficacy of our method on sim-to-real environment.
Introduction
Reinforcement learning (RL) is framework where an agent interacts with an unknown environment, receives a feedback from it, and optimizes its performance accordingly [44, 3]. There have been attempts of learning a control policy directly from real world samples [28, 49, 36, 21]. However, in many cases, learning from the actual environment may be slow, costly, or dangerous, while learning from a simulated system can be fast, cheap, and safe. The advantages of learning from simulation are counterbalanced by the reality-gap [18]: the loss of fidelity due to modeling limitations, parameter errors, and lack of variety in physical properties. The quality of the simulation may vary: when the simulation mimics the reality well, we can train the agent on the simulation and then transfer the policy the real environment, in one shot manner (c.g., [2]). However in many cases, simulation demonstrates low fidelity which leads to the following question: Can we mitigate the differences between real environments ("real") and simulations ("sim") thereof, from both, and performs well in the real one?
train agent that learns
In this work, we propose to learn simultaneously on real and sim, while controlling the rate in which we collect samples from each environment and controlling the rate in which we use these samples in the policy optinization. This synergy offers speed-fidelity trade-off and harnesses the advantage of each domain. Moreover, the simulation speed encourages exploration that helps to accelerate learning process. The real system in turn improve exploitation in the sense that it mitigates the challenges of sim-to-real policy transfer, and encourages the learner to converge to relevant solutions.
This research conducted during internship in Bosch Center of AI.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

8,7
M1
RB(1)
8,
21
M
92
Agent π
(s. 7
32
RB
OK
5K
8,
MR
RB(K)
Figure 1: Mixing environments scheme. The agent selects an environment M, with probability q. and interacts with it. Simultancously, the agent chooses RB()) with probability By and samples from this replay buffer stored transition O which is used for estimating the TD error and update the policy parameters.
A general scheme describing our proposed setup is depicted in Figure 1. In a nutshell, there is a single agent interacting with K environments (on the left). Each sample provided by an environment is pushed into corresponding replay buffer (RB). On the right, the agent pulls samples from the RBs and is trained on them. In the sim-to-real scheme, K 2.
In the specific scheme for mixing real and sim samples in the learning process, separate probability measures for collecting samples and for optimizing parameters policies are used. The off-policy nature of our scheme enables separation between real and sim samples which in turn helps controlling the rate of real samples used in the optimization process. In this work discuss two RL algorithms that can be used with this scheme: (1) off-policy linear actor critic with mixing sim and real samples and (2) Deep Deterministic Policy Gradient (DDPG; [29]) mixing scheme variant based neural networks. We analyze the asymptotic convergence of the linear algorithm and demonstrate the mixing samples variant of DDPG sim-to-real environment.
The naive approach in which one pushes the state- cate-action-rewarte settion-rwarrd-eex-ttttteee next-state tuples into a single shared replay buffer is prone to failures due to the imbalance between simulation and real roll-outs. To overcome this, we maintain separate replay buffers for each of the environments (e.g.. in the case of single robot and simulator would have two replay buffers). This allows us extract the maximum valuable information from reality by distinguishing its tuples from those generated by other environments, while continuously improving the agent using data from all input streams. Importantly, although the rate of samples is skewed in favor of the simulation, the learning may be carried out using a different rate. In sense, the mechanism we suggest is version of the importance sampling technique [10].
Our main contributions in this work are follows:
We present a method for incorporating real system samples and simulation samples a policy optimization process while distinguishing between the rate of collecting samples and the rate of using them.
2. We analyze the asymptotic convergence of our proposed mixing real and sim scheme.
3. To the best of our knowledge, we provide for the first time theoretical analysis of the dynamics and properties of replay buffer such as its Markovity and the explicit probability measure induces by the replay buffer.
4. We demonstrate our findings simulation of sim-to-real, with two simulations where one is distorted version of the other and analyze it empirically.

---

2 Related Work
Sim-to-Real is a long investigated topic in robotics where one aims to reduce the reality gap between the real system and its digital twin implementation. A general framework where we transfer results from one domain to another is domain adaptation. In vision, this approach have helped gain state-of-the-art results [15, 43, 32, 9, 23, 42]. In our work, we focus on the physical aspects of the sim-to-real gap. Related to domain adaptation, is the approach of domain randomization, where the randomization is done in simulation in order robustify and enhance the detection and object recognition capability [45, 40, 19, 47]. Recently, James et al. [20] proposed a method where both simulation and reality are adapted a common domain. Andrychowicz et al. [2] extensively randomize the task of reaching a cube pose where one-shot transfer is achieved but with large sample complexity. Randomization may also be applied dynamics, e.g., [35], where robustness to inaccuracy in real world parameters is achieved.
Sim-to-Real:
Another approach in Sim-to-Real is how to change the simulation in the light of real samples. In [1I] the agent learns mainly from simulation but its parameters are updated to match the behavior in reality by reducing the difference between simulation and reality roll-outs. Our method is direct approach that incorporates phenomena that is difficult to simulate accurately. In Bayesian context, [39] provide a principled framework to reason about the uncertainty in simulation parameters. Kang et al. [22] investigated how real system and simulation data can be combined in training deep RL algorithms. They separate between the data types by using real data to learn about the dynamics of the system, and simulated data to learn a generalizing perception system. Our method mix real and simulation data by controlling the rate of streaming each data type into the learning agent.
Replay Buffer analysis: Large portion of RL algorithms use replay buffers [30, 33] but here we review only works that provide some amalysis. Several works study the effect of replay buffer size on the agent performance [51, 31]. Our focus is the effect of controlling the rate of collecting samples and the rate of using them in the optimization process. Fedus et al. [14] investigated the effect of the ratio between these rates on the learning process through simulated experiments, while our focus is on the theoretical aspects. Other works studied the criteria for prioritizing transitions to enhance learning [41, 34, 50J. In case of multiple agents that share their policy, Horgan et al. [16] argue in favor of shared replay buffer for all agents and a prioritizing mechanism. We, on the other hand, emphasize the advantage of separating replay buffers when collecting samples from different environments to enable mixing management the learning process.
Our proposed algorithm is based on the Stochastic Approximation method [27]. Konda and Tsitsiklis [25] proposed the actor-critic algorithm, and established the asymptotic convergence for the two time-scale actor-critic, with TD(A) learning-based based critic. Bhat- nagar et al. [6] proved the convergence result for the original actor-critic and natural actor-critic methods. Di Castro and Meir [13] proposed a single time-scale actor-critic algorithm and proved its convergence. Recently, several finite sample analyses were applied by [48, 52, 12] and more but these works have not analyzed the replay buffer asymptotic behavior while we do.
Stochastic Approximation:
3
Setup
We model the problem using Markov Decision Process (MDP: [38]), where S and A are the state space and action space, respectively. We let P(s'|s, a) denote the probability of transitioning from state S to state s' E S when applying action a E A. The MDP measure P(s'ls,a) a) and the policy measure Ty (a]s) induce together a Markov Chain (MC) measure (s((s) (P& is matrix form). We consider a probabilistic policy To (a)s),, parameterized by 0 E e C Rd which expresses the probability of the agent to choose an action given that it is in state We let μi denote the stationary
distribution induced by the policy Te. The reward function is denoted by r(s, a). Throughout the paper we assume the following.
Assumption 1. 1. The Q is compact. 2. The reward (r((--) S 1 for all E S,a E A. Assumption 2. For any policy wa, the induced Markov chain of the MDP process {87)-20 is irreducible and aperiodic.
The goal of the agent is to find a policy that maximizes the average reward that the agent receives during its interaction with the environment [38]. Under ergodicity assumption, the average reward
3

---

over time eventually converges to the expected reward under the stationary distribution [3]:
Σr=o
r(s
7 lim T-000
Exppo [r(s, a)].
(1)
T
The state-value function evaluates the overall expected accumulated rewards given a starting state and policy
00
WT (8) 2 R
Σ (r(s,, 07) - 176) 80 8, T8
where the actions follow the policy 0z ~ (0(-117) and the next state follows the transition probability 8y+1 ~ P( a. Denote v be the vector value function defined in (2). Therefore, the vectorial Bellman Equation (BE) for fixed policy (T(-)-) is v
?e Pov", where is vector of rewards for each state [38]. We recall that the solution to the BE is unique up to an additive constant. (due to In order to have unique solution, we choose state s* to be of value i.e., V** Assumption 2, 8 can be any of E S).
In our specific setup, consider model where there are K MDPs, denoted by Mg. all share the same state space S, action space A, and reward function r(s, a). The environment dynamics, though, different, and are denoted by transition function Pk (-.-)..)). Together with shared policy (- each Mj is induced by a state transition measure Po. (s' [s) and a stationary distribution 10, Let 1 EBgupt, [r(s, a)] and define the average reward over K environments,
K
f = E B awwar (s, a)]
Σ Bame,ks k=1
(3)
where /3 is a distribution which will be defined in Section The following assumption resembles Assumption for K environments.
Assumption 3. For any policy no, the induced Markov chain of MDP My is irreducible and aperiodic for all k I. K.
We define Vi to be the throughput of Mg and it is defined as the number of samples MDP M provides for unit time. In sim-to-real context, this setup can practically handle several robots and several simulation instances. We assume for the sim-to-real scenario that v, > Vr-
Since the samples from real arrive at a lower throughput than the sim, if we push the samples into two separate Replay Buffers (RB; [30, 33]) based on their sources, we can leverage the relatively searce, but valuable samples that originated in the real system. This observation is the main motivation for our "Mixing Sim and Real" scheme, presented in the next section.
Mixing Sim and Real Algorithm
In order to reconcile the dynamics disparity, we propose our Mixing Sim and Real Algorithm with Linear Actor Critic, presented in Algorithm and described in Figure 1. We consider K environments, modeled as MDPs, M,.... MR. where the agent maintain a replay buffer (B(k) for each MDP, respectively. For the sake of analysis simplicity, we replace {F*} with the following random variable. The agent chooses an environment to communicate with according to Categorial .g..... .K) where q è 191.....9K: 9: 2 O, and Σ; 9:
The agent collects transitions (st, ag r., s from the chosen environment and stores them in the corresponding (i).). In order to approximate the rates { correctly, we choose q: = vs/ Σx Vs for the agent to interact according to the rates.
We train the agent in an off-policy manner. The agent selects RB(j) for sampling the next batch for training according to ~ (S),
where 3 2 [3 3x], B5 2 O, and Σ, B3 = This distribution remains static, and hence the selections in time are i. In addition, the 33 distribution that selects which samples to train over should be different than the q distribution that controls the throughput each environments pushes samples to the RB. In that way, scarce samples from the real environment can get higher influence on the training.
We note that one could remove this restriction and think of other schemes in which the replay buffer selection distribution changes over time based on some prescribed optimization goal, cost, etc.

---

Algorithm Mixing Sim and Real with Linear Actor Critic
Initialize Replay Buffers RB(k) with size N and initialize & for k I,... K.
2: Initialize actor parameters fu critic parameters Up and average reward estimator 77o-
3: for r O.... do
4: Sample ~ q. interact with Mj according to policy тe, and add the transition
{Si, Ri,t, Fi,ts $1,4+1] RB(i). Increment &, + +.
5: Sample j ~ 3 and choose Nuuth transitions from RB(j) denoted as (T) Naa.
6: 8(0²) fz 1/, + (((*=) Ut ((8*)Tur
7:
Update average reward
Th++1 7/ + a Nun
( Σ; f- Thr)
8: Update critic u++1 = 0, 0 Nonn E0 50)())6(18
1
9: Update actor 0,+1 = r(0 - a Nosth Σ. o0==vo log To (²)))))
10: end for
Once a RB selected, the sampled batch is used for optimizing the actor and the critic parameters. In this work, we propose two time scale linear actor critic optimization scheme [25], which is an RB-based version of [6] Algorithm. We analyze its convergence properties in Section 5. We note, however, that other optimization schemes be provided, such as DDPG [29], which we use in experiments.
We define tuple of indices (k, where & corresponds to (B(k) and corresponds to the n-th sample in this (k).... In addition, corresponds to time ((k; n) where this is the time when the agent interacted with the k-th MDP and the resample was added to RB(k). Let (T. & (5km ak.n, Tk.n. n,n be transition sampled at time from RB(k). Whenever it is clear from the context, we simply use Ô.
The temporal difference (TD) error 5(Ō) is a random quantity based on single sampled transition from (B),), 8(0) = r(ā, ā) 77 + d(3') v - (33 v, where V.T (s) = d(s) v is a lincar approximation for (s), d(s) E Rd is feature vector for state s and v Rd is a parameter vector. In Algorithm 1, average reward, critic and actor parameters are updated based on the TD error (see lines 7- 9). Note that for the actor updates, we use a projection I'(.) that projects any 0 E Rd to compact set O whenever 0 O.
In order to gain understanding of our proposed setup, in the next section we characterize the behaviour of the iterations in Algorithm 1.
5 Convergence Analysis for Mixing Sim and Real with Linear Approximation
The standard tool in the literature for analyzing iterations of processes such as two time scale Actor- Critic in the context of RL is SA; Stochastic Approximation [26, 7, 4]. This analysis technique includes two parts: proving the existence of fixed point, and bounding the rate of convergence to this fixed point. By far, the most popular methods for proving convergence is the Ordinary Differential Equation (ODE) method. Usually, the iteration should demonstrate either some monotonicity property, or a contraction feature in order for the iteration to converge.
Although in practice such algorithms (after some tuning) usually converge to objective value, it is not always guaranteed. To achieve that in a stochastic approximation setup, the main known result shows that the iteration can be decomposed into a deterministic function, which depends only on the problem parameters, and a martingale difference noise, which is bounded in some way.
In this section we show that the iterations of Algorithm converge to stable point of a corresponding ODE. We begin with showing that the process of sampling transitions from RBs is Markov process. Afterward, we show that if the original Markov chain is irreducible and aperiodic, then also the RBs Markov process irreducible and aperiodic. This property required for proving the convergence of the iterations in Algorithm using SA tools. We conclude this section with showing that if in some
5

---

sense sim is close to real, then the properties of the mixed process are close to the properties of both sim and real.
5.1 Asymptotic Convergence of Algorithm 1
Let RB(k) be a replay buffer storing the last N transitions from MDP Let RB, be the state of RB(k) at time T, i... RB, (k) _ {Or.1...., OK.N where Ok,n = fsk. ag,nr Tk.ms k is a transition tuple pushed at some time (kk, < r. We denote the collection of all RB, (k) as
K
U (k). We define I, and J, be i..d random processes based on / and J, respectively. We k=1
define Y, to be the process induced by Algorithm i.c.
K
Y, = U (R. 1==1
(4)
The next lemma states the Y, is Markovian. The proof is deferred to the Supplementary material A.1. Lemma 1 (Y, induced by Algorithm is Markovian). 1. The random process Y, is Markovian. 2. Under Assumption 3, there exists some T' > such that Y, is irreducible and aperiodic for 2 T'.
Next, we present several assumptions that are necessary for proving the convergence of Algorithm 1. The first assumption is standard requirement for policy gradient methods.
Assumption 4. For any state-action pair (s, a). Th (als) is contimuously differentiable in the param- eter 0.
Proving convergence for a general function approximation is hard. In our we demonstrate the convergence for a linear function approximation (LFA; [4]). In matrix form, it can be expressed as V = tu where E TR (s|xd The following assumption is needed for the uniqueness of the convergence point of the critic.
Assumption 5. The matrix has full rank. 2. The functions ((s) are Liphschitz s and bounded. 3. For every RRd, tu # e where a vector of ones.
In order to get a with probability I using the SA convergence, the following standard assumption is needed. Note that in the actor-critic setup need two time-scales convergence, thus, in this assumption the critic is a 'faster' recursion than the actor. Assumption 6.
The step-sizes {a²), {a²), {a t > 0 satisfy Σn a = =Z at Σr a =0. (, Σx (a)),, Σx (a < 30 and a o(a
We define the induced MC for the time n(k;,) with corresponding parameter (=(k.n)- For this parameter, we denote with Pi(k.)) the transition matrix at that time and the corresponding state distribution vector Pe(k,n) (both induced by the policy Te Finally, we define the following diagonal matrix Se(R,m) diag(Peck, and the reward vector (z(ke,) with elements (t(kk,)(s) = E. TBek.n) (als))(s,a)) a). Based on these definitions we define
K ΣΣ k=I n1
3k Suk.n) (Pak..)) - I), by
K ΣΣ k=In=1
A
S((k,n) (reck.) - 70,xe)
(5)
where I is the identity matrix and e is a vector of ones. The intuition behind A, and by is the following. For an online TD TD(OO)learning under stationary policy we have a fixed point at the solution to the equation & D(P I) - D(r - 7) = O ([4]; Lemma 6.5). our case, since we have K RBs where each one with N samples entered at different times, we have superposition of all these samples. When x., P((k.)) - m40 for all index n. We let So, e diag(uo.x) and define
K Σ SaS (Po.k - I), by Σ
Ag 2
SaSa, (re.k ne.ee).
For proving the convergence of the critic, we assume the policy fixed. Thus, for each RB the induced MC is one for all the samples in this RB, so the sum over N disappear for Ag and by. Now we are ready to prove the following theorems, regarding Algorithm 1. We note that Theorems 2 and state the critic and actor convergence.

---

Theorem 2. (Convergence of the Critic to fixed point) Under Assumptions 1-6, for any given and { as in the updates in Algorithm 1, we have 1/ + 110 and V- + v" with probability I, where v" is obtained as a unique solution to TT Ag + 40 bg =0. 0.
The proof for Theorem 2 follows the proof for Lemma 5 in [6], see more details in the supplementary material A.2. For establishing the convergence of the actor updates, we define additional terms. Let Z denote the set of asymptotically stable equilibria of the ODE 0 I' i'( eñja) and let Z be the e-neighborhood of 2. Let V (5) La TT(ā) (r(3, @) 70.k + Eg (x a))(") (T T=ae) and
define
K
_ Σ 8k Σ (0) ((3) Vov - VoV
k=
Theorem 3. (Convergence of the actor)
Under Assumptions 1-6, given e > 0, 36 > 0 such that for 0,, T = 0 obtained using Algorithm 1, if supo, - || < 6, then 0, + Z' as T + oc with probability one.
The proof for Theorem 3 follows the proof for Theorem 2 in [6] and is given in the supplementary material A.3.
5.2 Sim2Real Asymptotic Convergence Properties
In this section we analyze the convergence properties of the Mixing Sim and Real algorithm we use. The main idea is that if sim and real are close in their dynamics through the MDP transition matrix many properties of their MDPs under the same policy are close well. Moreover, we show that under the assumption of sim close to real, any process derived from both processes is close to both sim and real.
Assumption 7. (Closeness of sim and real). For all s, E S, a E A, we have (P,('(18,)) (s a)l 6,2r.
The following theorem states that Assumption 7 holds then the convergence points of sim, real, and the mixed process (as defined in Algorithm 1) converge to close points.
Theorem Consider a policy no(als) and Assumptions 2, and 7. Then, for each 8, s' E S, a E A,
and v0 E θ we have:
The induced MC of sim and real, MC, and MC,, satisfy |Pd (s'ls) (s'lss) S Bp & |A/Gar
Let De E [((()- where its elements are identical to the first (IS) - 1) x ( - 1)
elements of Po- The corresponding stationary distributions satisfy (n((s) (s)) s Bμ -
Sp minege VSR,, where Rm is the largest eigenvalue of the matrix P 3. The convergence points for the average reward and value functions under the policy for sim and real satisfy 1|v% 7 < B, A B [S] and V[v -f.ll S By.
The proof for Theorem 4 is the supplementary material B. Based on this Theorem, follows immediately that any convex combination of "close" enough sim and real share the same properties as both sim and real. We defer to supplcmentary material the precise statement.
Experimental Evaluation
In this section we evaluate the performance of our proposed algorithm two Fetch Push environments [37], one acts as the real environment and the other is the simulation environment 3, Although our theoretical results are on the proposed mixing scheme with linear function approximation, in this section we focus on non-linear methodologies, i.e., using neural networks. We set K meaning there is only one real and one simulation environments. We denote by gr the probability of collecting samples from the real environment and by B, the probability of choosing samples from the real environment for the optimization process. We are interested in demonstrating the effect of different q. and B, values on the learning process. In addition, we investigate different mixing strategies for combining real and sim samples:
'The code for the experiments is available at: Sirchudlealleetterogggehe.
https://github. http://ttiuuuucoodiiiiieitooocct wwt/sdiaastro/////occccccccccccc
7

---

(a)
Fet
(B)
Push Different a, values
90K
83K
79K
Succees rate- -eeal
0.7
Success rite- Reel
Real edfopdes
60K
0-0.1 9.50.5 Mixed
4 A.000, (Sim only)
50K
4-0.3, B-0.5, Mixed
0.3
0.
40k
Q-0.77 q.007799.003
a 9-0.5, 0.m05 Mixed
0.4
30K
9-0.9, A.0.5, Mixed
m-100, A.1100 (Heel only)
0.2
20K
0.2
16K
100
150
25K 50K 75K.
100K
255 50K
75K
100K
Epochs
Roal episodes
Sim episodes
Figure 2: "Real only", "Sim only", and "Mixed" strategies with fixed 8, and different q values. (a) Success rate in the real environment vs. number of epochs. Each epoch corresponds to 100 episodes, mixed with real and sim episodes. The success rate is computed every epoch over 10 test episodes. (b) Success rate in the real environment vs. number of real episodes (c) Number of real episodes vs. number of sim episodes. The size the markers corresponds to the increasing success rate. For all graphs, we repeated each experiment with 10 different random seeds and present the mean and standard deviation values.
1. "Mixed": real and sim episodes are collected according to Algorithm 1.
2. "Real only": The agent collects and optimize only real samples (i.e., qr and 3, = 1).
3. "Sim only": The agent collects and optimize only sim samples (i.e., Qr = and B, = 0).
4. "Sim first": At the beginning, the agent collects and optimizes only sim samples. When the success rate in the sim environment reaches 0.7, we switch to sampling and optimizing only using real samples.
5. "Sim- deiedeependntt: At the beginning, the agent collects and optimizes only sim samples. When the success rate in the sim environment reaches 0.7, we switch to the "Mixed" strategy.
In the Fetch Push task, a robot arm needs to push an object on a table to a certain goal point. The state is represented by the gripper, object and target position and pose, as well as their velocities and angular velocities*. The action specifics the desired gripper position at the next time-step. The agent gets reward of -1, if the desired goal was not yet achieved and 0 if it was achieved within some tolerance. To solve the task we used our mixing sim and real algorithm and replaced the linear actor-critic optimization scheme (lines 6-9 in Algorithm 1) with DDPG [29] together with Hindsight Experience Replay (HER; [1] optimization scheme. We created the real and sim environments using the Mujoco simulator [46]. The difference between the environments is the friction between the object and the table. We preceded the following experiments with an experiment to depict region of friction parameters where training the task using only sim samples and using the trained policy in the real environment does not solve the task (see supplementary material Section C.3).
We emphasize that we evaluate the performance in each experiment according to the success rate in the real environment, as this is the environment of final interest. In addition, we seek for mixing strategies that achieve the lowest number of real samples since usually they are costly and harder to get than sim samples.
Different 9, values: We fix optimization parameter 3, = 0.5 and test different collection parameter g, = 0. 0.1, 0.3, 0.5, 0.7, 0.9. Results are presented Figure 2. We notice that when the agent trained using "Sim only" strategy (q, 0), it fails to solve the task in real (Figure 2a). Next, when the agent is trained using "Real only" strategy (q- 1). the task solved. However, for achieving 0.9 success rate, "Real only" requires approximately 20K real episodes and to increase it to success rate of I, it requires approximately 40K real episodes (Figures 2b and 2c). Observing the qr values in- betweenee, we that g, 0.1 achieves the best performance it uses fewer (Ne 10K) real episodes to achieve high success rates compared to the "Real only" strategy. Notice that as gr increases the
'The final dimension 28 after removing non- inniimoommttve dimensions.

---

Fetch Push Different P. (e (5)
(a)
12K
0.9
11K
0.8
0.8
10K
0.7
Success rate: Real
0.7
0==0.1, 8,=0.111 Mixed
0.6
0.6
Real episodes
9-001. 8,0011, Mixed
Success
0.5
0.5
0-0.1. 8-0.5, Mixed
9-0011 8,-0.3, Mixed
0.4
0.4
q, Mixed
0.3
48
0--0,1 8,009, Mixed
0.3
0.2
0.1
0.1
50
100
150
0 2.5K SK 10K
23K 50K 75K 100K Sim episades
Epochs
Real episodes
Figure 3: The "Mixed" strategy with fixed or and different 3, values. (a), (b) and (c) descriptions the same as in Figure? 2. In (c), the size of the markers corresponds to the increasing success rate: 0.1, 0.3, 0.5, 0.7,0.9.
performance deteriorates. This phenomenon can be explained due to the mixed samples distribution. When q, is low, most of the data distribution is based sim, and real samples do not change much, but only "fine tune" the learning. When increases, the data distribution is composed of two different environments which may confuse the agent.
Different 8 values: In this experiment, we fix q. = 0.1 and test for 3, = 0.1, 0.15, 0.3, ..9.. Results are presented in Figure 3. When 3, is low and equals gr. the agent fails to solve the task (Figure 3a). However, when 3, is higher than qr. the performance improves where no significant differences are observed for 8, 0.3, 0.5, 0.7. For 8, 0.7, the algorithm achieves the best performance: high success rate of 0.9 while using fewer real episodes and fewer sim episodes compared to other B, values (Figures 3b and 3c). Interestingly, when B, is too high (with respect to 4r. i.e., 3, = 0.9) the performance deteriorates.
Different Mixing Strategies: We tested different mixing strategies. "Mixed", "Sim first" and "Sim- dependent" as described above. Results are presented in Figure 4. Using the "Sim-dependent" strategy reduced the required real and sim episodes to achieve 0.9 success rate comparing to the "Mixed" strategy with the same 9r and 3, values (Figure 4c). When using "Sim first" strategy, we observe that although in the beginning of the learning it uses only sim samples, once it switches to use only real samples, the agent requires many more real episodes to achieve success rate of 0.9 (compared to the "Mixed" and "Sim-dependent" strategies; Figures 4b and 4c). Although the most common approach is to train a policy in simulation and then use as an initial starting point for the real system, we see that applying the mixing strategy after transferring the policy to real can reduce further the required real episodes while maintaining high success rate.
7 Conclusions and Future Work
In this work we analyzed a mixing strategy between simulation and real system samples. By separating the rate of collecting samples from each environment and the rate of choosing samples for the optimization process, we were able to achieve a significant reduction in the amount of real environment samples, comparing the common strategy of using the same rate for both collection and optimization phases. This reduction is of special interest since usually the real samples are costly and harder achieve. We believe this work can lead a new line of research. First, finite sample analysis for our proposed algorithm can reveal its exact sample complexity. Comparing it to the sample complexity of learning only on real environment can emphasis the advantage of using the mixing strategy. Second, other replay buffer prioritization schemes can now be theoretically analyzed using the dynamics and properties of replay buffers we have developed. Third, our approach limited to the online case, where new samples are collected during training. Adapting our approach to the offline case can discover new venues in the offline RI. research. Fourth, learning the real samples collection rate and adapting it during training can further improve our approach.

---

Fetch Push Different Strategles
40C
0.6
0.2
30K
Success rate Rell
9 1, R.0.5. Mixed
8.005. Hixed
8.6
q.00 R.m..5,
0.5
20k
g-- 1, R.O.5, Simddgeenent
q-+0 2, R,-O., Sm-dappedent
0.4
q-- 3, R-O.5. Sm dependent
first
10k
0.2
100
150
10K
20X
10K
40K
25K SOK 70k
100K
Epochs
Real egasodes
5i epteodes
Figure 4: Comparing strategies: "Mixed", "Sim-dependent" and "Sim first". (a), (b) and (c) descrip- tions the same in Figure 2. can be clearly seen in (c) that "Sim first" requires the most number of real episodes to achieve a high success rate. In addition, (b) and (c) demonstrate that for the same (gr, (,) tuple, for example (g, 0.2, 0.5), "Sin-dependent" strategy achieves higher success rates with less number of real episodes, compared to the "Mixing" strategy.
8 Acknowledgements
Research was conducted under ISF grant number 2199/20.
References
[1] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in neural information processing systems, pages 5048-5058, 2017.
[2] OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob Mc- Grew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1)33-20, 2020.
[3] D. Bertsekas. Dynamic programming and optimal control. Athena scientific Belmont, MA, 2005.
[4] Dimitri P Bertsekas and John N Tsitsiklis. Neuro-dynamic programming. 1996.
Athena Scientific,
[5] Shalabh Bhatnagar and Shishir Kumar. simultaneous perturbation stochastic approximation- based actor-critic algorithm for markov decision processes. IEEE Transactions Automatic Control, 49(4):592-598, 2004.
[6] Shalabh Bhatnagar, Richard S Sutton, Mohammad Ghavamzadeh, and Mark Lee, Natural actor-critic algorithms. Automatica, 45(11)2241122482 2009.
[7] Vivek Borkar. Stochastic approximation: a dynamical systems viewpoint, volume 48. Springer, 2009.
[8] Vivek Borkar and Sean Meyn. The ode method for convergence of stochastic approximation and reinforcement learning. SIAM Journal on Control and Optimization, 38(2):447-46,, 2000.
[9] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In Adyances in neural information processing systems, pages 343-351, 2016.
[10] James Bucklew. Introduction to rure event simulation. Springer Science & Business Media, 2013.
10

---

[11] Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference Robotics and Automation (ICRA), pages 1973-8979. IEFE, 2019.
[12] Gal Dalal, Balázs Szörényi, Gugan Thoppe, and Shie Mannor. Finite sample analyses for td (0) with function approximation. In Proceedings of the AAAI Conference Artificial Intelligence, 2018.
[13] Dotan Di Castro and Ron Meir. A convergent online single time scale actor critic algorithm. The Journal of Machine Learning Research, 11:367-410, 2010.
[14] William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International Conference Machine Learning, pages 3061-3071. PMLR. 2020.
[15] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. aoveiriadversariall training of neural networks. In Domain Adaptation in Computer Vision Applications, pages 189-209. Springer, 2017.
[16] Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv:/803.0000 1803.00933, 2018.
[17] Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
[18] Nick Jakobi, Phil Husbands, and Inman Harvey. Noise and the reality gap: The use of simulation in evolutionary robotics. In European Conference Artificial Life, pages 704-720. Springer, 1995.
[19] Stephen James, Andrew Davison, and Edward Johns. Transferring end-to-end visuomotor control from simulation to real world for multi-stage task. arXiv preprint 17X7vv:707.02200 2017.
[20] Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to- sim: Data-efficient robotic grasping via randomized-to-canoen Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12627-12637, 2019.
adaptation networks. In
[21] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scal- able deep reinforcement learning for vision-based robotic manipulation. arXiv preprint 18Xiv: 10293, 2018
[22] Katie Kang, Suneel Belkhale, Gregory Kahn, Pieter Abbeel, and Sergey Levine. Generalization through simulation: Integrating simulated and real data into deep reinforcement learning for vision-based autonomous flight. In 2019 International Conference on Roboties and Automation (ICRA), pages
IEFE, 2019.
[23] Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. Proceedings of the 34th International Conference on Machine Learning- -Volume 70, pages 1857-1865. JMLR. org, 2017.
[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[25] Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances in neural information processing systems, pages 1008-1014. Citeseer, 2000.
[26) Harold Kushner and G George Yin. Stochastic approximation and recursive algorithms and applications, volume 35. Springer Science & Business Media, 2003.
11

---

[27] Harold Joseph Kushner and Dean S Clark. Stochastic approximation methods for constrained and unconstrained systems, volume 26. Springer Science & Business Media, 2012.
[28] Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research, 37(4-5):421-4351 2018.
[29] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
2015.
[30] Long-Ji Ji Lin. Reinforcement learning for robots using neural networks. Technical report, Carnegie- Univ Pittsburgh PA School of Computer Science, 1993.
[31] Ruishan Liu and James Zou. The effects of memory replay in reinforcement learning. In 2018 56th Annual Allerton Conference Communication, Control, and Computing (Allerton), pages 478-485. IEEE, 2018
[32] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning- Volume 37, pages 97-105. JMLR. org, 2015.
[33] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 2013.
[34] Yangchen Pan, Muhammad Zaheer, Adam White, Andrew Patterson, and Martha White. Orga- nizing experience: a deeper look at replay mechanisms for sample-based planning continuous state domains. arXiv preprint arXiv: 1806.000242 2018.
[35] Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 1-8. IEEE, 2018
[36] Lerrel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA). pages 3406-3411 IEEE, 2016.
[37] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint
2018.
[38] Martin L Puterman. Markov Decision Processes. Wiley and Sons, 1994.
[39] Fabio Ramos, Rafael Carvalhaes Possas, and Dieter Fox. Bayessim: adaptive domain random- ization via probabilistic inference for robotics simulators. arXiv preprint 1rXiv:1906.011 2019.
[40] Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without single real image. arXiv preprint arXiv: 161 .04201, 2016.
[41] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint
2015.
[42] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2107- 2116, 2017.
[43] Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. dirt-t approach to unsupervised domain adaptation. arXiv preprint arXiv: 1802.08735 2018.
[44] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
12

---

[45] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation the real world. In 2017 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages
23-30. IEFE, 2017.
[46] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
[47] Quan Vuong, Sharad Vikram, Hao Su, Sicun Gao, and Henrik I Christensen. How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? arXiv preprini arXiv: 1903. 1774, 2019.
[48] Yue Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite time analysis of two time-scale actor critic methods. arXiv preprint arXiv:2005.0 01 2020.
[49] Ali Yahya, Adrian Li, Mrinal Kalakrishnan, Yevgen Chebotar, and Sergey Levine. Collective robot reinforcement learning with distributed asynchronous guided policy search. In 2017 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 79-86.
IEEE, 2017.
[50] Daochen Zha, Kwei-Herng Lai, Kaixiong Zhou, and Xia Hu. Experience replay optimization. arXiv preprint arXiv: /tt///966088888 2019.
[51] Shangtong Zhang and Richard S Sutton. A deeper look at experience replay. arXiv preprint arXiv:1712.01275. 2017.
[52] Shaofeng Zou, Tengyu Xu, and Yingbin Liang. sample analysis for sarsa with linear function approximation. arXiv preprint arXiv:1902.0223:: 2019.
13


Reviews:
[{"review_id": "qwM2Jsh8kJq", "reviewer": "Reviewer_rbtG", "summary": "The paper addresses the challenge of sim-to-real transfer, i.e., how to use simulations for robot learning in the real world. It proposes the use of information from rollouts in both the simulated and the real world. The underlying technique is reminiscent of importance sampling. While the approach is explained for transfer between two environments (sim and real) it is applicable to any situation that involves K environments. At a basic level, the approach keeps track of K individual Replay Buffers which are used for RL. An agent chooses with some probability an environment and then with yet another probability samples a replay buffer to update the policy. In effect, the data from both the simulated and the real environment are used for training albeit with different mixing probabilities. The core contribution (in my opinion) of this paper is a theoretical proof that guarantees convergence of this mixing scheme. The proof shows that sampling transitions is a Markov process and that under some (reasonably benign) conditions it converges to a fix point. The approach is rigorous and the appendix includes all the individual steps of the proof. Two critical assumptions that are made in the proof is that the real and simulated transitions are close to each other (defined as difference of probabilities is below a threshold). In addition, the current proof assumes a linear approximation of the value function. The approach is evaluated on the Fetch Push environment in OpenAI Gym. In the experiments, however, a nonlinear function approximation is used which is not in line with the theoretical results. In addition, only simulation is used -- no real-robot experiments have been performed. To this end, the difference between the simulated and the \"real\" environment was the friction coefficient.       ", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Sim to real transfer is arguably the holy grail in ML for robotics and progress along this line of research would be of great interest to the NeurIPS community. The approach presented in the paper is not necessarily novel (e.g. combining real and simulated has been tried before) but the authors present a simple method that is built upon a rigorous foundation. The rationale is very much in line with importance sampling and actually could even be extended to more environments. This could be interesting in settings where there is a simulated environment, a simplified real-world setting, and a high-fidelity real-world setting (costly samples). The approach addresses the question of different sampling speeds (throughput) in these different domains. The strongest aspect of the paper is the careful derivation of a theoretical convergence proof for the proposed method. \n\nWhile I cannot guarantee correctness of the full proof, I check the individual derivation steps for soundness and was impressed by the detailed analysis in the appendix. The authors also do a good job of clearly communicating which assumptions are being made here. It would however have been interesting to discuss what ramifications these assumption have in practice for robot learning tasks (not just their theoretical background). The proof itself is non-trivial and very interesting. Some of the methodology could be used for other similar proofs going forward. Despite the theoretical rigor, the paper is easy to follow and well-written.\n\nUnfortunately, the paper fails short when it comes to the empirical evaluation. To begin with, there is no \"real\" environment in the experiments. The OpenAI Gym Push environment is used twice with different friction parameters. This doesn't do the task justice and just flatly needs to be rejected as a sim-to-real experiment. Real world environments include a variety of aspects that would have put the theoretical assumptions underlying the proof to test. I would have understood such an experiment, if the paper was differently framed. But the title is literally \"Sim and Real\" and up to the evaluation section the reader is left to believe that the theoretical propositions will be evaluated in the real world. The fact that only the friction between the object and the table. Hence, only a small difference along a single parameter exists between the simulated and the real world. No differences in actuation, sensing, noise levels, time deltas, wear-and-tear, etc. The authors explicitly talk about reduction in sampling requirements for robotic manipulation. Given the presented empirical results, I can't see how we can draw this conclusion at the moment.          \n\nTo summarize, the paper addresses an important issue and proposes an interesting theoretical framework that (if true) allow for improved performance when combining real-world and simulated data. However, the performed experiments do not allow for any sort of conclusion at the moment. Some assumptions are still not well explained. For example, the difference in transition functions between the real and simulated world needs to be bounded. What does that mean in practice? How small does this bound have to be in practice? Especially given the claims made in the beginning of the paper. I still salute the authors for the effort put into a rigorous theoretical analysis of the problem. My recommendation is to setup a set of real-world experiments that show both the strength and the limitations of the method. This could be a very strong paper, if these changes are made.  \n\nEdit after rebuttal: Thank you to the authors for addressing our concerns. However, I maintain my score. The paper in its current form is not ready for publication. The main challenge is that the topic of concern is not realistically addressed in the paper. Since the title says \"Sim and Real\", it is important to clearly specify and address the challenges inherent to sim2real problems in robotics. That, in my opinion, has not been achieved in this paper. An interesting theoretical formulation has been introduced, but there is no grounding of this fomalization that allows us to believe that this would solve (or even partially address) the sim2real problem. In the experiment, a mapping between two slightly differing variants of the same simulation is performed - definitely not sim to real. Many times in robotic sim2real tasks, multiple aspects (difference in geometry between sim and real, diff in parameters, diff in dynamics, invalid assumptions about rigidity of objects, etc.) compound in complex ways and render solutions, as the one proposed here, impossible. There is nothing unique in the proposed theoretical or practical implementation that suggests that a future extension of this framework would allow us to solve the true sim2real problem. My recommendation to the authors is really to ground this approach in a real-world experiment in which your assumptions can be tested compared the real world and in which the performance of your method can be reasonably evaluated.   ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the approach are not sufficiently discussed. These should be discussed not only at the theoretical level but also with regards to real-world robotics/intelligent systems implications. That hinges however on a realistic empirical evaluation that has not been provided so far. No discussion of the societal impact was provided.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "lULwB4bvG8", "reviewer": "Reviewer_r88y", "summary": "The paper presents a reply-buffer based algorithm to learn simultaneously from multiple environments. The target application is learning in a mixture of simulated and real environments but the theoretical analysis is conducted in the context of multiple environments. The proposed approach is sim-real-mix which maintains a reply buffer for each of the available environments; actor and critic updates are computed with a TD error which results from a random sampling from the available reply buffers according to a static categorial distribution.  \n\nThe proposed sim-real-mix algorithm (theoretical analysis limited to linear actor-critic) is proved to converge to fixed points for both the actor and the critic. Additionally, bounds on the fixed points are given in case the case of two environments (i.e. sim and real) for which a distance between the transition probabilities is upper bounded.\n\nAdditionally, the authors propose some experimental evaluations of their sim-real-mix algorithm, testing it on two slightly different instances of the same simulated environments. The idea is to simulate the situation which occurs when learning from two environments (e.g. one sim environment and one real environment) with the aim of solving a task in one environment (i.e. the real environment) while taking advantage of data produced by both environments, assumed to share some similarities. The experimental evaluation aims at understanding how the proposed sim-real-mix algorithm behaves when varying the categorical distributions used for sampling (Figure 2 and 3). Additionally, authors compare the proposed strategy with a standard sim2real fine tuning (i.e. learn in sim and fine-tuning in real) and with a modified sim2real (i.e. learn and sim and fine-tune with the proposed sim-real-mix strategy). Interestingly it is observed that fine-tuning with sim-real-mix data is better than fine-tuning with real data only. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The submitted paper is original, sound, clearly written and significant to the field. The relevance to field is mainly theoretical but I don't exclude that in the future this theoretical analysis can be useful for practical considerations. The major limitation of the submitted paper is that the theoretical results hold only for linear function approximation even though interesting applications would require the extension to non-linear neural networks. Another important limitation is that the main results on sim/real convergence (Section 5.2) rely on the assumption of being able to measure the sim-real closeness (as defined in Assumption 7) which however can be very impractical. The sim-real closeness is given as |P_sim(s'|s, a) - P_real(s'|s, a)| < epsilon which is impractical since it's in general difficult to measure the state of a real system. On real systems the state is often a function of on non-directly-observable quantities and therefore the sim-real closeness would be more practically characterized in terms of observations only, rather than relying on states.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No considerations needed.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "lKQKAxnILBo", "reviewer": "Reviewer_jQKF", "summary": "The paper combines simulation and the real world data by maintaining a replay buffer for both. The agent selects both the environment as well as the replay buffer with respective probabilities and uses that to estimate the TD error and update the policy parameters. They test this on a simple and well known `FetchPush` task and provide a theoretical convergence analysis. \n\nOn a high level, the paper is about how to mix and balance simulation and real world data to increase the probability of the transfer of policies from simulation to the real world. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Adding real world data to the learning process whenever makes a lot of sense. In reality, there is never a zero-shot transfer so every time you train a policy in simulation you have to verify if it transfers to the real world and that means running experiments on the real world. This also means you get real world data generated through the testing and use that to improve your model or simulation parameters. Even if the policy fails to solve the task in the real world right way, the collected data is still useful to providing a snapshot of real world environment. \n\nThe paper focuses on the theoretical analysis more and therefore uses a simplified task as well as representation of the policy i.e linear actor critic. Experimental analysis on various ways to combine real and simulation data chosen based on categorical distribution shows that design choices (q_r - sampling real world environment probability and beta_r - sampling real world replay buffer probability) matter.  \n\n- For a fixed beta_r, when q_r is low, most of the data comes from sim and role of real world is only to \"fine tune\" the learning. When q_r increases, the data distribution is composed of two different environments which may confuse the agent. \n- When beta_r is too high (with respect to qr, i.e., beta_r = 0.9) the performance degrades and that choosing beta_r > qr is preferable.\n- Applying the mixing strategy after transferring the policy trained entirely in sim, to real can reduce further the required real episodes while maintaining high success rate.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The work focusses on simplified scenarios e.g. the environment `FetchPush` is a relatively easier environment and already has very little reality gap due to fewer contacts and task of nudging the object along the table (no lifting needed). The policy consequently also seems simpler requiring linear actor critic. It would make sense to try out slightly harder tasks especially where contacts are made and broken and the task requires more than just nudging the object. It would not only help testing on more challenging problem but also makes collecting more real world data harder and slow because of the complexity of the problem. \n\nI think the choice of mixing real and sim data makes sense and has been of interest to many preceding works in this direction so overall this isn't entirely new. While they have provided a way to combine environment and replay buffer to show the trade-offs of each but I feel on a challenging tasks you'd have far less real world data and in the end need a strategy that is trained entirely on sim first and fine tuning a bit on the real world data. However, this also involves tuning the simulator so that fewer real world data is needed which in itself isn't as straightforward. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "VQySEVj2T9f", "reviewer": "Reviewer_7FTV", "summary": "This is a paper on sim-to-real transfer, proposing a new method for combining data from both the sim and real domains in order to obtain a better tradeoff between sample efficiency and quality of learning. This is a natural and good idea, although it appears novel in this proposed form. \n\nThe technical contribution is to devise a scheme to store data in multiple replay buffers, corresponding to the different sim and real sources, then using this data in a mixed linear actor critic scheme by suitably sampling the source and data points online. This paper attempts to present an analysis of the mixing scheme to argue convergence despite the difference in environments, as long as the sim and real domains are not too different in a certain sense. Finally, there is an empirical evaluation involving multiple simulations of a simple robotic pushing task.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a timely problem, which the authors of the paper handle competently. The key technical steps have to do with controlling the collection and sampling processes, and the consequent effect on approximations within the actor-critic scheme. The fact that the agent is actively switching between different environments and using different replay buffers to update the policy calls into question the convergence of such an algorithm. This is mitigated by noting that the environments are not too dissimilar and by tuning the ratios of sampling rates.\n\nI have a few different points to query, starting with the experimental framing. The version of this work that utilises the \"real\" environment has only one sim and one real environment, where the real environment is really a more noisy simulation of the sim environment. With this, the initial experiments show that a certain level of sampling (q_r and \\beta_r) are required for good performance. However, this seems delicate. I note the phenomenon wherein increasing q_r actually makes the learning deteriorate (unless it is a very low level indeed, at 0.1). The explanation given is that the mixed nature of the samples confuses the agent, but this really is the test and raises the question of whether a more informed schedule is needed for the mixing. This is acknowledged in the mixing section, where the conclusion is that introducing mixing after transferring to real is beneficial. I would have liked to have seen a lot more detail on these options and whether the proper procedure requires optimization of #5 by changing the interleaving of mixing, sim and real.\n\nOn a similar note, it seems like one issue here is that the sim environment might not be capturing the real diversity of phenomena in the real world. So, it would be good to understand the behaviour of such a scheme when there are multiple sim environments (i.e., K >> 2), representing significant diversity. This would address two types of questions - how do the relative numbers for q_r and \\beta_r impact performance in this more realistic setting, and how well does the procedure scale when the real environment is substantially different from sim but different sim environments capture different parts of this difference well.\n\nThe next point that concerns me is the assumption about the nearness of sim and real. This paper rests on the strong assumption that the only difference between the two is a benign form of noise, which is why assumption 7 is reasonable. However, in real applications - especially in robotics - there are other forms of differences. For instance, there is state-action dependent noise such as objects behaving significantly differently in low or high parameter regimes, or collisions and deformations being wrongly simulated, and so on. In these cases, there will not be a single \\epsilon_s2r that makes sense (or it would be too large to be useful). If this assumption were to be relaxed to be more realistic, I wonder how much this changes the claims of Theorem 4. We are not just asking what a larger \\epsilon means, which is clear. Instead, we are asking if assumption 7 were to be redefined in terms of different states being at different levels of fidelity, and we consider this in the convergence analysis, what is the effect on the training dynamics.\n\nThe fact that the sim2real experiment is really a sim2sim' makes it such that these issues have not arisen to a large extent in the experiments. The paper would be stronger if the experiments really did consider a *real* real environment (with realistic deviations between sim and real), so that one could get an empirical understanding of such issues. \n\nLastly, on a stylistic note, I found it a bit confusing that distributions and random variables are called by the same symbols in different places, e.g., \\beta in the algorithm description. Could this be revisited to clarify?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As my comments make clear, the paper could be strengthened in terms of consideration of limitations and the plausibility of assumptions. I say this while acknowledging that the core analysis is rigorous and appears sound - but the overall aim is to address what is ultimately a practical issue about the real world. A section on these issues would be good to include.\n\nThis is a methodological contribution to learning algorithms, which does not immediately impact societal issues. ", "review_text": "", "extra_scores": {"ethical_concerns": "-", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The subject of this paper is sim2real transfer, and it tackles this with two different parts. Firstly, it proposes an algorithm for sampling interactions from two different replay buffers, a simulated and a real one, the objective being to sample far less samples from the real environment, where interactions are costly. The algorithm is evaluated in an experimental part. Secondly, the paper contains a theoretical part with a congergence analysis of the provided algorithm.

This paper always was on the fence and was discussed by 4 reviewers, who essentially agreed on the strengths and weaknesses of the paper, but who disagreed on how to weight them and on the final decision.

All reviewers agreed that the algorithm itself was not sufficiently novel and did exist in some form in the literature. They also agreed on the weaknesses of the experimental part, in particular the simplicity of the chosen tasks and environments.

It then boiled down to whether the theoretical insights (convergence properties) are of interest to the community and whether they justify accepting. Here, reviewers were split 50/50.

The AC agreed that this paper is on the fence but judges that the theoretical derivations are of value and merit publishing in NeurIPS.
This paper was discussed between the AC and SAC.
Decision: Accept (Poster)