PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
Authors: Anish Agarwal
Abdullah Omar Alomar
Varkey Alumootil
Devavrat Shah
Dennis Shen
Zhi Xu
Cindy Yang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep Reinforcement Learning
Offline Reinforcement Learning
Model-based Reinforcement Learning

Abstract: We consider offline reinforcement learning (RL) with heterogeneous agents under severe data scarcity, i.e., we only observe a single historical trajectory for every agent under an unknown, potentially sub-optimal policy. We find that the performance of state-of-the-art offline and model-based RL methods degrade significantly given such limited data availability, even for commonly perceived "solved" benchmark settings such as "MountainCar" and "CartPole". To address this challenge, we propose PerSim, a model-based offline RL approach which first learns a personalized simulator for each agent by collectively using the historical trajectories across all agents, prior to learning a policy. We do so by positing that the transition dynamics across agents can be represented as a latent function of latent factors associated with agents, states, and actions; subsequently, we theoretically establish that this function is well-approximated by a "low-rank" decomposition of separable agent, state, and action latent functions. This representation suggests a simple, regularized neural network architecture to effectively learn the transition dynamics per agent, even with scarce, offline data. We perform extensive experiments across several benchmark environments and RL methods. The consistent improvement of our approach, measured in terms of both state dynamics prediction and eventual reward, confirms the efficacy of our framework in leveraging limited historical data to simultaneously learn personalized policies across agents. 

PerSim: Data-efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
Anish Agarwal MIT anish90smit..m edu
Abdullah Alomar MIT ealomar@mit.eem edu
Varkey Alumootil MIT varkey@mit. edu
Devavrat Shah MIT devavratemit.ed edu
Dennis Shen
Zhi Xu
Cindy Yang MIT cxy990mit. edu
MIT deshen@mit.edi
MIT
zhixu@mit.edi edu
Abstract
We consider offline reinforcement learning (RL) with heterogeneous agents under severe data scarcity, i... we only observe a single historical trajectory for for every agent under unknown, potentially sub-optimal policy. We find that the performance of state-of-the-an offline and model-based RL methods degrade significantly given such limited data availability, even for commonly perceived "solved" benchmark settings such as "MountainCar" and "CartPole". 'To address this challenge, we propose PerSim, a model-based offline RL approach which first learns personalized simulator for each agent by collectively using the historical trajectories across all agents, prior to learning policy. We do by positing that the transition dynamics across agents can be represented as latent function of latent factors associated with agents, states, and actions; subsequently. we theoretically establish that this function well-apper cell-approximated by flowrrak"" decomposition of separable agent, state, and action latent functions. This representation suggests simple, regularized neural network architecture to effectively learn the transition dynamics per agent, even with scarce, offline data. We perform extensive experiments across several benchmark environments and RI methods. The consistent improvement of our approach, measured in terms of both state dynamics prediction and eventual reward, confirms the efficacy of our framework in leveraging limited historical data simultancously learn personalized policies across agents.
Introduction
Reinforcement learning (RL) coupled with expressive deep neural networks has now become generic yet powerful solution for learning complex decision-making policies for agent of interest; it provides the key algorithmic foundation underpinning recent successes such as game solving 34
and robotics [30 (22). However, many satte-of-thh-all RL methods are data hungry and require the ability query samples will, which is infeasible for numerous settings such healthcare, autonomous driving, and socio-cconomic systems. As a result, there has been rapidly growing literature on "offline which focuses on leveraging existing datasets to learn decision-maie policies.
RI." 31 24
32
15
Within offline RL, consider regime of severe data scarcity: there multiple agents and for each agent, we only observe a single historical trajectory generated under an unknown, potentially sub-optimal policy; further, the agents are heterogeneous, i.e., each agent has unique state transition dy- namics. Importantly, the characteristics of the agents that make their transition dynamics heterogencous are latent. Using such limited offline data simultaneously learn good "personalized" policy for each agent is challenging setting that has received limited attention. Below, we use prevalent example
35th Conference Neural Information Processing Systems (NeurIPS 2021).

---

granity
Pushi
-
(a) MountainCar
(b) CartPole
(c) HalfCheetah
Figure 1: t-SNE visualization of the learned latent factors for the 500 heterogeneous agents, Colors indicate the value of the modified parameters in each environment (e-g., gravity in MountainCar). There is an informative low-dimensional manifold induced by the agent-specific latent factors, and there is natural direction the manifold along which the parameters that characterize the heterogeneity vary continuously and smoothly.
from healthcare to motivate and argue that tackling such challenge is an important and necessary step towards building personalized decision maainn engines via RL in a variety important applications.
A Motivating Example. Consider a pre-existing clinical dataset of patients (agents). Our goal to design a personalized treatment plan (policy) for each patient moving forward. Notable challenges include the following: First, each patient only provides single trajectory of their medical history. Second, each patient is heterogeneous in that they may have varied response for a given treatment under similar medical conditions; further, the underlying reason for this heterogencous response across patients is likely unknown and thus not recorded in the dataset. Third, in the absence of an accurate personalized "forecasting" model for patient's 's medical outcome given treatment plan, the treatment assigned is likely to be sub-optimal. This is particularly true for complicated medical conditions like T-Cell Lymphoma. We aim address the challenges laid out above (offline scarce data, heterogeneity, and sub- optimal policies) so as to develop a personalized "forecasting" model for each patient under any given treatment plan. Doing so will then naturally enable ideal personalized treatment policies for each patient.
Key question. Tackling a scenario like the one described above in principled manner is the focus of this work. In particular, we seek answer the following question:
"Can we leverage scarce, offline data collected across heterogeneous agents under unknown, sub-op sub-optimal policies to learn personalized policy for each agent?"
Our Contributions. As our main contribution, we answer this question in the affirmative by developing a structured framework to tackle this challenging yet meaningful setting. Next, we summarize the main methodological, theoretical, algorithmic, and experimental contributions in our proposed framework.
Methodological personalized simulators. We propose a novel methodological framework, PerSim, to learn a policy given the data availability described above. Taking inspiration from the model-based RL literature, our approach is to first build personalized simulator for each agent. Specifically, we use the offline data collectively available across heterogeneous agents learn the unique transition dynamics for each agent. We do this without requiring access to the covariates or features that drive the heterogeneity amongst the agents. Having constructed a personalized simulator, we then learn personalized decision-maaing policy separately for each agent by simply using online model predictive control (MPC) 16 9
Theoretical- learning across agents. As alluded to carlier. the challenge in building personalized simulator for each agent is that only have access to single offline trajectory for any given agent. Hence, each agent likely explores a very small subset of the entire state-action space. However, by viewing the trajectories across the multitude of agents collectively, we potentially have access to relatively larger and more diverse offline dataset that covers much richer subset of the state-action space. Still, any approach that augments the data of an agent in this manner must address the possibly large heterogeneity amongst the agents, which is challenging as we do not observe the characteristics that make agents heterogeneous. Inspired by the literature on collaborative filtering for recommendation systems, we posit that the transition dynamics across agents can be represented as a latent function of latent factors associated with agents, states, and actions. In doing so, we establish that this function is well-approximated by "low-rank" decomposition of separable agent, state, and action latent functions (Theorem D]. Hence, for any finite sampling of the state and action spaces, accurate model learning for each agent with offline data- genterted from any policy-can be reduced to estimating low-rank tensor corresponding to agents, states, and actions. As such, low-rank tensors can provide a useful algorithmic lens to enable model learning with offline data in RI. and we hope this work leads to further research studying the relationship between these seemingly disparate fields.

---

Shorithmic-regularii via a factor model. As consequence of our low-rank representation result, we propose natural neural network architecture that respects the constraints induced by the factorization across agents, states, and actions (Section is this principled structure, which accounts for agent heterogeneity and regularizes the model learning, that ensures the success of our approach despite access only scarce and heterogeneous data. Further, we propose natural extension of our framework which generalizes to unseen agents, i.c., agents that are not observed in the offline data.
Experimental-ettene benchmarking. Using standard environments from OpenAl Gym, we ex- tensively benchmark PerSim against four offte--f-the-rrr methods: model-based online RL method (CaDM) [9], two model-free offline RL method (BCQ and CQL) [S 25], and a model-based offline RL method (MOReL) 23 Below, we highlight six conclusions we reach from our experiments. (i) Despite access to only single trajectory from each agent (and no access to the covariates that drive agent heterogeneity). PerSim produces accurate personalized simulators for each agent. (ii) All benchmarking algorithms perform sub- sub-optimally for the data availability we consider, even on simple baseline environments such as MountainCar and Cartpole, which are traditionally considered to be "solved". (iii) PerSim is able to robustly extrapolate outside the policy used to generate the offline dataset, even if the policy is highly sub-optimal (e.g., actions are sampled uniformly at random), (iv) To corroborate our latent factor representation, agent-specific latent factors correspond very closely with the latent source of heterogencity amongst agents; we re-emphasize this is despite PerSim not getting access to the agent covariates. (v) We find that augmenting the training data of offline model-free method (e.g.. BCQ) with PerSim--ennrated synthetic trajectories results in a significantly better average reward. (vi) As an ablation study, if we decrease the number of observed trajectories, PerSim consistently achieves a higher reward than the other baselines across most agents, indicating its robustness to data scarcity. For visual depiction of the conclusions, see Figure | for the learned latent agent factors and Figure 2 for the relative prediction accuracy of the learned model using PerSim versus
find that across all environments, the learned
29
Related Work. Due to space con- straints, we present short overview of the related literature. A more detailed review is provided in Appendix B
15-step
15
30-step
step
50-step
step
step
Well GIDN PEPS-CO0 NerSin
M
M
M
M
M
There are two sub-fields within RL that are of particular relevance: (i) model- based online RL and (ii) model-free of- fline RL. (i) In the model-based RL lit- erature, the transition dynamics (sim- ulator) is learnt and subsequently uti- lized for policy learning. These meth- ods have been found to have far bet- ter data efficiency compared to their model-free counterparts
1
Figure 2: Visualization of prediction accuracy of the various learned models for CartPole. Actual and predicted states are denoted by the opaque and translucent objects, respectively.
(50 TCC. II
26 21 However, the current model-basee literature mostly focuses on the setting where one can
adaptively sample trajectories in an online manner during model-learing, A few recent works 22 36 have considered agent heterogeneity. We compare with 22 given its strong performance in handling heterogeneous agents. (ii) In the model-free offline RL literature, one uses pre-recorded dataset to directly learn policy, i.e., without first learning a model. Thus far, the vast majority of offline RL methods are model-free and designed for settings that allow access numerous trajectories from a single agent, i.e., no agent heterogeneity [ 24 methods suffer if agent heterogeneity is introduced,
25 To study how much offline compare with both [5] and [25] given their strong performance with offline data. We choose these baselines as they come from a well-established literature but their abilities to simultaneously handle agent heterogeneity and sparse offline data has yet to be studied.
28 32
Model-based offline RL is still a relatively nascent field. Two recent works 2 have shown that, in certain settings, model- based offline methods can outperform their model-free counterparts on benchmark environments. In [53] authors exhibit this using existing online model-based methods with minimal changes. However, both works restrict attention to the setting where there is only one agent (or environment) and a large number of observations from that agent are available. Hence, we study how these methods perform when given only sparse data from heterogeneous agents, and we find their performance does suffer. Extending these modell-baeed offline RL methods work with sparse data from heterogeneous agents, possibly by building upon the latent low-rank functional representation we propose, remains interesting future work.
3

---

2 Problem Statement
We consider the standard RL framework with N heterogeneous agents. We index agents with n e [N Formally, we describe our problem as a Markov Decision Process (MDP) defined by the tuple (S, Pu. Rn.Tn.Mur Hn.Ma). Here, S and A denote the state and action spaces, respectively, which are common across agents. For every agent n, Pn. (s'''s,,) is the unknown transition kerne,, (s,a) is the immediate reward received, "/n E (0,1) is the discounting factor, and H,, is the initial state distribution.
Observations. We consider an offline RL setting where we observe a single trajectory of length T for each of the N heterogeneous agents. Formally, for each agent n and time step I, let 8n (t) E S, a, (t) E A. and (t) E denote the observed state, action, and reward. We denote our observations as D (8 ((),,(())))) :n [
Goals. We state our two primary goals below.
"Personalized" trajectory prediction. For a given agent and state-action pair (s.a) 8 x A, we would like to estimate E[ss', (,, = (s.a)], i.c., i.c.. given the observations D, we would like to build "personalized" simulator (i.e., a model of the transition dynamic) for each agent n.
"Personalized" model-basee policy learning. To test the efficacy of the personalized simulator, we would like subsequently use it to learn good decision decision-making policy for agent n, denoted as Tr S A, which takes as input a given state and produces a corresponding action.
A causal inference lens. We note that this problem can be thought of as a counterfactual prediction problem. Our goal is to build a personalized simulator that answers the following question: "what would have happened had the agent took sequence of actions other than the one we observe in the dataset?". Broadly speaking, our aim in PerSim is to answer such questions by observing how other agents behave under varied sequences of actions. More generally, we hope PerSim serves to further link causal inference with offline RL, which fundamentally regarded as a counterfactual inference problem 31
3 Latent Low-rank Factor Representation
To address the goals, we introduce a latent factor model for the transition dynamics. Leveraging latent factors have been successful in recommendation systems for overcoming heterogeneity of users. Such models have also been shown to provide "universal" representation for muntii-dmmesiooall exchange- able arrays [5 18). Indeed, our latent model holds for known environments such as MountainCar.
Assume S C i.i.... the state is D-dimensional. Let 8nd refer to the d-th coordinate of 8n. We posit the transition dynamics (in expectation) obey the following model: for every agent n and state-action pair (s,a).
E| (S,, (s..)))))()))))=======)))))>)))) (s.))====()==((( ff((O..Pe.a))
where s'ms denotes the d-th state coordinate after taking action a. Here, 6, ER e R w,, E RR for some di d2,d3 21 1 are latent feature vectors capturing relevant information specific to the agent, state, and action; fa Th Rd2 x Rd - IR is a latent function capturing the model relationship between these latent feature vectors. We assume fa is L-Lipschitz and the latent features are bounded.
Assumption 1. Suppose 9, E [0.1]4. P.. [0.1] we E [0,1]4s, and fa L-LLipschiz with respect to its arguments, i.e., (Ps (wl (0
-
wa 12).
(s,a)) Theorem Suppose Assumption 1 holds and without loss of generality, let < d2. Then for alrallde [D] and any 8> 0, there exists h N x A-RR R, such that (n,,n,,,) -i ue (e(())o(s.d))v with < C5- (:05((1+tt)) (da) and liha- Thha-fallas fall= S 2L5, where C' is an absolute constant.
For notational convenience, let fa: [N] S A-
+ I be such that
Theorem I suggests that under the model in ①. the transition dynamics are well approximated by a low-rank order-three functional tensor representation. In fact, as we show below, for a classical non-linear dynamical system, it is exact.
An example. We show that the MountainCan transition dynamics X
exactly satisfies this low-rank tensor representation. In MountainCar, the state s, = [S2] consists of car (agent) n's position and velocity, i.e., SC R2; the action an is a scalar that represents the applied acceleration, i.e., AÇR R. For
For any positive integer N. let [N] NJ.....N..

---

car parameterized by gravity In. the (deterministic) transition dynamics given action an are
s/1 5n1 + gn
On
$/2= =822 - -on +0n.
2
Proposition 1. In MountainCar, r=
Model Learning Tensor Estimation. Consider any finite sampling of the states S and actions Á A. Let X [X E [X((((IXX be the order-four tensor, where Xnsad = fa(on. Ps,wwa)) Hence, to learn the model of transition dynamics for all the agents over S, AA, it sufficient to estimate the tensor, X' from observed data. The offline data collected for given policy induces a corresponding observation pattern this tensor. Whether the complete tensor is recoverable, determined by this induced sparsity pattern and the rank of Y. Notably, Theorem | suggests that X' is low-rank under mild regularity conditions. Therefore, the question of model identification, i..., completing the tensor X, boils down to conditions on the offline data terms the observation pattern that it induces the tensor. In the existing tensor estimation literature, there are few sparsity patterns for which the underlying tensor can be provably recovered, provided X' has low-rank structure. They include: (i) each entry of tensor is observed independently at random with sufficiently high-probabilityy [6|35] 43]; (ii) the entries that are observed are block-structured 42 However, the most general set of conditions on the sparsity pattern under which the underlying tensor be faithfully estimated, i.e., the model is identified for our setup, remains an important and active area research.
4 PerSim Algorithm
We now detail our proposed algorithm which is composed of two steps: (i) build a personalized simulator for each agent n given offline observations D, which is comprised of single trajectory per agent; (ii) learn personalized decision-making policy using MPC.
Step 1. Learning Personalized Simulators. Theorem
layer
suggests that the transition dynamics can be represented as low-rank tensor function with latent functions associated with the agents, states, and actions. This guides the design of simple, regularized neural network architecture: we use three separate neural networks to learn the agent, state, and action latent functions, i.e., we remove any edges between these three neural networks. See Figure M for visual depiction of our proposed architecture. Specifically, to estimate the next state for a given agent n and state-action pair (s,a) E A. learn the following functions:
wgpat Apper
lelkkon layer
Intond Guiters
Figure 3: Neural Network Architecture
1. An agent encoder yu [N] -N)
R", parameterized by v. which estimates the latent function associated with an agent, i.e., ([n)= = ga (n:z).
state encoder go :S - Dxr parameterized by d. which estimates latent functions, where each vector is associated with the corresponding state coordinate, i.e., e(s) (7(s,1), .... "(,
ge(5;0 (s;Ø).
3. An action encoder Ihe : R² parameterized by 0 which estimates the latent function associated with the action, i.c., w0 (a) gw (a;0).
Then, our estimate of the expected d-th coordinate of the next state is given by (ssnnd an) (
(a). We optimize our agent, state, and action encoders by minimizing the squared loss: L(s,a,n,s' s/ 20.000 -b=
Step 2. Learning a Decision-making Policy. We use MPC to select the next action, common
practice in the literature [29] Since the offline data may not span the entire state-action space, planning using a learned simulator without any regularization may result in "model exploitation" [31] Thus, when choosing the action via MPC, we choose the first action from the sequence of actions with the best average reward across an ensemble of M simulators. our experiments, we find that the simple technique of using the state difference instead of the raw state improves performance. For a detailed description of the algorithm, including the pseudo-code, please refer to Appendix D
5

---

5
Experiments
In this section, through systematic collection of experiments on variety of benchmark environments, we demonstrate that PerSim consistently outperforms state-of-the-at model-based and offline model- free RL algorithms, in terms of prediction error and reward, for the data regime we consider.
5.1 Setup and Benchmarks
We evaluate PerSim on three benchmark environments from OpenAl Gym [: MountainCar, CartPole, and HalfCheetah. A detailed description of each environment can be found in Appendix D
Heterogeneous Agents. We introduce agent heterogeneity across the various environments by mod- ifying the covariates that characterize the transition dynamics of an agent. This is in line with what has been done in the literature 29] to study algorithmic robustness to agent heterogeneity. The range of parameters we consider for each of the three environments is given in Tf Appendix L for example, we create heterogeneous agents in MountainCar by varying the gravity parameter of a car (agent) within the interval (0.01,0.035)
For each environment, we uniformly sample 500 covariates (i.e., heterogeneous agents) from the parameter ranges displayed in Table of Appendix E. We then select five of the 500 agents to be our "test" agents, i.e., these are the agents for which we report the prediction error and eventual reward for the various RL algorithms. Our rationale for selecting these five agents is as follows: one of the five is the default covariate parameter in environment; the other four are selected so to cover the "extremes" of the parameter range. Due to space constraints, we show results for three test agents in this section; the results for the remaining two agents can be found in Appendix F We note that the conclusions we draw from our experiments continue to hold over all test agents we evaluate on.
Offline Data from Sub-optimal Policies. To study how robust the various RL algorithms are the "optimality" of the sampling policy used to generate the historical trajectories, we create four offline datasets of 500 trajectories (one per agent) for each environment as follows:
(i) Pure policy. For each agent, actions are sampled according to fixed policy that has been trained to achieve "good" performance. Specifically, for each environment, we first train a policy in an online fashion for few training agents (see TableIfoo for details). We pick the training agents to be uniformly spread throughout the training range to ensure reasonable performance for all agents. See Appendix for details about the average reward achieved across all agents using this procedure. For MountainCar and CartPole, we use DQN 64 to train the sampling policy; for HalfCheetah, we use TD3 [1] The policies are trained to achieve rewards of approximately -200, 120, and 3000 for MountainCar, CartPole, and HalfCheetah, respectively. Then for each of the 500 agents, we sample one trajectory using the policy trained on the training agent with the closest parameter value. (ii) Random. Actions selected uniformly random. (Hi/iv) Mane-e--2/PPrree uniformly at random with probability 0.2/0.4, respectively, and selected via the pure policy otherwise.
For Pure-e-20/40, actions are selected
The "pure policy" dataset has relatively optimal transition dynamics for each agent compared to the other three policies. This is likely the ideal scenario when we only have access to limited offline data. However, such an ideal sampling procedure hardly met in practice. Real-world data often contains at least some amount of "trial and error" in terms of how actions are chosen; we model this by sampling a fraction of the trajectory at random.
Benchmarking Algorithms. We compare with four state-of-the-art RL algorithms; (i) one from the online model-based literature; (ii) two from the offline model-free literature; (iii) and one offline model-based approach. In Appendix D we give implementation details.
Vanilla CaDM + PE-TS CaDM 22). As aforementioned, we choose CaDM baseline given its superior performance against other popular model-hased (and meta-learning) methods in handling heterogeneous agents. CaDM tackles heterogeneity by learning a context vector using the recent trajectory of a given agent, with a common context encoder across all agents. In our evaluation, compare against two CaDM variants discussed in 29 with respect to both model prediction error and eventual reward.
BCQ-P+BCL [5] BCQ is an offline model-free RI. method that has been shown to exhibit excellent performance in the standard offline setting. In light of this, we consider two BCQ baselines: (i) BCQ- Population (or BCQ-P), where a single policy is trained using data from all available (heterogencous) agents, i.e., all 500 observed trajectories: (ii) BCQ-Agent (or BCQ-A), where a separate policy is learned for each of the test agents using just the single observed trajectory associated with that test agent. We compare PerSim against both BCQ variants with respect to the eventual reward in order to study the effect that data scarcity and agent heterogeneity can have on standard offline RL methods.

---

Table I: Prediction error.
MountainCar
CartPole
HolfCheetah
Data Method
0.01
0.025
0,035
(2/0.5)
(10/0.85)
(10/0115)
(0.3/1.7)
(1.7/0.3)
(0.3/0.3)
PerSim
0.025 (0.97)
(.014
(0.80)
0.01 (1.00) 0.01 (1.00)
(.035((.00)
1.194 (0.92)
4.064 (0.67)
(0.81)
Pure
Vanilla CaDM
(..49)00.7))
(-1844
(-30. I)
0.403 (0.71) 0.039 (0.98)
2.531 (-0.53)
3,902 (0.47)
(.851
6,308 (0.38)
PE-TS CaDM
(-1.9))
(-154)-
(-0.18)
0.031 (0.99)
0.00
0.319 (0.65)
3,147 (0.61)
5,270 (0.68)
(0.57)
Random
PerSim
(1.00)
0.01 (1.00)
0.01 (1.00)
0.014 (0.98) (1.00)))))
0.152 (0.88)
1.194 (0.92)
4.064 (0.67)
(0.81)
Vanilla CaDM
(0.43)
(-2.7 (.134
0.172 (0. 10) 0.098 (0.64)
3.307 (-0.73)
4,030 (0.44)
4.121 (0.43)
4,446
PE-TS CaDM
0.242 (0.31)
0,101 (0.87)
0.075 (0.97)
0.564((1.36
(.216)0:45)
(-1.0)
2.735 (0.73)
(0.69)
4.141 (0.77)
Pure- Pure- 2-20
Peršim
(1.00)
0.02 (1.00) 0.04 (1.00)
(.0(L.
0.01 (1.00) 0.048 (0.98)
1.172 (0.92)
4.283 (0.66)
(0.84)
Vanilla CaDM
0.227 (0.31) 0.101 (-U.37)
(-2.25)
0.270 (-0.98) 0.058 (0.94)
2.193 (-0.43)
3.613
3.455
6,046 (0.48)
PE/TS CaDM
(-2.57)
(0.75)
(0.89)
0.639 (-1.21)
0.148 (0.38)
(-0.56)
2.913 (0.70)
(0.65)
(0.65)
PorSim
(1.00)
0.06 (0.99) 0.03 (1.00)
(..00)
(..00)
0.018 (1.00)
(0.94)
4.021 (0.66)
3,742 (0.85)
Vanilla CaDM
0.199 (0.54)
0.1.7 (0.38)
0.233 (-0.88) 0.035 (0.99)
3.286 (-0.62)
3.685 (0,49)
(0.48)
6,000 (0.39)
PS CaDM
(-2.97) (-606) (0.55)
0.051 (0.98) 0.010
0.411 (0.55)
3.021 (0.67)
5.075 (0.61)
(0.62)
CQL is considered one of the state-of-the-aae offline model-free RL. methods Similar to BCQ, we consider two CQL baselines: CQL-P and CQL-A, which are defined analogously to BCQ-P and BCQ-A.
CQL-P CQL-A 25
MOReL is recent work that proposes a model-based approach for offline RL. MOReL does not consider a setting where data is collected from heterogeneous agents, unlike CaDM and PerSim. However, we include MOReL as a baseline to provide more comprehensive evaluation. Similar BCQ and CQL, we consider two MOReL baselines: MOReL-P and MOReL-A, which are defined analogously to BCQ-P and BCQ-A.
MOReL-P MOREL--
5.2 Model Learning: Prediction Error
The core of PerSim is to build personalized simulator for each agent. This is also the case for two of the methods we compare against, Vanilla CaDM and PE-TS CaDM. Thus, for each of these algorithms, we first evaluate the accuracy of the learned transition dynamics for each agent, focusing on long-hor long-horizon model prediction. Specifically, given an initial state and unseen sequence of 50 actions, the task is to predict the next 50-step state trajectory for the test agents. The sequence of 50 actions are chosen according to an unseen test policy that is different than the policies used to sample the training dataset. Precisely. the test policies are fitted via DQN for MountainCar and CartPole and via TD3 for HalfCheetah for the agent with the default covariate parameters. The test policies were such that they achieved rewards of 150. 150, and 4000 for MountainCar, Cort and HalfCheetah, respectively.
For each environment and for PerSim, Vanilla CaDM, and PE-TS CaDM. we train four separate models using each of the four offline datasets described earlier. We repeat the process 200 times (totaling 200 trajectories for each test agent) and report the mean soot-mean-square error (RMSE) over all 50-steps and over all 200 trajectories. In addition to RMSE, we provide the median R.2 (in parentheses within the tables) to facilitate a better comparison in terms of relative error. The results are summarized Table for MountainCar, CartPole, and HalfCheetah. Further experimental results regarding prediction error can be found in Appendix F.
PerSim Accurately Learns Personalized Simulators. Consistently across environments, PerSim outperforms both Vanilla CaDM and PE-TS CaDM for most test agents. RMSE for PerSim is notably lower by orders of magnitude in MountainCar and CartPole. Indeed, in these two environments, R for PerSim is nearly one (i.c., the maximum achievable) across most agents, while for the two CaDM variants it is notably lower. In a number of experiments, the two CaDM variants have negative H² values, i.e., their predictions are worse than simply predicting the average true state across the test trajectory. For HalfCheetah, which has relatively dimenniiinii state space (it 18-dimensional), PerSim continues to deliver reasonably good predictions for each agent despite the challenging data availability. Though PerSim still outperforms CaDM in HalfCheetah, note CaDM's performance is more comparable in this environment.
For a more visual representation of PerSim's prediction accuracy, refer back to Figure 2 in Section I which shows the predicted and actual state for different horizon lengths in CartPole; there, we see that PerSim consistently produces state predictions that closely match the actual state observed from the true environment. Additional visualizations across environments are provided in Appendix F.T]
Rethinking Model-based RL for Our Setting, Altogether, these results indicate that existing model- based RL methods cannot effectively leamn model of the transition dynamics simultancously across
7

---

all heterogeneous agents (e.g., CaDM sometimes has negative R² values) if only given access to sparse offline data. It has been exhibited 19 53 31 that certain model-based methods that were originally targeted for the online setting can potentially still deliver reasonable performance with offline data and minimal algorithmic change. Our experiments offer evidence that model-based RL methods, even those which are optimized work with heterogeneous agents, do not provide a uniformly good "plug-in' in" solution for the particular data availability we consider. In contrast, our latent factor approach consistently learns reasonably good model of the dynamies across all agents.
Latent Factors Capture Agent Heterogeneity. The poor performance of standard model-based RL methods for the data availability setting we consider emphasizes the need for new principled approaches. Ours is one such approach, where we posit a low-rank latent factor representation of the agents, states, and actions. To further validate our approach, we visualize the learned latent agent factors associated with the 500 heterogeneous agents in each environment in Figure Tf Section [ Pleasingly, across environments, the latent factors correspond closely with the heterogeneity across agents.
5.3 Policy Performance: Average Reward
In this section, we evaluate the average reward achieved by PerSim compared several state-of-the-art model-based and model-free offline RL methods. For the PerSim and the two CaDM variants, we follow (29 in utilizing MPC to make policy decisions on top of the learned model. We include an additional baseline, "True env + MPC", where we apply MPC on top of the actual ground-truth environment: this allows us to quantify the difference in reward when using MPC with the actual environment versus the learned simulators. For the model-free methods (e.g., BCQ and CQL variants), we can directly apply the policy resulted from the learned Q-value. Likewise, we directly apply the policy trained by MOReL. We use the trained policy for this model- -bssed method instead of utilizing MPC to faithfully follow the method described in [3
We evaluate the performance of each method using the average reward over 20 episodes for model- based methods and the average over 100 episodes for model-free methods. We repeat each experiment five times and report the mean and standard deviation. Tablee[preeen the results for MountainCar. CartPole, and HalfCheetah. Further experimental results regarding reward are given in Appendix F22
As high-level summary, in most experiments, PerSim either achieves the best reward or close it. This not only reaffirms our prediction results in Section B.2] but also is particularly encouraging for PerSim since the policy utilized is simply MPC and not optimized as done in model-free approaches and in MOReL. Furthermore, these results corroborate the appropriateness of our low-rank latent factor representation and our overall methodological framework as a principled solution to this challenging yet meaningful setting within offline RL. In what follows, we highlight additional interesting conclusions.
"Solved" Environments are Not Actually Solved. MountainCar and CartPole are commonly per- ceived as simple, "solved" environments within the RL literature. Yet, results in Table ddemnnttrate that the offline setting with scarce and heterogeneous data impose unique challenges. and undoubtedly warrants new methodology. We find state-of-the-art model-based and model-free methods perform poorly on some of the test agents in MountainCar and CartPole. In comparison, PerSim's 's performance in these environments is close to that of MPC planning using the ground -ruth environment across all test agents, thereby confirming the success of learning the personalized simulators in Step 1 of our algorithm. In certain rare cases (c.g.. MountainCar test agent 0.01) where other baselines have a comparable performance to PerSim, we see that these basclines even outperforms True env + MPC. This indicates that the bottleneck in these experiments using MPC for policy planning rather than the learned simulator in PerSim.
PerSim Robustly Extrapolates with Sub-optimal Data. Crucially, across all environments and offline data generating processes, PerSim remains the most consistent and robust winner. This is much desired property for offline RI. In real- world applications, the policy used to generate the offline dataset is likely unknown. Thus, for broad applicability of a RL methodological framework, is vital that it is robust to sub-optimality in how the dataset was generated, e.g., the dataset may contain si significant amount of "trial and error" (i.e, randomized actions). Indeed, this one of the primary motivations to use RL in such settings in the first place.
As mentioned earlier, the four offline datasets correspond to varying degrees of "optimality" of the policy used to sample agent trajectories. We highlight that PerSim achieves uniformly good reward, even with random data, i.c.. the offline trajectories are produced using totally random actions. This showcases that by first learning a personalized simulator for each agent, PerSim is able to robustly extrapolate outside the policy used to generate the data, however sub-optimal that policy might be. In contrast, BCQ and CQL. are not robust to such sub-optimality in the offline data generation. For example, for HallCCeetth, BCQ achieves reasonable performance primarily when trained on "optimal"
8

---

Table 2: Average reward.
MountainCar
CarPole
HaltCheetah
Data Method
0.01
0.025
0.035
(2/0.5)
(100.85)
(100.15)
(0.3/1.7)
(1.7/0.3)
(0.3/..)
PerSim
-56.80=1.88 1.83
-1894426,44
-210.64:.27
199.720.58
193.824428
192.042.28
1984 763
997.0:1403
714.74114
Vanilla CaDM
-106.32:44.1
117
-471 82:330
168.0=19.7
190.826.80
18.10::00.7
50.31:71.77
1134.0:81.1
11.39:171
PE-TS CaDM
-74.23:1.6.5
13.3.31:3.3
55..0::0.00
92.30:1:44.8
193.618.30
127.5:9955
481.1::252
503.71:181
553.0:1:27
BCQ-P
67.6001:22.3
267
198
366.2...9.3
181.21
18.0
549.8:1:22
2006 1:3
66.118192.88
Pure
BCQ-A
44.79:10.08
177
.5000000000
65.40:16..5
79.20:169.5
132.1185.0
262.7:1
-139.0:1:36
165.6.183.1
CQL-P
4176,1
2316.4#26.
16.99+17.9
17.7
190.9=0,7
170.2=00.6
-353,5=78.4
-55.66771.9
147
CQL-A
44.60=0000
-500.0::.00
099.3300.7
122.8=63.4
179.4=20.6
193.6#5.4
105.00==005
-257.9#35.9
-279.6::3444
MOReL-P
1.1.00#1.1
-500.0:0.00
-50.000.00
35.80:1144
96.60+:24.1
66.40:224.55
-1297 559
-1256
-1470 #727
MOReL-AA
333.00133.5
550.00....0
50..001..00
33.70.1..8
27.50:10.1
10.10.10.7
-726.2.1499
124.6
.841.71..9.5
PerSim
55..70=5.63
-186.6::4225
-2101144.88
197.747882
193.0=6.60
185.7 3:49
2124 5118
9900
472.0:1:56.9
Vanilla CaDM
622.57-5.11
479.3121177
49..
15.7
178.6155.80
16.10
288.4:1.2.4
362.9155544
351.8:114.9
PE-TS CaDM
32.000=3.47
.500.0::.00
.500010.00
18.60:18..5
196. 33000
171.0-21.7
754.6:::42
744.54:281
767.44214
Rundom
BCQ:P
50..0::0.00
-500.0::.00
500.0:00000
57.91::53.0 44.80-34400
36.40++36.0
-1.460=0011
-1.750=+..22
-1.690=00119
BOQ-A
:0..0::0000
-500.::0.00
500.000.00
43.90:16.44
21.10::5.81
39.50::12.1
14%
13,0.3:11.00
51.9.5:11.77
CQL-P
5500.0::000
-500.0::000
500.000000
39/9:1229.9
77 27.90+226.2
148.7417.11
24,1.5:25.99
54,2.244.6.4
1672.4117766
CQL-A
-500.0:0.0
-5.0.0::0.0
-5.0.000.0
67.40:49.11
30.60-2:6.4
7.000
-0,700=0044
-2.800=#0.8
-0.600:10.3
MOReL-P
-5,000=0.3
-500.0=0.00
-5000000.0
50,20+
68.90 17.90+17744
40.40::1244
-102.3+:45.7
-181,0 -18.77
MOReL-A
395.01..99
.500.01..00
500.0000000
35.80.0.4
27.50.1:0.7
10.6010.22
130.8.1195
67..7.1.9..
365.55197.4
PerSim
-5.220=0.56
-191 6.770
199.743.99
199,8=0.24
199.
197.8#1.6
3186 6004
1032-2232
1121
Vanilla CaDM
-56.73=4.20
-463.2457.55
-478.433.8
1711+38.1
193.4=2.10
14.20=:1000
412.0=1522
11
159
PE-TSCCDM
30..61136.3
5500.01:..0
.0..001..00
98.30.142.9
198.610.40
14..1112200
:: 126
1125
1067
Pure-c-22 -0
BCQ-P
24421
288.6.1 196
158
38.90.1.3.2
155.1
86.10
354.:1352
106.7.17.11
385.91157.1
BOQ-A
180
-260.6:::1.4
62.4.5268.9
67.50:162.2
65.60-1:1.8
840.0:80066
376.8:102
84.66:533.3
230.1=+10.0
CQL-P
179.40=16.1
135
107.6+15.3
163.7#13.6
198.1
190.426.6
838.7+:24.5
3155 125
$39.9+3113
COL-A
14.70:20.1
-500.0::.00
550.0:00.0
42.40:11.9
199.::19
199.8:0.2
-15.50::9.0
-73.00=:26.3
-108.0:1:4.66
MOReL-P
15.500-156
-357.01 13.8
177
166.7133.6
197.74277
189.44..1
0:600:::099
-171.24125
-219.9:16.77
MOReL-A
14.60
-500.0::0.0
-500.0:0.00
41.0::11.11
1988822.1
199.910.22
-781.04:37.9
-613.1 1::99
-847.1
PerSim
-5.60020.55
1189.747.14
-200.342.26
1999-2.18
198.0
1.72
2590 -813
1016 2283
1365 =582
Vanilla CaDM
-55.23-0.76
481.7
496.224331
160.6246.6
191. 19..9:6.00
39.60=
465.6:::9.2
452.7 30
720.0:+74.9
PE/TS CaDM
20.
.500.0..0.0
500.010.000
97.90:16..6
1.40
14.5
1500 1:46
1218 221
1339 1,58.8
Pure-c-41
BCQ-P
$0.01
.373.61.80 180
352.0.211
28.90.16.80
31.80.1.5.9
18.5011111
78.25.1.00
173.81189
157
BCQ-A
94.87.10.88
-358.71.011
34.60.1..55
47.71
23.20.19.44
269.2.1.607
1181.51557.4
193.0.131.8
CQL-P
-61.40#2.1
-366.94:11.
12...+18..8
182.2=18.0
198.3+2.9
191.98..7
808.5+::40
1662 =221
-156.3+119
CQL-A
-44.00=0.0
-500.0::..0
-490.1499.
20.70::0.77
134.2+10.22
9.700#8.11
-6,200+229
-386.0::42.4
37.10=154
MOReL-P
-61.30=2.3
19.3
+428.5+21.0
178.9=18.7
197.9
190.6::9.3
6.500=:61.6
-114.2#72.2
119,99777.3
MOReL-
44.70
-500.01:.00
.99.1199.3
20.70:10.88
13..411122
10.80
13.5
1644.5118.8
179
The eDVCCC
-53.9524.10
-182.94:2299
-197.5+20.77
200.0::0.00
198.417.20
200.0::0.0
7459
428931:959
66675-1934
offline data (i.e., pure policy). This is because BCQ, by design, is conservative and is regularized to only pick actions that are close to what is seen in the offline data. In summary, PerSim's 's ability to successfully extrapolate, even with sub-optimal offline data, makes it a suitable candidate for rorl-world applications.
5.4 Combination with Model-Free Methods: PerSim=BCQ/CQL.
We explore whether simulated trajectories produced from PerSim can be used to improve the per- formance of model-free RI methods such as BCQ and CQL.. In particular, instead of using a single observed trajectory to learn an spent-sp policy, as is done in BCQ-A A and CQL-A, to generate 5 "synthetic" trajectories for that agent. We then use both the synthetic and the observed trajectories to train both BCQ and CQL (denoted by CerSim-BCQ- and PerSim -Oann--CL-,, respectively). Note that using model-based methods augment the data used by model-free method has been explored the literature 46
use PerSim
the learned model is accurate, improvements for BCQ and CQL are naturally anticipated. Table confirms that this is indeed the case for PerSim. Across all environments, augmenting the training data with PerSim results in a significantly better average reward for most test agents. Specifically, the performance of PerSim-BCQ-S and PerSim-CQL-S indicates that a personalized BCQ/CQL, policy trained on few PerSim-generatee generated trajectories is superior to: (i) single BCQ/CQL policy trained using

---

all 500 trajectories (e.g., BCQ-P); and (ii) personalized BCQ/CQL policy trained using a single for more details about the experiment.
observed trajectory (e.g.. BCQ-A). Refer Appendix F.4
Table 3: BCQ+PerSim
Environent Method
0:01
0.05
0.01
0.025
0.035
BCQ-P
-500.0=0.0
-5000000.00
-500.0::0000
-500.0==00.0
-500,0=0..0
MountainCar
BCQ-A
-300.0=0.0
-500.0+0.00
-500.000.00
-500.0::0.0
-500.0=0..0
PerSim-BOQL
15.2
.2.60017.000
2209.41:.266
.99.771: 1.89
-500.:::0.0
CQL-P
500.0::0.0
500.0:10.00
500.010000
5000.0::0.0
:00,00:0.00
CQL-A
-500,=0..0
-500.0+0000
-5000000000
-500.0=:0.0
-500.0=0.00
PerSim-CQL-S
44.70=0.11
-4.800=0.00
63.20:+0.11
-500.0::0.0
-500.000.0
Environment
Method
(2X0.5)
(10.00.55
(18.00.5)
(10/085)
(1090.15)
BCQ-P
44.80=434.0
58.21-58.00
56.92=56.00
57,914:33.0
36,40--36.0
CariPole
BCQ-A
43.90:2: 16.4
18,70 13.1
7,200++0,84
21.10::5.881
39.50-t:2 12.1
PerSim PerSim-nOQ.5
57.82.L199.8
CQL-P
39.90:1229.9
72.30:1:4..7
44.4
77.
77.90-1.26.2
148.7:117.1
CQL-A
67.40:t:49.1
9.300=20.1
16.30:1-999
30.60:t:26.4
7,000=119
ParSim-CQL-S
81.80:3344
198.3::1.3
200.0=0.0
135.5=11.1
190.0:2 14.2
Environment
Method
(0.3/1.7)
(L.7/0.3)
(0.310.3)
(1.7/1.7)
(1.0/1.)
BCQ-P
- 1.460
1.750.1..22
1.690.10.19
-1.790.10.21
1.720.1..20
HallCheetah
BCQ-A
-998.9= 108
-11..3::13.0
-159,5+:51.7
-35.734:7.22
-171.9:24115
PerSim-BOQ-5
571.8-+-0.3
22.19=16.2
-10.91+-47.4
64.30-22..00
157.0+31.6
CQL-P
481.5=25.9
442.24456.44
16.2.4417.6
25.4.5::19.6
-18.22233.4
CQL-A
2.500.1..0
PerSin-CQL:.
1674 1:5.2
38.5501.3599
402.24:113.0
76.70:1:239
94,20 102.8
5.5
Additional Experiments
We show how extend our method unseen test agents, i.e., agents for which we have no training trajectory. Our extension crucially relies on having learned good agent specific latent factor representation. Through case-study for the MountainCar environment, we verify that using this extension, PerSim can successfully simulate unseen agents.
Generalizing to Unseen Agents (Appendix 1:5
Robustness to Data Scarcity (Appendix F.6. We investigate how the number of training agents affects PerSim's 's performance. We find that as we decrease training agents from N =55 to N = 25, PerSim consistently achieves higher reward than the other baselines across agents. This ablation study directly addresses the robustness of PerSim to data scarcity, demonstrating that our principled framework is particularly suitable for the extreme data scarcity considered.
6
Conclusion
In this work, we investigate RL in offline setting, where we observe single trajectory across hetero- geneous agents under an unknown, potentially sub-optimal policy. This is particularly challenging for existing approaches even in "solved" environments such as MountainCar and CartPole. PerSim offers successful first attempt simultaneously learning personalized policies across all agents under this data regime; we do so by first positing a principled low-rank latent factor representation, and then using it to build personalized simulators in data-efficient manner.
Limitations and Potential Impact. Effectively leveraging offline datasets from heterogencous sources (i.e., agents) for sequential decision- makinng problems will likely accelerate the adoption RI.. However, there is much to be improved in PerSim. For example. in environments like HalfCheetah where the transition dynamics of each agent are harder learn, the performance of PerSim is not comparable with the online setting, where one gets to arbitrarily sample trajectories for each agent. Of course, our considered data regime is fundamentally harder. Therefore, understanding the extent which
improve performance, using low-rank latent factor approach methodology altogether, remains to be established. Additionally, a rigorous statistical analysis for this setting, which studies the effect of the degree of agent heterogeneity, the diversity of the samples collected, etc. remains important future work. We believe there are many fruitful inquiries under this challenging yet meaningful data regime for RL. Further. while PerSim is motivated by real-world problems, our empirical evaluation is limited to standard RI. benchmarks where data collection and environment manipulation
different
feasible. Though PerSim's 's performance on standard RL benchmarks encouraging, one cannot yet use PerSim as a out-of-the-hho solution for critical real-world problems, without first designing rigorous validation framework.
10

---

Acknowledgements and Funding Disclosure
We would like express thanks the authors of (29 Kimin Lee, Younggyo Seo, Seunghyun Lee, Honglak Lee, and Jinwoo Shin for for their insightful comments and feedback.
This work was supported in parts by the MIT-IBM project on time series anomaly detection, NSF TRIPODS Phase II grant towards Foundations of Data Science Institute. KACST project on Towards Foundations of Reinforcement Learning, and scholarship from KACST (for Abdullah Alomar).
References
[1] A. Agarwal, S. Kakade, A. Krishnamurthy, and W. Sun. Flambe: Structural complexity and representation learning of low rank mdps. arXiv preprint arXiv:2006. 10814, 2020.
[2] A. Agarwal, D. Shah, and D. Shen. Synthetic interventions, 2020.
[3] R. Agarwal, D. Schuurmans, and M. Norouzi. Striving for simplicity in off-policy deep reinforce- ment learning. 2019.
[4] R. Agarwal, D. Schuurmans, and M. Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pages 104-114. PMLR, 2020.
[5] D. J. Aldous. Representations for partially exchangeable arrays of random variables. Journal of Multivariate Analysis, 11(4):811-58, 1981.
[6] B. Barak and Moitra, Noisy tensor completion via the sum-of-squaree hierarchy, In Conference on Learning Theory, pages 417-445. PMLR, 2016.
[7] Z. I. Botev, D. P. Kroese, R. Y. Rubinstein, and P. L'Eeuyer. The cross-entropy method for optimization. In Handbook of statistics, volume 31, pages 35-59, Elsevier, 2013.
[8] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv: /606.0/540 2016.
[9] E. F. Camacho and C. B. Alba. Model predictive control. Springer science & business media, 2013.
[10] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. arXiv preprint arXiv: arXiv:/805. 1805. /2/ 14, 2018.
[11] I. Clavera, J. Rothfuss, J. Schulman, Y. Fujita, T. Asfour, and P. Abbeel. Model-based reinforce- ment learning via meta-policy optimization. In Conference on Robot Learning, pages 617-629. PMLR, 2018.
[12] M. Deisenroth and C. E. Rasmussen, Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference machine learning (ICML-I1), pages 465-472. Citeseer, 2011.
[13] C. Finn, P. Abbecl, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pages 1126-1135. PMLR, 2017.
[14] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2018.
[15] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep decereeiiforremmeetteeannng without exploration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
[16] C. E. Garcia, D. M. Prett, and M. Morari. Model predictive control: Theory and practice- survey, Automatica, 25(3):335-344 1989.
[17] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas. D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pages 2555 25655. PMLR, 2019.
[18) D. N. Hoover. Relations on probability spaces and arrays of random variables. Preprint, Institute for Advanced Study, Princeton, NJ, 2, 1979.

---

[19] M. Janner. J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-basee policy optimization, arXiv preprini arXiv: 1906.08266 2019.
[20] Jin, Z. Yang, Z. Wang, and M. Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137-2143, PMLR, 2020.
[21] L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arX1v::903.003774 2019.
[22] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakr- ishman, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 10293, 2018.
[23] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline rein- forcement learning. arXiv preprint
2020.
[24] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11784-1177 11794, 2019.
[25] A. Kumar, A. Zhou, G. Tucker, and Levine. Conservative q-leariin for offline reinforcement learning. arXiv preprint Stt1:2000604777777 2020.
[26] T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv: 102.002,00 2018.
[27] S. Lange. T. Gabel, and M. Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.
[28] R. Laroche, P. Trichelair, and R. T. Des Combes. Safe policy improvement with baseline bootstrapping. In International Conference on Machine Learning, pages 3652-3661. PMLR, 2019.
[29] K. Lee, Seo, S. Lee, H. Lee, and J. Shin. Context-aware dynamics model for generalization in model-based reinforcement learning. In International Conference on Machine Learning, pages 5757-5766. PMLR, 2020.
[30] S. Levine, C. Finn, Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-3333 2016.
[31] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning; Tutorial, review, and perspectives open problems. arXiv preprint arxiv:2005.016433 2020.
[32] Y. Liu, A. Swaminathan, A. Agarwal, and E. Brunskill. Provably good batch reinforcement learning without great exploration. arXiv preprint arXiv:2007.08202, 2020.
[33] Y. Luo, H. Xu, Y. Li, Y. Tian, T. Darrell, and T. Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. arXiv preprint
2018.
[34] V. Mnih, K. Kavukcuoglu, Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529, 2015.
[35] A. Montanari and N. Sun. Spectral algorithms for tensor completion. Communications on Pure and Applied Mathematics, 71 1:311-22 2018
[36] A. Nagabandi, I. Clavera, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn. Learning to adapt in dynamic. real-world environments through meta-reinforcement learning. arXiv preprint arXiv:1803.11347. 2018.
[37] A. Nagabandi, C. Finn, and S. Levine. Deep online learning via meta-learning Continual adaptation for model- rl. arXiv preprint arXiv: 2018
[38] Parmas, C. E. Rasmussen, J. Peters, and K. Doya. Pipps: Flexible model-based policy search
robust to the curse of chaos. In International Conference on Machine Learning, pages 4065-4074. PMLR, 2018.
12

---

[39] S. Stemundsson, K. Hofmann, and M. P. Deisenroth. Meta reinforcement learning with latent 2018.
variable gaussian processes. arXiv preprini
[40] J. I.Schiittweserr, I, Antonoglou, Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839);604-699, 2020.
[41] T. Seno. d3rlpy: An offline deep reinforcement library. d3r1py 2020.
bettpa///gtthne http:://gtttuub...ootaaauassssee www.ttocm//aaooooo
[42] D. Shah, D. Song, Z. Xu, and Y. Yang. Sample efficient reinforcement learning via low-rank matrix estimation. arXiv preprint arXiv:2006.066/ 2020.
[43] D. Shah and C. L. Yu. Iterative collaborative filtering for sparse noisy tensor estimation. In 2019 IEEE International Symposium on Information Theory (ISIT), pages 41-45. IEEE, 2019.
[44] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al. Mastering chess and shogi by self-play with a general enmeeralreifooon reinforcement learning algorithm. arXiv preprint arXiv: 2017.
[45] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354-333) 359, 2017.
[46] R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting, ACM Sigart Bulletin, 204):160- 163, 1991.
[47] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSI International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
[48] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[49] T. Wang and J. Ba. Exploring model-based planning with policy networks. arXiv preprint 19Xiv:1906.088 08649, 2019.
[50] T. Wang, X. Bao, I. Clavera, J. Hoang, Y. Wen, E. Langlois. S. Zhang, G. Zhang, P. Abbeel, and J. Ba. Benchmarking model-based reinforcement learning. arXiv preprint 190..w:1900.000000 2019.
[51] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911 1361,
[52] L. Yang and M. Wang. Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning, pages 6995-7004. PMLR, 2019.
[53] Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005. 13239, 2020.
13


Reviews:
[{"review_id": "opbSSyFN34", "reviewer": "Reviewer_Hdeq", "summary": "This paper proposes a new model-based offline RL approach that tackles a particular offline RL problem where the dataset is generated by multiple agents and the data for each agent is scarce. To learn good policies from such heterogeneous dataset with limited per-agent data, the authors propose to learn per-agent personalized simulators (per-agent dynamics models) using latent low-rank factor representations with three encoders that encode the agent information, the state, and the action respectively. The authors show that the transition dynamics can be approximated with the low-rank order-three functional tensor representation with mild assumptions. Through experiments on gym tasks with heterogeneous agent data, the proposed method is shown to outperform prior online MBRL (adapted to the offline setting) and offline model-free RL method (BCQ).", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper studies a relatively new problem in offline RL, which is also quite practical since, in the real world, we would mostly have data collected by multiple agents. The proposed method is also novel as it takes the idea from recommendation systems and uses it to learn the dynamics model in the limited data setting. The authors also justify their approach through theoretical analysis, which makes the method theoretically grounded. The paper is also well written and easy to understand. I think the paper will be of reasonable significance to the offline RL community.\n\nHowever, I do have a few concerns, which I will discuss as follows.\n\nFirst, while the authors show that the method can approximate the true dynamics model to some extent, it is unclear if it will achieve better generalization guarantees and sample complexity compared to model learning trained with the maximum likelihood estimation objective. I think it would be important to show such a theoretical guarantee if the authors want to argue that the proposed algorithm can handle scarce data setting better than previous methods. \n\nMoreover, I think the proposed problem setting is quite similar to the offline meta-RL setting, which also handles data collected by multiple agents with different dynamics and different reward functions. The authors should really discuss works in offline meta-RL and also compared the method to these works such as [1,2,3,4].\n\nFinally, it seems that the empirical evaluations are not thorough. The authors only compare the method to an online MBRL method and BCQ. I think it would be important to compare the method to previous offline MBRL methods [5,6,7] and more recent model-free offline RL method [8,9,10]. To conduct fairer evaluations, I think the authors should also make previous offline RL methods conditioned on the task/agent. Furthermore, I'd like to see if the method can perform well on standard offline RL benchmarks such as D4RL, especially on domains with undirected multitask data such as antmaze and kitchen in D4RL.\n\nGiven the above comments, I would vote for a weak reject.\n\n[1] Mitchell, E., Rafailov, R., Peng, X. B., Levine, S., & Finn, C. (2021, July). Offline Meta-Reinforcement Learning with Advantage Weighting. In International Conference on Machine Learning (pp. 7780-7791). PMLR.\n\n[2] Dorfman, R., & Tamar, A. (2020). Offline meta reinforcement learning. arXiv e-prints, arXiv-2008.\n\n[3] Li, L., Huang, Y., & Luo, D. (2021). Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning. arXiv preprint arXiv:2102.10774.\n\n[4] Li, L., Yang, R., & Luo, D. (2020). Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization. arXiv e-prints, arXiv-2010.\n\n[5] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning. arXiv preprint arXiv:2005.05951, 2020.\n\n[6] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13239, 2020.\n\n[7] Argenson, A., & Dulac-Arnold, G. (2020). Model-based offline planning. arXiv preprint arXiv:2008.05556.\n\n[8] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779, 2020.\n\n[9] Sinha, S., & Garg, A. (2021). S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning. arXiv preprint arXiv:2103.06326.\n\n[10] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "RazBkYH07S", "reviewer": "Reviewer_6hLD", "summary": "This paper proposes a new offline dataset setting: agents under different dynamics with rare sampling data (1 trajectory in the paper) for each; along with a latent factor decomposition method that motivated from the recommendation system domain. The method is interesting, the experiment of this paper is thorough and the writing is overall good.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The task is new as I know. The scenario and the motivating example are convincing and intriguing. The methodology is motivated by the usage of low-dimension latent factors decomposition in the recommendation domain and is interesting to be practiced in the RL scenario. The related works are mostly adequately cited.\n\nQuality: The methodology is overall sound under the given assumption. The experimental results are thorough and seem good but are actually weak under the high-dimensional env (see detailed comments below) and may need more baseline methods. The work is complete, although it has some potential improvements.\n\nClarity: The writing is overall clear and well organized.\n\nSignificance: The problem seems to be new and is important to the application in practice (as the healthcare example that the authors proposed). However, this paper only works on a small range of simulated data. On these simulated environments, the proposed method gets good and robust results. The proposed thinking on the low-rank factor representation in RL is worth further considering.\n\nDetailed comments:\n\nPro:\n\n1. The problem is new and motivated; the proposed method solves the problem appropriately.\n2. The idea for applying low-rank composition in RL is well put in an intuitive scenario.\n3. The experiment and the analysis are thorough and the results are better than compared baselines.\n\nCons:\n\n1. The assumption seems strong and may not be applicable in many real-world scenarios. In the paper the author explained their rationality for these assumptions by showing that it holds for a simple synthetic environment (MoutainCar). Maybe they can put it in a better context. Since all the experiments are conducted under simulated environments where the difference between the dynamics is the covariate, I am not much surprised by the result that they can share a low-rank representation. But, is it sensible for a real-world problem, e.g., the healthcare example?\n\n2. The evaluation results on Halfcheetah is weak, although the proposed method beat all the baselines. Just seeing the last row of Table 2, where the oracle return of Halfcheetah takes a large gap with all compared methods and is less convincing. Also, the author should explain why the results on the last two dynamics (1.7/0.3, 0.3/0.3) are worse than the first (0.3/1.7) while all the environments contribute 1 trajectory to the dataset.\n\n3. The comparison to baseline contains online MBRL and offline MFRL methods. So, why not offline MBRL methods are compared? Yes you mentioned that they are not designed not for this multi-task setting but BCQ is also not designed for it. Also, BCQ is not a good baseline since it has been outperformed by many other algorithms, such as CQL.\n\nConsidering the quality of the paper is overall good and the problem is new, I vote for a weak accept in this stage.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations: the most limitation is the assumption of the method which needs further experiments on real-world problems. Also, the results on a high-dimensional environment are not that convincing.\n\nSocial Impact: The healthcare example is really attractive but when the latent personalized dynamic is well quantified, inappropriate usage of such models can lead to targeted attacking for specific people. However, these problems should be considered when it is appropriately designed for that application, not in this version.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "LjPRDCIxZEY", "reviewer": "Reviewer_p946", "summary": "The paper proposes PerSim, a model-based off-line reinforcement learning algorithm targeting a setting where each recorded trajectory is produced by acting in an environment with slightly different transition dynamics. PerSim first learns a simulator for each of those environments, taking advantage of a low-rank model similar to a recommender system to account for similarities between the environments. It then uses the simulator for a specific environment to perform model predictive control. The experiments compare against strong baselines, both model-free and model-based, and demonstrate gains both in terms of the average reward and in terms of simulator accuracy (in the latter case).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an important problem of using off-line reinforcement learning in a setting where data is scarce due to different demonstrations being recorded in different versions of the environment. While the experiments are performed on popular OpenAI gym environments, the authors provide a convincing motivating application in terms of personalized medical treatment planning. The architecture choice using latent factors with low-rank approximation makes a lot of sense. About half of the paper length is covering experimental evaluation, which I think is appropriate. The experiments are very comprehensive, demonstrating clear gains of PerSim against the baselines. I am not familiar with the CaDM and BQN baselines chosen for evaluation, but the selection seems appropriate in that it uses one model-free and one model-based baseline, with the former trained either on all trajectories in the dataset, or only the one recorded in the specific environment. Overall, this looks like a polished piece of work with a solid contribution.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "DGYulSDCQeP", "reviewer": "Reviewer_6vKu", "summary": "The authors propose a novel model-based method for solving offline RL with heterogeneous agents. The authors show that under some continuity assumption, the dynamics model could be well approximated by a low-rank order-three functional tensor representation. In practice, it could be computed by the outputs of three neural networks, and one of which deals specifically with agent information. The proposed algorithm achieves robust and competitive performance on three control benchmarks and outperforms two state-of-the-art offline RL algorithms.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. The idea of approximating the agent-specific dynamics model with the low-rank decomposition is novel. The authors provide theoretical justification for such representation, although it holds under some continuity assumption on the dynamics function. The additional example on MountainCar dynamics further solidifies the theory and provides good intuition on the algorithm design. \n\n2. The experiment is comprehensive. The evaluation results show that the proposed algorithm achieves robust and competitive performance under various evaluation conditions and various data collecting methods, although the reviewer does have some further questions regarding the experiment setting (see below).\n\n3. The paper overall is well organized. The presentation is clear, and the introduction of the method is detailed. It seems easy to reproduce the results and the authors also provide the source code.\n\nWeakness/Minors:\n\n1. Regarding the experiment setting: the current selected control tasks seem relatively easy compared to the main-stream benchmarks. Although the authors have already shown that even the state-of-the-art offline RL algorithms could fail on tasks as easy as MountainCar (under the proposed data collection scheme), it would be great to see how PerSim performs on more challenging tasks with more complicated dynamics, such as Hopper or Walker (HalfCheetah is considered relatively easy comparing with these two), or even tasks with high dimensional state space such as Ant. \n\n2. The current presentation of the result table seems overwhelming at the first glance. It would be great if the authors could remind the readers about the evaluation contexts in the captions. Also, it would also be helpful if methods with statistically significantly better results could be highlighted.\n\n3. The additional experiment on the scarcity of the data (e.g., 25 vs. 250 agents' trajectories) is reassuring. However, the reviewer is confused with the fact that only the random data collection scheme is included in the results. How is the performance different in the cases with the other data collection schemes?\n\n4. It seems like that the training of PerSim relies on the assumption that the index of the agent is known (i.e., the input $n$ to the agent-specific network $g_u$). It may be a reasonable assumption under the current setting where each agent only has one trajectory but it could raise problems when the number of trajectories of each agent in the dataset is not fixed (then one needs to manually label the trajectories first, which seems to be extra information). Also, the reviewer may overlook but the reviewer does not follow how should the agent label be designed during test time?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have provided a comprehensive list of limitations of their work and the proposed algorithm does not seem to have any negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies an interesting and challenging offline RL problem, where the agents are heterogeneous and there is only a single trajectory for every agent under a potentially non-optimal policy. The authors propose PerSim, where a personalized simulator is learned for each agent based on the trajectories from all agents, and then a policy is made based on MPC over simulators ensemble.
The technical novelty of the paper lies in the finding that the transition dynamics across agents can be represented as a latent function and thus can be nicely approximated by a low-rank decomposition of the latent functions of agent, state, actions. In general, the contributions of this paper are clear and sufficient.
The majority of reviewers provide acceptance recommendations for this paper. The only reviewer of score 5 raised some questions about the generalization ability and sample complexity of the method, problem setting, and the evaluation, which I think have been well answered with detailed revision plans. Thus I think this paper can be accepted.  
Decision: Accept (Poster)