Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a new model-based offline RL approach that tackles a particular offline RL problem where the dataset is generated by multiple agents and the data for each agent is scarce. To learn good policies from such heterogeneous dataset with limited per-agent data, the authors propose to learn per-agent personalized simulators (per-agent dynamics models) using latent low-rank factor representations with three encoders that encode the agent information, the state, and the action respectively. The authors show that the transition dynamics can be approximated with the low-rank order-three functional tensor representation with mild assumptions. Through experiments on gym tasks with heterogeneous agent data, the proposed method is shown to outperform prior online MBRL (adapted to the offline setting) and offline model-free RL method (BCQ).
main_review: The paper studies a relatively new problem in offline RL, which is also quite practical since, in the real world, we would mostly have data collected by multiple agents. The proposed method is also novel as it takes the idea from recommendation systems and uses it to learn the dynamics model in the limited data setting. The authors also justify their approach through theoretical analysis, which makes the method theoretically grounded. The paper is also well written and easy to understand. I think the paper will be of reasonable significance to the offline RL community.

However, I do have a few concerns, which I will discuss as follows.

First, while the authors show that the method can approximate the true dynamics model to some extent, it is unclear if it will achieve better generalization guarantees and sample complexity compared to model learning trained with the maximum likelihood estimation objective. I think it would be important to show such a theoretical guarantee if the authors want to argue that the proposed algorithm can handle scarce data setting better than previous methods. 

Moreover, I think the proposed problem setting is quite similar to the offline meta-RL setting, which also handles data collected by multiple agents with different dynamics and different reward functions. The authors should really discuss works in offline meta-RL and also compared the method to these works such as [1,2,3,4].

Finally, it seems that the empirical evaluations are not thorough. The authors only compare the method to an online MBRL method and BCQ. I think it would be important to compare the method to previous offline MBRL methods [5,6,7] and more recent model-free offline RL method [8,9,10]. To conduct fairer evaluations, I think the authors should also make previous offline RL methods conditioned on the task/agent. Furthermore, I'd like to see if the method can perform well on standard offline RL benchmarks such as D4RL, especially on domains with undirected multitask data such as antmaze and kitchen in D4RL.

Given the above comments, I would vote for a weak reject.

[1] Mitchell, E., Rafailov, R., Peng, X. B., Levine, S., & Finn, C. (2021, July). Offline Meta-Reinforcement Learning with Advantage Weighting. In International Conference on Machine Learning (pp. 7780-7791). PMLR.

[2] Dorfman, R., & Tamar, A. (2020). Offline meta reinforcement learning. arXiv e-prints, arXiv-2008.

[3] Li, L., Huang, Y., & Luo, D. (2021). Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning. arXiv preprint arXiv:2102.10774.

[4] Li, L., Yang, R., & Luo, D. (2020). Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization. arXiv e-prints, arXiv-2010.

[5] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning. arXiv preprint arXiv:2005.05951, 2020.

[6] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13239, 2020.

[7] Argenson, A., & Dulac-Arnold, G. (2020). Model-based offline planning. arXiv preprint arXiv:2008.05556.

[8] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779, 2020.

[9] Sinha, S., & Garg, A. (2021). S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning. arXiv preprint arXiv:2103.06326.

[10] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes PerSim, a model-based off-line reinforcement learning algorithm targeting a setting where each recorded trajectory is produced by acting in an environment with slightly different transition dynamics. PerSim first learns a simulator for each of those environments, taking advantage of a low-rank model similar to a recommender system to account for similarities between the environments. It then uses the simulator for a specific environment to perform model predictive control. The experiments compare against strong baselines, both model-free and model-based, and demonstrate gains both in terms of the average reward and in terms of simulator accuracy (in the latter case).
main_review: The paper addresses an important problem of using off-line reinforcement learning in a setting where data is scarce due to different demonstrations being recorded in different versions of the environment. While the experiments are performed on popular OpenAI gym environments, the authors provide a convincing motivating application in terms of personalized medical treatment planning. The architecture choice using latent factors with low-rank approximation makes a lot of sense. About half of the paper length is covering experimental evaluation, which I think is appropriate. The experiments are very comprehensive, demonstrating clear gains of PerSim against the baselines. I am not familiar with the CaDM and BQN baselines chosen for evaluation, but the selection seems appropriate in that it uses one model-free and one model-based baseline, with the former trained either on all trajectories in the dataset, or only the one recorded in the specific environment. Overall, this looks like a polished piece of work with a solid contribution.
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a novel model-based method for solving offline RL with heterogeneous agents. The authors show that under some continuity assumption, the dynamics model could be well approximated by a low-rank order-three functional tensor representation. In practice, it could be computed by the outputs of three neural networks, and one of which deals specifically with agent information. The proposed algorithm achieves robust and competitive performance on three control benchmarks and outperforms two state-of-the-art offline RL algorithms.  
main_review: Strength:

1. The idea of approximating the agent-specific dynamics model with the low-rank decomposition is novel. The authors provide theoretical justification for such representation, although it holds under some continuity assumption on the dynamics function. The additional example on MountainCar dynamics further solidifies the theory and provides good intuition on the algorithm design. 

2. The experiment is comprehensive. The evaluation results show that the proposed algorithm achieves robust and competitive performance under various evaluation conditions and various data collecting methods, although the reviewer does have some further questions regarding the experiment setting (see below).

3. The paper overall is well organized. The presentation is clear, and the introduction of the method is detailed. It seems easy to reproduce the results and the authors also provide the source code.

Weakness/Minors:

1. Regarding the experiment setting: the current selected control tasks seem relatively easy compared to the main-stream benchmarks. Although the authors have already shown that even the state-of-the-art offline RL algorithms could fail on tasks as easy as MountainCar (under the proposed data collection scheme), it would be great to see how PerSim performs on more challenging tasks with more complicated dynamics, such as Hopper or Walker (HalfCheetah is considered relatively easy comparing with these two), or even tasks with high dimensional state space such as Ant. 

2. The current presentation of the result table seems overwhelming at the first glance. It would be great if the authors could remind the readers about the evaluation contexts in the captions. Also, it would also be helpful if methods with statistically significantly better results could be highlighted.

3. The additional experiment on the scarcity of the data (e.g., 25 vs. 250 agents' trajectories) is reassuring. However, the reviewer is confused with the fact that only the random data collection scheme is included in the results. How is the performance different in the cases with the other data collection schemes?

4. It seems like that the training of PerSim relies on the assumption that the index of the agent is known (i.e., the input $n$ to the agent-specific network $g_u$). It may be a reasonable assumption under the current setting where each agent only has one trajectory but it could raise problems when the number of trajectories of each agent in the dataset is not fixed (then one needs to manually label the trajectories first, which seems to be extra information). Also, the reviewer may overlook but the reviewer does not follow how should the agent label be designed during test time?

limitations_and_societal_impact: The authors have provided a comprehensive list of limitations of their work and the proposed algorithm does not seem to have any negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new offline dataset setting: agents under different dynamics with rare sampling data (1 trajectory in the paper) for each; along with a latent factor decomposition method that motivated from the recommendation system domain. The method is interesting, the experiment of this paper is thorough and the writing is overall good.
main_review: Originality: The task is new as I know. The scenario and the motivating example are convincing and intriguing. The methodology is motivated by the usage of low-dimension latent factors decomposition in the recommendation domain and is interesting to be practiced in the RL scenario. The related works are mostly adequately cited.

Quality: The methodology is overall sound under the given assumption. The experimental results are thorough and seem good but are actually weak under the high-dimensional env (see detailed comments below) and may need more baseline methods. The work is complete, although it has some potential improvements.

Clarity: The writing is overall clear and well organized.

Significance: The problem seems to be new and is important to the application in practice (as the healthcare example that the authors proposed). However, this paper only works on a small range of simulated data. On these simulated environments, the proposed method gets good and robust results. The proposed thinking on the low-rank factor representation in RL is worth further considering.

Detailed comments:

Pro:

1. The problem is new and motivated; the proposed method solves the problem appropriately.
2. The idea for applying low-rank composition in RL is well put in an intuitive scenario.
3. The experiment and the analysis are thorough and the results are better than compared baselines.

Cons:

1. The assumption seems strong and may not be applicable in many real-world scenarios. In the paper the author explained their rationality for these assumptions by showing that it holds for a simple synthetic environment (MoutainCar). Maybe they can put it in a better context. Since all the experiments are conducted under simulated environments where the difference between the dynamics is the covariate, I am not much surprised by the result that they can share a low-rank representation. But, is it sensible for a real-world problem, e.g., the healthcare example?

2. The evaluation results on Halfcheetah is weak, although the proposed method beat all the baselines. Just seeing the last row of Table 2, where the oracle return of Halfcheetah takes a large gap with all compared methods and is less convincing. Also, the author should explain why the results on the last two dynamics (1.7/0.3, 0.3/0.3) are worse than the first (0.3/1.7) while all the environments contribute 1 trajectory to the dataset.

3. The comparison to baseline contains online MBRL and offline MFRL methods. So, why not offline MBRL methods are compared? Yes you mentioned that they are not designed not for this multi-task setting but BCQ is also not designed for it. Also, BCQ is not a good baseline since it has been outperformed by many other algorithms, such as CQL.

Considering the quality of the paper is overall good and the problem is new, I vote for a weak accept in this stage.
limitations_and_societal_impact: Limitations: the most limitation is the assumption of the method which needs further experiments on real-world problems. Also, the results on a high-dimensional environment are not that convincing.

Social Impact: The healthcare example is really attractive but when the latent personalized dynamic is well quantified, inappropriate usage of such models can lead to targeted attacking for specific people. However, these problems should be considered when it is appropriately designed for that application, not in this version.
review_title: null

Meta-review: This paper studies an interesting and challenging offline RL problem, where the agents are heterogeneous and there is only a single trajectory for every agent under a potentially non-optimal policy. The authors propose PerSim, where a personalized simulator is learned for each agent based on the trajectories from all agents, and then a policy is made based on MPC over simulators ensemble.
The technical novelty of the paper lies in the finding that the transition dynamics across agents can be represented as a latent function and thus can be nicely approximated by a low-rank decomposition of the latent functions of agent, state, actions. In general, the contributions of this paper are clear and sufficient.
The majority of reviewers provide acceptance recommendations for this paper. The only reviewer of score 5 raised some questions about the generalization ability and sample complexity of the method, problem setting, and the evaluation, which I think have been well answered with detailed revision plans. Thus I think this paper can be accepted.  
Decision: Accept (Poster)