Offline Adaptive Policy Leaning in Real-World Sequential Recommendation Systems
Authors: Xiong-Hui Chen
Yang Yu
Qingyang Li
Zhiwei Tony Qin
Wenjie Shang
Yiping Meng
Jieping Ye
Venue:  (2021)
Keywords: Reinforcement learning
Recommendation system

Abstract: The training process of RL requires many trial-and-errors that are costly in real-world applications. To avoid the cost, a promising solution is to learn the policy from an offline dataset, e.g., to learn a simulator from the dataset, and train optimal policies in the simulator. By this approach, the quality of policies highly relies on the fidelity of the simulator. Unfortunately, due to the stochasticity and unsteadiness of the real-world and the unavailability of online sampling, the distortion of the simulator is inevitable. In this paper, based on the model learning technique, we propose a new paradigm to learn an RL policy from offline data in the real-world sequential recommendation system (SRS). Instead of increasing the fidelity of models for policy learning, we handle the distortion issue via learning to adapt to diverse simulators generated by the offline dataset. The adaptive policy is suitable to real-world environments where dynamics are changing and have stochasticity in the offline setting. Experiments are conducted in synthetic environments and a real-world ride-hailing platform. The results show that the method overcomes the distortion problem and produces robust recommendations in the unseen real-world.

Under review as a conference paper at ICLR 2021
OFFLINE ADAPTIVE POLICY LEANING IN REAL- WORLD SEQUENTIAL RECOMMENDATION SYSTEMS
Anonymous authors Paper under double-blind review
ABSTRACT
The training process of RL requires many tril--nnd-eroors that are costly in real- world applications. To avoid the cost, a promising solution is to learn the policy from an offline dataset, e.g., to learn a simulator from the dataset, and train optimal policies in the simulator. By this approach, the quality of policies highly relies on the fidelity of the simulator. Unfortunately, due to the stochasticity and unsteadiness of the real-world and the unavailability of online sampling, the distortion of the simulator is inevitable. In this paper. based on the model learning technique, we propose new paradigm to learn RL policy from offline data in the real-world sequential recommendation system (SRS). Instead of increasing the fidelity of models for policy learning, we handle the distortion issue via learning adapt diverse simulators generated by the offline dataset. The adaptive policy is suitable real-world environments where dynamics are changing and have stochasticity in the offline setting. Experiments are conducted in synthetic environments and a real- world ride-hailing platform. The results show that the method overcomes the distortion problem and produces robust recommendations in the unseen real-world.
INTRODUCTION
Recent studies have shown that reinforcement learning (RL) is promising approach for real-world applications. e.g., sequential recommendation systems (SRS) (Wang et al.) 2018:) Zhao et al.) [019
Cai et al. 2017). which make multiple rounds of recommendations for customers and maximize long-term recommendation performance. However, the high trial-and-error costs in the real-world
obstruct further applications of RI. methods (Strehl al. 2006 Levine et 2018).
Offline (batch) RL is to learn policies with static dataset collected by behavior policies without additional interactions with the environment (Levine et al. 2020 (Siegel al.) Wang et al., 2020)] Kumar et al. 2019). Since it avoids costly trial-and-errors in the real environment, offline RL algorithms are promising to cost-sensitive applications (Levine et al. 2020). One scheme of offline RL is learning simulator from the dataset. In this way, RL policies can be learned from the simulator directly. Although prior works on model-based learning have achieved significant efficiency improvements in online RL by learning dynamics models (Kaiser et al. 2020 Wang al.. 2019 Heess et al.; Luo et al., 2019)). building an accurate simulator is still diffccltt, especially in offline RL. In particular, the offline dataset may not cover the whole state-action space, and there is no way for sampling in the real-world to recover the prediction еггоr of the learned simulator. The learned policies tend to exploit regions where insufficient data are available, which causes the instability of policy learning (Kurutach et al. 2018 Zhang et al. 2015 Viereck et al. 2017). By overcoming the problem, recent studies in offline basel RL Yu et al. 2020 2020]]iidmbb Kidambi et al.) (2020) have made significant progress in MuJoCo environments (Todorov et al.) 2012). These methods learn policies with uncertainty penalties. The uncertainty here is function to evaluate the confidence of prediction correctness, which often implemented with ensemble techniques (Lowrey al.) [2119] Osband et al., 2018). By giving large rewards penalty YYu et al.) [020) or trajectory truncation (Kidambi et al. (2020) with large uncertainty dynamics models, policy exploration is constrained in the regions where the uncertainty of model prediction is small, so that to avoid optimizing policy to exploit regions with bad generalization ability.
However, in real-world applications like SRS, several realistic problems of the current offline learning methods are ignored. First, take SRS example, customer behaviors (i.e., the environment)

---

Under review as a conference paper at ICLR 2021
non-stationary and thus change across different periods and locations. Therefore, besides the prediction error induced by model approximation, the transitions in the offline dataset also can be inaccurate in the future (Krueger et al. 2019 Chen et al. 2018 Zhao et al. 2018: Thomas et al. 2017 Li & de Rijke| 2019). Second, different from traditional RI. environments which are overall deterministic (Brockman al.; (0T6), real-world environments often introduce stochasticity. For example, after recommending production to customer, it is hard to model the user's ' decisions without stochasticity (e.g., buying it or not), even with large enough data. Hidden confounder fac- tors (Forney et al. [Bareinboim et al., 2015) Shang et al. (0199) obstruct the deterministic predictions. As result, the uncertainty regions are drastically icreased and thus the exploration of policy learning are obscured.
In this paper, instead of constraining policy exploration in high-confidence regions, we study to handle the offline issue by learning to adapt. We propose an adaptive policy which is trained to make optimal actions efficiently in regions with high confidence for model prediction. While in regions with large uncertainty, the policy is trained to identify the representation of each dynamics model and adapt the optimal decisions on the representation. When deploying the policy in the environment, the policy identifies the dynamics in the real-world through interaction, then adapt its behavior. The module to represent the dynamics models is named environment-come extractor and adaptive policy should be learned in a diverse simulator set and thus can generalize to unknown situations. As a solution, we propose to use model-learning techniques with augmentation approaches to generate simulator set cover real-world situations. In this way, with a sufficiently large simulator set, the learned policy can adapt robustly in unknown real-world environments.
extractor. The
To learn the adaptive policy and environment-contex extractor, we first analyze and formulate the environment context representation problem in SRS. In SRS scenarios, the recommendation platforms interact with customers. Each customer can be regarded as an environment in the view of the RL paradigm. The environments include a two-level structure: in the high-level structure, a recommendation platform serves customers from multiple domains (e.g., different cities and countries). In the low-level structure (i.e., for each domain), there are numerous customers with different behaviors, and the behaviors are dependent on the domain they current in. Although there have been recent interests in learning the representation of environment parameters based on the agent's 's trajectories in the robotics domain (Peng et al., 2018 Akkaya et al. 2019: Zhu et al. 2018) Sadeghi et al.), the ietrromment-conter. representation problem in SRS has never been proposed and the two-level environment structure makes the environment context agnostic based on a single customer's 's trajectory without considering the domain he/she current in. As solution, we use special network to embed the domain information and show that the additional domain information is necessary for representing the environment contexts in SRS.
As the result, we propose Offline learning with Adaptive Policy in sequential Recommendation Sys- tems (OapRS),
new paradigm to solve an offline problem that policies can applied to real-world applications without any additional online sample. By learning to adapt with the representation of dy- namies, OapRS is suitable to real-world scenarios in which environments are non-stationary and have stochasticity. As far we know, this is also the first study on reality-gap and the environment-context representation problem in SRS. We conduct experiments in a reproducible synthetic environment and real-world recommendation scenario: the driver program recommendation system of a ride-hailing platform. Our empirical evaluations demonstrate that OapRS can learn reasonable environment contexts and makes robust recommendations in unseen environments.
2 RELATED WORK
Reinforcement learning (RL) has shown to be a promising approach for real-world sequential recommendation systems (SRS) (Wang et al. 2018 Zhao et al., 2019 Cai et al. 2017) to make optimal recommendations with long-term performance. However, numerous online unconstrained trial-and-errors RL training obstruct the further applications of RL in "safety critical" SRS scenario since it may result in large economic losses (Levine et al. [020 Gilotte et al.) Theocharous et al.) 2015 Thomas et al. (2117). Many studies propose to overcome the problem by offline (batch) RL (Lange al. 2012)).
Most prior works on offline RL introduce model-free algorithms. To overcome the extrapolation error, which is introduced by the mismatch between the offline dataset and true state-action oceu- pancy (Wang et al., 2020), these methods are designed to constrain the target policy to be close to the behavior policies (Wang et al. 2020 Kumar al. 2019 Wu et al. 2019). apply ensemble
2

---

Under review as a conference paper at ICLR 2021
Consitrzzy region Leamell polisics
(a) domain randomization
(b) OapRS
(c) consistency region penalty
Figure 1: Illustration of the OapRS compared with other methods. The line denotes the optimal trajectory of the learned policies. There are several policies in domain randomization and OapRS since the methods learn to adapt in multiple simulators. The gray oval denotes the consistency region.
methods for robust value function estimation (Agarwal et al. 2020 re-weight with importance sampling (Liu et al., (019). Most recent studies have shown that learning robust policies from the approximation of dynamics models has the potential to take actions outside the distribution on behavior policies and thus can converge to better policies (Kidambi et al. 2020: Yu et al.) 2020). To remedy the extrapolation error, these methods learn policies from dynamics models with uncertainty penalty. In particular, Kidambi et al. (2020) constructs terminating states based on a hard threshold on uncertainty, while Yu et al. (2020) uses soft reward penalty to incorporate uncertainty. The uncertainty is often computed via the inconsistency of the ensemble dynamics model predictions on each state-action pair to evaluate the confidence predictions on next states. The penalty constrains policy exploration and optimization in the regions with high consistency for better lower bound performance in the deployment environment (Kddmmi et al. 2020 Yu et al. 2020). However, in SRS, the uncertainty estimation via the inconsistency will introduce two unexpected factors: (1) inconsistency of predictions may come from the non non-stationary of the environment. (2) the oracle dynamics model itself may have high uncertainty (comes from the stochastic of the environment). As a result, uncertainty estimation with these compounding factors may constrain the policy to explore in small region and thus hard to optimize.
An alternative to learning from dynamics model and deploying in real-world environments is domain randomization techniques, which is popular solution in the robotic domain (Tobin et al. 2017]] Sadeghi & Levine 2017 Tobin et al. 2018 James et al. 2019: Peng et al., |018]|Akaa) AAkaay et al. 2019 Nagabandi al.). In general, the framework trains a policy in numerous simulators, and then
deploys it to unknown target domains directly. The numerous simulators are constructed by sampling different environment parameters, which are modeled by human experts with laws of physics in the real-world. The algorithms can train a uniform policy (Tobin et al.) 2017] [Sadeghi & Levine 2017] Tobin et al., 2018 James et al., 2019) to maximize the expectation of the long-term rewards of the simulators, or train adaptive policy with online system identification (OSI) techniques (Peng et al., 2018 Akkaya et al. 2019 Yu et al.) to extract the representation of the dynamics of the simulators and maximize the performance metric of each simulator. In this paper, we adopt similar idea of OSI to construct the sitiromment-prammettt
extractor and the adaptive policy. However, in the offline setting, generating simulator set by random sampling directly from the parameter space is not only hard since we do not have the exact knowledge of representation of environment parameters to build the simulator, but inefficient because we only have access to the offline dataset to estimate the real-world dynamics.
The difference between the aforementioned methods and OapRS is shown in Figure Compared with previous methods learned by uncertainty penalty (Kidambi et al., 2020 Yu et al.) 2020) to constrain policy learning (in Figure [(c]). we learn to adapt decisions for all possible dynamics transitions (which are learned by ensemble) in the states out of the consistency regions (in Figure [(b).. While learning in the entire dynamics space an ideal solution to learn a robust policy (in Figure [(a).. for less training cost and better efficiency, the dynamics set is generated by an offline dataset and model-lcarning algorithms with ensemble methods. In this way, we construct a consistency region implicitly in which optimal trajectories are similar for each simulator, while the domain randomization learns to adapt in the whole space without considering the knowledge of the offline dataset.
In SRS, the non- con-stationarity and stochasticity problems have been studied recently. On one hand, Zhao et al. (2018); Chen et al. (2019) propose to handle the non -otttoatinn problem by optimizing the policy with offline data directly and correct the policy by learning with additional online feedback
3

---

Under review as a conference paper at ICLR 2021
data from customers. Our work focus on learning an adaptive policy from offline data directly, which is important for safety critical applications. On the other hand, the stochasticity problem can be modeled with confounder factors. Recent works show policy can make better decisions considering hidden confounder factors when optimizing policy (Shang et al.] (019 Forney et al.] [Bareinboim] et al., 2015). In this work, consider stochasticity by learning dynamics models with hidden confounder factors (Shang et al. 2019).
3 BACKGROUND AND NOTATIONS
In the standard RL framework, an agent interacts with an environment described as a Markov Decision Process (MDP) (Sutton & Barto 2018). The agent learns a policy π (To]ss)). which chooses action a E A conditional on a particular ar state s; E S, at each time-step t E {0, (0.1, T). S and A denote the state and action spaces, respectively. The reward function rt -((s,aa)) E R. evaluates the immediate performance of action ae given state &t. The goal of RL is to find an optimal policy x², which maximizes the multi-step cumulative discounted reward (i.e., long-term performance). The where denotes the discount
objective is to maximize JT(") max; Brrx(rrin,e)) 30==0"ree
factor. p( [ p) is the probability of generating trajectory := [so. ao..... -1 sT] under the policy π and a transition dynamics p($:41 (st, at), where 'I' denotes the trajectory length. In particular, P(T | T) := (u(ss) IT= p(s Se. )2) (St, at), where (u(Ss) is the initial state distribution. A common way to find optimal policy nT" is by optimizing the policy parameters with gradient ascent along (T)) SRS problem can be naturally formulated as an MDP. In general, the recommendation system is policy æ, while the environment consists of numerous human eustomers. During the learning process, the policy interacts with customers to execute the recommended action o, based on the customer state St.
In the offline RL problem, we assume that we are only given a static dataset D = {(s,, dg 8:4 I) j==1 collected by some unknown policies. The reward function r given by human experts. The goal is to output sub-optimal policy π² to maximize Jp with the static dataset.
SRS problem can be naturally formulated as an MDP. In general, the recommendation system is the policy æ, while customers can be regarded as the environment p. During the learning process, the policy interacts with customers to execute the recommended action az based on the customer state st A set of recommendation business metrics define the immediate performance T' based on customers' feedback. this work, we consider the non-stationarity of environment w.r.t time periods, which is common in SRS applications (Thomas et al. (017). We demonstrate the phenomenon in Figure 14 We use subscript Pt to denote the dynamics model at time-step t so that the dynamics may be different in different timesteps. We also model the stochasticity with hidden confounder (Shang) et al. 2019). In particular, there is a hidden policy n. for each timestep & to output confounder factors: xzf" n.. (a: [st, at). The dynamics model output the next state based on 8t, d and the confounder x.H that is 8:41 = Pr(#,, a,, HH) ). For the notation simplification, the following we still use s' pr(s, a) to denote the dynamics model with hidden confounder factors.
4 OAPRS: OFFLINE LEARNING WITH ADAPTIVE POLICY IN SEQUENTIAL RECOMMENDATION SYSTEM
In the offline setting, the learned policies tend to exploit regions where the model is distorted since insufficient data are available Kurutach et al. (2018). Prior offline model-based techniques have shown great potential in learning better policy without strict constraints in action selections (Yu et al. 2020 Kidambi et al. 2020). However, the robust offline policy learning in real-world applications is still challenging for following reasons: 1) In real-world applications like SRS, environments are non-stationary, thus, besides model learning error, the transition information of the dataset can be distorted too; 2) It is hard to predict the future states with high determinacy since there many hidden confounders in SRS (Shang et al. 2019). Since the environment has stochasticity, learning by consistency penalties will obstruct the policy into small region for exploration.
To output a robust policy in stochastic environments, we first study to handle the distortion problem without relying the consistency penalty and propose the learn to adapt framwork (in section 4.1). Then we formulate and analyze the environment-come an environment-comext extractor and adaptive policy optimization method (in section 4.2. Finally,
representation problem in SRS and propose
4

---

Under review as a conference paper at ICLR 2021
discuss important practical details to improve the robustness of simulators and policy in offline dataset (in section 4.3)
4.1 OFFLINE ADAPTIVE POLICY LEARNING
In this work, we assume that given a specific reward function v, any dynamics of environment E T can be fully characterized with enviromment-context vector 2 Z (c.g., the friction coefficient in robotics), where T denotes a set of dynamics and Z denotes the space of the context vectors. Formally, there is mapping function : T 2. We call 6 an environment- context extractor. We define the optimal environment-context extractor o" the one that satisfies: Omg- E II, Vp (T max, (T), where म := (a(a (t)) sx) is an adaptive policy and II denotes the policy class. We discuss the input of Φ in the next section. Besides, we define the optimal adaptive policy T the one that satisfies V/ T, (T = max; (T).. With the optimal o² and - given any p in 'T, the adaptive policy can adapt to make the best decisions via the output of environment-contexxt To achieve this, given dynamics model set T, we optimize e and n by the following objective function:
6" = arg max PTT (J)( 0,
where ~ denotes a sample strategy to draw dynamics models p from the dynamics set T w.r.t P[p] >> 0, Vp E T. The strategy we used in OapRS is uniform sampling. When deployed in the real-world p", OapRS infers the environment contexts by o" (ot) and makes the best action by Tz... If all the variant of dynamics models in the real-world T, fall into T, we can deploy the policy robustly via the optimal d² and T
To learn the robust o² and T.. for real-world deployment, the crucial problem is to construct a set of dynamics {p} to cover the real-world dynamics set T,- One ideal solution is to construct all possible dynamics model in S x A x S (or all possible context vectors in 2 and the corresponding simulators when we have the knowledge of the environment context) and optimize e and via equation (Tobin
iet al.
2017
Sadeghi & Levine| 2017: Tobin et al. 2018 James et al.) 2019 Peng et al. 20188 Akkaya
ct al.
2019 However, in real-world applications like SRS, we have no exact knowledge to formulate
customer behavior, and thus space 2 is unknown. Besides, since the real-world fall in a subspace of the dynamics space, constructing all possible dynamics model directly costly and inefficient for policy learning. We propose to regard model-learning algorithms G an efficient generator to map dataset to a dynamics model where the latent enviromment-comtee (although does not match exactly). Then we construct a diverse dynamics set by augmenting on G and D. In particular, for augmenting on D. we divide the offline dataset by the domain knowledge (e.g., can split the dataset by city, country, or time period in production recommendation systems); for augmenting on G, we can select several model-learning algorithms or different hypee-parameters for the same algorithm (Kurutach al. 2018) to generate simulators.
is near the real-world: G(D) -
We list the OapRS framework in Algorithm|] ! The framework is robust to the distortion of simulators since our goal is to generate dynamics model in which the environment contexts are near the distribution of the real-world instead of recovering the real-world dynamics exactly. The method is also robust to stochastic environments since learns to adapt to all possible situations in the state out of the consistency region instead of doing penalty in the state with multiple situations. (as illustrated in Fib) Additionally, we point out that the framework is orthogonal to other techniques since the reliable model- methods can reduce the requirement of dynamics augmentation to cover the real-world parameter space.
4.2 OPTIMIZATION OF ADAPTIVE POLICY WITH ENVIRONMENTICONTEX" EXTRACTOR
In this section, we formulate the invironment-contes representation problem in SRS. We first define an environment-context dependent dynamics as follows: p(s a,, a,,2), 21), where is the environment-context vector. Before learning the representation 2 by Φ(2 (2)))) the question is: What is an suitable input to & for representation learning.
In the SRS scenario, the recommendation policy interacts with customers in multiple domains. Each domain consists of numerous customers. Taking the ride-hailing platform as example, the platform
5

---

Under review as a conference paper at ICLR 2021
Algorithm 1 OapRS framework
Input: d, as environment-contee extractor, parameterized by yp; Adaptive policy network I&
parameterized by 0; Offline expert data D; Model learning algorithm G.; Rollout horizon
mminibathh m for policy training
Process:
divide offline dataset into N partitions {D) and generate M x N dynamics model {p} by ensemble of M model learning technique G for 1, 3, ... do random select the dynamics model p² in {o} and sample 81 from D which is used to train p initialize buffer D Pollout
for f=1,2....,II do
sample ,, from d<z)pp) and then sample a, from π [ar 24)
rollout one step $141, T++1 (s,,, ai) and then add ($1+1, St, Qe, 21) to Droilout
if mod m == 0 then
Update da and with equation [ by policy gradient (e.g., PPO) and then Drotlout t 0
end if end for end for
provides services in multiple cities (i.e., domains), and interacts with numerous drivers in each city (i.e., customers). To train a platform recommendation policy for customers, we regard each customer as an environment Different from traditional agent controlling, in SRS, customer's behavior is not only dependent on the character, but the domain he/she belongs to. For instance, passengers' demand will be very different with time and cities geographically. Drivers in different cities might have different engagement (i.e., online time) due different demand and supply, which independent of their personas. Drivers' behavior will also change when the domain he/she belongs to changes. Based on the analysis, we propose that the input á should include personas information besides domain information.
In the robotics domain, environment representation approaches, called system identification, have been proposed recently (Peng et al. 2018 Akkaya al. 2019; Yu et al.). The policy incorporates an online system identification module o Ee St, To:)). which utilizes a history of past states and actions To:t [Su, ag ..., St-1; 0t-1, sa] to predict the parameters of the dynamics. Since p in SRS is also dependent on domain information, we follow the system identification technique and add the domain trajectory information To. = [So, Ag, S1, Ai,..,, St] to the input. That is: 2
N ((tt, a;-1; St, A,-1, 2t-1), where (Sy AQ_ AQ_1) == [(s, i.d {)) which includes state-action pairs at each time-step f. We use superscript d to denote the domain type.
Since the number of customers N can be large and vary from timestep to timestep, is impractical to feed (S A(_1) ) to the neural network directly. In this work, we propose State-Action Distributional variational AutoEncoder (SA-DAE) to solve this problem by inferring the latent embedding v of X. In the rest of this article, process based on the encoded variable u based on the two assumptions: First, we assume state-action pairs in X are i.i.d. sampled from distribution Per(S (s,a), a), which is parameterized by vigf for each time-step and domain d. Second, we also assume that the parameters of the distribution generated by a distribution pe(v)l)), parameterized by 0. It involves a latent continuous random variable v. which is generated from another prior distribution p(v). Then the generation of X includes three steps: (1) sample v from p(u); (2) sample 16 from distribution po (v/w) ; (3) sample pel (s, a) repeatedly to generate X.
use X == (Sf AS to for brevity. We first design the data generative
Our target is to learn an embedding model (v) X) parameterized by k, aligned with the posterior ap- proximation Po (vlX). Using Kullback-Leibler Divergence (KLD) as the measurement, the objective can be written as follows:
min Lane = min Ex-Dx [KLD (g.(v) [po(o) X))],
(2)
where the dataset Dx= (Xf d 9 < T} includes state-action pairs in all time-steps 0 < t < T and domains g. and the posterior (u X) the target distribution of gu (x)x). Under the assumption of i.i.d. sampling on X, the probabilities of ge(vv and po (X (w:) can be estimated

---

Under review as a conference paper at ICLR 2021
via likelihood:
(x(x|X) II
Uu (v/s(0) .a(1)
(3)
N
N II Pv (s"), pa po(())))
po(
(s())
a
(4)
where y denotes the parameters of distribution po- We give our theorem on the tractable evidence lower bound (ELBO) in Theorem[[] We leave the proofs in the Appendix
Theorem
The tractable ELBO of state-action distributional variational inference is:
[ (xl()
Σ log Pe (s(1l) llog po (al))u, (II (5)
Ex=Dx
Theorem give us a three-step pipeline to minimize the objective of equation 2 (1) sample batch of X = {ss a)} from expert dataset Dx; (2) infer latent code v via the likelihood probability of equation 3 (3) compute the reconstructed log-prot org-robbability of state-action pairs based equation and KL divergence between posterior and prior of v, and then apply the gradient to k. and
Finally, the function Gx embed the distribution of customer trajectory To: into the latent code trajectory Vy:-- After that, the extractor & can infer environment-contex both with and т. Then adaptive policy (a,) 2)) samples action based on 24.
4.3 ROBUST OFFLINE POLICY AND MODEL LEARNING IMPLEMENTATION
The dynamics model and policy are high-dimensional and complex to learn. We introduce three important implementation techniques to handle these problems (more details can be found in the Appendix F): (1) Semi dynamics model learning: To reduce the model learning complexity, with domain expert knowledge, we learn the n (8)s, a) and use hand-coded function map(s' [8, 8,a) to construct the left of the states, instead of learning predict s' directly; (2) Confidence penalty: To reduce the cost of dynamics model construction, we send policy with large negative reward when policy reach the unreal regions. The unreal region is judged by expert rule. (3) Domain reward normalization: Reward normalization is important for model-free RL algorithms (Engstrom et al.). However, different simulators may have different reward scales. Instead of rescaling the reward with a uniform scale, we rescale the reward for each domain.
EXPERIMENTS
In this work, we focus on the three research questions: (1) In SRS scenarios, can the adaptive policy of OapRS make more robust decisions than other learn-to-adapt methods do? (2) By adapting to the learned representation of the dynamics, can OapRS handle the distortion of the constructed dynamics models? (3) Does non-stcadiness and stochasticity in real-world applications actually affect the performance of offline learning? If so, does OapRS overcome this challenge? We conduct experiments in a synthetic environment and a real-world environment to answer these questions.
5.1 RESULTS IN SYNTHETIC ENVIRONMENT
To analyze the adaptability of OapRS and its robustness to distortion, we first conduct the experiments in synthetic environment in which the environment parameters z are configurable. The environment called the long-term eustomer satisfaction (LTS) problem, a sequential recommendation environ- ment comes from Google RecSim (le et al. 2019). Instead of constructing dynamics set by learning with augmentation (line in Algorithm l). for analyzing the results reasonably, we construct the training dynamics set by selecting directly, and control the difference of between the training set and the target domain to design different tasks. In particular, we construct the target dynamics with sest and select the training dynamics set by equidistant sampling parameters from the space and removing those I|2 stest] a. The details of environment and experiment setup of the following
can be found in Appendix B.1
7

---

Under review as a conference paper at ICLR 2021
We first test the adaptability of OapRS in SRS. The algorithms compared are (1) DR-UNI: The
domain randomization technique to learn a uniform policy ((Tobin et al. (017)); (2) DR-OSI: An
online system identification algorithm proposed by Peng al. (2018); ; (3) DIRECT: Train in the dynamics model where z nearest to the target domain and deploy directly. (4) Upper Bound: A policy trained in the target domain directly. We regarded three tasks: "LTSI", "LTS2" and "LTS3". The numbers after "LTS" denote the removing scale a. We report our results in Figure 2 First, the results DIRECT show that the performance degradation is severe in the tasks. Without considering the difference between training and deploying, the policy generates unpredictable behaviors. Second, all algorithms which consider learning from multiple dynamic models can improve the robustness in unknown domains. However, the algorithms that adapt by the representation of environments (OapRS and DR-OSI) reach better performance since they try to find the optimal policy in the representation of the environment instead of maximizing the expected performance in the training set. In addition, OapRS reaches the near-optimal performance and does better than DR-OSI in difficult tasks (c.g., LTS3). The results demonstrate that without considering the domain information, the representation can not be extracted well. more difficult tasks, the limitation of the representation ability restricts the performance of the adaptive policy. We show more results about the SA-DAE and the extractor in Appendix C
the upper bound performance. We construct
144
Dept5
304
OBECT
DRDECT
Upper Bound
EX6 025
190
0/8 439
0.5
100
199 1.5
219
406
850
3.
3.9
1.3 375 2m
(a) LTSI
(b) LTS2
(c) LTS3
Figure 2: Illustration of the performance in synthetic environments. The solid curves the mean reward and the shadow is the standard error of three seeds.
We then analyze the influence of dynamics distortion on OapRS. To mimic the process of offline model-based learning, we follow the previous experiment setting at "LTS3", but divide z into two part: ze and =ą- In particular, ze denotes the parameters that the behavior can be learned well by dynamics model, which are equidistant sampled as before. And zd denotes the distortion parameters, which are the same in all dynamic models if distortion does not exist. We generate the distorted dynamics parameter 2d by Ed ~ 2d + U--3, (3) for training, where 3 can be regarded as the level of distortion. Figure 3 shows the performance of OapRS in this setting. We can see in Figure (aa) that the deployment performance of OapRS with limited training set declines when the distortion level becomes larger, but the performance is still better than the compared methods. However, in Figure (bb we found that with enough sampled simulators, OapRS can overcome the distortion problem well. As a conclusion, in empirically, by augmenting enough diverse dynamics models, is possible to overcome the distortion problem via OapRS.
060
Urrestsp
dme-sap
(a) limited simulators
(b) unlimited simulators
Figure 3: Illustration of the performance in distored synthetic environments. The solid curves are the mean reward and the shadow is the standard error of three seeds. In limited setting, sample 3 for each simulator and fix it. In unlimited setting, we sample 33 for each simulator and each episode.
8

---

Under review as a conference paper at ICLR 2021
5.2 EXPERIMENTS IN WORLD APPLICATION
We finally test OapRS in the driver program recommenaation (DPR) of large-scale ride- ride-hailing platform in the real world (the detail of task setting can be seen Appendix B.2). The offline data comes from human policy. We construct the dynamics model set by DEMER (Shang et al.) 2019) for its ability to learn hidden confounders. For dynamies model augmentation, we divide the dataset into 34 partitions by cities and time periods. For better comparison of algorithms and demonstrate the offline learning performance in stochastic environments, we first conduct a semi-online experiment: We leave one city in the dynamics set out for policy training, and test the policy performance in the unseen city. We select three different cities for these experiments. The algorithms we compare are: (1) MOPO: an offline model-based algorithm by consistency penalties. The dynamics models also trained in the same way as OapRS Yuu et al. 2020); (2) BCQ: an offline model-free algorithm by constraining actions (Wang et al. 2020 (3) DEMER-policy train policy in one single dynamics model without additional constrains but select the part of dataset nearest to the target environments to learn the dynamics model. Here we select the dataset collected in the deployment city and the nearest time period to train the dynamics model (note: the dataset are unseen to other algorithms). It is the policy learning method in the original paper of DEMER (Shang et al. 2019). Figure ((a) Figure (bbb) and Figure [(c) show the results. We finally deploy the policy trained by MOPO and OapRS to the real-world and test the performance for days. The results is shown in [d) We deploy the policy from day 23 to day 29, Before deployment, drivers are recommended with the same human expert policy. We found that the performance improvement of the MOPO policy is 0.1%, which is similar the expert policy, while the improvement of OapRS is 6.9%, which is significantly better than the expert. The details of the results can be seen in Table
Gurd
Sreeto
(a) City A (semi-online) (b) City B (semi-online)
(c) City C (semi-online)
(d) City A online
Figure 4: Illustration of the performance in semi-online and online tests. In the semi-online setting, the Y-axis is the averaged long-term reward among the users in the environments. In the online test, the Y-axis is the averaged daily reward. The rewards of online data are rescaled.
6 DISCUSSION
In this paper, propose paradigm of model-based learning to solve the offline RL in real- world applications. We focus on two realistic problems in real-world SRS applications: (I) real-world environments are non-stationary; (2) real- world environments are often with stochasticity. Instead of learning by penalty, constraint or randomization, OapRS solves the problem by learning to adapt to diverse simulators generated by the offline dataset. We formulate and analyze the environment- parameter representation learning problem in SRS and optimize the adaptive policy with the learned environment representation. The experiment results show the adaptability of OapRS, the performance decline of offline RL algorithms faced with the two realistic problems, and how OapRS work well in this situation. Although we focus on the real-world application of SRS, we consider that OapRS can be a general paradigm to solve the offline RL problem in other complex environments.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. International Conference on Machine Learning, 2020.
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik's cube with robot hand. CoRR, abs/1910.07113, 2019.

---

Under review as a conference paper at ICLR 2021
Elias Bareinboim, Andrew Forney, and Judea Pearl. Bandits with unobserved confounders: A causal approach. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28: Annual Conference Neural Information Processing Systems 2015, pp. 1342-1350, Montreal, Canada, 2015.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abss1660.01144 2016.
Han Cai, Kan Ren, Weinan Zhang, Kleanthis Malialis, Jun Wang, Yong Yu, and Defeng Guo. Real- time bidding by reinforcement learning in display advertising. In Proceedings of the 10th. ACM International Conference Web Search and Data Mining, Cambridge, United Kingdom, 2017.
Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H. Chi. Top-k off-policy correction for a REINFORCE recommender system. In J. Shane Culpepper, Alistair Moffat, Paul N. Bennett, and Kristina Lerman (eds.). Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM 2019, Melbourne, VIC, Australia, February 11-15, 2019, pp. 456-464 464. ACM, 2019.
Shi-Yong Chen, Yang Yu, Qing Da, Jun Tan, Hai-Kuan Huang, and Hai-Hong Tang. Stabilizing reinforcement learning in dynamic environment with application to online recommendation. Yike Guo and Faisal Farooq (eds.). Proceedings of the 24th. ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1187-1196, London, UK, 2018. ACM.
Logan Engstrom, Andrew llaas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep RL: case study on PPO and TRPO. In Proceeding of 8th. International Conference on Learming Representations, Addis, Ethiopia.
Andrew Forney, Judea Pearl, and Elias Bareinboim. Counterfactual data-fusion for online reinforce- ment learners. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th. International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1156-1164, Sydney, Australia. PMLR.
Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé, Offline A/B testing for recommender systems. In Yi Chang, Chengxiang Zhai, Yan Liu, and Yoelle Maarek (eds.). Proceedings of the Eleventh ACM International Conference Web Search and Data Mining, pp. 198-206. ACM.
Gene H. Golub and Charles F. Van Loan. Matrix Computations (3rd Ed.). Johns Hopkins University Press, USA, 1996. ISBN 0801854148.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th. International Conference Machine Learning, pp. 1856-1865, Stockholmsmässsan, Sweden, 2018.
Nicolas Heess, Gregory Wayne, David Silver, Timothy P. Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, Montreal, Canada.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9()): 1735-1788 1997.
Eugene Ie, Chih- Wei Hsu, Martin Mladenov, Vihan Jain, Sanmit Narvekar, Jing Wang. Rui Wu, and Craig Boutilier. RecSim: A configurable simulation platform for recommender systems. CoRR, Abs//199.00447,, 2019.
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-Real via Sim-to-Simm SSim Data- efficient robotic grasping via randomized to to-canonical adaptation networks. In 2019 IEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 12627-12637, Long Beach, CA, 2019.
10

---

Under review as a conference paper at ICLR 2021
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model based reinforcement learning for Atari. In Proceeeding of the 8th. International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model- based offline reinforcement learning. CoRR, 0an/2005500951, 2020.
David Krueger, Tegan Maharaj. Shane Legg, and Jan Leike. Misleading meta-objectives and hidden incentives for distributional shift. Safe Machine Learning workshop at ICLR, 2019.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer. Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, Vancouver, Canada, 2019.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In Pruceeding of 6th. International Conference on Learning Representations, Vancouver, Canada, 2018.
Sascha Lange, Thomas Gabel, and Martin A. Riedmiller. Batch reinforcement learning. In Marco A. Wiering and Martijn van Otterlo (eds.), Reinforcement Learning, volume 12 of Adaptation, Learn- ing, and Optimization, pp. 45-73. Springer, 2012.
Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand- eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research, 37(4-5):421444 4):624 2018.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement Icarning: Tutorial, review, and perspectives open problems. CoRR, abs/2005.01 646,,,,,0.6,,,,,,, 2020.
Chang Li and Maarten de Rijke. Cascading non-stationary bandits:
Online learning rank in the non-stationary cascade model. In Sarit Kraus (cd.). Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pp. 2859-2865, Macao, China, 2019.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with state distribution correction. CoRR, hixx/194.004777 2019.
Kendall Lowrey, Aravind Rajeswaran, Sham M. Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model- based control. In Proceeding of 7th. International Conference on Learning Representation, LA,, USA, 2019.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-basee deep reinforcement learning with theoretical guarantees. In Proceeding of 7th. International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, New Orleans, IA,, 2019.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through reta-reiiforrement learning. In Proceeding of 7th. International Conference Learning Representations, New Orleans, LA.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- -iinchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 201, Montréal, Canada, pp- Montréal, Canada, 2018
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-Real transfer of robotic control with dynamics randomization. In Proceedings of the 35th. IEEE International Conference Robotics and Automation, pp. 1-8, Brisbane, Australia, 2018
11

---

Under review as a conference paper at ICLR 2021
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy metu-reinforcement learning via probabilistic context variables. In Proceedings of the 36th. International Conference Machine Learning, pp- 5331-5340, Long Beach, CA, 2019.
Murray Rosenblatt. Remarks on some nonparametric estimates of a density function. Annals of Mathematical Statistics, 27(3):832-837, 09 1956.
Fereshteh Sadeghi and Sergey Levine. CAD2RL: Real single-image flight without single real image. In Proceedings of the 13rd. Roboties: Science and Systems, Massachuseus Institute of Technology, Cambridge, MA, 2017.
Fereshteh Sadeghi, Alexander Toshev, Eric Jang, and Sergey Levine. Sim2Real viewpoint invariant visual servoing by recurrent control. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd. International Conference on Machine Learning. pp. 1889-1897, Lille, France, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs//107.006 003477777777... 2017.
Wenjie Shang, Yang Yu, Qingyang Li, Zhiwei Qin, Yiping Meng, and Jieping Ye. Environment reconstruction with hidden confounders for reinforcement learning based recommendation. In Proceedings of the 25th. ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 566-576, Anchorage. AK, 2019.
Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin A. Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In Proceeding of the 8th. International Conference Learning Representations, Addis Ababa, Ethiopia.
Alexander L. Strehl, Lihong Li, Eric Wiewiora, John Langford, and Michael L. Littman. PAC model-free reinforcement learning. In Machine Learning, Proceedings of the 23rd. International Conference, pp. 881-888, Pennsylvania, PA, 2006.
Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018
Georgios Theocharous, Philip S. Thomas, and Mohammad Ghavamzadeh. Personalized ad rec- ommendation systems for life-time value optimization with guarantees. In Qiang Yang and Michael J. Wooldridge (eds.), Proceedings of the 25th. International Joint Conference on Artificial Intelligence, pp. 1806-1812, Buenos Aires, Argentina, 2015. AAAI Press.
Philip Thomas, Georgios Theocharous, Mohammad Ghavamzadeh, Ishan Durugkar, and Emma Brunskill. Predictive off- policy evaluation for nonstationary decision problems, with applications to digital marketing. In Satinder P. Singh and Shaul Markovitch (eds.), Proceedings of the 3 /tt. AAAI Conference on Artificial Intelligence, pp. 4740 4745, CA, USA, 2017. AAAI Press.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Picter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In Proceed- ings of the 29th. IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 23-30, Vancouver, Canada, 2017.
Josh Tobin, Lukas Biewald, Rocky Duan, Marcin Andrychowicz, Ankur Handa, Vikash Kumar, Bob McGrew, Alex Ray, Jonas Schneider, Peter Welinder, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization and generative models for robotic grasping. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3482-3489, Madrid, Spain, 2018
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model- based control. In Proceedings of the 24th. IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, Vilamoura, Portugal, 2012.
12

---

Under review as a conference paper at ICLR 2021
Ulrich Viereck, Andreas ten Pas, Kate Saenko, and Robert Platt Jr. Learning visuomotor controller for real world robotic grasping using simulated depth images. In Proceedings of the 1st. Annual Conference Robot Learning, pp. 291-300, Mountain View, 2017.1.
Chengwei Wang, Tengfei Zhou, Chen Chen, Tianlei Hu, and Gang Chen. Off-policy recommendation System without exploration. In Hady W. Lauw, Raymond Chi-Wing Wong, Alexandros Ntoulas, Ee-Peng Lim, See-Kiong Ng, and Sinno Jialin Pan (eds.), Advances Knowledge Discovery and Data Mining 24th Pacific-Aste Conference, Singapore, 2020.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. CoRR, 0ass/1977.000S5 2019.
Xiting Wang, Yiru Chen, Jie Yang, Le Wu, Zhengtao Wu, and Xing Xie. A reinforcement learning framework for explainable recommendation. In 2018 IEEE International Conference on Data Mining, pp. 587-596. IEEE, 2018.
Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52, 1987.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. CoRR, abs/1911.1113.1 2019.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model- offline policy optimization. CoRR, abs/2005. 13239, 2020.
Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. Preparing for the unknown: Learning universal policy with online system identification. In Nancy M. Amato, Siddhartha S. Srinivasa, Nora Ayanian, and Scott Kuindersma (eds.), Robotics: Science and Systems XIII, Massachusetts Institute of Technology,... July 12-16, 2017 Cambridge, MA.
Fangyi Zhang, Jürgen Leitner, Michael Milford, Ben Uperoft, and Peter I. Corke. Towards vision- based deep reinforcement learning for robotic motion control. CoRR, 1..0991,, 2015.
Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin. Recommenda- tions with negative feedback via pairwise deep reinforcement learning. In Proceedings of the 24th. ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.
Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. Deep rein- forcement learning for page-wise recommendations. Sole Pera, Michael D. Ekstrand, Xavier Amatriain, and John O'Donovan (eds.), Proceedings of the 12th. ACM Conference on Recom- mender Systems, pp. 103, Vancouver, BC, Canada, 2018. ACM.
Xiangyu Zhao, Changsheng Gu, Haoshenglun Zhang, Xiaobing Liu, Xiwang Yang, and Jiliang Tang. Deep reinforcement learning for online advertising in recommender systems. CoRR, Abs/1909.03602. 2019.
Yuke Zhu, Ziyu Wang. Josh Merel, Andrei A. Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, and Nicolas Heess. Reinforcement and imitation learning for diverse visuomotor skills. In Robotics: Science and Systems XIV, Pittsburgh, PA, 2018.
13

---

Under review as a conference paper at ICLR 2021
A PROOF OF SA-DAE
design the data generative process based on the encoded variable with two assumptions: First, we assume that state-action pairs in X are i.i.d. sampled from distribution p& p3(5, a), which is parameterized by v. Second, we also assume that the distribution parameters &p are generated by distribution pe(1 (v/lr), parameterized by 0. It involves latent continuous random variable u, which generated from another prior distribution p(v). Then, the generation of X includes three steps: 1) sample from p(u); 2) sample & from distribution po(v))) 3) sample (s a) repeatedly to generate X. A comparison with VAE on directed graphical model is shown in Figure 5
VAE
SA-DAE
K
0
K
X
M
Inferring process Generating process
Figure 5: Comparison of SA-DAE and vanilla VAE through the directed graphical model. The circles denote the variable nodes. The rounded rectangle denotes the dataset nodes, in which the notation in the corner denotes the number of datasets. 0 denotes the approximation parameters of the generative model, and denotes the parameters of the variational approximation model.
Our target is learn embedding model qu (vl X) parameterized by which aligned with the posterior (X) Using KI. divergence (KID)) as the measure of distribution, the objective function can be written follows:
min Lend min Ex-Dx [KLD (1x(())))l)))(((x))
where the dataset Dx = (x4 e 9,0 < < 7} includes state-action pairs in all time-steps 0 T and domains 9, and the posterior X)) is the target distribution of (u|X). We provide the evidence lower bound (ELBO) of Equation equation in Lemma]
Lemma The ELBO of state-action distributional variational inference is:
ELBO Ex [xx(v [log Po(())) - KLD (g. ({xxx))))po(())
Proof A.1
KLD(ga (cX))]]p))
-By,(o(X) log (v)x))
po (v) X)
-Ene(v(xx) [log (x) log (v, + pog)) L(O, k;; X) + log (X)),
8
Since log Pa (X) is independent of (ulX), minimizing Equation equation is equivalent to maximize T.(0, Hc, X) in Equation equation 8
14

---

Under review as a conference paper at ICLR 2021
Based Bayes's 's theorem, have:
L(0,ng n; X) =B (x/ 9x (u|X) + log po
ON
BB log ON (u|X) + log (po (X(o)p((o))
log po(v) + log logpo(X(l))
(x/x())
9r. (x(
(v) (v) x) [log Pa (Xl)) KLD (qx (p)))))
Under the assumption that X is i.i.d. sampled from Dx, we obtain the evidence lower bound (ELBO)
objective:
max Ex-Dx (gl()) [log po (logoo(Xo)) (X KLD (g (()(())))
Theorem The tractable ELBO of state-action distributional variational inference is:
Σ logpe pe (s1)1) log pe (60)),,(00) (KL(
Ex-Dx (vlx))
(9)
Proof A.2 Under the assumption of i.i.d. sampling X, the probability of qx and pa (Xle)
can be estimated via likelihood;
(x(v|X)
II
(ls(() (uls(0,,(()
(a())
(10)
i=1
Po
Po
(()) a 10)0
II Pv (s") i=1
(o) po (o(l))
(11)
where di denotes the parameters of distribution Po-
In the reinforcement learning scenario, the action sampled conditionally on the state, thus the posterior can be separated by:
(0)
a (10
P9
1 (0,)) po (510)2)
Pa
(a po (Valu, slt) ( pv. (s()) (Welu),
-Py
(12)
where tha and tha denote the decoded parameters of the distribution. Then the ELBO objective can be written as:
Ex-Dx
E (x)x)
(Ogg)(X(()]
KLD (g. (vlx(xX))))))
N
log Po (2(0)o)
=ExwDx
E (x)(x
(g)
log po (s(0)o) + log (000,,))) + KLD (g.(v) (x)
=ExwDx
(y)()
be modeled with Gaussian distribution, the results are also Gaus- sian distribution Rakelly et al. (2019). Besides, po (a01p, s(0) = Pé.. (a(0) po (valu,, 86 and pa (s(0)p) Pa. (s(1)) pe (1b v), where v. and v. denote the decoded parameters of the distribu- tion. Thus, for any differentiable pv, and Po.., the ELBO objective is tractable.
Since qu (u|s(), al))
In summary, SA-DAE has three-step pipeline to minimize the objective of Equation equation 9
1. Sample batch of X from the dataset Dx;
2. Infer latent code e via the likelihood probability based on Equation equation 10
3. Compute the reconstructed log- probability of state-action pair based Equation equation 12 and Equation equation] IT] and compute the KL divergence between posterior and prior of 8, then apply the gradient with respect to and 0.
15

---

Under review as a conference paper at ICLR 2021
B EXPERIMENTAL SETTINGS
B.1 MULTIPLE-DOMAN MULTIPLE-CUSTONEE LONG-TERM SATISFACTION PROBLEM (LTS)
Long-term satisfaction (Choc/Kale) problem comes from a synthetic environment in the Google RecSim framework le al. 2019). In this environment, the recommendation system sends items of content to customers, and the goal is to maximize customers' engagement in multiple timesteps. The items of content are characterized by the score of clickbaitiness. The engagement of customers is determined by the clickbaitiness score of content and the long-term satisfaction score. The higher clickbaitiness score leads a larger engagement directly but leads decrease in the long- term satisfaction while the lower clickbaitiness score increases satisfaction but leads to a smaller engagement directly. Moreover, the long-term satisfaction is coefficient rescale the engagement of the given item of content. In particular, the value of engagement for customer i at timestep t is sampled from a Gaussian distribution N (14, of which is parameterized by
p = (aint + (1-wa) u;) m1) SAT, of = (atot + (1- ( o
where denotes the index of customer, at denotes the clickbaitiness score the given item. ple pi: of and of are hidden states of the customer i. SAT; denotes the long-term satisfaction score, which is updated by az:
SAT; == sigmoid (h; x NNEE))
NPE 7 N 2 (a) 0.5).
where NPF; denotes the net positive exposure score of the customer i, 7d denotes the memory discount of N PF; and ht denotes the sensitivity ratio of NPE to satisfaction. 7 and h3 are also hidden states in this environment.
For the LTS environment, the hidden states p ui: of. of hi and 7¹ define customer behaviors. To μ², to represent
construct the problem as the multiple-domain multiple-customer environment, the parameter of group information 2d, which are the same among customers in the same simulator, and the rest of the hidden states randomly sampled for each customer, represent the parameter of z.
In our scenario, group-behavior parameters can only be identified through multiple customers' trajecto- ries via an unkown projection function f. i.e., = = (Tox), where To. = [So. Ao. S._S A,-,,,,,-11 Ar-1, and S and A denote the state and action sets of all customers in the same domain. To model this scenario in the synthetic environment, we need to design extra observations for each customer to infer et We consider the simplest case: zd time-invariant and can identified by S, at any timestep 1. In implementation, each customer has an extra fixed observation sampled from Gaussian distribution of N w.). Then žạ is identifiable through multiple eustomers' observation, since 2 E, [o']-
St),
We construct the simulator in which He. = 14 as the test simulator and select the training simulators by equidistant sampling parameters μe from enviromment-comtex space and removing those near testing simulator. The observation of customers is sampled at w. = 4. Then we construct three tasks with different transfer difficulties via controlling the removed range of parameters from training sets }:
LTS1: (Ptrain) } (pp.. : Mo 14| > 2 S Me < 22, E
LTS2: {Ptrain] } {ppe Me 14) 2 366 < M. 22, Me N)
LTS3: (Ptrain) (Pr. :
Mee 14 446 < Pe 22, Pc N)
The environment-comeen space is set to p. [6, 22). taking 14 the middle point, and take the number of 2, and as the removed range for LTSI, LTS2 and LTS3, respectively.
In distortion test, follow the setting of LTS3 and select μ4 the distored parameter. In particular, for all simulators, po ~ 4+ U-3,, (3, Without considering distortion, μ is set to 4 for all customers in the simulators.
16

---

Under review as a conference paper at ICLR 2021
B.2 DRIVER PROGRAM SEQUENTIAL RECOMMENDATION PROBLEM
To demonstrate the effectiveness of the proposed method, we deploy OapRS in a real-world recom- mendation application, for a large-scale ride-hailing platform. The goal of the platform balance the demand from passengers and the supply of drivers, i.e., helping drivers finish more orders, and satisfying the more trip demand from passengers. The driver program recommendation (DPR) is one of the typical recommendation scenarios in the platform. In this scenario, to satisfy more demand from passengers, we would like to maximize the engagement of drivers via recommending reasonable items from the program. The engagement is characterized by the number of orders completed by each driver. The programs contain tasks for the driver to accomplish. If the driver completes the recommendation program, his/her engagement will increase. However, the diffieulty of the task has influences on the completion rate since drivers respond differently to tasks, which also affects the effectiveness of recommendations. The responses are also changing for driver over time. For example, the base number of demand is not in the same order of magnitude among the different cities, since the total passenger volumes vary. Hence, the drivers who have similar personas may behave differently in different cities, leading to diverse recommended programs.
The driver program recommendation can be modeled as an MDP. For simplification, we assume the influence among drivers can be ignored. It is reasonable since drivers almost have no ideas about other drivers' tasks. the DPR environment, we regard each day as a timestep. At timestep 4, the recommendation system policy sends program of = (ss() to driver i based on the observed feature s/. of denotes the program features to characterize the action. The reward rf is the number of finished orders in the day. Our goal is to maximize the expectation of long-term engagement of drivers in different cities.
We divide real-world data into seventeen cities with two different periods for simulator learning. The Simulator is learned by technique |Shang et al. (2019). Then the environment context is implied in the learned simulator models. In the semi-online test, we evaluate the policy performance with reality-gap via running the policy in the unseen city. The experiments allow testing the algorithms on the reality-gap through city and time period transfer, which is essentially environment context change.
C EXPERIMENTAL DETAILS IN LONG-TERM SATISFACTION PROBLEM
We first train a policy given oracle environment context the synthetic simulators, to test the effect of the environment contexts. Figure shows the action distribution of optimal policies in different contexts of l'c- It can be seen that the larger be leads to larger percentage of the higher clickbaitiness-soe score item selected. Thus, transferable policy should be aware of the differences of He and adapt its decisions for better performance.
Sound
Upper Bound
-208-6755-22 003 0.50 0.35
-160--35-335--322 350 L33
--100-61---6----- 606 CJ5 626 2.04
diicta
(kdbuliness score
(a) Mr 6 6
(b) Pe Pc=111 14
(c) Pe = 21
Figure 6: Illustration of generative action distribution for optimal policies in LTS3. The score is rescaled into (-1,1) from (0, 30). We sample 750 trajectories of different customers and draw the histogram of these sampled actions.
In the LTS environment, Za is only related to group state information S. Thus we train SA-DAE to reconstruct the state distribution instead of the state-action distribution. We draw 1000 customers for each simulator to the constructed state dataset Dx. qr (v)s(() is neural network which outputs the
17

---

Under review as a conference paper at ICLR 2021
Gaussian distribution parameters of The dimension of vector is 5 in this environment. We also model pe (1ha ()) with neural network, which outputs the parameters of Gaussian distributions. The prior of v is set to standard normal distribution, i.e., p(v) (0,
150
20
R--divrgence
20
train
(a) KLD training set
(b) KLD testing set
Figure 7: Illustration of KI. divergence of the training set and testing set in LTS3. The solid curves are the mean reward of three seeds. The dark shadow is the standard error, and the light one is the min-max range of three seeds.
We use KLD to measure the performance of reconstruction. Since pe (v% (v) also outputs the parameters of Gaussian distribution, we compute the KLD directly via the analytic expression of Gaussian distribution between pe (s/v) and N Wc). We test the KLD every 100 epochs. Figure shows that the KLD in the testing set finally converges to the range of 0.01 to 0.02. Figure g shows the reconstruction distribution is also correlated.
1.0
FCaI
fake
0.8
0.8
denstty
0.6
0.4
0.4
0.2
0.2
-5.0
-25
0.0
2.5
5.0
7.5
10.0
12.5
15.0
0.0
10
15
environment.contaatt
environment context
(a) histogram traning set (fte 6) epoch
(b) histogram in testing set (Nt = 14) at epoch
0
3.12
faka
0.14
faba
0.10
dansty
0.08
denstty
2.08
2.06
3.04
0.02
-2.5
3.0
environment-crteed 12.5
2.5
7.5
7.5
10.0
12.5
15.0
17.5
20.0
22.5
anvironment-contett
histogram in traning (Ju. 6) at epoch
(d) histogram testing set (ptc 14) epoch 8000
8000
Figure& 8: Illustration of the histogram about customer feature of o² in reconstructed and real data in the task of LTTS.
18

---

Under review as a conference paper at ICLR 2021
1.00
2-332
3.02
2.00
1.00
-0.32
-1006
-4.0
3.3086 5060 00400322
F00
0322
6004 000 0.03
-10
-35
0.3
055
3.0
15
-10
-25
0.8
02
per
(a) epoch 0
(b) epoch 2000
(c) epoch 4000
8.06
5166
6000
-0.04
-6006
-0430
poa dim
pea dm
(d) epoch 6000
(c) epoch 8000
Figure 9: Illustration of the visualization on u. The X-axis denotes the first principal component, and the Y-axis denotes the second one. Each cross point denotes the projection of the latent code for the state distribution. The numbers with the same color to the point denote the ground-truth of pe- Although gio (e X) is a Gaussian distribution, we only draw the mean the distribution for legibility.
To evaluate the embedding performance of SA-DAE, we project into vvo-dimensional vectors via principal component analysis (PCA) Wold al. 1987). We show the result of PCA in Figure It can be seen that, after 6000 epochs, the latent code can represented by the first principal component. And the value of pe linearly depends on the first principal component.
We finally test the robustness of environment cnvironment-parameeeee extractor to the noise scale we- The result can be seen in Figure I0] The results show that the environment-paranee extractor can also work well in large noise We situations.
X4
03-05)
740
28-301
26-001
200
69
510 025
1.0 1.2%
640
0/8
435
0.7%
1.00
3.9 213
399 3.5 2.0 400
9.7 3.99 3.34
We
W.
10
(c) We
Figure 10: Illustration of the performance in synthetic environments in different we. The solid curves are the mean reward and the shadow is the standard error of three seeds.
D PPO IMPLEMENTATION OF OAPRS
We use PPO Schulman et al. (211)) example and any policy-based methods (i.e., TRPO Schulman et al. (2013). SAC Haarnoja et al. (2018) can also be plugged into the policy learning part. The pseudocode of OapRS is summarized in Algorithm
19

---

Under review as a conference paper at ICLR 2021
Algorithm 2 OapRS implemented with PPO
Input:
Offline expert data D;
LSTM network 9 as environment-contex extractor, parameterized by yp Adaptive policy network and value function network Vé, parameterized by 0 Model learning algorithm G; Rollout horizon Hiiiibatch for policy training Training epochs M Process:
1: Divide the offline dataset into N partitions (D) and generate M x N dynamics model {o} by ensemble of M model learning technique DEMER (Shang al. 2019 G.
2: Pretrain the state-action distributional embedding model q, (2 on (D).
3: for 1, 2, 3, do
4: Random select the dynamics model p² in {p} and sample from D which is used to train p²
5: initialize buffer Drollout
6: for (=1,2....H do
7:
Sample 2, from ((=)pp) and then sample de from (ay 21)
8: rollout one step 9+11:1++1 pE(8), ax) and then add 19441:71411 St, 04: ze) to Drollout mod m = then
9:
10:
Compute the long-term reward estimates, R, for each time-step t.
Compute the advantage estimate, 9m based the value function (o z). where z is the output of extractor o...
Compute the gradient V and V of the PPO objective
T
arg max
Σ TED t=0
Σ
In (x)
E
LPPO 9s Uz, ar a..1,
1 T
St,RE- L.T
Apply Ve and V..
14:
Update value function by one-step gradient of the regression loss:
T
arg min 1
Σ
Σ
(x)
Σ
(Vo(sr, =(0) ar-1, ) 1r,
(Dm. T
TED
t
8t,4:-1E7
15:
Update the state-action distributional embedding model qu (v]X) on D Drotlout Dratlou 0
17:
end
end for
19:
end for
E EXPERIMENTAL DETAILS IN REAL-WORLD RIDE-HAILING PLATFORM
We collect offline data from the platform and construct real- world simulators by imitation from data [Shang et al. (2019). The results show that the order completion in the reconstructed simulators generally consistent with the real-world data.
We train the SA-DAE in the training set and test the reconstructed data distribution the unseen envi- ronment. The training dataset Dx comes from human expert data the training set. 4x (vls(*) (1) is neural network which outputs the Gaussian distribution parameters of v. The dimension of the vector u is 200 in this environment, Pa (thal &()))) (()) and po (Ba/v) are modeled with neural networks, which output the parameters the distributions. The action reconstruction is modeled with Gaussian distribution since it is continuous in the environment. However, the state space includes continuous and discrete features. For simplification, we assume the continuous features are independent of discrete features. Thus model them with Multivariate Gaussian distribution and categorical distribution respectively. The prior of is set to standard normal distribution, i.e., p(v) (0 1).
We KLD measure the reconstruction performance. Since the dimension of state-action space is high and the distribution complex, we use Kernel Density Estimation (KDE) Rosenblati (1956) to estimate the probability density function (PDF) of real and reconstructed data. Then the KLD of two
20

---

Under review as a conference paper at ICLR 2021
b
(c)
(d)
(c)
(f)
(h)
Figure 11: Illustration the daily finished-orde curves of the constructed simulator and real data. Each column shows the results of the same city. Each row shows the results in the same period.
datasets computed based on it. In particular,
Σ log fa(x) ft(
KLD(Da, = |Daal
(13)
mED
where ||Dall denotes the number of samples in the dataset, and fa and fo denote the PDF of real and reconstructed data estimated by KDE. We test the KLD every 100 epochs. Figure 12(a) shows that the KLD between the real data X and the reconstructed distribution pe (Xlv) steadily converges to 0.6, which demonstrates nontrivial reconstruction performance. Figure 15 shows histograms for examples of real and reconstructed data a single feature, which are also significantly correlated.
To evaluate the embedding performance of SA-DAE, we performed the hidden state prediction experiments Akkaya al. (2019). If the embedding variables store useful information about the distribution, for simple neural network, the KLD prediction error between arbitrary two datasets would be negatively correlated with the training epochs. We use another one-layer neural network to predict the KLD of two data pairs (X,,X) Xy), given their embedding variable (V,, uy). The neural network has one 32-unit hidden layer with tanh as the activation function and links linear layer to predict the KLD computed by Equation 13 The neural network is initialized and retrained for the same epochs, every 100 iterations of SA-DAF learning. Figure T2(b) shows the mean absolute error (MAE). The MAE has 26% improvement than the initial variable, which implies the embedding variable helpful to infer the relation of two distributions.
F ADDITIONAL DETAILS FOR OAPRS
F.1 MAPPING FUNCTION CONSTRUCTION
Given a specific SRS application, we can split the state space into the following parts:
Customer-related static states: the states which are invariant to the recommended actions and are fixed for each customer. For example, the age and gender of each driver. Since we model the process of interaction between the platform and customer to an MDP, the customer is fixed in trajectory and thus customer-related states are invariant;
Timestep-related static states: the states which are invariant to the recommended actions and are fixed for each time-step. For example, the weather for each day. We train and the dynamics model with the same time period as the collected dataset and thus the timestep- related states can be found from the dataset and fixed;
21

---

Under review as a conference paper at ICLR 2021
0.370
0.226
(L-divergence
1.4
0.002
0.326
250
630
350
100n
1253
1500
1793
2000
250
660
75a
1056
1250
1506
1750
200
trein epochs
train apochs
(a) KI. divergence
(h) mean absolute error
Figure 12: Illustration of the reconstructed and embedding performance SA-DAE. The solid curves are the mean reward of two seeds. The dark shadow is the standard error, and the light one the min-max range of two seeds.
Dynamics-related states: the states which are dependent on the given recommendation actions and hard to hand-code by rules. For example, the completed orders and GMV of driver day given the predicted completed orders and the recommendation action;
Rule-related states: the states which can be computed by rules given other states. For example, can compute the cost of driver in day, given the completed orders and the recommendation action, and we can compute the averaged GMV of driver by averaging the predicted completed orders of recent days. Instead of learning to predict all the states, the neural network is used to predict the "dynamics states".
After sampling initial state so from the dataset at the beginning, for each timestep f, the trajectories of the environment model can be sampled by the following steps:
Fix the customer-related static states: st scustmmer customer;
s6-1
2. Set the timestep-related static states 8 timestep with the state dataset
3. Predict the dynamics states by neural network model: St NN denotes the trained neural network;
dynemice
NN at-1), where
4. Update the constructable states by rule:
rule mapping(s:-11 01-1 8t 8
timestep dgnamica
5. Put them together: 5; customer ,t
timestep
dynamics
srule].
st
The framework of semi-dynamies model learning is general other problems since it is easy for human experts to construct some rules for some states with a well-defined relationship and any number of rules be embedded reduce the complexity of learning. The more rules have, the less learned states are needed. We list the different type of state of our application in Table [
Table 1: Different type of state in the state space.
Category
Dimensionality
Description
customer
54
Personas features
timestep
The weather and holiday features et al.
dynomics
3
The probability drive in the day, the number of com- pleted orders and the averaged GMV per order
rule
140
The historical recommendation actions and the dynamicc in 7 days, some statistics and other construable features.
22

---

Under review as a conference paper at ICLR 2021
F.2 AUGMENTATION TECHNIQUES OAPRS
In SRS, there are several common choices to split the dataset: First is split by periods since the customer often behaves differently across periods in the applications. The second is split by some identification category features in the application since it often represents the domain classes. In our application, we split the dataset by the month and city id.
Some general ensemble techniques like learning with different initial neural network parameters, different hyper-parameters, different learning algorithms can also be used for dynamics model augmentation. However, we do not implement these methods for resource limitation and find the approach works in our application.
F.3 DYNAMICS MODEL PERFORMANCE EVALUATION
We split the dataset into training set (80%) and testing (20%). the environment model is learned in the training set and evaluated in the testing set. Multiple metrics that are concerned by the operational staff can be used for evaluation. For example, the similarity of total completed orders of drivers in a city between the data run by the environment model and the corresponding real-world data. We use Pareto improvement (a new model where some metries will gain, and no metries will lose) of the metrics in the testing set to define the performance improvement of the environment model. The metrics we selected are the mean absolute percentage error (MAPE) of the following statistics: (1) the total cost for each day; (2) the total number of driver is driving for each day (i.e., the number of completed order is larger than zero); (3) the total GMV for each day.
F.4 DETAILED STRUCTURE AND OPTIMIZATION TRICKS
The network structure of OapRS is shown in Figure 13 In the implementation, we use two in- dependent network structures for the value function and policy learning. Moreover, there a skip-connection for the environment-awart layer reuse the original input features. The embedding layer and environment-awar layer are modeled with Multilayer Perceptron (MLP). The environment- context extractor layer is modeled with single-layer LSTM network Hochreiter & Schmidhuber (1997). We use the same network structure and hyperparameters in the two experiments, but the complexity of neural networks are different. Table reports the detailed settings.
In the DPR environment, we model the distribution of continuous states with Multivariate Gaussian distribution. The covariance of the Multivariate gaussian distribution should be a symmetric positive semi-definite matrix. We construct the covariance matrix based on Cholesky decomposition Golub Van Loan (196).. In particular, a symmetric positive-definite matrix Σ is decomposition of the form:
LL
where lower triangular matrix with real and positive diagonal entries, and L denotes the transpose of L. We use neural network with the softplus activation function to output the elements of L, where the softplus is used to output a positive value. Then, we regard the matrix LL" T as the covariance of the Multivariate Gaussian distribution. The hyperparameters of SA-DAE in the two experiments are shown in T
In addition, use several tricks in the training process of the environmentt-ontexx extractor and adaptive policy:
Larger entropy coefficient We set the entropy coefficient 0.02 for the LSTM-bassed policy and le-5 for MLP-based policy in the two problems. We found that the smaller entropy coefficient leads performance collapse after some iterations when training with LSTM- policy.
Multiple environments sampling We sample from all training environments uniformly for VJ((0) computation every iteration. The policy updates would be unstable and inefficient if just sampling from single environment.
23

---

Under review as a conference paper at ICLR 2021
of
Action: of
Environment-awar value layer: of = of)
Environment-wwarc policy layer: di n(x),
Value envirumment of o(ff_)=4))
Policy envirenment-contexr extractor: * of
Fully-conmected embedding:
Fully-comnected embedding: f(ae)
fla)
Concatenated vector: xf by, bf.ef-1.1
SA-DAE embedding: M.fo)/A-111 Sy)
Group state-nction set: A St
Action: di_.
State: st
Learnable layer
Parameter-free layer
Input
Output
Figure 13: Illustration of the network structure for OapRS.
time pariod 2
Mean order completion
1.2
0.9
10
15
day
Figure 14: Illustratonn of the mean order completion in the real-world ride-hailing application. The number of mean order completion is computed by the mean value of the order completion in a city per day. The result is scaled with a constant value. The legend "time period I" "nd "time period 2" denote the data collected in adjacent 22 days of the same city.
24

---

Under review as a conference paper at ICLR 2021
13
15
0.5
3.8
0.
10
0.2
0..
04
1.0
(a)
(b)
(c)
2.4
24
Scaldd Narrrr
Bastor
acaled valas
(d)
rell
real
20.00
rell
15.00
10.0
3.4
Nuslure
Scabal
g
(h)
37.
2.00
3.50
L
0 3.00
6.5
32
6.00
3.8
03
3.4
0.0
scded
(k)
(I)
Figure 15: Illustration of the histogram about customer features of reconstructed and real data.
25

---

Under review as a conference paper at ICLR 2021
Table 2: The hyper-parameters of OapRS for policy learning.
Hyperparameter
LTS
DPR
Discount factor
0.99
0.9
Horizon H
140
30
Entropy regularization coefficient
0.02
0.001
PPO clipping parameter
0.2
Learning rate
from le-4 to le-6
Batch size
30000
120000
Sample reuse
3 Adam le-6
Optimizer
L.2 regularization weight
Activation function of hidden layer Activation function of policy output Fully-connected embedding f Unit of environment-comtex extractor Environment-awee aware layer of V and π
leaky relu tanh
[128, 128, 128,32]
[512, 512, 256]
64
256
[128, 64]
[512, 256]
Table The hyper-parameters of OapRS SA-DAE.
Hyperparameter
LTS
DPR
Learning rate:
2e-5
le-6
Layer Normalization:
False
Optimizer:
Adam
L2 regularization weight activation function of hidden layer Embedding layer dx (w|s, a) Reconstructed layer pa (vblv) units of latent code
0.1
0,001
leaky relu
[512, 512]
[512, 512]
5 200
Table 4: The averaged scaled daily rewards of real-world deployment in RDP environments
OapRS
MOPO
before
0.9019
0.8857
after
0.9641
0.8865
improvement (%)
6.9%
0.1%
26


Reviews:
[{"review_id": "s1K09HEYOI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper proposes curtaining the impact of simulator not being faithful to reality in a few ways: (a) define a policy that takes environmental context as input, (b) define an environment context predictor, (c) training scheme that learns the adaptable policy as well as the context predictor. The idea is that the learned policy is adaptable to diverse simulators defined using offline data. So when it is deployed online, it can adapt to the dynamics of real world changing across time. They use sequential recommendation systems (SRS) as the application to demonstrate this.\n\nThe key idea of the paper seems to be to extend domain randomization techniques from robotics to SRS, and justify that it can additionally take care of non-stationarity (for free), on top of the usual modeling errors.\n\n\n1. The inability to sample in the real-world maybe restrictive for robotics, but is a moot point for SRS. There is almost no cost for exploration (you will have to show recommendations anyway and the cost of showing bad recommendations is almost nil at medium-large scale) and augmenting your existing data should be much much easier. Is there compelling evidence otherwise?\n\n>> solve an offline problem that policies can applied to real-world applications without any additional online sample.\n\n\n2. Is there a quantitative reference for non-stationarity of consumer behavior and its impact on revenue loss or something similar? Kruger et al. has no real world examples itself. Even if it is non-stationary, in real world systems, policies are retrained at a high enough frequency that this should not be an issue.\n\n3. The description before equation (1) seems wrong. If there is a \\forall \\rho, then equation (1) should have such a constraint. Taking expectation with respect to \\rho is not considered robust. This misunderstanding decreases confidence in the validity of the results presented.\n\n\n4. The paper is generally dense and hard to understand at several places. Some of the statements are imprecise (some words seem to be misused) and detract from the key contributions of this work. \n\n\n - For instance, the following should be rephrased and jargon should be minimized:\n\n>> Hidden confounder factors (Shang et al., 2019) obstruct the deterministic predictions. As a result, the high-confidence regions are drastically reduced and thus the exploration of policy learning are obscured.\n\n - Unclear what this is aiming to say (without explicitly discussing how policy 'exploration' is constrained):\n\n>> instead of constraining policy exploration in high-confidence regions\n\n - Unclear (what is inconsistent?):\n\n>> in regions with inconsistency\n\n - Unclear. What is the two level structure below. Is the example with domains/countries supposed to be a proxy for environmental context?\n\n>> the environments include a two-level structure: ... \n\n - Unclear (what is the special structure? how is the environment made context agnostic?):\n\n>> and a special environment structure makes the environment context agnostic\n\n - Unclear how stochastic environments have an influence in policy optimization without fixing a learning algorithm:\n\n>> However, in stochastic and non-stationary environments, the learned dynamics have high uncertainty and thus constrain the policy to exploration in a small region.\n\n - Unclear what the performance metric of each simulator is:\n\n>> maximize the performance metric of each simulator\n\n - Repeated use of suboptimality and optimality before formally defining it: Also unclear what does \"all possible transitions out of consistency region\" mean:\n\n>> to reach suboptimal policies (in Figure 1(c)), we learn to make optimal decisions for all possible transitions in\nthe states out of the consistency region\n\n\n - What is \"exploitation\" here, isn't this confounding/overloading an existing technical term:\n\n>> learned policies tend to exploit regions where the model is distorted\n\n\n - What is a penalty and how is it consistent, and how does it obstruct? If it is relevant to differentiate the proposed solution with respect to this weakness of prior works, more information and clarity is needed.\n\n>> Since the environment has stochasticity, learning by consistency penalties will obstruct the policy into a small region for exploration.\n\n>> without relying on the consistency penalty\n\n - Unclear:\n\n>> instead of constraining policy exploration in high-confidence regions\n\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "nAot8Cua_n9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an adaptive model-based off-policy learning algorithm with applications to Sequential Recommender Systems (SRS) in mind. The novelty in the work is that it extends on other model-based and adaptive algorithms to suit the needs of SRS where stochasticity and non-stationarity are a part of the problem--this any learned model has to be robust to distortion because of off-policy setting. The key idea is to construct an 'environment-parameter extractor' to identify the dynamics system to adapt at runtime. \n\nThe background and notations in Section 3 and 4 are well written and rigorous, however, a little more grounding to a sequential recommender system setting would help in two ways: make the paper more self-sufficient for a Recommender systems audience, and also motivate the distinction between other model-based and other adaptive policy learning methods. \n\nIn the experiments section, it is worth discussing what is the trade-off between choosing an application-specific dynamics set has with models that do not have access to this knowledge in a different application. Moreover, one could add experiments where a misspecified model class is used to define the dynamics and show how the performance degrades wrt other algorithms that do not require this assumption.\n\nIn Section 5.1, the last experiment with distortion parameter,  the choice of distortion per simulator makes more sense than having an unlimited number of simulators to test robustness to distortion and hence makes me wonder (again) about how this model would perform against other model-based/adaptive algorithms.\n\nOn a minor note, the authors should make the setup in section 3 a little more self-sufficient and motivate the problem from a recommender system perspective i.e. how stochasticity and non-stationarity come into play.\n\nOverall, I find the paper convincing and novel and would recommend acceptance. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iosakICWzlv", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies off-policy reinforcement learning for sequential recommendation. The basic idea is to summarize each possible environment dynamic (i.e., the state transition) into an environment context vector, and optimize policy with respect to this context vector accordingly. The proposed solution is evaluated on a simulated sequential recommendation environment, and a real-world ride-hailing platform, which considerably increases the creditability of the proposed solution.  \n\nThe problem setup is a mix, which affects the generalization of the proposed solution. On the one hand, the solution is general for off-policy reinforcement learning, as I do not find any strong dependency of the proposed problem setup (i.e., a MDP) and the target sequential recommendation problem. However, on the other hand, the details in the proposed solution are so specific, which makes me question the generalizability of the proposed solution. For example, the reward function is assumed to be given by human experts. But in the experiments, it seems it is just the reward is defined by human expert, e.g., in the ride-hailing platform the reward is defined as the number of completed orders, rather than the reward function, e.g., how the state and action pair maps to a numerical reward. A clarification is needed here. And do we assume any non-stationarity in the reward function at all? There are other manually defined heuristics to “reduce model learning complexity”, e.g., “a hand-coded function $map(s’|\\bar{s},s,a)$ to construct the left of the states”. And actually I did not find the details about this hand-coded function in Appendix F. If even the details were provided, I am not sure how to construct this mapping for new problems, e.g., beyond the ride-hailing problem, in general.\n\nThe key idea behind the proposed solution is to exhaust possible environment models from offline data, and estimate the policy for each estimated environment model. The environment models are created by splitting the offline data into non-overlapping partitions, and use a particular environment model learning algorithm (e.g., with different hyper-parameters) to generate different environment models. Hence, how to partition the offline data and how to estimate the environment model become vital; but there were not enough discussion or experiments on these two important aspects. The mentioned solution for dataset partitioning is “by domain knowledge” and environment model learning is by existing algorithms. And I also did not find any experiment studying how these two factors affect the proposed solution. \n\nAn autoencoder-based method is used for environment model identification, but I cannot exactly follow how the autoencoder informs the construction of environment-context extractor $\\phi$? For example, how should I understand this sentence: “the extractor $\\phi$ can infer environment-context both with $v$ and $\\tau$”? \n\nBy the way, in Algorithm 2, what is the purpose behind line 19-22? Why do we need to generate those new episodes from the learnt environment model? \n\n**Response after author rebuttal period**\nI highly appreciate the detailed explanations and discussions the authors have provided during this period. It indeed clarifies my concerns and helps me better understand the setting and proposed solution. However, my major concern about the generalizability of the proposed solution still remines, as there are too many design choices depending on domain knowledge. I would keep my original recommendation; and if the paper could be accepted, I would like to encourage the authors to discuss the limitations of the proposed solution.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "d-8GSqlCOS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses the problem of RL based sequential recommendation system and proposes a model-based learning method to solve the offline RL in real-world applications. It addresses the two challenging problems:  (i) real-world recommendation\nenvironments are usually non-stationary, and (ii) real-world environments recommendation are often with stochasticity. The proposed model tackles these two problems by learning to adapt to different simulators generated by the offline dataset. Experiments demonstrate the effectiveness of the proposed model against state-of-the-art baselines on one dataset. This paper could be improved in the following aspects:\n\n1. Implementation code is not released.\n2. More details of the deployment of the model should be provided.\n3. Only a few baseline methods are introduced with the proposed model. More RL based recommendation models and simulator models should be incorporated to demonstrate the effectiveness of the model.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is rejected.

The authors focus on offline RL for the sequential recommender system problem and propose an approach that:
* builds multiple models based on splits of the offline data using domain knowledge
* splits the policy into a context extraction system and context conditioned policy (similar to Rakelley et al.)

While R1 and R4 appreciate the changes, they both feel that the paper is not ready for publication at this time. R1's main concerns is the generalizability of the proposed solution because it relies heavily on manually defined rules and domain expert knowledge. R4 was concerned with the definition and precision of robustness. How is robustness quantified? Finally, many of the baselines were not built for partially observed environments, so it is unsurprising that they perform poorly. Baselines with recurrent policies would strengthen the paper.

Decision: Reject