Addressing Extrapolation Error in Deep Offline Reinforcement Learning
Authors: Caglar Gulcehre
Sergio Gómez Colmenarejo
ziyu wang
Jakub Sygnowski
Thomas Paine
Konrad Zolna
Yutian Chen
Matthew Hoffman
Razvan Pascanu
Nando de Freitas
Venue:  (2021)
Keywords: Addressing Extrapolation Error in Deep Offline Reinforcement Learning

Abstract: Reinforcement learning (RL) encompasses both online and offline regimes.  Unlike its online counterpart, offline RL agents are trained using logged-data only, without interaction with the environment.  Therefore, offline RL is a promising direction for real-world applications, such as healthcare, where repeated interaction with environments is prohibitive.  However, since offline RL losses often involve evaluating state-action pairs not well-covered by training data, they can suffer due to the errors introduced when the function approximator attempts to extrapolate those pairs' value. These errors can be compounded by bootstrapping when the function approximator overestimates, leading the value function to *grow unbounded*, thereby crippling learning. In this paper, we introduce a three-part solution to combat extrapolation errors: (i) behavior value estimation, (ii) ranking regularization, and (iii) reparametrization of the value function.  We provide ample empirical evidence on the effectiveness of our method, showing state of the art performance on the RL Unplugged (RLU) ATARI dataset. Furthermore, we introduce new datasets for bsuite as well as partially observable DeepMind Lab environments, on which our method outperforms state of the art offline RL algorithms. 


Under review as a conference paper at ICLR 2021
ADDRESSING EXTRAPOLATION ERROR IN DEEP OFFLINE REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning (RL) encompasses both online and offline regimes. Unlike its online counterpart, offline RL agents are trained using logged-data only, without interaction with the environment. Therefore, offline RL is promising direction for real-world applications, such as healthcare, where repeated interaction with environments is prohibitive. However, since offlin RI. losses often involve evalu- ating state-action pairs not well-covered by training data, they can suffer due the errors introduced when the function approximator attempts extrapolate those pairs' value. These errors can be compounded by bootstrapping when the function approximator overestimates, leading the value function to grow unbounded, thereby crippling learning. In this paper, we introduce a three-pane solution to combat ex- trapolation errors: (i) behavior value estimation, (ii) ranking regularization, and (iii) reparametrization of the value function. We provide ample empirical evidence the effectiveness of our method, showing state of the performance the RL Unplugged (RLU) ATARI dataset. Furthermore, we introduce new datasets for bsuite as well as partially observable DeepMInn Lab environments, on which our method outperforms state of the art offline RL. algorithms.
INTRODUCTION
Agents are, fundamentally, entities which map observations actions and can be trained with reinforcement learning (RL) in either an online or offline fashion. When trained online, agent learns through trial and error by interacting with its environment. Online RL has had considerable success recently: Atari (Mnih et al., 2015), the game of GO (Silver et al., 2017), video games like StarCraft II, and Dota 2, (Vinyals et al., 2019; Berner et al., 2019), and robotics (Andrychowicz et al., 2020). However, the requirement of extensive environmental interaction combined with need for exploratory behavior makes these algorithms unsuitable and potentially unsafe for many real world applications. In contrast, in the offline setting (Fu et al., 2020; Fujimoto al., 2018; Gulcehre et al., 2020; Levine et al., 2020), also known as batch RL (Ernst et al., 2005; Lange et al., 2012), agents learn from fixed dataset which is assumed to have been logged by other (possibly unknown) agents. See also Fig. 1 for an illustration of these two settings. Learning purely from logged data allows these algorithms be more widely applicable, including problems such healthcare and self-driving cars, where repeated interaction with the environment is costly and potentially unsafe or unethical, and where logged historical data is abundant. However these algorithms tend behave considerably worse than their online counterpart.
Although similar in principle, there are some important differences between the two regimes. While it is useful for online agents to explore unknown regions of the state space to gain knowledge about the environment and better their chances finding good policy (Schmidhuber, 1991), this is not the case for the offline setting. Choosing actions not well-represented in the dataset for
Cirilee
Agent
Environmentt
Agent
Logged
Figure 1: In online RL (left), the agent must interact with the environment gather data learn from. In offline RL (right), the agent must learn from a logged dataset.

---

Under review as a conference paper at ICLR 2021
offline methods would force the agent to rely function approximators' extrapolation ability. This can lead to substantial errors during training, as well as during deployment of the agent. During training, the extrapolation emrors exacerbated by bootstrapping and the use of max operators (e.g. in Q-learning) where evaluating the loss entails taking the maximum over noisy and possibly overestimated values of the different possible actions. This can result in a propagation of the erroneous values, leading to extreme over-estimation of the value function and potentially unbounded error; see (Fujimoto et al., 2019b) and our remark in Appendix A. As we empirically show in Section 4.2, extrapolation errors are a different source of overestimation compared to those considered by standard methods such as Double DQN (Hasselt, 2010), and hence cannot be addressed by those approaches. In addition to extrapolation errors during training, further degradation in performance can result from the use of greedy policies at test time which maximize over value estimates extrapolated to under-represented actions. We propose a coherent set of techniques that work well together to combat extrapolation error and overestimation:
Behavior value estimation. First, we address extrapolation errors during training time. Instead of ²", we estimate the value of the behavioral policy Q*s, thereby avoid the max-operator during training. To improve upon the behavioral policy, we conduct what amounts to a single step of policy improvement by employing a greedy policy at test time. Surprisingly, this technique with only one round of improvement allows us to perform significantly better than the behavioral policies and often outperform existing offline RL algorithms.
Ranking regularization. We introduce a max-margin based regularizer that encourages the value function, represented episodes higher than any other actions. Intuitively, this regularizer pushes down the value of all unobserved state-action pairs, thereby minimizing the chance of greedy policy selecting actions under-represented in the dataset. Employing the regularizer during training will minimize the impact of the max-operator used by the greedy policy at test time, i.e. this approach addresses extrapolation errors both at training and (indirectly) at test time.
deep neural network, to rank actions present in the observed rewarding
Reparametrization of Q-values. While behavior value estimation typically performs well, par- ticularly when combined with ranking regularization, it only allows for one iteration of policy improvement. When more data is available, and hence we can trust our function approximator to capture more of the structure of the state space and as a result generalize better, we can rely on Q-learning which permits multiple policy improvement iterations. However this exacerbates the overestimation issue. We propose, in addition to the ranking loss, simple reparametrization of the value function to disentangle the scale from the relative ranks of the actions. This reparametrization allows us to introduce regularization term on the scale of the value function alone, which reduces over-estimation.
To evaluate our proposed method, we introduce new datasets based on bsuite environments (Osband et al., 2019), as well as the partially observable DeepMind Lab environments (Beattic al., 2016). We further evaluate dataset (Gulcehre et al., 2020). We achieve a new state of the art (SOTA) performance on the RLU Atari dataset well as outperform existing SOTA offline RL methods our newly introduced datasets. Last but not least, we provide careful ablations and analyses that provide insights into our proposed method as well as other existing offline RL algorithms.
method as well baselines the RL Unplugged (RLU) Atari
Related work. Early examples of offline/batch RL include least-squares temporal difference meth- ods (Bradtke & Barto, 1996; Lagoudakis & Parr, 2003) and fitted Q iteration (Ernst et al., 2005; Riedmiller, 2005). Recently, Agarwal et al. (2019a), Fujimoto et al. (2019b). Kumar et al. (2019), Siegel et al. (2020). Wang et al. (2020) and Ghasemipour et al. (2020) have proposed offline-RI. algorithms and shown that they outperform off-the-shelf off-policy RI. methods. There also exist methods explicitly addressing the issues stemming from extrapolation error (Fujimoto et al., 2019b).
2 BACKGROUND AND PROBLEM STATEMENT
We consider, in this work, Markov Decision Processes (MDPs) defined by (S, R, Po, 7) where is the set of all possible states and AA all possible actions. An agent starts in some state So ~ po(-) where Po(:) is distribution over S and takes actions according to its policy ~ ( a E A. when in state s. Then it observes a new state s' and reward r according to the transition distribution P(s'|s, a) and reward function r(s, a). The state action value function Q" describes the expected
2

---

Under review as a conference paper at ICLR 2021
Extrapolation over-estimation
Extrapolation under-estimation
Q
a observed
a
a
a observed
a2
a
unobserved
unobserved
Figure 2: Two types of extrapolation error. Type is most dangerous offline RI., due the max operation. Type is difficult to address without additional interactions with the environment. Here, we aim to address Type A extrapolation errors.
discounted return starting from state and action a and following afterwards:
as) = E
Σ 1=0
r'r(s,, 0t) 80 s, = 3, (0 ~ (-18t))
(1)
and (s) (a) a) is the state value function. The optimal policy x², which we aim to discover through RL, is one that maximizes the expected cumulative discounted rewards, or expected returns such that Q" (s. 2 Q" (s, a) Vs, a, æ. notational simplicity, we denote the policy used to generate offline dataset TB' In the same vein, for state in offline dataset, we write 6o(s) (s) to denote an empirical estimate of V (*)(,, computed by summing future discounted rewards over the trajectory that is part of.
Approaches to RL can be broadly categorized as either on-policy or off-policy algorithms. Whereas on-policy algorithms update their current policy based data generated by that same policy, off- policy approaches can take advantage of data generated by other policies. Algorithms
the mold of fitted Q-iteration make up many of the most popular approaches deep off- off-policy RL (Mnih et al., 2015; Lillicrap et al., 2015; Haarnoja et al., 2018). This class of algorithms learns a Q function by minimizing the Temporal Difference (TD) error. To increase stability and sample efficiency, the of experience replay is also typically employed. For example, DQN (Mnih et al., 2015) minimizes the following loss function:
Co (0) ((, QQ(ss (r max (s' (ss,a')))
where represents experience replay, i.c. a dataset generated by some behavior policy. Typically, for off-policy algorithms the behavior policy is periodically updated to remain close to the policy being optimized. A deterministic policy can be derived by being greedy with respect to Q, i.e. by defining π (s)) = arg max, (s,aa). cases where maximization nontrivial (e.g. continuous action spaces), we typically adopt separate policy and optimize losses similar to: Lar
(8', (',a"") a²)l) In this case, is optimized separately in order to maximize En-s( x) (s. sometimes subject to other constraints (Lillicrap al., 2015; Haarnoja et al., 2018). Various extensions have been proposed for this class of algorithms, including not limited to: distributional crities (Bellemare et al., 2017). prioritized replays (Schaul et al., 2015), and n-step returns (Kapturowski et al., 2019; Barth-Maron et al., 2018; Hessel et al., 2017).
Bee,a...e"'>2 (o(ss.)) a) (r +
In the offline RL setting (see Figure 1, right), agents learn from fixed datasets generated via other processes, thus rendering off-policy RL algorithms particularly pertinent. Many existing offline RL algorithms adopt variants Equation (2) to learn value functions; e.g. Agarwal et (2019b). Offline RL, however, is different from off-policy learning in the online setting. The dataset used is finite and fixed, and does not track the policy being learned. When a policy moves towards a part of the state space not covered by the behavior policy(s), for example, one cannot effectively learn the value function. We will explore this in more detail the next subsection.
2.1 EXTRAPOLATION AND OVERESTIMATION IN OFFLINE RL
In the offline setting, when considering all possible actions for a next state in Equation (2), some of the actions will be -idt---diitiibttion (OOD), i.c. these actions were never picked in that particular state by the behavior policy used to construct the training set (hence not present in the data). In such
'Our proposed approach does depend Ty being coherent policy.
3

---

Under review as a conference paper at ICLR 2021
circumstances, we have to rely the current Q-network's ability to extrapolate beyond the training data, resulting in extrapolation errors when evaluating the loss. Moreover, the need for extrapolation can lead to value overestimation, as explained below.
Value overestimation (see Fig. 2) happens when the function approximator predicts a larger value than the ground truth. In short, taking the max over actions of several Q-network predictions, as in Equation (2), leads to overconfident estimates of the true value of the state. We will expand on this point shortly. Before doing so, it is worth pointing out that this phenomenon of overestimation was well-studied in the online setting (Van Hasselt et al., 2015; 5; 2018) and some prior works sought to address this problem (Van Hasselt et al., 2015; Fujimoto et al., 2018). However, in offline RL overestimation manifests itself in more problematic ways, which cannot be addressed by the solutions proposed in online RI. (Kumar al., 2019). To see this, let us consider Equation (2) again. The max operator is used to evaluate Qo for all actions in given state, including actions absent in the dataset (OOD actions). For OOD actions, we depend on extrapolated values provided by Qp. While being an extremely powerful family of models, neural networks will produce erroneous predictions unobserved state-action pairs, and sometimes, these will be artificially high. These errors will be propagated in the value of other states via bootstrapping. Due to the smoothness of neural networks, by increasing the value of actions in the OOD action-state's 's neighborhood, the overestimated value itself might increase, creating a vicious loop. Mainly we remark that, in such a scenario, typical gradient descent optimization can diverge and escape towards infinity. See Appendix A for a formal statement, and proof on this statement, though similar observations had been made before by Fujimoto et al. (2019b) and Achiam al. (2019). In the online setting, when the agent overestimates some state-action pairs, they will be chosen more often due to optimistic estimates of values, even in the off- off-policy setting where the behavior policy trails the learned one. The online agent would then act, collect data, thereby correcting extrapolation errors. This form of self-correction is absent in the offline setting, and due the overestimation from extrapolation, this absence can be catastrophic.
SOLUTIONS TO ADDRESS EXTRAPOLATION ERROR
We build towards a solution to the extrapolation emrors by i) using behavior value estimation to reduce training time extrapolation error, ii) ranking regularization the Q-networks to better handle test time extrapolation error, iii) reparameterizing the Q-function to prevent divergence of these predictions to infinity.
3.1 BEHAVIOR VALUE ESTIMATION
One potential answer to the overestimation problem is to remove the max-operator in the policy evaluation step by optimizing the alternative loss:
Lo (0) Re (Qo(s,a) (-+
This update rule relies on transitions (s, a, a,,, s'a)s collected by the behavior policy T and resembles the policy evaluation step of SARSA (Rummery & Niranjan, 1994; Van Seijen et al., 2009). Since the update contains no max-operator, and Qa are evaluated only on state-action pairs that are part of the dataset, the learning process is not affected by overestimation. However, the removal of the max-operator means the update simply tries to evaluate the value of the behavioral policy. The astute reader may question our ability improve upon the behavioral policy when using this update rule. We note, that when acting using the greedy policy T(s) = arg max, Qo a) we are in fact performing a single policy improvement step. Fortunately, this one step is typically sufficient for dramatic gains as we show in our experiments (see for example Fig. 9). This finding matches our understanding that policy iteration algorithms typically do not require more than few steps to converge to the optimal policy (Lagoudakis & Parr, 2003; Sutton & Barto, 2018, Chapter 4.3).
3.2 RANKING REGULARIZATION
Policy evaluation with Eq. (3) effectively reduces overestimation during training. But also avoids learning the Q values of OOD actions. Due to the lack of learning, these values likely erroncous, and many will err on the side of overestimation, thus harming the greedy policy. This is in contrast with the tabular case, where all OOD actions will have default value of 0
4

---

Under review as a conference paper at ICLR 2021
To robustify the policy improvement step, natural choice is to regularize the function approximator such that it behaves more predictable on unseen inputs. Forcing the neural network to output O for OOD actions might require very non-smooth behavior of the hewwrrkhence regularizer that asks the model only to assign lower values to state-action pairs that have not been observed during learning. We formulate this ranking loss which follows typical hinge-loss approximation (Chen et al., 2009; Burges et al., 2005) for ranking problems. Given a transition from the dataset (st, (r) this can be formulated as
choose less harsh
A
C(0) =
Σ max O
(o(ss,,a)) a²) + v,
(4)
While equation (4) does, in expectation, encourage lower ranks for OOD action, it can also have the adverse effect of promoting suboptimal behavior that frequent in the dataset. This because for any transition, proportional its frequency in the dataset, the regularizer pushes the value of all but the selected action down, promoting a policy that picks the selected action regardless of its value. To minimize this effect, weigh the regularization based on the value of the trajectory:
IAI
C(O) - exp (c0 G (>))))) 13
Σ O.igfe
max (Qo(),a.) Qo(s, + v,, (5)
where [E) is estimated by average over (8) in mini-batches. In all our experiments, we fix to be 5e - and 3 to be 2. The new formulation of the loss ensures that particularly on trajectories performed well in the dataset, trajectories that are likely for policy learned using behavior evaluation, the OOD action rank lower than observed actions. We note that our rank loss, when viewed through the lens of on-policy online RI,, can be related to ranking policy gradients (Lin Zhou, 2020) or to (Su et al., 2020; Pohlen et al., 201 8) who also used hinge loss as regularizer but with different goals from ours.
3.3 REPARAMETRIZAATION OF Q-VALUES
The overestimation of state-action values can be severe in offline RL, and can escape towards infinity (see for example appendix A). While behavior value estimation can be an effective way for suppressing this overestimation, when one iteration of policy improvement insufficient, one may want to bring back the max-operator and therefore the implicit policy improvement step of Q-learning. To better handle this scenario, we introduce a complimentary method to prevent severe over-estimation by bounding the values predicted by the critic via reparamcterization. Specifically, reparameterize the critic Qe a) aQa(s, a) given state- and action-indepente independent scale parameter a. This, in effect, disentangles the scale from the relative magnitude of values predicted, but also enables reduce the variance of the estimations, we update a by stochastic gradient descent, but with larger minibatches and a smaller learning rate.
impose constraints the scale parameter. To further stabilize the learning and
In our formulation, the "standardized" value Qo a) [-1, 1] is attained by using tanh activation function. Note that the tanh activation has the side effect of reducing numerical resolution for representing extreme values (as the tanh will be in its saturated regime), minimizing the ability the learning process to keep growing these values by bootstrapping on each other. We let a cxp(p) such that a > 0. Our parameterization thus ensures that Q-values are always bounded in absolute value by i.e. Q(s, a) E [-a, a]. The equations below show how critic scaling can be adapted into the Q-learning objective:
Lo (0, = (ssaa (aQo(s, -
(rr (r+ 7aa max Q0 (s' (s()''))
(6)
The introduction of a allows us to conveniently regularize the scale of Q values without disturbing the ranking between actions. More precisely, we introduce regularization term on a
C(a) = Elssotplus(aaosss.. G*
where C(a) represents soft-constraint requiring Q values stay close the performance of the behavioral policy, and thereby prevent gross overestimation. In Eq. (7), we rely on the softplus function constrain a only when Q0 a) ($)).
5

---

Under review as a conference paper at ICLR 2021
4 EXPERIMENTS
We investigate the performance of discrete offline RL algorithms on the three aforementioned open- source domains: Atari, DeepMind Lab, and bsuite. A question we are particularly interested in answering is: how does the lack of coverage of the state-action pairs affect the performance of each algorithm? In that context, we study each algorithms' robustness to dataset size (see Fig. 6), noise (sce Fig. 3 and 7), and reward distribution (Fig. 9 in Appendix), as they all affect the datasets' coverage of the state and action space.
Because we explore various ablations of our proposed approach, discussed in Section 3, we used a specific acronym for each potential combination. According to our naming convention, Q is for Q-learning and B is for behavior value estimation as the underlying RL loss, R indicates the use of the ranking regularization and r the use of reparametrization. In that vein, QRr refers to Q-learning with ranking regularization and reparametrization and BR stands for behavior value estimation with ranking regularization (see Appendix B.1.) We note that, both our DQN and R2D2 experiments used Double Q-learning ( (Van Hasselt al., 2015), but for our approach (and ablations of it) that rely on Q-learning, we used the vanilla Q-learning algorithm. More details for each experimental setup appear in Appendix D. We also provide more analysis and additional results in Appendix B.
We used an open-source Atari offline RL dataset, which is a part of RL Unplugged (Gulcehre et al., 2020) benchmark suite. We have created two new offline RL datasets for bsuite and DeepMind Lab, which we are going to opensource. The details of those datasets are provided in Appendix C.
4.1 BSUITE EXPERIMENTS
bsuite (Osband et al., 2019) is a proposed bench- mark designed to highlight key aspects of agent scal- ability such as exploration, memory, credit assign- ment, etc. We have generated low-coverage offline RL datasets for catch and cartpole as described by Agarwal et al. (2019a) (see Appendix C.1 for details).
cartpole
100
50
% Perforannc
In Fig. 3, we compare the performance of BRr and QRr with four baselines: DDQN (Hasselt, 2010), CQL (Kumar et al., 2020), REM (Agarwal et al., 2019 20)9a) and BCQ (Fujimoto sider two tasks, each in five versions defined by the amount of injected noise. The noise is injected into transitions by replacing the actions from an agent with a random action with probability c.
catch
100
al., 2018). We con-
Epsilon Noise (x 0.1)
DDQN
REM
BR
On the harder dataset (cartpole), ERr, the proposed method, outperforms all other approaches showing the efficiency of our approach and its robustness to noise. Two other methods, QRr (proposed by us as an ablation BRr) and CQL, also perform relatively well. The results for catch are similar, with the ex- ception that BCQ also improves performance which re-emphasises the importance of restricting behavior to stay close to the observed data. We have additional results on mountain cart car, where most algorithms be- have well except DDQN (see Appendix D.4).
BCQ
CQL
QRr
Figure 3: Bsuite Experiments: bsuite experi- mental results on two environments with respect to different levels of noise injected into the ac- tions in the dataset. The proposed method, BRr, outperforms
the baselines on cartpole. Meth- ods implementing a form of hehavior constraining (BCQ, CQL and our methods BRr and QRr) excel catch, stressing its importance.
4.2 ATARI EXPERIMENTS
Atari is an established online RL benchmark (Bellemare et al., 2013), which has recently attracted the attention of the offline RL community (Agarwal et al., 2019a; Fujimoto al., 2019a) arguably because the diversity of games presents a challenge for offline RL methods. Here, we used the experimental protocol and datasets from the RL Unplugged Atari benchmark (Gulcehre et al., 2020). We report the median normalized score across the Atari games, and the error bars show a bootstrapped estimate of the [25, 75] percentile interval for the median estimate computed across different games.

---

Under review as a conference paper at ICLR 2021
In Fig. 4, we show that QRr outperforms all baselines reported in the RL Unplugged benchmark as well as CQL (Kumar et al., 2020). While BRr performs well, this experiment highlights the potential limita- tion of doing a single policy improvement iteration in rich data regimes. Because in the considered setting there is enough data for the neural networks to learn reasonable approximations of the Q-value (exploit- ing the structure of the state space extrapolate for unobserved state-action pairs), one can gain more by reverting to Q-learning in order to do multiple policy improvement steps. However this amplifies the role of the regularization and in particular the reparametrization. Therefore, in this setting, QRI, which we proposed as an ablation to BRr outperforms
112
112
100
1992
100
98.9
803 REM 12N 64r
Figure 4: Atari results:
We compare our proposed QRr results against other recent State of Art offline RL methods on the Atari offline policy selection games from RL Unplugged benchmark.
1134
other techniques. Fig. 4 also shows the robustness of QRr's hyperparameters to different tasks.
Dartomarre
1.7
Ablation Experiments on Atari We ablate three different aspects of our algorithm on online policy selection games: i) the choice of TD backup up- dates (Q-learning or behavior value estimation), ii) the effect of ranking regularization, iii) the reparame- terization on the critic. We show the ablation of those three components in Fig. We observed the largest improvement when using ranking regularization. In general, we found that estimating the Monte-Carlo returns directly from the value function (we refer this in our plot as "MC Learning") does not work on Atari. However, behavior value estimation and Q-learning both have similar performance on the full dataset, however in low data regimes, the behavior value pol- icy considerably outperforms Q-learning (see Fig. 9 in Appendix B).
3
Modes
Figure 5: Ablations (online policy selection games): [LEFT] We compare behavior value esti- mation (B), DDQN and Monte Carlo approaches in offline RL Atari dataset. B and DDQN achieve similar median episodic returns, learning with Monte Carlo returns performs poorly. [RIGHT] We show various ablation studies (in terms of us- ing regularization, reparametrization and behavior value estimation). We found the most significant improvement from the ranking regularization term. Although the combination of ranking regulariza- tion and the reparameterization performs the best.
10'
Overestimation Experiments Q-learning can over-estimate due to the maximization bias, which happens due to the max-operator in the backups (Hasselt, 2010). In the offline setting another source of overestimation, as discussed in Section 2, OOD actions due the dataset's limited coverage. Double DQN (DDQN by Hasselt (2010)) is supposed address the first problem, but it is unclear whether it can address the second. In Fig. 6, we show that in the offline setting DDQN still over-estimates severely when we evaluate the critic's predictions in the environment. We believe this is because the second factor is the main reason of overestimation, which is not explicitly addressed by DDQN. However, Qr (vanilla Q-learning with reparametrization) and B are not effected from the reduced dataset size and coverage as much. In the figure, we compute the over-estimation error as Eie (inax(Q* (s, a) G (s), 0))² over 100 episodes, where (*(s) corresponds to the discounted sum of rewards from state s till the end of episode by following the policy т.
10a
Modian Coer-sstmabo
10*
10"
to
10"
10"
10"
10"1
100
% Dataset
Figure 6: Overestimation of Q-values with sub- sampled Atari datasets (% of Dataset). DDQN over-estimates the value of states severely whereas Qr and B reduce over-estimation greatly. We report median over-estimation error over online policy selection games on Atari.
Robustness Experiments In Appendix B.3 (see Figure 9), investigate the robustness of B and DDQN with respect to the reward distribution and dataset sizes. We found that the performance of more robust than DDQN to the variations on the reward distribution and the dataset size.
7

---

Under review as a conference paper at ICLR 2021
20.0
14-
17.0
257
150
22
Figure DeepMind Lab Results: We compare the performance of different baselines challenging DeepMind Lab datasets coming from four different DeepMind Lab levels. Our method, BR, consistently performs the best.
4.3 DEEPMIND LAB EXPERIMENTS
Offline RL research mainly focused on fully observable environments such as Atari, However, in complex partially observable environment such as Deepmind Lab, is very difficult to obtain good coverage in the dataset even after collecting billions of transitions. To highlight this, we have generated datasets by training an online R2D2 agent on DeepMind Lab levels. Specifically, we have generated datasets for four of the levels: explore object rewards many, explore object rewards few, rooms watermaze, and rooms select nonmatching object. The details of the datasets are provided in the Appendix C.2.
We compare offline R2D2, CQL, BC, and BR on our DeepMind Lab datasets. In contrast to Atari, BR performed better than QR according our preliminary results. Thus, here, decided only focus on BR. We use the same network architecture and hence, the models vary only is the loss function. We want compare there is a large amount of data stored during online training with online R2D2. In Figure 7, we show the performance of each algorithm on different levels. Our proposed modifications, BR and outperform other offline RL approaches on all DeepMind Lab levels. We argue that poor performance of R2D2 in the offline setting is due to the low coverage of the dataset. Despite having on the order 300M transitions, since the environment is partially observable and diverse, it is still not enough to cover enough of all possible state-action pairs. We present further results about dataset coverage on Deepmind Lab seekavoid where we showed that BR is more robust to the dataset coverage than other offline RL methods.
baselines' performance on our Deepmind Lab datasets when
level with dataset generated by fixed policy in Appendix B.2
5 DISCUSSION
In this work, we first highlight how, in the offline deep RL setting, overestimation errors may cause Q-learning to diverge, with weights and Q-value escaping towards infinity. We discuss using behavior value estimation to address this problem, which efficiently regresses to the Q-value of the behavior policy and then takes a policy improvement step at test time by acting greedily with respect to the learnt Q-value. The behavior value estimation oversteps the overestimation issue by avoiding the max-operator during training. We note that a single policy improvement step seems sufficient, especially in the low data regime, improve over the behavior policy and the policy discovered by double DQN. However, the max-operator used to construct the test time policy re-introduces overestimation errors that were avoided during training. We can address this issue by regularizing the function approximator with a ranking loss that encourages OOD actions to rank lower than the observed actions. This reduces overestimation at test time and improves performance. Nevertheless, we observe that behavior value estimation can be too conservative in rich data settings. In such scenarios, the function approximator can exploit more of the state and action space's underlying structure, leading to more reliable extrapolation. Therefore, it can be more lucrative to rely on Q-learning such scenarios, which can do multiple policy improvement steps, further constraining the function approximator. The resulting algorithm QRr, that is Q-learning with the ranking loss and reparametrization, outperforms all other approaches on the RL Unplugged Atari benchmark.
Overall behavior value estimation coupled with the ranking loss, is an effective algorithm for low data regimes. For larger data regimes, where the coverage is better, it is possible to achieve better performance by switching Q-learning and using reparametrization. The proposed methods outperform existing offline RL approaches on the considered benchmarks. As future work, we plan to extend our observations to the continuous control setup and towards more real-world applications.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam, Ethan Knight, and Picter Abbeel. Towards characterizing divergence in deep Q-learning. arXiv preprint arXiv:1903. 08894, 2019.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. optimistic perspective on offline reinforcement learning. 2019a.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. Striving for simplicity in off-policy deep reinforcement learning. Preprint arXiv: /eti:vv900.04543 2019b.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Gabriel Sarth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. In International Conference on Learning Representations, 2018.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801, 2016.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Mare Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In International Conference Machine Learning, pp. 449-458, 2017.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme Chris Hesse, et al. DotA 2 with large scale deep reinforcement learning, arXiv preprint arXiv:1912.0664. 2019.
Steven Bradtke and Andrew Barto. Linear least-squares algorithms for temporal difference learning. Machine Learning. 22:33-57 03 1996.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pp. 89-96, 2005.
Wei Chen, Tie- Yan Liu, Yanyan Lan, Zhi-Ming Ma, and Hang Li. Ranking measures and loss functions in learning to rank. In Advances in Neural Information Processing Systems, pp. 315-323, 2009.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint axXiv:2004.07219, 2020.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.0900 1802.09477, 2018
Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. Preprint arXiv:191.001708. 2019a.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062, 2019b.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. EMaQ: Expected- Max Q-Learning operator for simple yet effective offline and online RL. arXiv preprint arXiv:2007. 1091, 2020.

---

Under review as a conference paper at ICLR 2021
Albert Gu, Caglar Gulcehre, Tom Le Paine, Matt Hoffman, and Razvan Pascanu. Improving the gating mechanism of recurrent neural networks. arXiv preprint arXiv:1910.09890, 2019.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo. Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RL unplugged: Benchmarks for offline reinforcement learning. arXiv preprint arXiv:2006.13888 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with stochastic actor. arXiv preprint arX1v:1801.01290, 2018.
Hado V Hasselt. Double Q-leaming. In Advances in neural information processing systems, pp. 2613-2621, 2010.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. arXiv preprint arXiv: arXiv:1710.02298. 2017.
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcchre, Tom Le Paine, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979, 2020. URL //tp:v..orgiooooooooooo. 0tt5/2006.009979
Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrent experience replay in distributed reinforcement learning. In International Conference Learning Representations, 2019, URL https: /esttss://ppereeette http://///p/eeeeeeeeeeeoee net forum?ormm? wtt.:eeennooommmmdddd1letoyeeeee iii-:riddrrlyyy"".
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2014.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. Conference on Neural Information Processing Systems, pp. 117761-17711 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
Michail G. Lagoudakis and Ronald Parr. Least-squares policy iteration. Journal of Machine Learning Research, 4:1 107-19919 2003.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. Marco Wiering and Martijn Otterlo (eds.), Reinforcement Learning: State-of-the-AAt, pp. 45-73. Springer Berlin Heidelberg, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint ar Xiv:20005.0/643 2020.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. Preprint arXiv:1509.02971, 2015.
Kaixiang Lin and Jiayu Zhou, Ranking policy gradient. In 8th International Conference on Learning Representations, ICLR 2020, Addis Abuba, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL /ttp::::///peeevveie http:///wwviieeeeeeefeeeraeaeaaa
net wow.mmiiidrooddeEete...
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
10

---

Under review as a conference paper at ICLR 2021
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepezvari. Satinder Singh, et al. Behaviour suite for reinforce- ment learning. arXiv preprint arXiv: 10Xf::/908.03666 2019.
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural networks, 21 21(4):682-697, 2008.
Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Barth-Maron, Hado Van Hasselt, John Quan, Mel Večerík, et al. Observe and look further: Achieving consistent performance on Atari. arXiv preprint arXiv:/805: 1193, 2018.
Martin Riedmiller. Neural fitted Q iteration first experiences with data efficient neural reinforce- ment learning method. In João Gama, Rui Camacho, Pavel B. Brazdil, Alípio Mário Jorge, and Luís Torgo (eds.), European Conference Machine Learning, pp. 317-328, 2005.
Gavin Rummery and Mahesan Niranjan. On-line Q-learning using connectionist systems, vol- ume 37, University of Cambridge, Department of Engineering Cambridge, UK, 1994.
Tom Schaul, John Quan, Toannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv: 1.005995/...0..... 2015.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference simulation of adaptive behavior: From animals to animats, pp. 222-227, 1991.
Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Reprexentations, 2020.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676)3354- 359, 2017.
Andy Su, Jayden Ooi, Tyler Lu, Dale Schuurmans, and Craig Boutilier. Conqur: Mitigating delusional bias in deep q-learning arXiv preprint arXiv:2002. 12399, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. arXiv preprint 10XXvv15090066611 2015.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv: 1812.026440 2018.
Harm Van Seijen, Hado Van Hasselt, Shimon Whiteson, and Marco Wiering. theoretical and empirical analysis of expected SARSA. In 2009 ieee symposium on adaptive dynamic programming and reinforcement learning, pp. 177-184. IEEE, 2009.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czameeki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-333 2019.
Ziyu Wang, Alexander Novikov, Konrad Žoina, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. arXiv preprint arXiv:2006. 200::2006./13344 15134, 2020.
Ronald J Williams. Simple statistical folllwein- algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-2)) 1992.
11

---

Under review as a conference paper at ICLR 2021
A Q-LEARNING CAN ESCAPE TO INFINITY IN THE OFFLINE CASE
Remark 1. Q-learning, using neural networks as function approximator, can diverge in the offline RL setting given that the collected dataset does not include all possible state-actions pairs, even if it contains all transitions along optimal paths. Furthermore, the parameters (and hence the Q-values themselves) can espace towards infinity under gradient descent dynamics.
Proof. The proof relies providing particular instance where Qllarniig diverges towards infinity. This is sufficient to show that divergence can happen. Note that the remark does not make any statement of how likely is for this to happen, nor is providing sufficient conditions under which such divergence has to happen.
Let us consider simple deterministic MDP depicted in the figure below (left).
S4
ae, 1=1
a2, r=0
r
W
U
h
h
h4
S1
As or a1.
0
S3
ao or a
2
-1
as, =0
a2, 0
r
S2
S
a[0]
a[1]
a[2]
Depiction of the MDP
Depiction of the MLP
= {51, $2, $3, } is the set of all states, where Si is deterministically the starting state and S is the terminal state of the MDP. Let A = {ao, a1, 02) be the set of all possible actions. Let the reward function r(s, a) be for all action-state pair except r(31, ag) which 1. Let the transition probabilities P(s'|s, s, a) be deterministic defined by the depicted arrows. Le. for any state action pair only transitioning to one state has probability 1, while the rest has probability 0. For example, only (884 81, 00) 1, while P(ss|si, an) 0, P(22|81, an) (s2 81, un) 0. For 81, a only P(*2|81, a1) = 1 and so on and so forth.
First observation is that the optimal behavior is to pick action og (as it is the only rewarding transition in the entire MDP).
The features describing each state are given by single real number, where s 0, s2 1, - B, with 3 > > 0, where is the discount factor. Assume actions are provided to the neural network as one-hot vectors, i.e. dg j0, 0, 11" a1 = [0, 1, 0J a2 [1, O, 0 where we will refer afi as the i-th element of the vector that represents the action a. For example ag [2] = 11 and ou [ = 0.
Let us consider the Q-function parametrized MLP uses rectifier activations, and gets as input both the state and action, returning single scalar value which the Q-value for that particular state action combination. Rewriting the diagram in analytical form we have that for 8 E R and R3:
simple MLP (depicted in the figure above left). The
Qo(s,a) a) = =
w. r + +ureeu((a()) - 2s) + usrelu(a[1] 2s) + ugrelu( al2))
A note on initialization. The weights of the first layer are given as constants. The process would work if we leave them be learnable as well, but the analysis would become considerably harder. The exact value used, -2, 1, -1, are not important. In principle we care for the negative weights connecting to h22 hgg ha be larger in magnitude than those from ali] to h, and we care for the weight between a d[2] and h2 to be negative. They can be scaled arbitrarily small and do not need to be identical.
one-hot representation is the typical representation for action in discrete spaces
12

---

a V.... ... for some learning rate > O, the update creates a vicious loop that will increase the norm of w at every iterations, such that lim We D0. Given that the gradient on ug tracks w, it means that the path that takes action 02 Under review as a conference paper at ICLR 2021
What we will rely in the rest of the analysis is that the preactivation of h22 h3, h4 to be negative for state 82 and 83. This will be in the zero region of the rectifier, meaning no gradient will flow through those units. Since 83 $2 2 1 and afi] {0, 1]. it is sufficient for the weight from to h2, h3. h4 to be larger in magnitude than the weight from afi to b2. hgg h4- This ensures that for > 1, the Q-function is not function of u, as u, will get multiplied by 0. Also want the function never depend on ug to simplify our analysis, which is easily achievable if the weight going from a|2] to h4 negative.
Given the observations above, if we plug in the formula the different values of 8 and a; we get that:
Qn(s), ao)
11
Q,(s,1,a1)
1/2
Q0(81, 02)
(9)
Va E A, Va A,
Qa(s2,a) a)
Qa(53,
= +Bw
Note that this implies that
max, Q0( a) = w
(10)
max, Q0 a) Bw
Assume w > 0. And let the dataset collected by the behavior policy to contain the following transitions:
D {($1, Gn, 14), (s,, 0, s2), (82, 02 0, 8z)}
We can now construct the Q-learning loss that we will use to learn the function Q in the offline case which will be
C (. (Qo a) - O/ ())
(Q((1).8)) 50) - 1)) + (Qo(s), a1) max, (82 (Qo a2) - max, (/(53, a))² (ui - 1)2 + -wu >w')² + (w 73w')²
(11)
Note that relied on Eq. (10) evaluate the max operator and 0' is copy 0, that is used for bootstrapping. This is the standard definition of Q-learning see Eq. (2). In particular in this toy example 6' is numerically always identical to 8 (in general it can be trailing copy of 0 from steps back) and is used more to indicate that when we take a derivative of the loss with respect to f we do not differentiate through Q/- From Eq. (11) contributes to the gradient of u1, only the second transition contributes to the gradient of w2 and only the third transition contributes the gradient of w. We can not evaluate the gradient with respect to of the loss C over the entire dataset:
notice that only the first transition in dataset
V aa U1
V U12 (0 + yw)
(12)
V w (0 + 7Bw)
= =(1 = -1-788)
V w
(0+ 7Bw)
Note that we assumed > 0 and for simplicity we exploited that w/ = w numerically, to able to better understand the dynamics of the update. Given that 3 > 1 V.. will always be negative as long as w (and implicitly w') stays positive. Given that w, wT
7.
in the initial state $1 will have +00 as value. Note that all transitions along the optimal path of this deterministic MDP are part of the dataset.
Thhe fact that gradient gets propagated in the first layer is only important if we attempt to consider the case when the first layer weights learnable.
13

---

Under review as a conference paper at ICLR 2021
Also that given our example, the same will happen if rely on SGD rather than batch GD (as the different examples affect different parameters of the model independently and there is no effect from averaging). Preconditioning the updates (as for e.g. is done by Adam or rmsprop) will also not change the result as they will not affect the sign of the gradient (the preconditioning matrix needs to be positive definite). Neither momentum will not affect the divergence of learning, it will not affect the sign of the update.
This means that the provided MDP will diverge towards infinity under the updates on most commonly used gradient based algorithms.
B ADDITIONAL RESULTS AND ABLATIONS
B.1 ACRONYMS
In Table 1, we provided the acronyms for our models and their corresponding meanings.
Table I: Acronyms for our models and their expansions
Acronym
Meaning
B
Behavior Value Estimation
BR
Behavior Value Estimation with Ranking Regularization
BRr
Behavior Value Estimation with Ranking Regularization and reparameterization
Q
Standard Q-learning
QR
Standard Q-learning with Ranking Regularization
QRr
Standard Q-learning with Ranking Regularization and reparameterization
B.2 DEEPMIND LAB: THE EFFECT OF COVERAGE ON OFFLINE LEARNING
As mentioned in Section 4.3 we investigate the effect of coverage on the DeepMind Lab seekavoid arena 01 level. To do so, we have created another set of datasets which is gener- ated by using a fixed R212 snapshot with different noise levels when evaluating the trained snapshot in the environment for seekavoid_arena_.0 aarena011 level. We have used different cs in the <-greedy algorithm to create datasets with different noise levels. The also effects the coverage the dataset.
We compare R2D2, CQL, BC, B and BR on these DeepMind Lab datasets by using the same network architecture-thee the only change among the models the loss function.
We investigate the effect of coverage in the Deep- Mind Lab seekavoid_arent aariiddaareaa01111 level, by evaluating the policy with different e's for the epsilon-greedy in the environment and storing each episode in the dataset. Increasing the will increase the coverage of the dataset but also it will increase the noise in the dataset well. In Figure 8, we show the effect on the simple Deep Lab level. When = 0, BC works outperforms offline RL approaches, however increasing the level of noise deteriorates the perfor- mance of BC, and BR starts to performs better. Since the environment is deterministic, if the policy is deter- ministic as well, this corresponds to only having one single unique episode in the dataset. As we increase the epsilon the coverage in the dataset and diversity of the trajectories will increase as well. We trained all models by unrolling on the whole episode and trained using back propaaaapantioon through time.
Nodaa Lovel
Figure Effect of coverage in the dataset: compare offline RL models with varying the noise level in the environment. Increasing the noise level increases the coverage well. BC performs well with noise, however, BR performs significantly better as the noise increases. Let us note that, in all our experiments, R2D2 uses double Q-learning.
14

---

Under review as a conference paper at ICLR 2021
B.3 ATARI: ROBUSTNESS TO DATA
The robustness the reward distribution in the dataset is an important feature required deploy offline RL algorithms in the real-world. We would like to understand the robustness of behavior value estimation in the offiin RL setting. Thus, we first investigate the robustness of B in contrast to Q-learning with respect to the datasets' size and the reward distribution. In Fig. 9, we split out the dataset into two smaller datasets: i) transitions coming from only highly rewarding ii) transitions from only poorly performing episodes. We show that outperforms Q-learning in both settings.
100
94.1
110
DDON
DOON
100
80
Purfomance
59.6
80
74.8
70
40
80
90
20
14,0
13.5
Episodk: Roward Mean
Eskodi Rouuli Mean
Fltcrig
Dataset
Figure 9: Robustness Experiments: (left) We compare DQN and B in terms of their robustness the reward distribution on Atari online policy selection games. split the datasets in two bins: the dataset that only contains transitions that are coming from episodes that have episodic return less than the mean episodic return in the dataset ("Episoiic Reward < Mcan"), transitions coming from episodes with return higher than the mean return in the dataset (" ("Episodic Reward Mcan"), performs better than DQN in both cases. (right) Normalized scores of DQN and subsets of data from online policy selection games. performs comparatively better than DQN The Q-learning suffers more since coverage the datasel reduces causes more severe extrapolation error.
subsampling which
B.4 ON THE EFFECT OF REGULARIZATION
this Section we study the effect the regularization on the action gap and the overestimation error. In Figure 10, we show that increasing the regularization co-efficient for the ranking regularization increases the action gap across the Atari online policy selection games which can result to lower estimation error and better optimization.
In Figure 11, we show the effect of increasing the regularization on the overestimation of the Q- network when evaluated in the environment. We show the mean over-estimation across the games.
B.5 ONLINE POLICY SELECTION GAMES RESULTS
In Figure 12, we show the performance of different models with respect to the rewards they achieve the training.
15

---

Under review as a conference paper at ICLR 2021
0.0
0000
0.99
0.15
0.02
1000:
300.E8TE Cooee
Veer 00w.000110
0004
0.2
0.00
0.20
0.10
6.00
3,900
Figure 10: The Effect of increasing the ranking regularization on the action gap.
Naa Donestationn
Sactor burbiid
3.00
1-56105
Learner Stopa
Figure 11: The Effect of increasing the ranking regularization on the overestimation.
16

---

Under review as a conference paper at ICLR 2021
0.40000
Voore
Figure 12: The Raw Returns obtained by each baseline Atari online Policy Selection Games.
B.6 OVERESTIMATION ON ONLINE POLICY SELECTION GAMES
In Figure 13 and 14, we report the value error of B, and DQN's value error and squared value error respectively.
440000 200000
Figure 13: The value error computed in the environment by evaluating the agent and computed with respect to the ground truth discounted returns. The negative values indicate under-cstimation and positive values for over-estimation.
17

---

Under review as a conference paper at ICLR 2021
Figure 14: The squared value error computed in the environment by evaluating the agent and computed with respect to the ground truth discounted returns and reporting the mean squared values of the values.
C DETAILS OF DATASETS
C.I BSUITE DATASET
BSuite (Osband al., 2019) data was collected by training DQN agents (Mnih et al., 2015) with the default setting in Acme (Hoffman et al., 2020) from scratch in each of the three tasks: cartpole, catch, and mountain car. We convert the originally deterministic environments into stochastic ones by randomly replacing the agent action with uniformly sampled action with probability of e E {0, 0.1, 0.2, 0.3, 0.4, 0.5) (ie. e = 0 corresponds to the original environment). We train agents (separately for each randomness level and 5 seeds, i.e. 25 agents per game) for 1000, 2000, 500 episodes cartpole, catch and mountain _arr respectively. The number of episodes is chosen so that agents all levels can reach their best performance. We record all the experience generated through the training process. Then to reduce the coverage of the datasets and make them more challenging we only used 10% of the data by subsampling More details of the dataset are provided in Table 2. The results presented in the paper are averaged over the random seeds.
C.2 DEEPMIND LAB DATASET
DeepMind Lab (Beattie et al., 2016) data was collected by training distributed R2D2 (Kapturowski al., 2019) agents from scratch on individual tasks. First, tuned the hyperparameters of distributed version of the Acme (Hoffman et al., 2020) R2D2 agent independently for every task achieve fast learning in terms actor steps. Then, we recorded the experience across all actors during entire training runs a few times for every task. Training was stopped after there was further progress in learning across all runs, with resulting number steps for each run between 50 million for the easiest task (seekavoid.arena.. built a separate offline RL dataset for every run and every task. See more details about these datasets in Table 3.
and 200 million for some of the hard tasks. Finally
Additionally, for the seekavoid.arent awr.rraa.... 01 task we ran two fully trained snapshots of our R2D2 agents the environment with different levels of noise (e 0, 0.01, 0. 0.25 for e-greedy action selection). We recorded all interactions with the environment and generated different offline RL dataset containing 10 million actor steps for every agent and every value of c.
18

---

Under review as a conference paper at ICLR 2021
Environments
Number of episodes
Number of transitions
Average episode length
cartpole (e 0.0)
1000
710K
710
cartpole (e 0.1)
1000
773K
773
cartpole (c 0.2)
1000
649K
649
cartpole (c 0.3)
1000
607K
607
cartpole (E 0.4)
1000
672K
672
cartpole (E 0.5)
1000
643K
643
catch (e 0.0)
200
T.8K
9
catch 0.1)
200
1.8K
9
catch (e 0.2)
200
1.8K
9
catch 0.3)
200
1.8K
9
catch (c 0.4)
200
1.8K
9
catch 0.5)
200
1.8K
9
mountain (c 0.0)
50
10K
205
mountain car (c = 0.1)
50
10K
210
mountain (c 0.2)
50
22K
447
mountain car (e 0.3)
50
13K
277
mountain_car 0.4)
50
12K
250
mountain.car (E 0.5)
50
24K
494
Table BSuite dataset details.
explore object rewards few
explore objest rewards many
39300
2009
21000
1999
14000
60
309
100
320
rooms, select nonmatching object
23000
20000
503009
11:00
200000
18:00
159993
59383
snapshot/eekaweld arona 01-ops=0.01
hasppshattsokavvvkkkk arona, 01-cps=0..25
17580
15980
10000
7580
1000
20
13
Figure 15: DeepMind Lab Reward Distribution: We show the reward distributions for the DeepMind Lab datasets. The vertical red line indicates the average episodic return in the datasets.
19

---

Under review as a conference paper at ICLR 2021
Table 3: Deep.ind Lab dataset details. For training data, reward is measured the maximum training the average reward over runs for the same task. snapshot data, reward just an average over episodes recorded using the same level of noise.
Task
Episode Length
Datasets
Episodes (K)
Steps (M)
Reward
menskavoid_aeeaaa
300
667.1
200.1
39.0
seekavoid_ aaredaaarrea_000 snapshot (e 0)
300
66.7
20
40.4
seekavoid arena 01 snapshot (e = 0.01
300
40.1
seekavoid arena 01 snapshot (e = 0.1)
300
66.7
20
36.9
seekavoid arena 01 snapshot (c = 0.25)
300
2
66.7
20
29.7
exp_ore.objecttrevardtt
1350
178.3
240.7
51.5
explore.sbjjettrrrarrtee.-e
1800
334.1
601.4
64.5
coom__select.normatccooo rooms watermaze
180
2001.1
360.2
32.5
1800
201.8
363.3
48.8
D EXPERIMENT DETAILS
We used the Adam optimizer (Kingma & Ba, 2014) for all experiments. For details the used hyperparameters, refer to the Table 4 for bsuite, Table 5 for Atari, and Table 6 for DeepMind Lab. Our evaluation protocol is described below, in Section D.I. On Atari experiments, have normalized the agents' scores as described in (Gulcehre et al., 2020).
On Atari, in all experiments report the median normalized score along with the bootstraps estimates of 75th and 25th percentiles for the interquantile range estimates of the errors in the error bars as done by (Gulcehre et al., 2020).
Atari Hyperparameters: On Atari we directly used the baselines and the hyperparameters reported in (Gulcehre et al., 2020), to get the detailed Atari results on test set communicated with the authors. We have run additional CQL and our own models with ranking regularization and reparameterization. For CQL we have finetuned both the learning rate from the grid [Se 5, le 4, 3e 4] and the regularization hyperparameter a E [0.005. 0.05, 0.01, 0.1, 1]. For our own proposed models we have only tuned the learning rate from the grid 88 5, le 4, 3e 4] and the ranking regularization hyperparameter from the grid [0.005, 0.05, 0.01, 0.1, 1]. We have fixed the rest of the hyperparameters. As mentioned earlier, we have only used the online policy selection games for finetuning the hyperparameters. As a result of our grid search, we have used learning rate of le 4 for CQL and our models. We have used 0.01 for the hyperparameter of CQL. 0.05 seems to be the optimal hyperparameter choice for the ranking regularization hyperparameter.
DeepMind Lab Hyperparameters: On DeepMind Lab experiments, we tuned the hyperparam- eters of each model individually on each level separately. We have tuned the learning rate and the regularization hyperparameters for each model from the same grid that we have used for Atari. All our algorithms are n-step in DeepMind Lab experiments, where n is fixed to 5 in all our experiments. Thus both behavior value estimation and Q-learning experiments use steps of unrolls for learning.
D.1 EVALUATION PROTOCOL
To evaluate the performance of the various methods, use the following protocol:
We sweep over a small (5-10) sets of hyperparameter values for each of the methods.
2. We independently train each of the models on 5 datasets generated by running the behavior policy with 5 different seeds (ie. producing 25-50 runs per problem setting and method).
3. We evaluate the produced models in the original environments (without the noise).
4. We average the results over seeds and report the results the best hyperparameter for each method.
20

---

Under review as a conference paper at ICLR 2021
D.1.1 EVALUATION METHOD
To evaluate models (step 3. above), in the case of bauite and DeepMind Lab we ran an evaluation job in parallel to the training one. It repeatedly read the learner's 's checkpoint and produced evaluation results during training. We report the average of the evaluation scores over the last 100 learning steps.
In the case of the Atari environments, instead of averaging performance during the final steps of learning, we take the final snapshot produced by a given method and evaluate it on a '100* environment steps after the training finished.
Table 4; bsuite experiments' hyperparameters. The top section of the table corresponds to the shared hyperparameters of the offline RL methods and the bottom section of the table contrasts the hyperparameters of Online Offline DQN.
Hyperparameter
setting (for both variations)
Discount factor
0.99
Mini-batch size
128
Target network update period
every 2500 updates
Evaluation
0.4
Q-network:
an MLP
Q Q-network: hidden units
56, 56, num actions 2M learning steps
Training Steps
Hardware
Tesla V100 GPU
Replay Scheme
Uniform
Hyperparameter
Online
Offline
Min replay size for sampling
20,000
Training (for e-greedy exploration)
0.01
e-decay schedule
250K steps
Fixed Replay Memory
No
Yes
Replay Memory size
IM steps No
2M steps
Double DQN
Yes
Table 5: Atari experiments' hyperparameters. The top section of the table corresponds to the shared hyperpa- rameters of the offline RI. methods and the hottom section of the table contrasts the hyperparameters of Online Offline DQN.
Hyperparameter
setting (for both variations)
Discount factor
0.99
Mini-batch size
256
Target network update period
every 2500 updates
Evaluation e
0.4
Q-network: channels
32, 64, 64
Qenewwork: filter size
8 8, 4 x 4, 3 x 3 4, 2,
Q-netwrrk: stride
Q Q-network: hidden units Training Steps
512
2M learning steps
Hardware Replay Scheme
Tesla V100 GPU
Uniform
Hyperparameter
Online
Offline
Min replay size for sampling
20,000
Training (for e-greedy exploration)
0.01
e-decay schedule
250K steps
Fixed Replay Memory
No
Yes 2M steps
Replay Memory size
1M steps No
Double DQN
Yes
21

---

Under review as a conference paper at ICLR 2021
Table 6: Deepmind Lab experiments' hyperparameters. The top section of the table corresponds to the shared hyperparameters the offline methods and the bottom section the table contrasts the hyperparameters of Online vs Offline DQN.
Hyperparameter
setting (for both variations)
Discount factor
0.997
Target network update period
every 400 updates
Evaluation
0.1*
Importance sampling exponent
0.6
Architecture
Canonical R2D2 (Kapturowski et al., 2019)
Hyperparameter
Online
Offline
Hardware
4x TPUv2
4x Tesla V100 GPU
Training Steps
50-200M actor steps 120 (40 burn-in)
50K learning steps Full episode
Sequence Length
Mini-batch size
32
Training (for e-greedy exploration)
0.4, 0.4
Replay Scheme
Prioritized (exponent 0.9) 600K steps 12M steps
Min replay size for sampling
Replay Memory size
50-200M steps
D.2 ATARI OFFLINE POLICY SELECTION RESULTS
In Table 7, we show the performance of our baselines on different Atari Offline Policy selection games. We show that QRr outperforms other approaches significantly.
Table Atari Offiine Policy Selection Results: this table, list the median normalized performance different baselines.
Name
Normalized Score
BC
50.8
DDQN
83.1
CQL
98.9
BCQ
102.6
IQN 104.8
REM
104.7
QRr
108.2
D.3 DEEPMIND LAB DETAILED RESULTS
In Table 8, we have shown the results on the Deepmind Lab datasets. It is possible see from these numerica results that BR outperforms other approaches and still very competitive.
D.4 SSUITE DETAILED RESULTS
We generated datasets and performed experiments analogous these in Section 4.1 for mountain car environment. We present results for all three environments in Table 9, BRr outperforms all the baselines.
E REPARAMETRIZZING THE Q-NETWORK
In all reparameterized critic experiments we have used the tanh(-) activation function with refine gates help with optimization (Gu et al., 2019). We have not tuned the hyperparameters of the
22

---

Under review as a conference paper at ICLR 2021
Table 8: Detailed Results the DeepMind Lab: We provide the detailed results for each DeepMind levels along with the standard deviations.
BC
R2D2
CQL
B
BR
axplore..bjeet.reewadd...f few
1.8 1.0
19.8 1 4.0
23.8 +: 5.1
23.7 1 3.8
28.6 + 1.7
rewards many
2.9
8.5
9.3 2.5
7.6 3.1
13.4 1:
rooms_aatermaze
0.1 0.1
2.7 1.4
4.0 3.7
9.9 2.7
11.2 = 4.2
1.1 4.6
5.4
3.4 2.4
9.4 1: 6.3
10.4 9.6
seekavoid.arena.o0
28.02 + 7.6
4.7 4: 3.0
12.8 4: 10.7
4,4 + 0.9
17.07 + 10.1
seekavoid.arena.000 c 0.01
33.0 = 1.3
5.5 1.6
12.7 = 5.4
4.1 + 1.8
19.8 + 4.9
seekavoid arena 01, c 0.1
18.9 + 14.4
8.6 3.0
16.3 + 7.7
11.775 + 4.5
31.8 + 4.7
seekavoid arena 01, 6 0.25
17.46 7.5
13.5 5.1
13.5 + 5.06
9.0 0.25
25.57 7.0
Enviromments
DDQN
BCQ
REM
CQL
BRr
QRr
cartpole (c = =0.0)
203.5
244.3
383.7
354.6
933.8
358.3
cartpole (e =0:11 =0.1)
240.5
244.6
218.0
673.7
886.8
732.7
cartpole (c =0.2)
134.5
215.7
295.6
528.3
786.1
566.0
cartpole (e =0.3)
265.9
432.8
248.2
594.6
937.3
642.0
cartpole (e =0.4)
278.9
418.4
263.8
791.3
814.5
745.2
catch (c (I0)
-0.04
0.96
0.3
1.0
1.0
1.0
catch (c 0.1)
-0.19
0.85
0.18
1.0
1.0
1.0
catch 0.2)
0.08
0.91
0.34
1.0
1.0
0.99
catch (e 0.3)
-0.08
0.92
-0.05
1.0
0.99
1.0
catch 0.4)
-0.13
0.85
0.14
1.0
1.0
0.99
mountain_com = 0.0)
-196.5
142.0
-116.3
-12.33
-130.3
-128.7
mountain_ca (e = 0.1)
-231.5
-145.2
-167.0
-135.6
-127.1
141.5
mountain_cat (e 0.2)
-158.3
-161.1
-118.6
-120.0
-116.7
-40.3
mountain car (e 0.3)
-316.6
-180.9
-128.3
-154.8
-125.0
-13.44
mountain.ccm 0..4)
-125.1
-133.7
-127.6
-128.9
-127.1
-163.6
Table 9: BSuite mean results.
reparameterization in our experiments, have used four times larger minibatches to update the scale, since it is cheap to update single scalar and as shown in Algorithm 1, we have used twice smaller learning rate update the scale than the rest of the parameters of the network. This is heuristic, but we found this simple heuristic to work well across all the tasks that we have tried. Potentially it is possible to get better results by tuning the hyperparameters for reparameterization more carefully.
Algorithm 1 Algorithm Reparametrized Q-Network
Inputs: Dataset of trajectories D, batch size update 0: B1, batch size update 7: B2, and number of actors A.
Initialize Q weights
Initialize to 1.
Initialize target policy weights 0' +- for Msteps do
Sample transition sequences (St:++m Ge:t+mm Tte++m) from dataset construct a mini-batch size B.
Calculate loss [(t,, Ft, 8:41: 0. a)) using target network.
Update 0 with GD: f- 0 (.)
Update a with GD: a <- a If t mod ttrrget = 0, update the target weights and 0' + 0. a.
BB//BV,CCC
end for
As in Algorithm 1, there is stage of optimization to update the parameters of Q --ntwwork and the scale the Q values a. They both use different learning rates, is important to make sure that update the with smaller learning rate: 7/2 < m/
23

---

Under review as a conference paper at ICLR 2021
F RANKING REGULARIZER
We propose a family of methods that prevent the extrapolation error by suppressing the values of the actions that are not in the dataset. We achieve that by ranking the actions in the training set higher than the ones that do matter, we are rather interested relative ranking of the actions. Given d, is the action from dataset. For all j * and illustration purposes, the value iteration can be written as:
not in the training set. For the learned Q-function the absolute values of actions
E[max Q(s, a)] x E [[((Q(= ((s. E Maz] + E (P(Q(s,.)) x ((s.a;))Q(s e Max]
Maz] (B P(Q Q(sa)))) Maz) = aE [(((ss.a)=((..))))0)))a....))))) ai]] Max]
= ((P(Q(ssaa))((.)))))))))))
> ((s, =a (1[ [((0(s,a))
+ a/
[1 P(Q ((s,) aj)li e Max
at)lt Max]
+ E (1 P(Q (8, ar) ((s
where & is an irreducible noise, because we can not gather additional data on (s,, ay), and we don't know the corresponding reward for it. This causes extrapolation error which accumulates through the bootstrapping in the backups as noted by Kumar et al. (2019). We implicitly pull down the P(Q(s, at) * Qss, at)) by ranking the actions in the dataset higher which pushes up P(Q(s, a.) > Q(s az)). As result, the extrapolation error in Q-learning would also reduce.
F.1 PAIRWISE RANKING LOSS FOR Q-LEARNING
In this section, we discus the relationship between the pairwise ranking loss for Q-learning and the list-wise pairwise ranking losses.
Ptj sigm(()o(s, Qu(s,a;))
LAI II
T(aels)
P
i=0,e
[A|
Z Σ
II
Po
1= $=0.$$
A
R(0) = - log(Pe)
=
A log log(sigm(Q0(3,a)) (0(s,,,)))
A - Qo(s,a.))
We use a common approximation (Chen et al., 2009; Burges et al., 2005) the softplus-based log-likelihood is to use hinge-loss which can be seen as approximation:
c(0) = Σ
max (olls,))Q((ssa))
(13)
Imposing the constraint in Equation (13) can be harmful if the dataset has lots of suboptimal trajectories. Because this constraint will try to maximize the values of suboptimal actions in the
24

---

Under review as a conference paper at ICLR 2021
dataset. As result, similar to Wang et al. (2020), propose filtering function to impose that constraints only on rewarding transitions:
LA
C(0) cxp(G E.D E
Σ 1=0,i
max -Qo(s.))) + +x,0) 22.0))
(14)
F.2 RELATIONSHIP TO THE POLICY GRADIENTS
is possible to drive the foirmulation that we use for the ranking regularizer from the policy gradient theorem to show the relationship. The Ranking Policy Gradient Theorem formulates the optimization of long-term reward using a ranking objective as done in Lin & Zhou (2020). The proof below illustrates the formulation process. Let note that apply the ranking regularization the offline and off-policy data, such that thee formalism below only works when the behavior policy and target policy are equivalent, when the transitions coming from on-policy data. If the ranking regularizer used on the on-policy data it approximates the policy gradients, but it will not on the off-policy data.
Our construction is based on direct policy differentiation (Peters & Schaal, 2008; Williams, 1992)
where the objective function is to 0" arg max, J0)..
Ve.(()) V Σ. (s(
(15)
Vo log (og))
Vo log (I az)) (B(s)
Σ, po(T)
Volog Volog 84) (s)
ETT
Ve lor (r)
TR -Fg
log P(j) (B(
EETWST
log (sigm(p:))) G (8)
(16)
jーl softplus(p, (pt
Q(s, c9 (17)
E ET
rectifier (Q(s, (Q(s.a))QQ(saa))))
j=li
with baseline BynD 10 (s)] it will be,
ETNTY ΣS 1=1
Vo
(E rectifier (Q(s, a:) Q(s,a)))) (sn
m
Then apply the exp(-) transformation on (G (s) E EgED ( (s)) to impose this loss loss mostly on the rewarding trajectories, and we can turn the maximization problem to a minimization one with flip of sign:
-E,
T
Vo
Σ 1 rectifier (Q(s, 0.) Q(s, (()) exp (G (s) B
1=1
(19)
where the trajectory is series of state-action pairs from / = 1,.., T. i.e. (81, 82, 02 .... ST). The gradients in (19) is exactly the gradients of the ranking regularizer.
25


Reviews:
[{"review_id": "N8Jp-uVVMVC", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper focuses on the problem of Q value over-estimation in offline reinforcement learning and proposes three approaches (tricks) to help solve this problem. (1) estimate Q value of behavior policy avoiding max-operator in Q learning and take greedy action according to the behavior value estimation. (2) introduce ranking loss to push down the value estimation of all unobserved state-action pairs to avoid over-estimation. (3) use tanh operator to bound the range of Q value estimation, and learn a scale parameter with regularization term. The experimental results on several domains (Atari, Bsuite, Deepmind Lab) with discrete action space show performance better than existing algorithms.\n\nClarity:\nThis paper is generally written clearly, though I have several questions about the technique and experiments, which may need more clarification. Please see 'Cons' part for the detailed questions.\n\nOriginality:\nThe techniques of ranking regularization and re-parameterization of Q-values are novel in the literature of offline reinforcement learning. Behavior value estimation removing the max operator to alleviate over-estimation seems not novel, because many previous work in offline RL (e.g. SPIBB, ABM, CRR) use bellman operator without max operature for policy evaluation of target policy. Also, the one-step policy improvement in section 3.1 seems not novel, many previous work (e.g. BEAR, BCQ) sample the action according to the learned policy and take one of the sampled action with maximum value estimation at test time in the implementation.\n\nSignificance:\nThe main concern of the proposed method is whether it is theoretically sound and performs well in the other domains (such as domains with continuous action space) without much tuning of hyper-parameters (weight of regularization term when combining the proposed approaches). The three tricks are intuitive and might be useful in practice, but I am not sure whether the contributions are significant enough to match the acceptance bar of ICLR.\n\nPros:\n* This paper attempts to solve a significant problem (extrapolation error in offline RL).\n* The paper explains the intuition behind each proposed approach clearly.\n* The experimental results are good on several domains with discrete action space, better than the baseline methods.\n\nCons:\n* In section 1, \"Surprisingly, this technique with only one round of improvement ... often outperform existing offline RL algorithms\" seems a bit misleading and overclaiming. The proposed technique (1) can be better than behavioral policy only when the behavior policy is not deterministic and greedy with respect to the value estimation. And the experiment only verifies that it can outperforms the existing methods in some specific datasets collected on domains with discrete action. I doubt whether this technique can \"often outperform\" the existing algorithms (e.g. ABM, CRR, CQL) on continuous control tasks.\n* Overall, the proposed techniques (2) (3) are not quite convincing without the support of the theory. In equation (5), the specific formulation of the weight of regularization, e.g. $exp((G^B(s)−E_{s∼D}[G^B(s)])/β)$ is not well-motivated. Why we use exp function instead of another simpler monotonically increasing function? Why we need the coefficient β? How to choose the value of ν and β (the given value in the paper are just randomly picked)? In equation (7), the regularization for learning the scale parameter is not well-motivated either. Is it really necessary to have this regularization? Why square function here is better than other functions? Without a theoretical ground, all these design choices and value choices are just like heuristic or magic numbers. The experiments show these choices can work on some dataset (perhaps with much tweak and tuning), but we are not confident whether they can also work on new datasets.\n*In section 4.1 and 4.2, the performance of the proposed method is not significantly better than the baselines (the error bar of the proposed method overlaps with the error bar of the baselines).\n*In section 4.1 and 4.2, QRr and BRr are mainly considered, but in section 4.3, only B and BR are considered. I am curious whether BRr and QRr perform well in this domain? If not, could you please explain why different combinations of the three propose techniques perform differently in different domains? Is there any principle about which combination should be used in which kind of dataset?\n*In Figure 10 and 11, it seems that on Atari games, the larger weight of the ranking regularization means better performance. Then why \"0.05 seems to be the optimal choice for the ranking regularization hyper-parameter\"? Is the hyper-parameter value 0.05 used across all the datasets? Is the proposed approach sensitive to this value?\n*In Algorithm 1, is it a typo of using γ? In the main text, γ means the discounting factor, then what's the definition of $\\mathcal{L}(\\gamma)$? The details of the re-parameterization of Q network need more clarification.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MtENuUk-gjG", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##### Summary & recommendation \nThe paper proposes an offline RL algorithm, which consists of three techniques: behavior value estimation, ranking regularization, and reparametrization of Q function, to reduce overestimation errors. The algorithm is evaluated on several benchmark datasets. \n\nOverall, the paper is clearly written. The empirical results also seem promising. However, my main concern is that the proposed algorithm, especially with the behavior value estimation technique, has a big limitation for solving the offline RL problem (details come below). Moreover, I don’t think the paper provide enough justification for using all three techniques other than experiment results. It would have been better if the authors could discuss why we need all these three techniques (e.g. maybe behavior value estimation + ranking regularization is similar to behavior regularization?), rather than just combing these tricks to make the algorithm work empirically. I think there is still room for improvement before publishing this paper. Therefore, I recommend to reject the paper.\n\n##### Supporting arguments\nThe goal of an offline RL algorithm is to find a nearly optimal policy $\\pi$ from an offline dataset. However, the behavior value estimation technique is just learning the value function for the behavior policy. Even though we perform a single policy improvement step in the test time, it is generally not sufficient to obtain a nearly-optimal policy, especially when the behavior policy is far from optimal. \n\nThe authors mention “Fortunately, this one step is typically sufficient for dramatic gains as we show in our experiments (see for example Fig. 9). This finding matches our understanding that policy iteration algorithms typically do not require more than a few steps to converge to the optimal policy (Lagoudakis & Parr, 2003; Sutton & Barto, 2018, Chapter 4.3)”. I think this is not true. Even in tabular case, policy iteration requires a polynomial time w.r.t. the size of the state space and the action space (e.g., see Theorem 1.14 of [1]). I think it is possible to construct a family of MDPs and some behavior policies such that one step of policy improvement is not sufficient. In such case, I think the proposed algorithm would not work well. \n\nThe related work section should not just list previous works, but explain how the proposed algorithm is different from or similar to the existing algorithms. \n\nI have some questions to clarify my understanding of the paper: \n* I am not sure what is the propose of Appendix A? Maybe I missed some important points here, but it has already been shown that off-policy + function approximation + bootstrapping can diverge (Sutton and Barto’s book).\n* What is the exact definition of extrapolation error used in this paper? The paper mentions extrapolation over-estimation, extrapolation under-estimation, training time extrapolation error, and testing time extrapolation error, but I don’t see a clear definition of these terms. \n* Regarding the experiments: How were $\\nu$, $\\beta$, and $\\alpha$ selected? The algorithm introduces more hyper-parameters, so I wonder do you have any comments on hyperparameter search (e.g. do existing algorithms also require tuning extra hyper-parameters?). Do you have any reason why it needs a larger mini-batch and a smaller learning rate to update $\\alpha$? \n \n##### Minor comments\n* $\\theta’$ in Equation (2) is not defined \n\n[1] Alekh Agarwal, Nan Jiang, Sham Kakade, and Wen Sun. Reinforcement Learning: Theory and Algorithms. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Dws57_LhFLK", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper deals with offline aka batch RL for discrete actions. Three techniques ((i) behavior value estimation, (ii) ranking regularization, and (iii) reparametrization of the value function), which can be combined with each other, are presented. These techniques are compared with other methods in different experiments. Furthermore a new benchmark is being introduced.  It is claimed that in this new benchmark, the new techniques outperform state-of-the-art methods. Furthermore it is claimed that the presented method „behavior value estimation“, although it is only a one-step greedy optimization is typically already sufficient for dramatic gains.\n\nStrong points:\nThe abstract and the first part of the introduction (the first 1.5 pages) are very well written and the problems of offline RL are very well presented. Also very good is the consideration that the existence of a behavior policy is a restriction that does not apply to every given dataset, as expressed in the terms \"behavior policy(s)\" and \"coherent policy\". However, it is not specified in the text what exactly is meant by \"coherent policy\".\n\nWeak points:\nThe representation becomes increasingly unclear from page 2 onwards. None of the three techniques presented is sufficiently discussed and sufficiently tested. None of the statements is supported convincingly, although the paper already makes extensive use of references to the Appendix. There are 14 references in the main text to the Appendix and four to figures in the Appendix.\n\nRecommendation:\nIn its current form, the experimental part of the paper is immature. A uniform structure is missing. The statements are not sufficiently substantiated. Therefore I recommend to reject the paper. It seems that there is not enough space to present and sufficiently verify all three techniques.\n\nThe claim \"this one step is typically sufficient for dramatic gains as we show in our experiments (see for example Fig. 9)\" is not sufficiently substantiated, because one cannot speak of \"typically\", as only \"Atari online policy selection games\" are considered. And additionally Fig. 9 is located in the Appendix.\nThe meaning of the error bars in Fig. 3 and Fig. 7 is not explained.\nThe (on first sight) counterintuitive result that the performance of CQL and QRr at cart-pole is higher at 40% noise than at 0% must be explained in the text or caption.\nBecause the presented ideas are not sufficiently examined and supported, no sound knowledge is generated on which the reader can rely.\n\nQuestions:\nWhat is the meaning of error bars in Fig. 3 and Fig. 7?\nWhat is meant by \"BC\"?\n\nAdditional feedback with the aim to improve the paper:\nThe abbreviation BC is not introduced. It is unclear what is meant by it.\nIt is unclear what is meant by \"coherent policy“.\nWhat is meant by \"discrete offline RL algorithms“?\nit is not clearly described that discrete actions are required.\nIn the sum, i runs from 0 to 100 and is divided by 100, but from 0 to 100 we count 101 (unfortunately there are no line numbers in the manuscript, to locate the sum).\n\"5e - 2\" does not look nice, better is 0.05 or $5 \\cdot 10^{-2}$.\nIn Figure 8, the measurement results should not be connected by lines. Lines should only be used for fits or predictions of theory.\nIn Appendix F the text width is not respected.\t\nPlease do not use \\pm for the standard deviation, but for the specification of the uncertainty (aka error of the measurement) e.g. the standard error.\n\n-----------------------------------------\n(Dec 3.) Although I appreciate the feedback, my assessment of the paper remains unchanged.\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a new method for improving offline RL. AC thinks that the paper has a potential, but all reviewers suggest rejection as the current write-up is quite poor. This causes many misunderstandings of reviewers. The authors clarify some misunderstandings/concerns in the discussion phase, but did not update the draft accordingly. Hence, AC cannot suggest acceptance, given the current form.
Decision: Reject