Pseudometric guided online query and update for offline reinforcement learning
Authors: Haoran Li
Yang Weng
Venue: Submitted to ICLR 2023 (2023)
Keywords: Offline Reinforcement Learning
online query
optimal query
policy update

Abstract: Offline Reinforcement Learning (RL) extracts effective policies from historical data without the need to interact with the environment. However, the learned policy often suffers large generalization errors in the online environment due to the distributional shift. While existing work mostly focuses on learning a generalizable policy, we propose to adapt the learned policy to fit the online environment with limited queries. The goals include querying reasonable actions with limited chances and efficiently modifying the policy. Our insight is to unify these two goals via a proper pseudometric. Intuitively, the metric can compare online and offline states to infer optimal query actions. Additionally, efficient policy updates require good knowledge of the similarity between query results and historical data. Therefore, we propose a unified framework, denoted Pseudometric Guided Offline-to-Online RL (PGO2). Specifically, in deep Q learning, PGO2 has a structural design between the Q-neural network and the Siamese network, which guarantees simultaneous Q-network updating and pseudometric learning, promoting Q-network fine-tuning. In the inference phase, PGO2 solves convex optimizations to identify optimal query actions. We also show that PGO2 training converges to the so-called bisimulation metric with strong theoretical guarantees. Finally, we demonstrate the superiority of PGO2 on diversified datasets. 

Under review as a conference paper at ICLR 2023
PSEUDOMETRIC GUIDED ONLINE QUERY AND UPDATE FOR OFFLINE REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Offline Reinforcement Learning (RL) extracts effective policies from historical data without the need to interact with the environment. However, the learned pol- icy often suffers large generalization enrors the online environment due to the distributional shift. While existing work mostly focuses learning generaliz- able policy, we propose to adapt the learned policy to the online environment with limited queries. The goals include querying reasonable actions with lim- ited chances and efficiently modifying the policy. Our insight is to unify these goals via proper pseudometric. Intuitively, the metrie compare online and offline states to infer optimal query actions. Additionally, efficient policy updates require good knowledge of the similarity between query results and his- torical data. Therefore, we propose a unified framework, denoted Pseudometric Guided Peline-to-Onlingg RL (PGO?). Specifically, in deep Q learning, PGO² has structural design between the Q-neural network and the Siamese network, which guarantees simultaneous Q-network updating and pseudometric learning, promoting Q-network fine-tuning. In the inference phase, PGO2 solves convex optimizations to identify optimal query actions. We also show that PGO* training converges to the so-called bisimulation metric with strong theoretical guarantees. Finally, we demonstrate the superiority of PGO2 on diversified datasets.
INTRODUCTION
Offline Reinforcement Learning (RL) leverages large historical data learn the behavior policy, which seeks the optimality for sequential decision-making without any costs from interacting with the environments (Lange et al. 2012, Levine et al.) 2020). This promising feature significantly
promotes real- world RL applications, especially when explorative actions are costly. Thus, advances for offline RL have been made in robotic control (Lee et al. 2022), healthcare (Tang & Wiens) (20TT),
dialogue model (Jaques et al. 2020), recommendation Zhang et al. 2021), etc.
(iioo & Wang 2021), and E-commerce
Despite the achievement, the learned policy in offline RL may still suffer large extrapolation errors in online implementations (Dadashi et al. [021] Fujimoto et al. 2019 Lee et al. (022). The central reason is the data distributional shift between the offline and online environments. To address this issue, most solutions improve the algorithmic model, like adding constraints in the learning proce-
dure (Fujimoto et al. 2019 Wang et al., 2020), designing ensemble models (Agarwal et al. 2020 2020
Lee et 2022 An et al. (2221). or providing better value function estimation (Kumar et al. Dadashi et al.) 2021] [221]] Rezaeifar et al., 2022). While these methods provide appealing results, further improvements can be made when the agent can obtain online data. A classic setting is off- off-policy RL
(Levine et al. 2020 Munos et al. 2016) where data buffer continuously memorizes online data to
update the policy.
In many cases, it is often unacceptable to obtain large online samples based on the offline policy some random explorations. This is because the poor generalization of offline policy may cause high costs for online implementations, let alone random explorations. Therefore, we propose to actively produce query actions to the environment for online states. Subsequently, the environment can evaluate the query and provide feedback (i.e., rewards), which facilitates adapting the offline policy. The general goal is to utilize limited queries to achieve efficient policy adaptation.
More specifically, one needs to (1) produce proper query actions and (2) accurately modify the offline policy according limited query results. Basically, the state distributional shift challenges

---

Under review as a conference paper at ICLR 2023
goal (1). In addition, limited query results challenge goal (2). For example, for deep Q-learning it is hard to adjust a huge parameter set in the Q-neural network with few samples. To address these issues, one principled approach is to find proper similarity measure for the state-action pairs in Markov Decision Process (MDP). Then, for goal (1), we can infer query actions based on similar states and actions in the historical dataset. For goal (2), the differences between each online query with all the historical samples provide rich information that can be intelligently employed to update the Q-network. In general, we demand good pseudometric (Dadashi et al. 2021 to link offline datasets with online queries.
In particular, the well-defined pseudometric should guarantee that similar states must have a small difference between the expected rewards. In an MDP, the above definition is known as the bisimu- lation metric (Ferns & Precup 2014, Dadashi et al. 2021). To approximate the bisimulation metric, (Dadashi et al., 2021) has developed a pseudometric learning model that takes in the historical data and outputs the pseudometric, modified bisimulation metric the state-action space. Although this model benefits goal (1), goal (2) is still unreachable since the pseudometric learning and the Q-network updating are decoupled.
Therefore, we propose PGO": Pseudometric Guided Offline-to- RL. PGO" is a unified frame- work to learn the pseudometric, update the Q-network, and infer query actions. Specifically, PGO? employs Siamese networks to learn the similarity of the state-action pairs (Bromley et al. 1993] Dadashi et al. 202Tp. More importantly, we restrict the Q-network and the Siamese network to shar- ing parameters. Consequently, learning the pseudometric between each query result and all historical data updates the parameters of the Q-network. This updating scheme provides sufficient evaluations from limited query results, which enables the offline policy to quickly adapt to the online environ- ment. Moreover, the Q-network in PGO² is designated as Partially Input Convex Neural Network (PICNN) (Amos et al., (2017a). Thus, the inference of the action query has optimality by solving convex optimizations. For theoretical guarantees, we also show the theorems of convergence to the bisimulation metric and the global optimality for the query process. Finally, we observe significant improvements of PGO² in multiple RL tasks under the toniineet-- setting.
2 RELATED WORK
Offline RL. In the Introduction, we have presented review of different models to optimize the sequential decisions using offline data. In addition, some other works view the offline RL as a supervised learning model to generate sequences. (Janner et al. (021) employs trajectory Trans- former to predict the sequence of states, actions, and rewards. (Chen et al., 202T) also builds these sequences by masked Transformer. Finally, (Schweighofer et al.) [021] Sinha et al.) (022] study how the historical data can impact the learned policy in offline RL. However, these studies do not provide a reasonable approach for online queries and updates.
Online Implementations to Improve Offline RL. We categorize existing studies into the following groups. The first group requires the historical data to be informative that they can abstract prior knowledge, c.g., skills (Pertsch et al.] (2020), primitive behaviors (Ajay et al., 2020), and behavioral priors (Singh et al., 2020), for online implementations. However, may not have enough high- quality offline data. The second group admits imperfect data and employs online data to update the learned policy. (Nair al., 2020) proposes to restrict Kullback-Leibler (KL) divergence of the policy for both offline and online learning, which requires a certain amount of online data. (Lee) al. 2022) weighs the offline and online data based on the density ratio so that the offline Q- network can be fine-tuned Similarly, enough online samples are required to accurately estimate the density ratio. We make significant contributions by showing how a generalized pseudometric can guide the online action query and policy update, even with limited query opportunities.
State-action Similarity Metric for MDPs. The bisimulation metric uses rewards to determine the similarity of two states and/or actions (Ferns & Precup, 2014). Another similarity definition is the MDP homomorphism by considering both the reward and the transition probability (Ravindran &
Barto 2003). Many studies have been done to approximate these metrics (Castro 2020 Dadashi et al. 2021] van der Pol et al. 2020). Our pseudometric learning is similar to (Dadashi et al. 2021 to approximate the bisimulation metric in the aate-action space with theoretical guarantees. However, sufficient policy updating.
we have a unique structural design with convex optimizations to infer optimal queries and conduct
2

---

Under review as a conference paper at ICLR 2023
3 METHODS
3.1 BACKGROUND AND PROBLEM FORMULATION
tuple (S, r, P.7). where S the state space, A is the action space, r S x A x S + R is the reward function, P(s++1 (8k, az) is the state transition function to measure the probability from 8 to 8k+1 with a given action ag, and y E [0, 1) a discount factor. The agent can follow policy S + A to determine action for given state. In the optimization of RL, the general goal is to find an optimal policy to maximize E,, Lk= 1 r aks 88+1)] 1]]
An MDP can be formalized
Q-learning can solve the optimization in an iterative manner. For each policy и, the Q-function is value function for the state-action pair: (8) = Ei r**( ags 8k++1 = 8, Og = a)]- solve the optimization max, (= a) obtain the policy function for all states. With advances in deep learning, neural networks are used to approximate the Q-function, e.g.. Deep Q-network (Mnih al., 2013). To obtain the optimal policy, one can iteratively update the Q-network:
Then, the optimal policy
(=-1(skaas) (skk,, ax) + (r((sk, a&) + max Qe - Qe(*k ax)):
(1)
where f and & are the indices for the episode and state-action pair, respectively. a is the learning rate. This updating can happen both offline and online. For example, in the offline RL, we can compute equation using state transition sample (Sk: ag 8841, Tk) E DD where we denote T = r(8k, ak, 8k+1) and the set of N samples as D = {ss a; r;)
In this paper, focus on offline training and online queries with start state that may be dis- similar to the states in 'D e.g., an unseen block in the grid world (Crook & Hayes 2003). We add the notation indicate online data and formally define the problem as follows.
Input: historical dataset and online start state $1.
Query Task:
use DD learn offline Q-network Q(-). Then, D, Q(-), and 81 to generate sequential and optimal query actions (ā1, ... ,au ). where M is small integer. Note that repetitions
happen for the query action sequence.
Update Task: use D and the query results D { āi, R:) to update the offline ((-) and produce an online ((-).
. Output: D and Q(-) for online implementations.
We will illustrate how to use limited resources to achieve optimal queries and updates, shown in the We term the framework as Pseudometric Guided Oflline-to-Online RL (PGO?).
left part of Fig. 1
3.2 PGO ARCHITECTURE AND OFFLINE TRAINING
(Dadashi et al. 2021 proposes a pesudometric learning framework to approximate the modified bisimulation metric (i.c., pscudometric) in the state-action space. Specifically, they introduce Siamese neural network & to learn the pesudometric (s;, a; 0;; a,) O+ a;) T (sy: a;) I| where || is the Euclidean distance. By the definition the proposed bisimulation metric in (Dadashi et al. 202T) (see also in Section they design another Siamese network V to compute the bootstrapped estimate of the state bisimulation metric. This metric measures the similarity between states with randomly selected actions from (A). i.c., a uniform distribution over the action space. Then, they propose a concurrent learning scheme to optimize the following loss functions:
L E (|(0(s;; a.) -D(sj, a a;)ll lr:
ryl ( T ((3)+1))l)
2
(2)
Ly = =E ( ((;s))
Σ
( up) - (s,,,,,,))
where (s,, a. Tis 8;+1) D, (s,, aj. Tj. 8/+1) ~ D. The pseudometric do is trained by D, which does not require Q-network for the offline RL.
3

---

Under review as a conference paper at ICLR 2023
Challenges
PGO?: Pseudometric Guided Offline-to-Online RL
Offline sample transition.
Online query and update.
transition
1. Data distributional shift.
M queries
Key 1. Pseudometric de (s, á, reasonable query.
2. Expensive queries.
Seffort
(s.a)
Key 2. Parameter sharing efficient update. PICNN
Policy-dependent
Key 3. PICNN local convexity.
PICNN
(s.
Key 4. Convex opt. + optimal query.
3. Limited queries vs. efficient policy update > optimality?
Pollcy-independent
PICNN-bundle
(s, a)
PICNN
(a
fa(s,
fa a)
Opt. action space.
Figure 1: The proposed framework.
We improve the above learning process by coupling the pseudometric learning and the Q-network training in a parameter-sharing manner. The intuition is that Q(s, a) and ((ss, a) both serve as evaluations the state-action pair (s, a), although Q(s, a) is the value function and (ss, a) an feature embedding related to the reward differences. Therefore, we restrict the Q(-) and (()) to share some common features. In general, parameter-sharing enjoys two benefits. First, training each network improves the convergence of the other network they have similar outputs. Second, updating the network is cheap in the online mode. For example, one online query result can help to largely re-tral e--train the t network by comparing the query sample with all samples in D.
To elaborate on the architecture of the two networks, we note that their output differences are due to the different policy distributions. Q--) considers the policy by the learned Q-function while Φ(-) considers a uniform distribution over the action space. Thus, we consider policy-dependent and policy-independent features for both networks. The poliey-independent features common to evaluate the state-action pair no matter what the subsequent policy is taken, c.g., the instant reward. Thus, we make these features common with shared parameters, as shown in the blue block at the bottom of Fig. [ Correspondingly, the policy-dependent features have different parameters, as shown in the orange blocks at the bottom of Fig. []
Finally, these two features are summed to jointly output the final values. The summation is due to the weighted summation formula to evaluate the state-action pairs, e.g., equation 0
In the offline training of PGO², due to the parameter-sharing design, (--), (-).), and Q(-) should be trained concurrently to guarantee efficient updating. We demonstrate the specific training process as follows. First, in each iteration, the three networks are updated via random samples from D, like experience replay (Mmih et al.) 2015) for off-po RL. Second, with the selected samples, parameters of (-)) and V--) can be directly trained by minimizing we train Q--) according to the temporal difference in equation [ In particular, we need to solve the optimization maxa (e(5x+1, a) to seek the optimal actions for Sk+11
loss in equation 2 Third,
In vanilla Deep Q-network (Mnih et al., 2013), the optimization can be easily solved since the output of the Q-network is the action-Q pairs, which helps to do maximization by comparison. However, in our PGO?, the action is treated as the input for the Q(-). Thus, max, (e( a) is the optimization with respect to the input of neural network. We employ the Partially Input Convex Neural Network (PICNN) AAmos et al. 2017b) to model (s,,, a). Specifically, with non- negative weight and Rectified Linear Unit (RcLU) activations, to be convex in a. Then, we can utilize the gradient- based method or the Projected Newton method (erttskkas 1982) to solve the convex optimization max, (s(++))
make (s, a). PICNN,
The above optimization is an inference problem for PICNN (Amos et al. 2017b). Similarly, query- ing action online is another inference process. Specifically, to seek optimal query action, compare the online state with state-action pairs in D to find the optimal query action. Thus, the inference is related to not only Q(-) but also (c)). To guarantee to find the optimal query action,
4

---

Under review as a conference paper at ICLR 2023
propose bundle of PICNNNs model - (-)) such that each output neuron of - (s, a) is convex in input action a, shown in the bottom part of Fig. In Section 3.3 we show the PICNN-bul -undde brings convex optimization for online inference.
Algorithm PGO Offline Training
Input: training dataset DD
Initialize: discount factor y E [0, 1), maximum episode T, PICNN for Q-network - Q a., target network Q (+) Q-),, PICNN bundle for (s, a), neural network for (a), and update interval To for Q (+), (-). and (.()..
while S T do Sample random minibatch B D for (mm, am: I'm Sm+1) E B do
b Experience replay.
Solve the optimization maxa Q (8m+1, to obtain am+11 Vm Tm 7Q' (8m+1, am++1)
Train Q(-) using training data {8m+1: am+1 Jn where (8m+1: am+1 Imm are the input and {3 Jm are the output.
Stack the minibatch B1 {B1, B3. if t mod To O then (.) Q-)
b Update Q' (-)- Use m, am; I'm 8m+1) E Bi to train (()) and v.()) by solving the optimization in b Update D(-) and (-)..
equation 2
Output: trained neural networks Q(-), ((-), and (-).
Finally, training instability, a common issue for Deep Q-Learning, also exists in our training pro- cess. In particular, one needs to guarantee stable policy for several training episodes improve convergence. In PGO², the policy instability comes from the continuous update of both Q(-) and (()) with common parameters. To address this issue, we follow (Fan et al.. 2020) and employ target Q-network that is copy of the Q-network but updated occasionally. This guarantees stable policy before the target Q-network is updated. Similarly, we restrict ((-) and (-()) to be updated frequently the target Q-network to achieve stable policy. In general, propose the offline training algorithm for PGO² in Algorithm
3.3 2 ONLINE INFERENCE FOR OPTIMAL QUERIES
In an online mode, PGO2 needs infer reasonable query action for given online state $1. Since 8 may be an Out- Of-Distribution (OOD) sample, one cannot directly infer the query action by Q(-). Consequently, our strategy is to find the closest state-action pair (8c. a.) in D based on the learned metric da. By Dadashi et al. (2021), learning de has good convergence to the unique fixed point pseudometric to measure the bisimulation state-action pairs in MDPs. Thus, it is reasonable to utilize da optimization objective infer good online actions. Specifically, we propose the following optimization:
a arg max Q(se, a), ā arg min (s acj 8), (s..
(3)
where ae is our target query action. The first optimization in equation 3 can be casily solved as convex optimization. Then, we can input each (s.., a.)) pairs and the online initial state 81 to do (::-) solve the second optimization. With the PICNN-bundle to model ((-). We prove the second optimization is also convex for each state-action pair in D in Proposition - Then, can also utilize the gradient-based method to solve the optimization and find a.. For example, the bottom right of Fig. visualizes the gradient-based optimization in the 2-dimensional space. After the optimization, we set ā1 = a. as the first query action for $1.
After performing the query action ă1. the agent will move to 82. The online environment will evaluate (ø1, ā1, 32) and produce the reward F1. The online sample ($1, م1 82, can facilitate to update (--), (-),, and Q(-), discussed in Section 3.4 Based the updated networks and &2, can continue the optimization in equation B and produce 02. Iteratively, we can obtain the query M
dataset D { (8:; āi, $
1, f.) ) === accomplishing the Query Task.
5

---

Under review as a conference paper at ICLR 2023
3.4 PGO² ONLINE TRAINING FOR EFFICIENT POLICY UPDATES
To efficiently update the Q-network, we start from the decomposed architecture in Fig. Specifi- cally, the poliey-independent features play key role evaluate the instant online rewards that may be completely different from the offline rewards. Therefore, we need to significantly and frequently update the corresponding parameters. Next, the update is cheap as one can train one query result versus all samples in D, bringing accurate and stable function approximation. Thus, we update the related parameters each time after an online query.
Basically, we only enable the parameters (b-) and (-(-) trainable, which include the policy- independent parameters for Q(-) due to the parameter-sharing, Then, we retrain PGO? by using the query result (e.g.. (&;, 0, 8; Fi) for the jth query) versus all the samples in D, which is beneficial for the next action query.
The parameters of policy-dependent features for Q(-), however, should not be frequently updated to maintain stability. Thus, after obtaining D, we utilize all samples in D to retrain the parameters for policy-dependent features. This fine-tunes (---) to make it applicable for online implementations. which ends the Update Task.
We summarize the online algorithm for PGO in Algorithm 2
Algorithm PGO Online Inference and Update
Input: training dataset D, trained neural networks Q(-), ((), and (-), and an online start state 81.
Initialize: maximum number of query actions M.
while < MM do
For online state 8m, use gradient- badde method to solve the convex optimization in equation and obtain amn-
Perform the online action 7m and obtain the reward Fm and the next state 8m++1 Retrain ((.) and (-) in PGO2 by (Sm. am, Sm+1 Pm) and all the samples in D.
8.
Sm++1
D [D, [8, d.,, 1,
Enable the policy-dependent parameters trainable in Q(-). Retrain Q(-) using all the data in D. Output: trained neural networks Q(.), (()), and (-) for online environments.
THEORETICAL ANALYSIS
In this section, we provide theoretical support for pseudometric learning and online inference. First, Dadashi et al. (202T) provides that iterative calculations from samples can converge to the fixed point of the pseudometric. Let d(-:-) be pseudometric defined over the state-action space and F be an operator for the pseudometric such that:
F(d)(s), 015 a2) [r; ryl + " 8441 81+1, u) if 81, 01 82, a2 8i, ai 8j, aj otherwise,
[((s), a1; 82, .2)
where (s;, a; Tx) and (s), aj, 85+1, T5) are samples from D. Then, Dadashi et al. 202 introduces the following proposition show the convergence.
Proposition 1. Suppose sufficient coverage of the state-action space: Ee > 0 such that for any state- action pairs (st, a.), (3,, a,) E (S x A) (S A), (s a.), (s,, aj) are sampled with at least probubility c, then the repeated application of F converges to the fixed point d", ie., the bisimulation metric over the state-action space.
Proposition guarantees to learn a good pseudometric in the iterative manner in Section 3.2 an approximate version of the operator F. Notably, our pseudometric learning consistent between offline and online environments. Thus, if the offline data samples are not sufficient as required in Proposition I the online queries further improve the convergence. Secondly, we show that our query mechanism can produce optimal query action based our PICNN-bundle model for ((.).
Proposition Assume Q(-). ((.), and do(:;-) are defined in Section trary state from D, mina do , az; 85; â.) convex in à E A, where À is defined in the proof.
3.2 Let s; be an arbi- arg max, Q(s;, a), and 8j be an online state. Then, the optimization

---

Under review as a conference paper at ICLR 2023
Proof. In the optimization mina (s; ax: $,, a), the inputs 8,, a, and 8, are fixed to the Siamese neural network. Then, we simplify the objective as L(ā) = Σx (fa(a) cx)² where K is the number of dimension for ((-). fü is output the tt entry in o a), and Cg is the 1*th constant entry in $(s,, a:). Let h be a perturbation direction of a and & be a small step size. Then, we can calculate the second-order derivative based on the chain rule:
d²
d
Σ (fi(a + th) - -c) d
dt2 1= L(a + th) dt t=0= 2
dt fu(ā + th),
= 2 > dt l + th))² + (Fn(a) - -x) de
K
dt2 o fa (ā + th).
Then, we define Ā (álv] S k S (á) I cx). Since we utilize PICNN to model
-f, L1 I k S K, the concavity leads to de Skla + th) < 0. Thus, Vā E Á.
2e Lا + th) > 0 and the convexity is proved.
The global optimal solution minimize I(ā) lies in the boundary of A, i.e., 0Å = {alıi < k S K. fa(( (ā) = cx). Although is hard to quantify the region of Á in Proposition 2 we can initialize āo = 0. The reason is that we can easily make sure v1 < l S K, (0) = 0 and C+ fa.))) > 0 as long as a; * 0 due to the weight positivity in PICNN. Thus, åo E Á. Consequently, we can utilize the gradient-based method to iteratively search the global optimal solution from åg to the boundary point in AA, as shown in the bottom right part of Fig. | Finally, the above optimization is solved multiple times, and each time we input one offline state-action pair to the second optimization in equation 3
Thus, the inference procedure brings optimal query actions.
5 EXPERIMENTS
5.1 SETTINGS
Datasets. We use the following datasets for testing. (1) Grid world. We simulate a 20 x 20 grid and let the agent search for the correct path to the target grid. Further, in the offline setting, we restrict the bottom right region of the grids to unseen states that the agent never explores, shown as the black region in Fig. 2a Subsequently, in the online scenario, the start state & lies in one of the unseen grids. PGO then helps to generate action queries that enable the agent to quickly move to the target grid. (2) Maze2D (Fu al., 2020). Similarly. Maze2D an environment that requires the agent to navigate to the target location. We conduct the same treatment as grid world to restrict the unexplored areas in the offline dataset. Then, the agent starts at the unexplored location in the online phase. (3) Gym domain (Fujimoto et al. 2019). The data is gathered from OpenAI Gym benchmark tasks (Walker environments). We randomly delete 20% of the historical benchmark data and use the state in the deleted dataset as the start state for online testing.
(4) Atari games (Bellemare et al. 2013). We evaluate the performance of the agent for the image-based Atari game (Breakout). We conduct the same treatment gym domain for the offline data and the online state initialization. (5) Electric system (Li et al. 2022). Electric systems transmit electricity from generators to loads. Proper controls are essential to guarantee system stability. Offline RL. provides chance learn sub-optimal policy with changing system states. We conduct the simulation for the 200-node system (Engincering Texas A&M University 2016) based on set of system states. For the online testing, we start at an out-ol--ammp sample state to evaluate the learned policy.
Benchmark methods. We introduce the following benchmark methods for comparisons. (1) Q- Learning with Bootstrapping Error Accumulation Reduction (Kumar et al. 2019) (BEAR). BEAR reduces the bootstrapping error and enables stable training that robust to data distributional shifts. (2) Conservative Q-L.carning (CQL. (Kumar et al.] 2020). CQL estimates a conservative Q value that is a lower bound of the true value. Thus, CQL. prevents the overestimation of the Q values and increases the generalizability of the learned policy to online implementation.
(3) Pseudometric Learning Offfine RL (PLOFF) (Dadashi et al. 2021). PLOFF proposes to learn pseudometric with the historical dataset. Then, the pseudometric works bonus term to restrict the state-action pairs to always stay close to the support of the historical data. (4) Offline-to-Online RL with Balanced Experienced Replay and Pessimistic Q-Ensemble (O'RL) Lee et al. (2022). OºRL reweights the offline and online samples and utilizes Q-value ensemble to reduce overfitting.
7

---

Under review as a conference paper at ICLR 2023
Metric. We utilize the normalized score (Fu et al. 2020) between 0 and 100 to evaluate the perfor- mance of different methods. The normalization facilitates the comparison across tasks. The normal- ized score is evaluated in the online implementation after the offline training and online querying. Specifically, we consider E {20, 40, 60, 80, 100) query actions in the query phase. In addition, report the average results over seeds to guarantee robust performance.
5.2 PSEUDOMETRIC GUARANTRES OPTIMAL QUERY ACTIONS
In this subsection, dometric for online queries. Specifically, we need to validate the optimality of the query action for unseen grid in the online phase. Therefore, Fig. 2 The rest 2 methods have similar performance to the CQL method. With multiple trials, the color of the grid implies the frequency of states in the online testing.
the grid world data demonstrate the effectiveness of the learned pseu-
plot the query actions for different methods in
Sageigg mound
Taggn
Experpplay
Gi minza
pelr
(a) The 20 20 grid world dataset.
The offline and online policy of PGO
camn
(c) The offline and online policy of CQL.
(d) The offline and online policy of PLOFF
Figure 2: Results of different methods for the grid world dataset.
The results imply that starting from unseen state, the proposed PGO2 can successfully find the target grid (i.e., the green block) while CQL and PLOFF cannot. The key reason is that PGO can produce the optimal query action based the comparison the states in the historical dataset. For the first start state (8, 0). the query mechanism in PGO² will find the closest state (7,0) 0) and infer the query action "left" that is the optimal action for (). This action helps the agent to go out of the unexplored region and find the known path to the target grid. Similarly, when the start state is (14, 9). PGO² will find the state-action pair, (14, 13) and "up", in the historical dataset, which guides the agent to the target grid. However, for CQL and PLOFF, we directly apply the offline trained policy network to find the action, which eventually fails due the overfitting of the neural network-based approximator.
5.3 OPTIMAL QUERIES AND FAST UPDATES BRING THE BEST ONLINE POLICY In this subsection, we conduct comprehensive tests for different tasks and methods. Specifically, we restrict M 60 and report the performance of the online policy obtained after the offline training and the online query. Fig. illustrates the results of the normalized scores. Generally, PGO has the best performance in the online environment. For grid world data, Section 5.2 shows that PGO can find correct query actions based on the most similar state-action pairs in the historical dataset. For other methods, however, their agents will get stuck in the unexplored region. For more complex tasks, PGO" is still better since it can efficiently use online information. Finally, O²RL also has the mechanism to largely make use of the online query results. However, since our query number M is limited compared to the historical dataset, 2RRL does not have significant improvement.
8

---

Under review as a conference paper at ICLR 2023
OQU
PLOFF O'RL
100
Normalized soore
70
world
Gym
Datasets
Figure 3: Results of different methods for online tests different datasets.
5.4 SENSITIVITY ANALYSIS: SMALL NUMBER OF ONLINE QUERIES IS ENOUGH
We further study the problem of how many queries are needed to produce a high-level online policy. Thus, we vary M and test different datasets. Fig. demonstrates the results. We have the fol- lowing observations. First, M increases, PGO and OaRL have significant improvements while the rest methods only have slight improvements. This because PGO² and OaRL have effective mechanisms update the policy based the online data. However, the improvement for PGO² is much higher since PGO² can better use the online queries results based our unique designs of parameter-sharing between the Q-network and the pseudometic learning network. Second, when small, PGO? still brings relatively large scores, which further validates the model's effectiveness.
126
125
105
100
COl
PLOEY H6-OP
630
TEAS
He-0/Tt
35
No mombodooddooo oc3r
90
Nomnllzzd S001
30-
30
85
30-
Nomellod
60
80-
71
755
75g
70
50
10
number
numbor
M
number
The result for Maze2D. (b) The result for Gym.
(c) The result for Atari (d) The result for Electric.
Figure 4: Results of sensitivity analysis for different datasets.
5.5 COMPARISONS TO PRE-TRAINING-BASHH OFFLINE TO ONLINE RL
In this subsection, we treat the pure offline RL models as pre-training for online implementations. Then, we continue the model updates in the online phase using query results. We fix M 100 and the results are shown in Table Although different models have the post-training process after the online results, our 2GO2 still achieves the best performance due to its efficient utilization of online query results.

---

Under review as a conference paper at ICLR 2023
Table 1: Results of different methods for offline pre-training and online post-training.
DATASET
PGO
BEAR
CQL
PLOFF
O'RL
MAZE2D
92.3
66.4
85.2
79.7
85.2
88.2
72.3
73.5
73.8
78.9
ATARI
83.2
75.6
67.8
74.5
73.8
ELECTRIC
94.2
78.6
76.2
88.6
82.3
CONCLUSION AND FUTURE WORK
We propose PGO?: Pseudometric Guided Offline-to-Online Reinforcement Learning. PGO2 is effective with (1) optimal online action queries and (2) efficient online Q-network updates. We demonstrate that the effectiveness comes from the coupling between pseudometric learning and pol- icy network learning. addition, we enforce input convexity the framework guarantee query optimality. Finally, test different datasets demonstrate the superiority of PGO?. Possible fu- ture directions may include (1) the development of evaluations on the transferability from offline to online after the online query and (2) model improvement in the face of sparse rewards.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104 114. PMLR, 2020.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline prim- itive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010. 2020.
B. Amos, L. Xu, and Z. Kolter. Input Convex Neural Networks. International Conference on Machine Learming, pp. 146-155, 2017a.
Brandon Amos, Lei Xu, and Zico Kolter. Input convex neural networks. In International Confer- ence on Machine Learning, pp. 146-155, 2017b.
Gaon An, Seungyong Moon, Jyng-Hyun Kim, and Hyun Oh Song. Uncertainty-basee offline inforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Dimitri P Bertsekas. Projected newton methods for optimization problems with simple constraints.
SIAM Journal on control and Optimization. 20(2):221-246, 1982.
Jane Bromley, James W Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. Signature verification using a "siamese" time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence, ((04):669-688) 1993.
Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 10069-10076, 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 14:150044-5000 2021.
Paul Crook and Gillian Hayes. Learning in a state of confusion: Perceptual aliasing grid world navigation. Towards Intelligent Mobile Robots, 4, 2003.
10

---

Under review as a conference paper at ICLR 2023
Robert Dadashi, Shideh Rezacifar, Nino Vieillard, Léonard Hussenot, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning with pseudometric learning. In International Conference Machine Learning. pp. 2307-2318. PMLR, 2021.
Engineering Texas A&M University. Illinois 200-Bus System: ACTIVSg200. 2016. URL
wtw.:iidddng......mm... tamu, wtt..ccceceeeeeeerrro-o---------
activsq2007
Jianqing Fan, Zhaoran Wang. Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep q- learning. In Learning for Dynamics and Control, pp. 486-489. PMLR, 2020.
Norman Ferns and Doina Precup. Bisimulation metrics are optimal value functions. In UAI, pp. 210-219. Citeseer, 2014.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052 2062. PMLR, 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 14:1273-1286, 2021.
Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Shane Gu, and Rosalind Picard. Human-centric dialog training via offline rein- forcement learning. arXiv preprint arXiv:2010.05848, 2020.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q- learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 13:1179-1191, 2020.
Sascha Lange. Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning, pp. 45-73. Springer, 2012.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. cofline-to-onlingg reinforcement learning via balanced replay and pessimistic g-ensemble. In Conference on Robot Learning, pp. 1702-1712. PMLR, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Haoran Li. Hanghang Tong, and Yang Weng. Domain adaptation in physical systems via graph kernel. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 868-87 876, 2022.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 132.55002 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 8(754)):529-533 2015.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off-policy reinforcement learning. Advances in neural information processing systems, 29, 2016.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.09339, 2020.
Karl Pertsch, Youngwoon Lee, and Joseph J Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010. 1944, 2020.

---

Under review as a conference paper at ICLR 2023
Balaraman Ravindran and Andrew G Barto. Relativized options: Choosing the right transformation. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 608- 615,2003. 2003.
Shideh Rezaeifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning as anti-exploration. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8106-8114, 2022.
Kajetan Schweighofer, Markus Hofmarcher, Mariis-Constantin Dinu, Philipp Renz, Angela Bitto- Nemling, Vihang Patil, and Sepp Hochreiter. Understanding the effects of dataset characteristics on offline reinforcement learning. arXiv preprint arXiv:2111.0477 2021.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine, Parrot: Data-driven behavioral priors for reinforcement learning. arXiv preprint arXiv: 2020.
Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4rl: Surprisingly simple self-supervision for offline reinforcement learning in robotics. In Conference on Robot Learning pp. 907-917. PMLR, 2022.
Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical con- siderations for healthcare settings. In Machine Learning for Healthcare Conference, pp. 2-35. PMLR, 2021.
Elise van der Pol, Thomas Kipf, Frans A Oliehoek, and Max Welling. Plannable approximations to mdp homomorphisms: Equivariance under actions. arXiv preprint arXiv:2002. /963, 2020.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778. 2020.
Teng Xiao and Donglin Wang. A general offline reinforcement learning framework for interactive recommendation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 4512 4520, 2021.
Yang Zhang, Bo Tang, Qingyu Yang, Dou An, Hongyin Tang, Chenyang Xi, Xueying Li, and Feiyu Xiong. Bcorle(A): An offline reinforcement learning and evaluation framework for coupons al- location in e-commerce market. Advances in Neural Information Processing Systems, 34:20410- 20422, 2021.
12


Reviews:
[{"review_id": "y6MeMQ2guK", "reviewer": "Reviewer_Ggyx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for learning policies first from offline dataset and then from a limited number of interactions with the real environment. The special feature of the method is that it learns a pseudometric that corresponds to the similarity between states in the environment. Then, when during online interactions the agent faces some unseen initial states, it selects the action by retrieving the most similar state in the existing offline dataset. The experimental results demonstrate promising behavior in several environments in the situations when part of the initial state space is hidden.", "summary_of_the_review": "I am leaning toward the rejection of the paper mainly because I find the setting and its motivation (missing part of the states space) not very convincing given the nature of the experiments where states were manually deleted. The nature of the offline datasets that are required for training should be clarified as well. Besides, I think the clarity of the paper could be improved and experiments could be done more thoughtfully to include ablations of the components of the method.", "strength_and_weaknesses": "Strengths:\n- The problem of adapting offline policies with the small amount of online interactions is certainly interesting and deserves attention.\n- The authors test their method on multiple diverse domains. \n- Several diverse baselines are used in the comparison.\n- Experimental results include experiments for understanding qualitative behavior of the agent.\n- Experimental results show the sensitivity of the method to the number of online trajectories.\n\nWeaknessess:\n- It seems that the paper uses an assumption that part of the initial states are hidden from the agent during offline training. While I understand that a problem of such distribution shift can occur naturally due to the fact that the dataset is limited, the experimental settings are such that part of the states are manually deleted from the offline training. I find this setting somehow artificial. It would be more interesting and convincing if the method would demonstrate its advantages in the more realistic settings in addition to the manually constructed test case. Moreover, this assumption should be stated clearly at the beginning of the paper, currently it becomes clear only when reading about the results.\n- It is not clear from the paper what kind of offline dataset is required for training. From section 3.2 it sounds that the dataset should contain uniformly random actions. Is this the case? How realistic is it to collect such a dataset in practice? How would the method perform if this assumption is violated? How was the dataset created? Is the data part of an existing offline RL benchmark? Also, one of the claims about the proposed method is that it does not require as much offline data as other methods but no information about the sizes of the offline datasets are provided.\n- I found some parts of the paper not clear and the writing needs a bit more work. For example, on the first page \"seeks sub-optimiality\" (do you mean optimality?), \"centric reason\" (central reason?), \"query reasonable actions\" (what does it mean to query an action?). In terms of mathematical details, I find some parts confusing. For example, r: S x A, but then r(s_k, a_k, s_{k+1}), \\Phi function in equation 2 sometimes takes s and a and sometimes only s. The figures in the experiments do not have sufficient explanations, for example, while figure 2 looks interesting, it is not clear to me what color is representing (the text says \"probability of the state in the implementations\", what is implementation? Is it during online fine-tuning? Or are states sampled during offline training? Or is it during the deployment?) etc.\n- The proposed method is quite complex and includes many moving parts. It would be beneficial for the reader to understand how various components contribute to the performance of the method if introduced independently. Usually in offline RL, applying the policies from online RL directly results in very poor performance. Would this also be the case with the proposed method if no online updates are done? Or is this problem mitigated with the elements of representation learning? Also, how were the different hyperparameters set? For example, frequencies of updates of various parts of the network.\n- The proposed method with pseudometric learning sounds like it is from a group of representation learning methods in RL. Related work should include works on it. Maybe even a representation learning baseline could be beneficial. \n- Limitations of the method are not discussed.\n\nQuestions and comments\n- Some part of the method deals with finding the maximum of Q(s, a) where a is the input to the network. Wouldn't methods like DDPG already address such a problem? \n- I think the trick with having a \"target\" Q network is much older than 2020 and was introduced as part of the DQN method.\n- Theoretical analysis does not need to include the proposition from the existing work if it is not required for other proofs.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MlihpNK0J4s", "reviewer": "Reviewer_pNvF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for offline reinforcement that relies on limited queries to the environment for dealing with out-of-distribution actions during the online learning phase. The authors use a state-action (learned) pseudometric to compare online and offline states and determine which actions are best to query. Part of this paper's method is the use of a siamese network to simultaneously update the Q-values and the pseudometric.", "summary_of_the_review": "Overall an interesting algorithm that is well motivated. There are a number of issues around clarity and notation, but I think most of these should be fixable during rebuttal. The question around reproducibility should also be addressed.\n\nI am on the fence with this paper, but leaning towards an accept. The final decision will be contingent on the authors' response during the discussion period and the other reviews.", "strength_and_weaknesses": "# Strengths\nThis paper presents an interesting algorithm that builds on the work of [Dadashi et al., 2021] for more efficient online environment interactions when training from an offline dataset. \n\nThe use of pseudometrics is well motivated and reasonably well evaluated.\n\n# Weaknesses\nThe main weakness of this paper is in the clarity of presentation, outlined below.\n\n## Algorithmic issues\n1. Why is the method restricted to a single start state $\\tilde{s}_1$? What about an initial state distribution?\n1. Figure 1 helps calrify the method, but it is not clear where $\\Psi$ fits in this figure.\n1. The idea of coupling metric and Q-value learning in a parameter-sharing manner has already been done in [MICo](https://papers.nips.cc/paper/2021/hash/fd06b8ea02fe5b1c2496fe1700e9d16c-Abstract.html), so probably worth citing.\n1. In the second paragraph of page 4 it says \"$\\Phi(\\cdot)$ considers a uniform distribution over the action space.\" but this is not entirely correct, since in (2) the actions are sampled from $\\mathcal{D}$ (replay buffer) when computing $\\mathcal{L}_{\\Phi}$.\n1. One point that is not clear to me is how critical the use of PICNN is to this method. While I can appreciate the desire to have a convex optimization problem, it is not entirely clear to me how critical this choice is to the performance of the algorithm.\n1. In the last paragraph of page 4 it proposes using the projected Newton method to find $\\max_a$. If $\\mathcal{A}$ is small, couldn't you just do a simple iteration over all actions?\n1. In the last line of page 4, why does using a bundle of PICNNs \"guarantee to find the optimal query action\", if they're all outputting different estimates for the same $a$?\n1. In the first paragraph of section 3.3 it says \"one cannot directly infer the query action by $Q(\\cdot)$\", but one can, in principle; if $Q$ generalizes well it might be reasonable.\n\n\n## Notational issues\n1. In the second paragraph of section 3.1, the term inside the expectation is wrong, as it is saying $s_k, a_k$ must be equal to the original $(s,a)$ for all $k$. This should only be the case for $k=0$.\n1. The maximization in the next line says $\\max_{\\pi}Q^{\\pi}(s, a)$, but it should be quantified to be for all $s$, or at least for all $s$ from some initial state distribution.\n1. In the \"Query Task\" bullet on page 3, are the optimal query actions an open-loop policy? The way it's written it appears to be the case, which seems strange.\n1. In the second line of equation (2) it is written $\\|\\| \\Psi(s_i) - \\Phi(s_j) \\|\\|$, but I believe this should be $\\|\\| \\Psi(s_i) - \\Psi(s_j) \\|\\|$.\n1. In Algorithm 1 in the innermost for loop, shouldn't the parameters of $Q'$ be $s_{m+1}$ and $a_{m+1}$ (instead of $m$)? Similarly, in the line below it shouldn't it be using $s_m,a_m$ (instead of $m+1$)?\n1. In the first paragraph of section 3.3 the phrase \"learning $d_\\Phi$ has good convergence to the unique fixed point pseudometric to measure the bisimulation of state-action pairs in MDPs.\" does not make much sense. Is the fixed point the bisimulation metric? \n1. A related point: it is assumed bisimulation is a \"reasonable\" thing to use, but bisimulation has not really been introduced properly at this point, so it's not clear _why_ it's reasonable.\n1. In  equation (3) is the $\\arg\\min$ only over $s_c$? It's not clear.\n1. In the last line of the derivation in proof of Proposition 2 in the $(f_k(\\tilde{\\mathbf{a}}) - c_k)$ term it seems like there's a $t\\mathbf{h}$ term missing inside the $f_k$.\n1. In the last line of the proof of Prop 2, it says $L(\\tilde{\\mathbf{a}} + t\\mathbf{h})=\\frac{d}{dt}$, ... what does that RHS mean? It appears to be incorrect o a typo.\n\n## Empirical evaluation\n1. In section 5.2, what does \"the probability of the state in the implementations\" mean? Probability of what?\n1. How stable are the trajectories in Figure 2? It would be good to superimpose multiple policy rollouts to evaluate this.\n1. Since the values in Figure 3 are normalized, you should use more robust metrics such as IQM, as presented in [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://proceedings.neurips.cc/paper/2021/hash/f514cec81cb148559cf475e7426eed5e-Abstract.html) (and [corresponding library](https://github.com/google-research/rliable).\n1. Continuing a point made above, I think it would be good to have an ablation study where PICNN is not used (e.g. convexity is not guaranteed). Although perhaps not theoretically principled, this may yield better empirical performance.\n\n\n# Questions / suggestions\n1. In the introduction it would be good to discuss connections with the findings from:\n    * [Georg Ostrovski, Pablo Samuel Castro, Will Dabney. The Difficulty of Passive Learning in Deep Reinforcement Learning. NeurIPS 2021](https://openreview.net/forum?id=nPHA8fGicZk)\n1. In the second paragraph of page 2 it says \"must have a small difference between the expected rewards\", but this is not accurate. It is the difference between the sum of discounted returns.\n1. In the **State-action Similarity Metric for MDPs** paragraph in section 2, you should also reference\n    * [Jonathan Taylor, Doina Precup, Prakash Panagaden. Bounding Performance Loss in Approximate MDP Homomorphisms. NeurIPS 2008.](https://proceedings.neurips.cc/paper/2008/hash/6602294be910b1e3c4571bd98c4d5484-Abstract.html)\n    * [Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, Mark Rowland. MICo: Improved representations via sampling-based state similarity for Markov decision processes. NeurIPS 2021](https://papers.nips.cc/paper/2021/hash/fd06b8ea02fe5b1c2496fe1700e9d16c-Abstract.html)\n1. In the second line of equation (2), could you have independent action samples for $s_i$ and $s_j$, instead of the same $u_k$ for both?\n1. In the last paragraph of page 4 what is meant by \"action-Q pairs\", and what is meant by \"helps to do maximization by comparison\"?\n1. In the last paragraph of page 4 what is meant by \"an inference problem for PICNN\"?\n1. In the last line of page 4, why is the output considered negative (e.g. $-\\Phi(s,a)$)?\n1. In Algorithm 1, wouldn't $\\gamma$ and $T$ be inputs, rather than values to initialize?\n1. In Algorithm 1, does $\\mathcal{B}_1$ start empty?\n1. Nit: In Algorithm 1 ity says \"Solving the optimization in equation 2\", but it's not actually _solving_ at each step, but approaching solution, right?\n1. In the paragraph below Algorithm 1 [Fan et al., 2020] is cited for the use of a target Q-network, but this is an idea that was introduced by Mnih et al., 2015, in the original DQN paper.\n1. In section 5.3, just to confirm: there is no learning happening during eval, correct?\n\n\n# Minor points\n1. In the second paragraph of the introduction it should say \"The **central** reason is the data distributional ...\"\n1. In the end of the first paragraph in page 2 I think it'd read better if it said \"we **require** a good pseudometric\" (instead of demand).\n1. In the start of section 3.2 use `\\citet` to avoid having parentheses around the noun.\n1. In the second line of section 3.2 you can remove \"(i.e. a pseudometric)\".\n1. In the third line of section 3.2 there is a typo: \"pesudometric\"\n1. In the second line of page 4 it should say \"$\\Phi(s, a)$ is **a** feature embedding...\"\n1. In the second-to-last paragraph of page 4, it should read \"trained by minimizing the loss**es** in equation 2\"\n1. In the last paragraph of page 4 it should say \"output of the Q-network **are** the action-Q pairs\"\n1. The sentence right before section 3.3 does not need to start with \"In general\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6U_KF6nDWi", "reviewer": "Reviewer_dmti", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the offline RL with online query setting, i.e., given a dataset and online access to the environment, how to effectively learn the optimal policy. It proposes Pseudometric Guided Offline-to-Online RL (PGO2). The main idea is to learn a pseudo metric that measures the closeness from the support of the training data in the offline training stage, and using this metric to query nearby actions in the online setting to gather more data. Experiments on different datasets show better performance compared with purely offline algorithms. ", "summary_of_the_review": "The current version of the paper is a clear reject due to:\n\n- The low clarity in the paper.\n\n- The limited novelty in the method, and no theoretical justification of the online exploration part is given. \n\n- The weak baselines for the empirical setting.", "strength_and_weaknesses": "Strength:\n\n- The setup this paper studies is kind of interesting. There is some statistical limit of learning from purely offline dataset, and sometimes online query under some costs is available. The setting is of practical relevance and should be relevant to various settings.\n\n- Empirical results show better performance compared with purely offline setting, though straightforward. \n\nWeakness:\n\n- This paper is very difficult to read, as it does not provide a clear motivation of the method, and why the pseudo-metric would be helpful for online exploration stage. The main sections is a mix of methods, algorithms and implementation details, which makes it difficult to follow.\n\n- The novelty is limited, as the notion of using this type of pseudo metric is proposed in [1], and this paper utilizes it the same way as the original paper in the offline stage. The use of it in the online stage seems to help query the new data? However, it is super unclear to me why we want to query nearby data points? It is very different from most online exploration literature. There is no justification here.\n\n- The empirical results are limited, most of the algorithms are purely offline setting, there are some works such as utilizing offline RL dataset for pre-training. Could the authors add more comparable baselines?\n\n\nRef: \n[1]. Robert Dadashi, Shideh Rezaeifar, Nino Vieillard, Le ́onard Hussenot, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning with pseudometric learning. In International Conference on Machine Learning, pp. 2307–2318. PMLR, 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject