The Provable Benefit of Unsupervised Data Sharing for Offline Reinforcement Learning
Authors: Hao Hu
Yiqin Yang
Qianchuan Zhao
Chongjie Zhang
Venue: ICLR 2023 poster (2023)
Keywords: offline reinforcement learning
unsupervised learning
data sharing

Abstract: Self-supervised methods have become crucial for advancing deep learning by leveraging data itself to reduce the need for expensive annotations. However, the question of how to conduct self-supervised offline reinforcement learning (RL) in a principled way remains unclear.
In this paper, we address this issue by investigating the theoretical benefits of utilizing reward-free data in linear Markov Decision Processes (MDPs) within a semi-supervised setting. Further, we propose a novel, Provable Data Sharing algorithm (PDS) to utilize such reward-free data for offline RL. PDS uses additional penalties on the reward function learned from labeled data to prevent overestimation, ensuring a conservative algorithm. Our results on various offline RL tasks demonstrate that PDS significantly improves the performance of offline RL algorithms with reward-free data. Overall, our work provides a promising approach to leveraging the benefits of unlabeled data in offline RL while maintaining theoretical guarantees. We believe our findings will contribute to developing more robust self-supervised RL methods.


Published as a conference paper at ICLR 2023
THE PROVABLE BENEFITS OF UNSUPERVISED DATA SHARING FOR OFFLINE REINFORCEMENT LEARNING
Hao Hu Yiqin Yang"; Qianchuan Zhao?, Chongjie Zhang!
Institute for Interdisciplinary Information Sciences, Tsinghua University meppartment of Automation, Tsinghua University
[huh22, yangy 1q 9) @mails. ts singuua. edd.u..eeddd..... .n (chongjie, zhaoqo stigggua .du.. cn
ABSTRACT
Self-supervised methods have become crucial for advancing deep learning by leveraging data itself to reduce the need for expensive annotations. However, the question of how to conduct self-supervised offline reinforcement learning (RL) in principled way remains unclear. In this paper, we address this issue by in- vestigating the theoretical benefits of utilizing reward-free data in linear Markov Decision Processes (MDPs) within a semi-supervised setting. Further, we propose novel, Provable Data Sharing algorithm (PDS) utilize such reward-fii free data for offline RL. PDS uses additional penalties on the reward function learned from labeled data prevent overestimation, ensuring a conservative algorithm. Our re- sults on various offline RL. tasks demonstrate that PDS significantly improves the performance of offline RL algorithms with reward-free data. Overall, our work provides a promising approach to leveraging the benefits of unlabeled data in of- fline RL while maintaining theoretical guarantees. We believe our findings will contribute to developing more robust self-supervised RL methods.
INTRODUCTION
Offline reinforcement learning (RL) is promising framework for learning sequential policies with pre-collected datasets. It is highly preferred in many real-world problems where active data col- lection and exploration is expensive, unsafe, infeasible (Swaminathan & Joachims 2015 Shalev- Shwartz et al. (2016) Singh et al. 2020). However, labeling large datasets with rewards can be costly and require significant human effort (Singh et al., 2019 Wirth al., 2017). In contrast, unlabeled data can be cheap and abundant, making self-supervised learning with unlabeled data an attractive alternative. While self-supervised methods have achieved great success in computer vision and nat- ural language processing tasks (Brown et al.) 2020) Devlin et al. 2018 Chen et al. 2020), their potential benefits offline RI. are less explored.
Several prior works (Ho & Ermon 2016 Reddy et al. 2019 Kostrikov et al. 2019) have explored demonstration- bastdd approaches eliminate the need for reward annotations, but these approaches require samples to be near-optimal. Another line of work focuses on data sharing from different datasets (Kalashnikov et al., 202T Yu al. 2021a). Still, it assumes that the dataset can relabeled with oracle reward functions for the target task. These settings can be unrealistic in real-world problems where expert trajectories and reward labeling are expensive.
Incorporating reward-free datasets into offline RL is important but challenging due to the sequential and dynamic nature of RL problems. Prior work (Yu et al., (022) has shown that learning to predict achieve good results. However,
rewards can be difficult, and simply setting the reward it's 'ss unclear how reward- preurdddreeitioon methods affect performance and whether reward-free -ree data can provably benefit offline RL. This naturally leads the following question:
How can we leverage reward-free data improve the performance of offline RL. algorithms in principled way?
"Equal contribution.

---

Published as a conference paper at ICLR 2023
To answer this question, conduct theoretical analysis of the benefits of utillizigg unlabeled data in linear MDPs. Our analysis reveals that although unlabeled data can provide information about the dynamics of the MDP, it cannot reduce the uncertainty over reward functions. Based on this insight, we propose a model-free method named Provable Data Sharing (PDS), which adds uncer- tainty penalties to the learned reward functions maintain conservative algorithm. By doing so, PDS can effectively leverage the benefits of unlabeled data for offline RL while ensuring theoretical guarantees.
We demonstrate that PDS can cooperate with model-free offline methods while being simple and efficient. We conduct extensive experiments on various environments, including single-task domains like MuJoCo (Todorov et al. 2012) and Kitchen (Gupaa et al.. 2019), as well as multi-task domains like AntMaze and Meta- World (Yu et al.. 2020a). The results show that PD$ improves significantly over previous methods like UDS Yu et al. 2022) and naive reward prediction methods.
Our main contribution the Provable Data Sharing (PDS) algorithm, novel method for utilizing unsupervised data in offline RL that provides theoretical guarantees. PDS adds uncertainty penalties to the learned reward functions and can be easily integrated with existing offline RL algorithms. Our experimental results demonstrate that PDS can achieve superior performance various locomotion, navigation, and manipulation tasks. Overall, our work provides promising approach to leveraging the benefits of unlabeled data in offline RL while maintaining theoretical guarantees and contributing to the development of more robust self-supervised RI. methods.
2 RELATED WORK
Offline Reinforcement Learning Current offline RL methods (Levine et al. 2020) can be roughly divided into policy constraint-based, uncertainty estimation-based, and model-based approaches. Policy constraint methods aim to keep the policy close the behavior under probabilistic dis- tance (Siegel et al.] [0200] Yang et al.] 2021 Kumar et al., 2,200 Fujimoto & Gu [222 Yang et al.
2022
Fujimoto et al. [019 Hu al. [022 Kostrikov et al.] 2202; Ma et al. [021 Wang al.
2021
Uncertainty astimation-hasee methods attempt to consider the Q-value prediction's 's confi-
dence using dropout or ensemble techniques (An et al., [021 Wu et al.) (021). Last, model-based
methods incorporates the uncertainty in the model space for conservative offline learning (Yu et al.
20206
20216
Kidambi et al. 2020).
Offiine Data Sharing Prior works have demonstrated that data sharing across tasks be ben- eficial by designing sophisticated data-sharing protocols (Yu et al.) 2021a). For example, previous studies have explored developing data sharing strategies by human effort (Kalashnikov al. 2021), inverse RL (Reddy et al.] 2019 Li et al. (020), and estimated Q-values (Yu al. 2021a). However, these data sharing works must assume the dataset can be relabeled with oracle rewards for the target task, which is strong assumption since the high cost of reward labeling. Therefore, effectively incorporating the unlabeled data into the offline RL algorithms essential. To solve this issue, some recent work (Yu et al.) 2022) proposes simply applying zero rewards to unlabeled data. In this work, we propose a principle way to leverage unlabeled data without the strong assumption of reward relabeling.
Reward Prediction It is widely observed that reward shaping and intrinsic rewards can accelerate learning in online RI. (Mataric 1994 Ng al. 1999 Wu & Tian 2017] Song et al., 2019 Guo et al. 2016 Abel et 2021). There are also extensive works that studies automatically designing reward functions using inverse RL (Ng et al., 2000; Fu et al. 2017). However, there is less attention on the offline setting where online interaction is not allowed and the trajectories may not be optimal.
PRELIMINARIES
3.1 LINEAR MDPs AND PERFORMANCE METRIC
We consider definite-horizon discounted Markov Decision Processes (MDPs), defined by the tuple
(S, A. P, r),,),, with state space S, action space A, discount factor y E [0, 1), transition function
P : A (SS), and reward function
A - [0, r.ax). To make things more concrete,
2

---

Published as a conference paper at ICLR 2023
consider the linear MDP (Yang & Wang 2019: Jin et al. 2020) follows, where the transition kernel and expected reward function are linear with respect to feature map.
Definition 3.1 (Linear MDP). We say episodic MDP A, P. 7) is linear MDP with known feature map Φ S x A - Rd if there exist unknown measures = (Mi,...,Pa) over and an unknown vector 0 IRd such that
Pis' 88 a) = (b(s, a), m(6'), r(s,a) = (((ss)))))
for all (, 8) E S x A x S. And we assume ||0(s, a)ll I 1 for all (8,a, ESEEXxX and max(]la(S) 2, |0 [2] I Vd, where [|x(S)] = Is [Ix(s)]
A policy ग : S + ((A) specifies decision-making strategy in which the agent chooses actions adaptively based the current state, i.e., Or TT- 8c). The value function V : R and the action-value function (Q-function) Q* S x A +
IR are defined as
V (8) Em
>'r(s;, az) so = s}
("(s, Ex Σ
2'r(ss, (t)
80 8, 8, - a). (2)
t=0
where the expectation is with respect to the trajectory T induced by policy We define the Bellman operator as
(s)) a) = Ber-p())s,a)) [r(s, a) + +s)))
(3)
We use .'' "Q" and V to denote the optimal policy, optimal Q-function, and optimal value function, respectively. We have the Bellman optimality equation
(s) = max Q" (s, atA
Q" = (BV")
(4)
Meanwhile, the optimal policy x² satisfies x (- s) = argmax(Q" (s, (
(Q" (1)))))
where the maximum is taken over all functions mapping from S to distributions over A. We aim to learn policy that maximizes the expected cumulative reward. Correspondingly, define the performance metric as
SubOpt(T, 8) = V (s) Vs).).
(5)
3.2 PROVABLE OFFLINE ALGORITHMS
In this section, we consider pessimistic value iteration (PEVI; et al. 202T) as the backbone algo-
rithm, described in Algorithm 2 It is representative model-free offline algorithm with theoretical guarantees. PEVI uses negative bonus I(-,)) over standard Q-value estimation Q -) (iV) to
reduce potential bias due to finite data, where B some empirical estimation of B from dataset D.
Please refer to Appendix I more details of the PEVI algorithm.
We use the following notion of uuccerttainty quantifier as follows to formalize the idea of pessimism. Definition 3.2 (E-Uncertainty Quantifier). We say r: S x A IR is a E-uncertainty quantifier for B and V if with probability &, for all (s,a) S x A.
(s. (alaaa--BBBii)) (s)) & T(s.a).
3.3 UNSUPERVISED DATA SHARINGG
We consider unsupervised data sharing in offline reinforcement learning. We first characterize the quality of the dataset with the notion of coverage coefficient (Uehara & Sun 2021). defined as below.
Definition 3.3. The coverage coefficient Ct of dataset D {(s,, T defined as
N
Ct = sup
(6r ar (r))((r))))))) a7) 2 C. IE ar)ss,, ax)
80 =n]] .l.vsEs
3

---

Published as a conference paper at ICLR 2023
The coverage coefficient Cf is common in offline RL literature (Uehara & Sun 2021 Jin et al. 2021 Rashidinejad et al. 2021). which represents the maximum ratio between the density of empirical state-action distribution and the density induced from the optimal policy. Intuitively, it represents the quality of the dataset. For example, the expert dataset has a high coverage ratio while the random dataset may have low ratio.
We denote Do the origin labeled dataset, with coverage coefficient Cj and size No. And denote the unlabeled dataset as Dj, with coverage coefficient Cf and size N1. Note that it is possible that the unlabeled data comes from multiple sources, such multi-task settings, and still use Dj to represent the combined dataset UM D; from M tasks for simplicity.
4 PROVABLE UNSUPERVISED DATA SHARING
How can we leverage reward-fres data for offline RL? A naive approach to learn the reward function from labeled data via the following regression
i- argmin Σ (fo(sr,a)) -rr,)
V 1012
this learned reward function to label unsupervised data. However, this approach can lead to suboptimal performance due to overestimation of the predicted reward rg Yu et al. 2022). which undermines the pessimism in offline algorithms.
where ay) o(sT a,) T in linear MDPs. Then
To address this issue, propose a data-sharing algorithm called Provable Data Sharing (PDS). We start by analyzing the uncertainty in learned reward functions and add penalties for such uncertainty leverage unlabeled data. In Section 4.2] we show that PDS has a provable performance bound consisting of two parts: the offline error, which is tightened compared to no data sharing due to additional data, and the error from reward bias. The performance bound of PDS is provably better than no data sharing as long as the unlabeled dataset has mediocre size quality. We also ex- tend our algorithm in linear MDPs to general settings Section 4.3 and propose using ensembles for reward uncertainty estimation. We demonstrate the effectiveness of PDS by integrating it with IQL (Kostrikov et al., 220I), and present Algorithm 3 which is simple and can be easily integrated with other model-free offline algorithms.
4.1 PROVABLE DATA SHARING
To address the issue of potential overestimation of predicted rewards, we first analyze the uncertainty in learned reward functions. In the context of linear MDPs, the reward function can be learned via linear regression, and the uncertainty of the parameters is characterized by the elliptical confidence region, as shown in Lemma 4.1 This confidence region is important as it allows us to give a more accurate estimation of the reward function while keeping the overall algorithm pessimistic.
Lemma 4.1 Abbasi-- Yadkori et al. 2011)). Let V Tmux /2 /2log] dllog (1 + No ),
Abbasi-Yadkori
vI + M @(sr, ar (668,,)))
C(6) = fDE Rd 100 10-011 0l S sa
(8)
where θ is the minimizer in Equation parameter for the reward function.
then have P(O* E C(0)) 2 1 1 8, where 0 is the true
Proof. Please refer to Theorem in Abbasi- Yadkori et al. 201T) for detailed proof.
provides useful insight: the uncertainty of the learned reward function in linear MDPs only depends on the quality and size of labeled data. Based on this insight, we propose a two-phase algorithm that guarantees provable performance bound, shown in Theorem 4.3 The simple reward prediction method can compromise the pessimistic estimation of the algorithm, while UDS may result in reward bias that is too large. Our algorithm consists of two phases: in the first phase,
Lemma 4.1
4

---

Published as a conference paper at ICLR 2023
construct pessimistic reward estimator that finds the reward function in the confidence set that leads to the lowest optimal value. In the second phase, we conduct standard offliee RL with the given pessimistic reward function.
To solve the challenge of finding the best parameter in the confidence set, which is a bi-level op- timization problem, we propose using simpler method that maintains the pessimistic property of the offline algorithms. This method involves using pessimistic estimation, which allows us to keep the algorithm pessimistic while avoiding the computational challenges of the bi-level optimization problem. Formally,
F(s,a) - max m -
where A vl + + - o(s, ar (s(;) a,)
We adopt the pessimistic estimation in Equation (9) because it provides a lower bound for reward functions the confidence set C()), as guaranteed by the following lemma derived from Cauchy- Schwartz inequalities.
Lemma 4.2. For any 8 C(8). 0 - ((s s s)
(10)
When labeled data searce or there sigmificant distributional shift between the labeled and unlabeled data, Equation (9) degenerates to O, which is equivalent to the UDS algorithm (Yu et al, 2022).
Algorithm Provable Data Sharing, Linear MDP
Labeled dataset Do {(s,, ar F, ) unlabeled dataset D) {(rr, ar ) 2: Require: Confidence parameter 3,
Require:
3: Learn the reward function 0 from D/ using Equation 7
4: Construct the confidence set C()) using Equation (8)
5: Construct the pessimistic reward over the confidence set
+- argmin Ve. BEC(6)
(II)
where Va is the estimated value function from Algorithm and dataset Do U Dj with reward relabeled with parameter 9.
6: Annotate the reward in Do U with parameter 0.
7: Learn the policy from the annotated dataset D, UD using Algorithm
V. क. + PEVI(D, D.D.).
(12)
8: Return
4.2 THEORETICAL ANALYSIS
The following subsection analyzes how the provable data-sharing (PDS) algorithm can enhance the performance bound by leveraging unlabeled data. To be specific, present the following theorem.
Theorem 4.3 (Performance Bound for PDS). Suppose the dataset Do, D1 have positive coverage coefficients C ct, and the underlying MDP a linear MDP. In Algorithm
we set
1, 1, 2 A. Tmax, B cdvC1
1-7 r Tmaxx Si = log [4d(No + N1)
2d.No
C2 = log
(1-2)8
8
5

---

Published as a conference paper at ICLR 2023
where 0 is absolute constant and ó E (0, is the confidence parameter. Then with proba- bility 26, the policy generated by PDS satisfies for all s E E,
2CT max
4r max
dCc2 No.Ol
Su;Opt(Fi;))
(1-7)²
7)²
NoCJ NiC
1- 1
Proof. Please refer to Appendix B for detailed proof.
The performance bound of PDS is composed of two terms. The first is the offline error, which is inherited from offline algorithms. This bound is improved when additional unlabeled data with size N1 and coverage C is available. The second term is the reward bias, which arises due to uncer- tainties in the rewards. Notably, this term is equivalent the performance bound of linear bandit with rewards in the range [0, /'max/(1 -)]] As the number of unlabeled data approaches infinity, the uncertainty of the dynamics decreases zero, and the RL problem becomes linear bandit prob- lem. The theorem demonstrates that PDS outperforms UDS, which suffers from a constant reward bias, and naive reward prediction methods, which lack pessimism and therefore do not offer such guarantees. Moreover, we demonstrate the tightness of the bound by constructing an "adversarial" dataset that matches the bound's suboptimality (see Appendix F))
To better understand the benefits of unlabeled data, we define the suboptimality bound ratio (SBR) of an offline algorithm A the ratio of the suboptimality bound obtained by the policy learned with additional unlabeled data to the suboptimality bound of the policy learned using labeled data alone. Mathematically, the SBR of given by:
SBR(A) = (T.Opt(F.A(Do. D.))
(13)
SubOpi (T.(D). Ø))
where SubOpt is the tight upper bound on suboptimality. The SBR provides a measure of the benefit of unlabeled data to the offline algorithm, with smaller SBR indicating greater benefit from the unlabeled data. Applying this definition to PDS, we obtain the following corollary.
Corollary 4.4 (Informal). The SBR of PDS satisfies
NoC. Noc + N
2(1 -1-11 7) cvd
SBR 2
(14)
Ginte sample term
asymptotic
where e is the constant in Theorem
4.3 and we ignore the logarithmic factors.
When does unlabeled data improve the performance of offline algorithms? Corollary 4.4 al- lows analyze the relative performance of PDS under different conditions. The first term of the bound depends on the qualities and amounts of both labeled and unlabeled datasets. the unlabeled dataset has mediocre number of samples or data quality, the first term will be sufficiently small. The second term affects the asymptotic performance when the unlabeled data approaches infinity, and it depends on the discount factor and the dimension of the problem. PDS improves over no data-sharing algorithms asymptotically in larger problems or longer horizons. For a more detailed discussion, please refer to Appendix
4.3 PRACTICAL IMPLEMENTATION
This subsection outlines the practical implementation of PDS in general MDPs. We employ L ensembles .,,.... Hr to estimate uncertainty, which are learned using Equation pessimistic rewards, we use the following pessimistic estimation:
To estimate
a(s,a) max (u(s, a) - ko(s,a))0)
(15)
where (ss.a) a) I. (s, standard deviation, respectively. Here, a hyperparameter used to control the amount of pes- simism. We also use the minimum I. ensembles for the pessimistic estimation, which is
/ (f a) u(s, a))² are the mean and

---

Published as a conference paper at ICLR 2023
linked Equation
15
following An al. (2021) Royston (9822)
shown in Equation (6)
E min fe, (s, pls.a) 4
((s,)),
(16)
j=l,..,
where @ is the CDF of the standard Gaussian distribution.
The appropriate value of k for each domain can be difficult to determine. To address this issue, we observe that the amount of pessimism required for different domains is proportional to the difference in mean rewards between labeled and unlabeled data. Leveraging this insight, we propose a simple and efficient automatic mechanism for adjusting the value of the parameter. Specifically, suggest method that adjusts & based on the difference in mean rewards, as given by Equation (17
F(s,a) max min fo, (s, a) ko a),
I=.....
where k= == a max(# - P..0)
(17)
| +
where μ = - + Ein ((s,, a.). Ein (s, s, a,) are the mean reward of labeled and (pre- dicted) unlabeled data, respectively. We use 25 and L 10 in all experiments.
Then we can plug in any model-free offline algorithms. Here we use IQL (Kostrikov et al. 2021 as the backbone offline algorithm, but we emphasize that it can be easily integrated with other model- free algorithms. The details of our algorithm is summarized in Algorithm 3 in Appendix A.2
5 EXPERIMENTS
In this section, we aim to evaluate the effectiveness of pessimistic reward estimation and answer the following questions: (1) How does PDS perform compared to the naive reward prediction and unlabeled data sharing (UDS) methods in single locomotion and manipulation tasks? (2) How does PDS behave in multi-task offline RL settings compared baselines? (3) What makes PDS effective?
Algorithm
door-open
door-close
drawer-open
drawer-close
average
UDS
16.2+11.1
0.0++0.0
30.4+60.4
0.4
57.2++30.8
Rew Pred
26.4
14.3
102.6:140.2
182.2:10.4
105.5
PDS
25.5 4.8:
114.3
153.8
182.81104
119.1+18.1
No Share
0.0+0..0
29.6
175.0=11.66
52,4+31.0
Oracle
20.6
113.245.7
135.64436.5
182.6:10.4
113.0#19.6
Table 1: Experiment results for multi-task robotic manipulation (Meta- experiments. Num- bers are averaged across five seeds and we bold the best-performing method that does not have access to the true rewards.
Env
Tasks / Dataset type
UDS
Rew Pred
CDS+UDS
PDS
medium-play (3 tasks) directed 15.8
26.2.13.7
40.6
40.0-1..6
medium-pla -lay (3 tasks) / undirected
19.6+2.5
27.242.9
37.245.1
29.2:44.1
Antmaze
medium-diverse (3 tasks) directed
8.743.3
20.243.8
33.3#11.5
53.2+3.6
medium-diverse (3 tasks) undirected
9.7++1.3
4..
40.743.5
4..5+
Table 2: Experiment results for AntMaze tasks with normalized score metric averaged with five random seeds.
Single-task domains and datasets. To address Question (1), we empirically evaluate the PDS algorithm on the hopper, walker2d, and kitchen tasks from the D4RL benchmark suite (Fu et al. 2020). We use 50 labeled trajectories with varying amounts of unlabeled data of different sizes and qualities. This experimental setup is motivated by real-world problems where labeled data is often scarce, and additional unlabeled data may be readily available.

---

Published as a conference paper at ICLR 2023
Task
Labeled Size
Unlabeled Size
UDS
Rew Pred
PDS
Oracle
medium 50K
medium 10.M
58.7
64.813,2
73.9=8.4
66.322.1
Hopper
medium 50K
medium / 0..4M
57.3+1.4
68.9++4.0
77.8+7.4
67.4+4.22
medium 50K
medium 0.6M
5..6=1.4
68.5+2.1
75.9+2.4
68.2+2.1
expert 50K
random 0.1M
53.313.8
27.9415.1
42.719.8
18.145.9
random / 50K
expert / 0.1M
4.3::0.4
84.7
92.319.8
40.1
medium 50K
medium/ / 0.
7.8
71.412,9
76.1
74.6+t2.3
Walker2d
medium 50K
medium 0.4M
1.4
70.914.11
00.1
77.7+2.44
medium 50K
medium / 0.6M
7..8
79.914422
1..1
79.122.7
expert/ 50K
random 0.
25.7=11.11
2.7+0.2
39.5+10.0
22.6#112
random / 50K
expert / 0.1M
0.4:0011
95.3+2.3
1014=332
15.
Table 3: Experimental results with normalized score metric averaged with five random seeds.
Multi-task domains and datasets. We investigate Question (2) by evaluating PDS several multi-task domains. The first set domains we consider is Meta- Meta-World (Yu et al. 2020a). where adopt the same setup in CDS (Yu et al. 2021a) and evaluate PDS four tasks: door open, door close, drawer open, and drawer close. The second domain is the AntMaze task in D4RL, which consists of mazes of two sizes (medium and large) and includes and 7 tasks, respectively, corresponding to different goal positions. For detailed description of the experimental setting, please refer Appendix D
Comparisons. To ensure a fair comparison, we combine UDS with IQL (Kostrikov et al. 2021) the same underlying offline RL method PDS. In addition to UDS, train naive reward pre- diction method and the sharing-all-tuu--ewwarr method (Oracle), and adapt them with IQL. In all experiments, we set the hyperparameters 25 and L 10 for our method.
5.1 EXPERIMENTAL RESULTS
Results of Question (1). We evaluated each method the hopper, walker2d, and kitchen domains and found that PDS outperformed the other methods on most tasks and achieved competitive or better performance than the oracle method (TableB). Notably, PDS performed well when the labeled and unlabeled datasets had different data qualities, which we attribute to its ability to capture the uncertainties induced by this distribution shift and maintain pessimistic algorithm. The prediction method performed well when the unlabeled dataset had high quality, and UDS performed well when the unlabeled data had low quality. PDS combined the strengths of both methods and achieved superior performance.
Results of Question (2). Multi-task settings exhibit greater distributional shifts between labeled and unlabeled data due to the differing task goals. We evaluated PDS and the other methods on the AntMaze and Meta-World domains (Tables 2 and [ and found that PDS's performance was comparable to the oracle method and outperformed the other methods. UDS performed relatively poorly on the Meta-Workh dataset, possibly due to the high dataset quality, which made labeling with zeros induce large reward bias. On the multi-task AntMaze domain, PDS outperformed both UDS and the naive prediction method, especially on the diverse dataset. These results aligned with observation single-task domains that PDS performs better when the distribution shift between datasets larger.
Results of Question (3). We conducted experiments on the hopper and walker2d tasks with vari- ous penalty weights investigate the effect of uncertainty weights in PDS (Figure []. The results shows that PDS can interpolate between UDS and the reward prediction method and offered better trade-off to balance the conservation and generalization of reward estimators, resulting in better per- formance. Also. PDS reduces the variance from reward prediction and is close to the oracle method, indicating its ability to reduce the uncertainties from the variance of reward predictors while keeping the reward bias small, shown in Figure 2
8

---

Published as a conference paper at ICLR 2023
Wallerid Random Expert
Hopzer Random Expert
Walker Expert-Random
Hopsen Expert Raadom
100
PD6 UpS
PDS
UnS
Riw Prd
Reward Frod
Narmsload
Rormallued
Nermalood
PDS
-2H
PDS
Roward Prod
Revard Prod
Weglt
Weight
Woglt
Peralty
Figure 1: Impact of penalty weight & on the performance. We evaluate PDS on Hopper/Walker2d- Labeled (Exper/Randomm--nllabeee (Expert/Random) tasks with various k.
Drawer Open
Expert
Drawer Close
Medium-Pli Play
10
Standard Deviation
Standard Deviation
109
M
UDS
10P
PDS
10-1
Reward Pred
10-1
10-2
No Sharing
Sharing All
10-2
10-3
0.25
0.75
0.00
0.25
0.50
0.75
1.00
Million Steps
Million Steps
The standard deviation of the state values learned with IQL. The reward prediction method suffers from large deviation. The oracle method's is and PDS's deviation is mediocre, and sharing and UDS methods enjoy the smallest deviation.
Figure 2:
Discussion of PDS, UDS, and Reward Prediction: PDS can be seen as a generalization of both UDS and the reward prediction method. UDS the penalty weight k to infinity, while the reward prediction method sets it to zero. However, UDS introduces a high reward bias, and the reward prediction method ruins the pessimism of offline algorithms. In contrast, PDS offers trade-off be- tween bias and pessimism by adaptively adjusting k. To verify that overestimation is the main factor for the suboptimal performance of the reward prediction method, we conduct additional ablation studies, as shown in Appendix
CONCLUSION
In this paper, we show that incorporating reward-free data into offline reinforcement learning can yield significant performance improvements. Our theoretical analysis reveals that unlabeled data provides additional information about the MDP's dynamics, reducing the problem to linear bandits in the limit and improving performance bounds therefore. Building upon these insights, pro- pose a new algorithm, PDS, that leverages this information by incorporating uncertainty penalties on learned rewards ensure conservative approach. Our method has provable guarantees in the- ory and achieves superior performance in practice. In future work, it may be interesting to explore how PDS can be further improved with representation learning methods, and extend our anal- ysis to more general settings, such as generalized linear MDPs (Wang et al. 2019) and low-rank MDPs (Ayoub et 2020 Jiang al. 2017).
ACKNOWLEDGEMENTS
This work is supported in part by Science and Technology Innovation 2030 "New Generation Artificial Intelligence" Major Project (No. 2018AAA0100904) and the National Natural Science Foundation of China (62176135).

---

Published as a conference paper at ICLR 2023
REFERENCES
Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances neural information processing systems, 24:2312-2322 2011.
David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho. Michael Littman, Doina Precup, and Satinder Singh. On the expressivity of markov reward. Advances in Neural Information Processing Systems, 34:7799-7812, 2021.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offinne inforcement learning with diversified q-ensemble. Advances in Neural Information Processing Systems, 34, 2021.
Alex Ayoub, Zeyu Jia, Csaba Szepesvari, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In International Conference Machine Learning, pp- 463-474. PMLR, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamners Advances neural information processing systems, 33:1877-1901, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Jacob Devlin, Ming- Mei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
2018.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint arXiv:1710.11248. 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2106.06666 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 2052-2062. PMLR, 2019.
Xiaoxiao Guo, Satinder Singh, Richard Lewis, and Honglak Lee. Deep learning for reward design to improve monte carlo tree search in atari games. arXiv preprint arXiv:1604.0700
2016.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint 19Xiv:1910.11999 1196,, 2019.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Hao Hu. Yiqin Yang, Qianchuan Zhao, and Chongjie Zhang. On the role of discount factor in offline reinforcement learning. arXiv preprint S.Xiv:220060038883 2022.
Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. Con- textual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning, pp. 1704-1713, PMLR, 2017.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pp. 2137-2143. PMLR, 2020.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference Machine Learning, pp. 5084-5096. PMLR, 2021.
10

---

Published as a conference paper at ICLR 2023
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski,
Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic re- inforcement learning scale. arXiv preprint arXiv:2104.08212, 2021.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-
based offline reinforcement learning. arXiv preprint arXiv:2005.05951 2020.
llya Kostrikov, Ofrr Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. arXiv preprint arXiv:1912.05033 2019.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2110.06161 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Alexander Li, Lerrel Pinto, and Pieter Abbeel. Generalized hindsight for reinforcement learning. Advances in neural information processing systems, 33:7754-7767. 2020.
Xiaoteng Ma, Yiqin Yang. Hao Hu, Qihan Liu, Jun Yang, Chongjie Zhang, Qianchuan Zhao, and Bin Liang. Offline reinforcement learning with galue-based episodic memory. arXiv preprint arXiv:2110.0979 2021.
Maja Mataric. Reward functions for accelerated learning. In Machine learning proceedings 1994, pp. 181-189. Elsevier, 1994.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Ieml, volume 99, pp. 278-287, 1999.
Andrew Y Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, vol- ume 1, pp. 2, 2000.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein- forcement learning and imitation learning: A tale of pessimism. arXiv preprint arXiv:2103.1200 2021.
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv:1905.11 11008, 2019.
JP Royston. Expected normal order statistics (exact and approximate): algorithm as 177. Applied Statistics, 31(2),161-5, 1982.
Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295, 2016.
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08396, 2020.
Avi Singh, Larry Yang, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. In Antonio Bicchi, Hadas Kress-Gazit, and Seth Hutchinson (eds.), Robotics: Science and Systems XV. University of Freiburg, Freiburg im Breisgau, Germany, June 22-26. 2019, 2019, doi: 10. 15.//557 2019.X..03..
Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint arXiv:2010.14500, 2020.
Shihong Song, Jiayi Weng, Hang Su, Dong Yan, Haosheng Zou, and Jun Zhu. Playing fps games with environment-aware hierarchical reinforcement learning. In IJCAL, pp. 3475-3482, 2019.

---

Published as a conference paper at ICLR 2023
Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. The Journal of Machine Learning Research, 16(1):1731-1755 2015.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226, 2021.
Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offlin reinforcement learning with reverse model-based imagination. Advances in Neural Information Processing Systems, 34:29420-29432 2021.
Yining Wang, Ruosong Wang, Simon S Du, and Akshay Krishnamurthy. Optimism in reinforcement learning with generalized linear function approximation. arXiv preprint 191.Xi::112204416.. 2019.
Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, al. A survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1-46, 2017.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08144 140, 2021.
Yuxin Wu and Yuandong Tian. Training agent for first-person shooter game with actor-critic currieu- lum learning. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
Lin Yang and Mengdi Wang. Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning, pp. 6995-7004. PMLR, 2019.
Yiqin Yang, Xiaoteng Ma. Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent 2021.
reinforcement learning. arXiv preprint
Yiqin Yang. Hao Hu, Wenzhe Li, Siyuan Li, Jun Yang, Qianchuan Zhao, and Chongjic Zhang. Flow to control: Offline reinforcement learning with lossless primitive discovery. arXiv preprint arXiv:2212.01105, 2022.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100. PMLR, 2020a.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13239 2020b.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems, 34:1 1501-11516,, 2021a.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-basee policy optimization. arXiv preprint arX1v:2102.08363, 2021b.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement learning. arXiv preprint arXiv:2202.01741, 2022.
12

---

Published as a conference paper at ICLR 2023
A ALGORITHM DETAILS
A.1 PESSIMISTIC VALUE ITERATION (PEVI, ET AL. 202
In this section, we describe the details of PEVI algorithm.
In linear MDPs, we can construct BV and I' based on D as follows, where BBV is the empirical estimation for BV. For given dataset D {(0r, ar, (r)))==== we define the empirical mean squared Bellman error (MSBE) as
N M(w) = Σ (r. + (8 a7) ww)) + Al
Here À >> is the regularization parameter. Note that w has the closed form
N
1 = A
((5 a7) (rr + ((+++1)))
N o(s ar))s,))
where A Al +
(18)
Then simply let
BV (0, w)
(19)
Meanwhile, construct based D
r(s,a) = (o(s, B- - A
(20)
Here > is the scaling parameter. The overall PEVI algorithm is summarized in Algorithm
Algorithm 2 Pessimistic Value Iteration, PEVI
1: Require: Dataset D = {(5, Tr, $r+1) )
2: Initialization: Set (-)) 0 and construct I(.)).
3: while not converged do
4: Construct (BV (-.-)
5: Set Q(..)) + (B) - ....).
6: Set (--1-) + argmax, E, ((.-))
7: Set (-)) ER (
8: end while
9: Return V
A.2 IQL WITH PROVABLE DATA SHARING
In this section, we give detailed description of our IQL+PDS algorithm.
13

---

Published as a conference paper at ICLR 2023
Algorithm 3 IQL+PDS algorithm, General MDPs
Labeled dataset Do, unlabeled dataset 1 Input: Parameter at 3, k, 7.
Input:
Output: policy TT..
1: Learn reward functions as in Equation (
2: Construct pessimistic reward estimation in Equation (7
3: Relabel unsupervised dataset D2 and combine with the labeled dataset Do.
4: Initialize v, 8, 0, Φ
5: for each gradient step do
6: Φ + v xy V (vb),,
(v( = E,,a [V(()]
- [*))
7: 0 - x (o)
LQ(0) Bs,a, 02gs,a) Qo(s.a))
8:
i - of + (1 10--a. 0
9: end for
10: for each gradient step do
@ f- Φ A VoLn((),
L (Ø) Es,a exp (3(Qp(s,a) V=(s))) log(T)(a(s))
12: end for
B PROOF OF THEOREM 4K
Proof. From Equation
11 in Algorithm
1 we have
V S Ve. v0 C(6),
(21)
where 0 is the pessimistic estimation of 0.
Let & be the event 0 E C()), then we have P(E,) 2 1 - 5 from Lemma 4.1
Let E be the event where the following inequality holds,
(BB) (B) a) <T (T ES A
(22)
then we have P(E2) 21 8 from Lemma C.3
Condition on & Eg, we have.
V
Vol"
Vo.
Vo-
V
Vg
D/
V
V
138"
V
V/
Vo.
8
VØ
VT
V
V
V
V
+ V - Vo
0
47
d²
2CT max
d:
Noc
(1 71
NoC + NiC
where the first inequality follows from Lemma
The second inequality follows from Equa-
tion (TT), and the last inequality follows from Lemma
C.5 and C.I
From the union bound, we have that the above inequality holds with probability of - 25.
C ADDTIONAL LEMMAS AND MISSING PROOFS
Lemma C.1. Under the event Lemma C we have
V (8) Vo(8) 5 2ermax
dà
(1 7)2 CiN'
14

---

Published as a conference paper at ICLR 2023
with probability 1 - 6, for all |101|3 < d.
Proof. Let
6(s,a) a) = r(s,a) +)Bx-P((aa) V(s') ((s, a),
(23)
From the definition (s, a) and (s). we have
8(5,a) (.) ass,a) - B.V.V. a))
(24)
Under the condition of Lemma
C it holds that
f ö(5,a) S 21' a), for all s, a.
(25)
Then we have
Vg (s) Vo(s)
r(s,a) + ,² (5)) Eana [((..a) ((8,a) + , (5') @(s,,0) Banx' [Q(s,a)) - Eant [0(s.a)
E Earr PP (1)a,)
= -Raw (-13aa)
Baa
- (5')) (((s,a) + (V')) - [(33,a) +/ ( [ (s')-
(Q(s,) ) (.1s) R(-1s) 1
E Eaoo (11..a) ((s,a)] +
((s,,),, (, - - +... A
Ex
80(5,, az) 10000
+ E
Σ 7' (Ō(se) -, - -F(-180)
90 =
r==
S
y 18(8,, ax)
1000000000000 =
80 = s) (T(, ax)
<2E,
-2
a)) AA- ((=))(s) 80 =
Here the first inequality follows from the fact that R-11s) argmax, (Ā(--). x-(s)) and the second inequality follows from Equation 25 By the Cauchy-Schward inequality, we have
E
r' (0(sE, ar) TA - -((()s =8]
1
Eyr' Tr
50 =
E TT (o(s,
80
7
1
<
Tr Exr (p(s,
8o A-A1)
(26)
15

---

Published as a conference paper at ICLR 2023
for all S. Then have
V (+) Vo(s) s 2
(T) "(ss,
( at) 1/2 80
2/3
(Ire,s (T+ N. x)), x))
$ 1 1-7
23
Σ
2,(s)
(27)
1
Ct N. 2y(s))
Here ()y(s) are the eigenvalues of Ex-.s for all E S, the second inequality follows from
Equation (6).. Meanwhile, by Definition 3.1 we have (((())))) < 1 for all (8,a) E x A. By
Jensen's inequality, we have
[Dart S E, [[ a))(s, 1lop so =s] <
(28)
for all E S. As Ex is positive semidefinite, we have As(s) E [0, 1] for all E S and all j E [d].
Hence we have
28
V8 (8) -6) (s))
Ci - (y(ss)
2/3
Σ
2=1 1+
(1 71 V CiN
2cr max
dàc
(29)
for all x S, where the second inequality follows from the fact that (y [0, for all S and all j E [d], while the third inequality follows from the choice of the scaling parameter 0.
Then we have the conclusion in Lemma Lemma C.2. Under the event in Lemma
C.3
we have
Vo(s) V (s) (s) 0
(30)
with prohability for all [10113 S d.
Proof. Similar the proof of Lemma C.I let
ö(s, a) = r(s, a) + "E -
(31)
we have
((s) (3) =Ban [(5,,a)
-BanR +
[Q(s,a) r(s, 7V(6))
=
(V(s') - V$ (6')
+ B
Ex ())s + 77BonR ((55)-V*(8) (s/)
Ex [)(ss))] +...
Ex
(ss, az) 30
Then under the condition of Lemma C
it holds that
0 5 5(8,a) < 28 a), for all 8,
(32)
Then we have the result immediately.
16

---

Published as a conference paper at ICLR 2023
Lemma C.3 (g- (E-Quantffiers). Let
X == 1, = c. cVarr Vs, l= (2dN/(1 - -))).
(33)
Then = B. (d(s, a) 1 a)) 1/2 are E-quantifiers with probability at least &. That is, let En be the event that the following inequality holds,
(BB)(s.)) (BV) a)l ST=
ESxA
(34)
Then have P(E2) -
Proof. we have
BV BV a( (o::(w---- (w
N
= 0 a) w - -(s
Σ
(r(rr + yV 7 (00+11)
N
N
N
= a(s w - @(8
Σ
2-0 w
+ + (0
2
- w
Σ
o (r(r + (+++1)))
7=1
T=1
ii)
(ii)
(35)
Then we bound (i) and (ii), respectively. For (i), we have
= a) (( Al)w
|
S Vmax V A a),
(36)
where the first inequality follows from Cauchy-Schwartz Schwartz inequality. The second inequality follows from the fact that ||A |A op S A and Lemma C.4
For notation simplicity, let e, Tr (V(((())) o5
then we have
(6i) o(s a) 11 Σ dter
N
Er cr aa- 110(8,
7=1
N
Σ
Φ cr 1a-1
-
(37)
7=1
The term (iii) is depend on the randomness of the data collection process of To bound this term, we resort to uniform concentration inequalities to upper bound
Σ o(s a.) cv()
sup V
where
VRR = (V, : + 10, Vinal with < 10,
(38)
where V(s; w. 8) max.(ōls, T)T Tw 3 Vøls,a) TE- For all 0, let
Alee; RG B, X) be the minimal cover if V(R, B, A). That is, for any function V E V(R, B, A), there exists function Vi EN((e R, B, A), such that
sup ((s) - vt() Se. SES
(39)
17

---

Published as a conference paper at ICLR 2023
Let R6 Vmaxx v /d/A, Bo = 23, it is easy to show that each iteration, Va E V(Ro, Bu, 2). From the definition I. we have
= BV - B (
(ols) (1) do SEc.
(40)
Then we have
(r + +V - BV) - (r+ s 27c.
(41)
Let ct = 7(57; + (s,+1) - -V a), we have
(ii)² = || Σ o- 11 S 21|
+ + Σ (et -c)ll
N
$ 2||
Σ
Φ el 11.2 1 + +8y22²
Σ 1
N
K 2
Σ
+ -
+872223/2
11 From the assumption for data collection process, it is easy to show that Ep(c, [ O, where F--1 a(((s; (ra, 8841) (441)==13) is the o-algebra generated by the variables from the first step. Moreover, since E, 2V we have e, are 2Vmxx-Sube -ubb--uuaaa conditioning on F--1. Then we invoke Lemma C.7 with Mo X and
remains to bound || |
Mr I o(,,, a,) a()) TT) For the fixed function V: S [0, Vmax]. we have
PD Σ ar)
(V .
8V2 max log ( det(A) (cc(4)11222)
( det(A (X 1)1/22)
(42)
for all 8 (0, 1). Note that (|o(s & for all (s, ES x b Definition 3.1 We have
N
||A = | Σ (5 ar (r)((,,,)))))) a
<
|ø(s., (o)s,, llop SX+N,
where || !lop denotes the matrix operator norm. Hence, it holds that det(A) S (A + N) and det(A . 1) = X4, which implies
PD
(5 (4) (r). (/1))/ (.))))
4V2 max (2 log(1/8) + d 1og(11+ + N/A))
A_
Σ ((8 (7) e (V)) (
N
> 8V2 log (1 det(A)2/2
det(() 1)1/2 11112) Sd.
PD
Therefore, we conclude the proof of Lemma
Applying Lemma C.3 and the union bound, we have
Po sup VEN(()
((8. (+))) A
4V2 - (o-lo
(43)
Recall that V E V(Ro Bo, A),
where Ro = Vmax v Nd/A, Bo = 23, 1, 8 c
-==c-VVmrr dV (44)
Here e >> is an absolute constant, & E (0, 1) is the confidence parameter, and ( = log(2dV / 15) is specified in Algorithm 2 Applying Lemma C.6 with dV ///max// /N, we have
log [=) < d log + 1/2 + d2 log + 32c² d
s d log(1 4/ + d² log(1 32c² ///2
(45)
18

---

Published as a conference paper at ICLR 2023
By setting 8 = E/\W(E)|, we have that with probability least 1 - E.
N
(b.: u-) er ( Σ
8V max (2
+ 4d log(64c² (29(6422.1/22330) + d loggl + N) + 4d²)
8V man C(4 + log (64c²)).
(46)
Here the last incquality follows from simple algebraic inequalities. We set 2 1 to be sufficiently large, which ensures that 36 + log(64c") S c² /4 on the right-hand side of Equation (46). By
Equations (37 and (46), it holds that
[i) S c/2 dV dVmmx o(z a) 1os, 3/2 v a l0s, a)
(47)
By Equations 20 35
36 and (47 for all (s, a) E E x A, it holds that
(BB)) (B (BV) (s, a) s (Vmux vd + 8/2)
S T(s,a) a)
(48)
with probability at least ç. Therefore, conclude the proof of Lemma
Lemma C.4 (Bounded weight of value function). Let Vmaa T'maa/ (1 7)- For any function
V + [0, Vmax], we have
Nd
||l||| S Vinaa Vd, ||||||||||||||||| S VIna
Proof. Since
w ((s,a) (M, + ( a)(s
have
M + ((si/V(s'))) V(5')(('') Mds'
Tmax vd + 7Vmava
Vmnax Vd.
For G. we have
| = ||A Σ (r + (V(+4)))
|| dr (r + VV
N
< VV max
Σ
'
T=1
N
V max
6J 1/2 -
max
0 A A-1
N
Tr(A -1 Σ
A
Nd
< Vimax
A
19

---

Published as a conference paper at ICLR 2023
Lemma C.5. For policy πT" and any reward function parameter A E C()), have
2r max
d²
IV
V
NoCo
Proof. From the definition, we have
(ro(s, a) - rg((,,)) (-(((s (0, o(s, O
1 A| || a)
$ a aaavllss a(s - ((s.a),
Where the first inequality follows from the Cauchy-Schwartz inequality and the second inequality uses the fact that C(o). Then have
E
7((o(s,,) r0(8,a)) 80
S Exg
7' - r)(s,
500
S a.Ex
80
2r max
d²
<
c. No
Noco
where the second to last inequality follows similarly Equation follows from the fact that a 1
26
27 and the last inequality
Note that such choice of a is sufficient for Lemma to hold since
No vd
No
V + Tiax
| 2 8
+ dlo (1 +
= Timx
2 2 + d dlog (1 +
d
< 1 Timxx
2 log
d dlog 2No
2No
Timxx
dlog
2No
< 2Fmax
dlog
max V dG2,
where the inequalities holds for sufficiently small 8 0 and d 2 2.
C.I TECHNICAL LEMMAS
Lemma C.6 (e-Covering Number (Jin et al. 2020 ) For all h E [H] and all é > O, we have
log [N[ R, B, d log(lg 4R/e) + d log / /////////////))) (=*
Proof of Lemma C.6 See Lemma D.6 in Jin et al. (2020) for detailed proof.
Lemma C.7 (Concentration of Nelf-Noraalz Processes (Abbasi- Yadkori et al. 201T)). Let
{F, tzo be a filtration and ) be RR-valudd stochastic process such that et is F. F-measurable
20

---

Published as a conference paper at ICLR 2023
for all t 1. Moreover, suppose that conditioning Fi-1, Et is a zero-mean and o-sub Sub-Gaussian random variable for all t 2 1, that is,
Ele Ft-1) =0 O,
E (((xp(ACe) Ft-1] exp(X2o212))
Meanwhile, let {0r be an -a-uul stochastic process such that Φ: is Fi-1 m--easuurrreeee for all t 2 1. Also, ler Mo E BRdxd be a deterministic pnsitive-definite matrix and
M, = Mo Σ 0.D.
for all I 1. For all >> holds that
det(M det
Σ dsc.
S 20
log
Mr¹
8
for all t with probability at least 8. 8.
Proof. See Theorem of Abbasi- Yadkori et al. (2011) for a detailed proof.
21

---

Published as a conference paper at ICLR 2023
D EXPERIMENTAL SETTINGS
D.1 MULTI-TASK ANTMAZE
We first divide the source dataset (c.g., (rimmzzz-mmdiu-ddives---))))) into the directed dataset or the undi rected dataset. The trajectories in the undirected dataset are randomly and uni- formly divided into different tasks. contrast, the di directed dataset is associated with the goal closest to the final state of the trajectory. Then, for each subtask (e.g., goal=(0.0, 16.0),, we rela- the corresponding undi rected or undi rected dataset according to the goal. Finally, we keep the rewards in the target task dataset (labeled dataset) while setting the rewards in the other task dataset to (unlabeled dataset). We visualize the directed or undire ed dataset in Fig- ure We can find that the distribution shift issue in the directe ed dataset is more severe than the undi i rect ed dataset, which further challenges the unlabeled data sharing algorithms.
directed
cirected anomazee medium diverse 12: gool 22al-1160.0. 22,00
directed wntmaateetnneen mediumdddiiiid gealo
+
+
30
30
undricttoo art att.aaoocdddnneeeeeeeeee.. qual-16.0 16.01
undimettes g1a.-1160,
=120.
+2
10
38
Figure 3: Visualization of multi-task antraae-needumm-diverr--- datasets. The purple dots denote the transition of reward 0. The yellow dots denote the transition near the goal of the sub-task and the reward is +1.
D.2 MULTI-TASK WORLD
We consider the same setup as in CDS Yu et al. 202 and four tasks: door open, door close, drawer open and drawer close, which is shown in Figure 4 We use medium-replay datasets with 152K transitions for task door open and drawer close and use 10 expert trajectories for task door close and drawer open.
(a) Door Open
(b) Drawer Close
Figure 4: Visualization of subtasks in Meta- World.
22

---

Published as a conference paper at ICLR 2023
E MORE DISCUSSION ON THE RELATIVE PERFORMANCE OF UDS
Based on the discussion in the main text, we can summarize the factors that affect the relative performance of PDS algorithms follows.
scenarios
large number of unlabeled data
high quality of unlabeled data
long horizon
large dimension
finite-sample
term
asymptotic
term
Table 4: Scenarios where PDS has better relative performance.
However, it is worth noting that the asymptotic term is dependent on the backbone offline algorithm we choose. For model-based algorithms like [eehara & Sun (202T), the performance bound seale ((d) and thus the problem's 's dimension does not affect the asymptotic term. The dependence over the discount factor is also dependent on the choice of the backbone algorithm. However, is known that the lower bound of offline RL algorithms in linear MDPs scales as O((1 >)1..)) this asymptotic term must scales at least as O(11 7).5)) and thus negatively depends on the discount factor. That is, the larger the discount factor, the better the relative performance of PDS.
F DISCUSSION ON THE PERFORMANCE BOUND
F.1 CONSTRUCTION OF ADVERSARIAL EXAMPLES
In this section, show that there is MDP and an "unfortunate" dataset such that the subopti- mality of Algorithm 3 matches the performance bound in Theorem 4.3. We first focus on the case without data sharing. The same techniques only need to show that all inequalities Theorem 4.3 can become equality. Suppose we have an MDP with one state and N actions. d of them are optimal actions, with feature map
be used to match the reward-learning bound. We
((s,a;) = -,0,.. Vd.
and let the optimal policy be uniform over d actions. Such construction makes Σxx,s / so that inequalities in Equation (25)~(27) become equalities. Then we let samples from other actions match the confidence upper bound while samples from the optimal action match the confidence lower bound, and we also need all the samples to be symmetric over different dimensions (this is required inequality used in the proof), such that the confidence bound inequalities in Lemma C.3 also become equalities. Then, in this case, the suboptimality bound is matched and the bound in Theorem 4.3 is tight.
by the
F.2 REWARD BIAS OF UDS
In this section, we show that UDS suffers from constant reward bias.
Lemma F.I. By labeling all rewards in the unlabeled dataset to zeros, we have
N1 Ep, [|r(s,))]]]
ED. [r((s, a) -uups(s, No + N,
That is, the reward bias does not vanish as long as the ratio of labeled data size and unlabeled data size keeps constant.
23

---

Published as a conference paper at ICLR 2023
Proof.
Fpo+D,1lrss,0) -fuDs((,))) = No
Ni No + Ni
ED. a) r((s,a)]++ +
ED, - 01]
No + Ni
Ni
No Ni ED, (s,,
G How BAD IS THE REWARD PREDICTION BASELINE ACTUALLY?
We conduct ablation studies for the reward prediction method from three aspects, including model size. ensemble number, and early stopping, which are shown in Table 5 Table 6 and Table For the model size, 256*3 denotes the 256 hidden neurons with three hidden layers. As for the early stopping, Epoch Number 3 denotes traversing the entire training dataset 3 times. (We find the Epoch Number=3 enough to reduce the prediction error to small range, and increasing epochs leads to overfitting))
We conduct experiments setting where the quality of the reward labeled and reward-free data dif- fers significantly. For example, ) data from expert datasets and 0.1M reward-free data from random datasets. We set the PDS as the default parameter in all comparisons, including the model size 256*2, the training epoch 3, and the ensemble number 10. All experimental results adopt the normalized score metric averaged over five seeds with standard deviation.
IM) denotes 50K reward-labeled
The experimental results show that fine-tuning reward prediction can improve its performance in some cases, Nevertheless, well-tuned reward prediction method still performs poorly compared to PDS in general. We hypothesize that this is because the "test" (reward-free) dataset may have large distributional shift from the "training" (reward-labeled) dataset, violating the i.i.d. assumption in supervised learning.
Model Size (Reward Prediction)
256*3
512*3
256*4
512*4
PDS
wallkerdd-expprt((0K))mommoooo walleerzd-aanoomm((KKe-eepptttt 1M)
1.8=10.3
11.2#2.0
1.2::0.2
13.3
77.611..1
95.1+-1.9
96.5++1.6
99.4+-2.0
97.2++2.2
105.1+1.2
27.8+11..7
31.6=15.2
36.6+14.8
36.3+11.8
6..5
hopperraadomm(OOKKeeeeettOO....
72.9+19.77
78.9-16.7
92.1+15.7
92.3
93.8
Table 5: Ablation for the model size. We adopt various model sizes for the prediction network in the Reward Prediction baseline, while PDS adopts the default parameter model size 256*3.
Epoch Number (Reward Prediction)
2
PDS
valleee2dexxperr(30K)yrrddoonno MM)
2.94
39.6-14.1
1.8+H0.7
77.6
91.2+11.
101.2+1.99
95.1
105.1=11.2
lopperccxper((00KKrrmmddomm....
35.919.8
31.0+1..2
27.8
61.5-16.2
42.4+11.8
57.8+12.2
84.8+10.5
93.8
Table 6: Ablation for the early stopping. We adopt various Training Epochs for the prediction network in the Reward Prediction bascline, while PDS adopts the default parameter epoch number 3.
24

---

Published as a conference paper at ICLR 2023
Environment
Reward Prediction (Ensemble=10)
PDS
walkerzddeepprr(55k)aaadodddd
18.9+3.6
77.6++8.1
91.4
105.112.1
Hopper-xppet(000K>rradommm... IM)
39.4415,7
6..5:16.2
(bpperrraddomm@kkKxxeeerrr....
78.4
93.8-L4.9
Table 7: Ablation for the ensemble. PDS adopts the default parameter ensemble number 10 and we adopt the same ensemble number for the prediction network in the Reward Prediction baseline.
25


Reviews:
[{"review_id": "ohofRKlCc1o", "reviewer": "Reviewer_ZyTC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an offline RL method for the setting where the agent is additionally provided a dataset of reward-free transitions. Different from prior work, this method imputes the rewards for those transitions using a lower confidence bound. Empirically, the proposed method outperforms a prior method that uses 0 for these reward-free transitions. The paper provides theoretical results showing that the suboptimality of the learned policy can be decreased by using these reward-free transitions.", "summary_of_the_review": "Overall, I think the proposed method seems quite promising -- a really simple way for making use of reward-free data. I appreciate that the paper contains theoretical results, too, to supplement the empirical results. I have a few concerns/questions noted above. If these are addressed, I will vote for accepting the paper.", "strength_and_weaknesses": "Strengths\n* The proposed method is very simple\n* Empirically, the proposed method achieves great results\n\nWeaknesses\n* I'm not sure that the main theorem actually proves the claim in the title (that unsupervised data sharing is provably useful). The issue is that Theorem 4.3 only provides a _bound_ on the suboptimality. The bound decreases when we make use of unlabeled data.\nHowever, if the bound isn't tight, then decreasing the bound doesn't necessarily mean that the unlabeled data is useful. As a toy example, the bound $\\frac{2 r_\\text{max}}{1 - \\gamma} + N_0 + N_1$ is also a valid (and vacuous) bound on the suboptimality. This bound _decreases_ if we reduce the amount of labeled and unlabeled data, suggesting that we should throw away all the data. \n* I'm unsure whether the analysis about the suboptimality ratio in Eq 13/14 is solid, for the same reason as above. If this bound isn't tight, then this ratio might be meaningless.\n* The clarity of the writing could be improved (see details below)\n* I'm unsure how bad the reward prediction baseline actually is, _if it is very well tuned_. For example, it might be important to use high-a capacity prediction network; it might be important to use the same ensemble as the proposed method; it might be important to use early stopping to prevent overfitting. I think that showing that _even a very tuned_ reward prediction method can fail would strengthen this paper.\n\n\nAdditional comments\n*  Title -- I'm not sure \"Provable\" is being used correctly. Typically it refers to a claim that could be proven; saying \"this provable theorem\" is grammatically correct, if redundant. I'd recommend changing the title to something like \"The Provable Benefits of...\" or \"Theoretical Gaurantees for ....\" or \"A Theoretical Justification for ...\" [Though, see comment above about the correctness of these claims.]\n* \"obviating\" -> \"reducing\" -- Presumably, the method still needs some annotations.\n* \"merit\" -- Which merit? Perhaps rewrite as \"Like self-supervised methods in other areas of ML, we would like self-supervised RL methods that can ...\"\n* \"in a supervised manner\" (in abstract and intro) -- I found this statement a bit strange, as it it were arguing that offline RL were equivalent to supervised learning. Perhaps behavioral cloning methods can argue this, but I don't think most offline RL methods are \"supervised\" in this way.\n* \"is highly preferred\" -- Preferred to what alternative?\n* \"can be costly and requires huge human efforts\" -- Cite.\n* \"CV, NLP\" -- No need to define acronyms if they aren't used again\n* \"which is even worse than labeling the unsupervised dataset with all zeros\" -- I might be misremembering the UDS paper, but this seems like a misrepresentation of this prior work. I'd recommend a more generous statement (e.g., \"Prior work has shown the learning to predict the rewards can be challenging, and that the simple approach of setting the reward to zero can achieve good results.\")\n* \"How can we leverage unlabeled data\" -- Perhaps clarify that \"unlabeled\" means reward-free, not action-free.\n* \"reduced to some sort of linear bandits\" -- I didn't understand this during the first pass; perhaps add more context, or wait to introduce this to the theoretical section.\n* \"as shown in Algorithm 3\" -- Broken reference? Algorithm 3 doesn't appear in the main text.\n* \"The coverage coefficient represents the maximum ratio ...\" -- How does this relate to the ratio of the state occupancy measures, $\\rho(s, a)$?\n* \"How to leverage\" -> \"How can we leverage\"\n* \"we have the following lemma\" -- I would highly recommend adding a few sentences describing what the result will say and providing some intuition, before stating the formal result.\n* \"Lemma 4.1\" -> \"Lemma 4.1 [Abbasi-Yadkori]\" -- Make clear that this is not a contribution of this paper\n* Eq. 8 -- Where is $\\hat{\\theta}$ defined?\n* \"model-based approach with different amounts of pessimism\" -- This \"strawman\" approach is described on page 4 and in the introduction. I found it a bit odd because it was unclear whether this was referring to a specific prior method. I'd probably just cut this, and direct the comparison to be with UDS/CDS.\n* \"summarized in Algorithm 1\" -- Add a paragraph introducing the method first. Something like \"We now introduce our method. We do ... We summarize our approach in Algorithm 1.\"\n* \"set as in Equatoin 11\" -- Broken reference? Eq 11 hasn't appeared yet.\n* \"bi-level optimization problem\" -- I'd recommend writing out what this problem is. As written, Eq 11 doesn't look like a bilevel optimization problem.\n* Eq. 9 -- Why do we need to clip the reward $\\hat{r}$ to be positive? Is this motivated by the theory?\n* Lemma 4.2 -- I'd recommend writing out the result in words, before stating the math.\n* Eq. 11 -- Where is $\\hat{V}$ defined? (I.e., is it the solution to some other optimization problem/procedure?)\n* Theorem 4.3 -- Does this make the linear MDP assumption? If so, I'd recommend mentioning it.\n* Theorem 4.3 -- Does this theorem also apply to simple (non-pessimistic) reward prediction methods?\n* Theorem 4.3 -- When is this bound non-vacuous? E.g., if $\\sqrt{d^2 \\zeta_2 / (N_0 C_0)} > 1$, then this is vacuous; so, when will this square root be much less than 1?\n* \"This means that ...\" -- Great intuition!\n* Eq 15, Eq 16 -- I was confused which method was used in the analysis, and which was used for the main experiments.\n* For all experiments, I'd recommend increasing the number of random seeds to 5.\n* Table 2 -- It's a bit non-standard to show separate performance numbers for different goals. I'd recommend just showing the standard evaluation numbers, so that they are comparable with the numbers reported in prior work.\n* Fig -- Increase figure resolution. What is the \"Learn\" line?\n* Fig 2 -- I was a bit confused what to learn from this plot; at face value, it seems to indicate that UDS is the best\n* \"statistics(exact\" -- Missing space in one of the citations.\n* \"Training agent for first-person\" -- Missing journal/identifier in this citation.\n* Table 4 -- This table seems to indicate that CDS+UDS is the strongest baseline. I'd highly recommend including the strongest baselines in the main text.\n* Table 4 -- What does the \"Learn\" method correspond to?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "a_FjfOG7wdJ", "reviewer": "Reviewer_dc2J", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes provable data sharing (PDS) for leveraging reward-free data for offline RL. The algorithm amounts to constructing a pessimistic estimate of the reward function, relabeling all available data with those pessimistic rewards, and then running standard offline RL. This is shown to be provably efficient in linear MDPs and extended to the deep RL setting via an ensemble of reward models to estimate the pessimistic reward. Empirical results show improvements over MLE reward models and the UDS algorithm which labels all new data with zero reward.", "summary_of_the_review": "Overall, I really liked the core idea of the paper and think that the presentation is solid. There are two key weaknesses that I raised in my review. The issue with the theory should be easy to resolve. The issue with hyperparameter tuning seems to be somewhat more fundamental and should be addressed, but is not a fatal flaw. I will rate the paper as a weak reject for now, but am happy to improve the score to accept if the authors address the issues I raised.", "strength_and_weaknesses": "### Strengths\n\n1. The paper provides a novel and simple approach to the important problem of leveraging reward-free data for offline RL. In particular, it is novel and useful to show that a simple reward penalty can suffice to incorporate reward-free data into offline RL.\n\n2. The paper provides a theoretical analysis of the algorithm in linear MDPs that shows the provable benefits in a simplified setting.\n\n3. The paper also is able to scale the algorithm up to a deep RL variant that shows strong empirical performance on a substantial variety of tasks.\n\n### Weaknesses \n\n1. The theory is not quite complete. The proof of the main novel contribution bounding the gap between the estimated reward and true reward that yields the asymptotic term in the result is omitted \"for simplicity\" in Appendix B/C. I think that this shouldn't be a serious problem, as intuitively the result seems to hold, but a complete proof ought to be provided. In particular, my understanding is that the bound that is omitted is to bound the difference between the V^{\\pi^*}_{\\theta^*} - V^{\\pi^*}_{\\tilde \\theta} (also there is a typo in the appendix, in the last line of the align in appendix B, it should be $ d^2$ in the first term instead of $d$ to match the main text, but this is also the term where the proof is missing). Intuitively, this can be bounded by unrolling $\\pi^*$ and bounding the difference between true and estimates rewards at each step, yielding some term that depends on $ \\alpha $ times the sum of the feature norms, but this needs to be worked out explicitly. It is also a bit confusing to me why so much space in the appendix is spent proving Lemmas C.1 and C.3, which seem to not be novel to this paper and could perhaps be cited from prior work(let me know if I missed some detail that makes them different). \n\n2. There is an issue with hyperparameter tuning in the experiments. Essentially, the two main baselines are UDS and prediction which correspond to special cases of the proposed PDS algorithm (UDS is PDS with $ k = \\infty$ and prediction is PDS with $k = 0$). This itself is not a problem, as it is nice to have more general algorithms with improved capabilities. The problem is that it seems that the experiments rely on (potentially extensive) tuning of the $ k$ hyperparameter for PDS while the baselines have no such hyperparameter since they correspond to special cases. First, I would like to see a better description of how the hyperparameter was tuned. Appendix G lists the final values that were used, and they seem to be unique for almost every task, suggesting that some tuning method was used, but no explanation is given. Second, I think there ought to be a more straightforward discussion in the presentation of the algorithm that PDS is a generalization/unification of UDS and prediction and that thus it is not surprising that tuning $k$ yields better results than either baseline.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2xmioLURC2E", "reviewer": "Reviewer_5Qve", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work target the setting of offline reinforcement learning with some extra data (without reward) sharing setting. This work proposes a revised reward function to keep the pessimistic property of the algorithm and design the algorithm in linear MDP and general MDP cases. Through different experiments, it shows the performance is better than baselines with some other data-sharing methods.", "summary_of_the_review": "This is a very interesting work that targets data sharing in offline RL. However, the experimental performance of the proposed main technical reward design is not convincing enough to show that it can work better than other baselines since it uses different parameters $k$ in different games. The experiments didn't show that there exists a universal $k$ that works generally better than the baselines.", "strength_and_weaknesses": "Strength:\n1. It proposes a different reward estimation method to keep the pessimistic property of the algorithm and evaluate the proposed method PDS to 3 baselines in many different experimental settings to show the performance.\n2. It has a reasonable theoretical analysis of the intuition of why data-sharing works better than no data-sharing in linear MDP cases.\n\nWeaknesses:\n1. The theoretical results do not support the main contribution of this work. It seems the main contribution of this work is the designing of the estimated reward minus a confident interval and it works better than other data-sharing methods such as letting the reward be 0 or using a learned reward function to label the reward in the extra dataset. So the theoretical results should support that the proposed reward design in this work will have better results than other data-sharing methods. However, the theoretical results only support that with reward designing, it works better than no data-sharing.\n2.  Seeing the main contribution of changing $r(s,a)$ to the reward designing in Equation 15, we know that the proposed reward has an additional parameter $k$ that can be tuned with comparisons to other baselines. But from my understanding, $k$ should keep the same for all the experiments to show that there exists some universal constant that will work in a lot of settings but not need to be tuned a lot for all games. However, it seems different $k$ is used in different games when the proposed methods is conducted to compare with other baselines. This seems not fair. For instance, DQN use the same parameters for all 57 Atari games and works well [1].\n\n[1] Mnih, Volodymyr, et al. \"Human-level control through deep reinforcement learning.\" nature 518.7540 (2015): 529-533. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster