Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization
Authors: Kun LEI
Zhengmao He
Chenhao Lu
Kaizhe Hu
Yang Gao
Huazhe Xu
Venue: ICLR 2024 poster (2024)
Keywords: Offline-to-Online Fine-tuning
On-policy Learning
Robot Learning
Reinforcement Learning
Primary area: reinforcement learning
TL;DR: We seamlessly integrate offline and online learning through an on-policy RL algorithm, attaining SOTA performance in simulated and real-world environments across both phases, all without the need for additional regularization.

Abstract: Combining offline and online reinforcement learning (RL) is crucial for efficient and safe learning. However, previous approaches treat offline and online learning as separate procedures, resulting in redundant designs and limited performance. We ask: *Can we achieve straightforward yet effective offline and online learning without introducing extra conservatism or regularization?* In this study, we propose Uni-O4, which utilizes an on-policy objective for both offline and online learning. Owning to the alignment of objectives in two phases, the RL agent can transfer between offline and online learning seamlessly. This property enhances the flexibility of the learning paradigm, allowing for arbitrary combinations of pretraining, fine-tuning, offline, and online learning. In the offline phase, specifically, Uni-O4 leverages diverse ensemble policies to address the mismatch issues between the estimated behavior policy and the offline dataset. Through a simple offline policy evaluation (OPE) approach, Uni-O4 can achieve multi-step policy improvement safely. We demonstrate that by employing the method above, the fusion of these two paradigms can yield superior offline initialization as well as stable and rapid online fine-tuning capabilities. 
Through real-world robot tasks, we highlight the benefits of this paradigm for rapid deployment in challenging, previously unseen real-world environments. Additionally, through comprehensive evaluations using numerous simulated benchmarks, we substantiate that our method achieves state-of-the-art performance in both offline and offline-to-online fine-tuning learning. [Our website](uni-o4.github.io)

Published as a conference paper at ICLR 2024
UNI-04: UNIFYING ONLINE AND OFFLINE DEEP REINFORCEMENT LEARNING WITH MULTI-STEP ON- POLICY OPTIMIZATION
Kun Lei' Zhengmao Hel4* Chenhao Lu
Kaizhe Hu Yang Gao
Huazhe Xu
Shanghai Qi Zhi Institute. Tsinghua University, IIS. Shanggaai Al Lab. 4 The Hong Kong University of Science and Technology (Guangzhou). kun980 gma com, mrazhe_xu@ma il tsinghua edu. cn
ABSTRACT
Combining offline and online reinforcement learning (RL) is crucial for efficient and safe learning. However, previous approaches treat offline and online learning as separate procedures, resulting in redundant designs and limited performance. We ask: Can we achieve straightforward yer effective offline and online learn- ing without introducing extra conservatism or regularization? In this study, we propose Uni-O4, which utilizes an on-policy objective for both offline and online learning. Owning to the alignment of objectives in two phases, the RL agent transfer between offline and online learning seamlessly. This property enhances the flexibility of the learning paradigm, allowing for arbitrary combinations of pre- training, fine-tuning, offline, and online learning. In the offline phase, specifically, Uni-O4 leverages diverse ensemble policies to address the mismatch issues be- tween the estimated behavior policy and the offline dataset. Through a simple of- fline policy evaluation (OPE) approach, Uni-O4 can achieve multi-step policy im- provement safely. We demonstrate that by employing the method above, the fusion of these two paradigms can yield superior offline initialization well as stable and rapid online fine-tuning capabilities. Through real-world robot tasks, we highlight the benefits of this paradigm for rapid deployment in challenging, previously un- seen real-world environments. Additionally, through comprehensive evaluations using numerous simulated benchmarks, we substantiate that our method achieves state-of-the- toofthh-art art performance in both offline and offline-to-online fine-tuning learn- ing. Our website: htttps://ii kun, github. 10/ /u////nno/
INTRODUCTION
Imagine scenario where reinforcement learning robot needs function and improve itself in the real world, the policy of the robot might go through the pipeline of training online in a simulator, then offline with real-world data, and lastly online in the real world. However, current reinforcement learning algorithms usually focus on specific stages of learning, which sophisticates the effort to train robots with single unified framework.
Online RL algorithms require substantial amount of interaction and exploration to attain strong performance, which is prohibitive many real-world applications. Offline RI., in which agents learn from fixed dataset generated by other behavior policies, is potential solution. However, the policy trained purely by offline RL usually fails to acquire optimality due to limited exploration or poor out-of-disttibution (OOD) value estimation.
natural idea is to combine offline and online RL, which entails a common paradigm of using an offline RL algorithm to warm-start the policy and the value function with subsequent online RL stage to further boost the performance. Although the paradigm of pre- pre-training and fine-tuning widely adopted in other machine learning domains such computer vision (He al.] 2022) and natural language processing (Devlin et al. 2019), its direct application to RL is non-trivial. The reasons can be summarized as follows: firstly, offline RL algorithms require regularization
"Equal contribution.

---

Published as a conference paper at ICLR 2024
(referred to conservatism (Kumar al. 2020) policy constraints (Kostrikov et al. 2021) for RL algorithms to avoid erroneous generalization to OOD data. However, when the policies and value functions trained with these regularizations are used as initialization for online learning, they could lead to fine-tuning instability due to the distribution shift. Previous works have attempted to address
these challenges by employing conservative learning. introducing additional policy constraints (Nair
et al. 2020 Kostrikov et al. [222]] Zheng et al. [022 Wu et a..). Q-ensemble (Lee et al.| IZhao
al. 2023b), or incorporating other value or policy regularization terms (Nakamoto et al. 2023
Zhang
et al. 2023 Li et al. 2023) during the online learning stage. However, these methods may still
suffer from an initial performance drop or asymptotical suboptimality (Wu et al., Nakamoto et al.)
2023
Li et al. 2023) in the online stage due to the conservatism inherited from offline RL training.
MejiCa Lecametion
The objective of offline-to-online RL. al- gorithms is to strike trade-off between fine-tuning stability and asymptotical op- timality. Challenges arise from the inher- ent conservatism during the offline stage and the difficulties associated with off- policy evaluation during the offline-to- online stage. To provide a clearer under- standing, we track the average values of the V and Q functions during fine-tuning with four different methods. As depicted in Figure ((b) SAC (off- foffeppllcyy and CQL (conservatism) are used to fine-tune the policy trained by CQL offline. Off2on (Hong et al.) 2022) is Q ensemble-based fine-tuning method. CQL shows minimal improvement over Q value due to over-conservatism, leading to suboptimal- ity. The Q value of Off2on and SAC improve faster, but both of them face performance drops and unstable training during the initial stage. In contrast, Uni-O4 presents steady and rapid improve- ment in the V values the fine-tuning performance progresses, emerging as a favorable choice for achieving both stable and efficient fine-tuning.
80
8
poliyy Joum
CD. 10)
SAC
SAC
viizee
0fhine dilasct
20
Crvinamment (tSp]
50
100
200
Ondirrment (C513)
50
120
(a) Normalized Return
(b) Average valuc
Figure I: (a) Normalized return curves of online fine- tuning and offline initialized scores on all Mojoco tasks. (b) Average V or Q -values of value functions on Hopper and Walker2d -medium tasks.
Recently, the work on Behavior Proximal Policy Optimization (BPPO) (Zhuang et al. 2023 has made significant advancements by adopting the PPO (Schulman et al.) 2017) objective with advan- tage replacement achieve offline monotonic policy improvement. This has resulted in improved offline performance compared to one-step offlin algorithms (Brandfoobrener et al., 202T). How- ever, BPPO heavily relies on online evaluation to validate the improvement of the behavior policy, introducing risks and additional costs.
At the heart of this paper is an on-policy optimization method that unifies both offline and online training without extra regularization, which we term Uni-O4. Motivated by BPPO, we employ the on-policy PPO objective to optimize both offline and online RL, aligning the objectives in both phases seamlessly. Our approach leverages offline policy evaluation (OPE) stage to evaluate the updated policy and achieve multi-step policy improvement. OPE demonstrates appealing evaluation performance comparable to online evaluation methods. We further address the mismatch problem between the offline dataset and the estimated behavior policy by using ensemble policies and encour- aging policy diversity. In this way, Uni-O4 significantly enhances the offline performance compared to BPPO, without the need for online evaluation.
After offline optimization, standard online PPO learning stage follows leverage the pretrained policy and value function. Such an algorithm enables a seamless transit between offline and online learning while achieving superior performance and efficiency against current approaches in both offline and offline-to-onlin scenarios.
To summarize, main contribution lies in Uni-O4, which demonstrates the remarkable efficiency of the on-policy RL algorithm in offline and offline-to-online learning. Uni-O4 gets rid of the sub- optimality and instability issues in current offline-to-onliee literature. Because of the unified design, it can be scaled up to more complex fine-tuning see-tunnng settings, such as online (simulator) -offline (real- world)-and and online (rrll- which is useful for alleviating the fim-to-real gap in robot learning. We evaluate Uni-O4 across real-world legged robots and numerous D4RL (Fu et al. 2020) tasks. For simulated tasks, experimental results show that Uni-O4 outperforms both SOTA offine and offline-
2

---

Published as a conference paper at ICLR 2024
to-online RL algoritims. Furthermore, Uni-04 excels in real-world experiments, showcasing its ability to fine-tune policies and adapt to challenging unseen environments with minimal interaction, surpassing SOTA sim2real and offtine-to- tifine-to-online methods.
PRELIMINARIES
Online reinforcement learning. We consider the Markov decision process (MDP) M {S, A, p. do: 7}. with state space S, action space A, reward function r(s, a), transition dynam- ics p, initial state distribution (s)))) and discount factor y.
In this paper, we consider on-policy RL algorithms, which typically utilize an estimator of the policy gradient and plug it into a stochastic gradient ascent algorithm. Among these algorithms, PPO (Shhulman et al. 2017) introduces a conservative policy iteration term with importance sampling:
Ju (TT) = (.sppp()) (-|s) [min (r(TT) Am, (s, a), clip (r(T), 1 - e, + Am, (s, (s), (I)
where Pr is the stationary distribution of states under policy n, An, (8t, az) is the advantage function where subscript & denotes the policy iteration number, r(π) = x(a]))
denotes the importance
(a(s))
sampling ratio between the target policy π and behavior policy Tg. clip(-) is conservatism operation
that constrains the ratio, and hyper-parameter is used to adjust the degree of conservatism.
Offiine reinforcement learning. Offline RL focuses on addressing the extrapolation error due to querying the action-value function with regard to the OOD actions (Fujimoto et al.) 2018 Kumar cet al., 2020). Implicit Q-lcarning (IQL) (Kostrikov et al., (222)) revise the SARSA objective as an asymmetric L2 loss to estimate the maximum Q-value over in-distribution data, followed by an advantage-basee policy extraction. Specifically, the losses of the state-value and Q-value functions follows:
L(V)- (D Li (((ssa)-((())
L(Q)= E (... (r(ss a) 7V())) Q(s,
(3)
where Q denotes the target Q function Mnih et al. 2013). and L L5(u) is the expectile loss with
intensity 7 E [0.5, 1): L (u) = --- 1(u < =. Based on dataset support constraints, IQL has the capability to reconstruct the optimal value function (Kostrikov et al. 2021), i.e., lim, Q+ a)
Q*(s,a), In this work, we exploit the desirable property to facilitate multi-step policy optimization and recover the optimal policy. Q- and V, are the optimal solution obtained from Equation 3 and We use Q- and V- denote the value functions obtained through gradient- gradient-based optimization.
Based on one-step policy evaluation, Zhuang et al. (2023) employ an on-policy PPO objective to perform offline monotonic policy improvement:
J& (T)) (Esppp))) (-1s) [min (T(T) Arr (s. a), clip (r(#), e,1++ Ar, (s,
(4)
The only distinction between Equation 4 and is that the state distribution is replaced by PD (+) El P (st = s s)D) and P (8t = sDD) is the probability of the f-th state equaling to s the offline dataset. Due to the advantage of this objective, BPPO is able to surpass the performance of the estimated behavior policy न... However, they update the behavior policy T& by querying online evaluation, which contradicts the purpose of offline RL.
Offline policy evaluation. Off-policy evaluation techniques. such as fitted Q evaluation (FQE) Paine et al.) (020)). weighted importance sampling (WIS) (Voloshin et al.] 2019), and approxi- mate model (AM) (Jiang & Li 2016), are commonly employed for model selection in offline RL. Typically, these methods require partitioning the offline dataset into training and validation sets to evaluate policies trained by offline RL algorithms. The selected hyperparameters are then used for retraining. In this work, we propose straightforward offline policy evaluation method based on AM and one-step Q evaluation without data partition and retraining. Given an offline dataset, we train estimated dynamics model I using the maximum likelihood objective:
(s'{s,,)).
(5)
The estimated transition model A is employed to perform H-step Monte-Carlo rollouts (So, 88,00) an) } for offline policy evaluation.
3

---

Published as a conference paper at ICLR 2024
3 METHOD
In this section, we formally introduce our method, Uni-O4, which offers various training paradigms, including pure offline, affline-to-onling and online-to-ofllie to-ooiii settings. In the offline-to- online setting, our framework comprises three stages: 1) the supervised learning stage, 2) the multi- step policy improvement stage, and 3) the online fine-tuning stage, as illustrated Figure 2
3.1 ENSEMBLE BELAVIIO CLONING WITH DISAGREEMENT-BASEE REGULARIZATION
We aim to avoid extra conservatism and off-policy evaluation during both offline and offline-to- online RL learning. To this end, our entire method is built upon PPO algorithm. We start by re- covering the behavior policy Jy that collects the offline dataset. A straightforward approach used in Zhuang et al. (2023) is optimizing the behavior cloning objective to approximate empirical policy tt. However, this approach leads to a mismatch the state-action support between the estimated itg and Th due to the presence of diverse behavior policies in the dataset D. To address this mismatch issue, we propose an ensemble behavior cloning approach with disagreement regularization, aiming learn diverse behaviors initialization for policy improvement. Specifically, learn set of to recover the behavior policy πT. To encourage diversity among the policies, we jointly train them using the BC loss augmented by the negative disagreement penalty
policies II, { ....... }
.......
between each policy t and the combined policy (((#g)
Proposition Given the dataset and policies II. the distance over #tg(-ss) and f({#)(-1s))) can be expressed DRL.
(1(+1)) ((#26 ((((()())))))))))))()))())))(( where (()) is the normalized coefficient. Z(0)
The average KL divergence over (-1s) and (-1s)())))) can be approximated by sampling (Schul- man et al. 2015a). In the offline setting, we further approximate this distance constraint by sam- pling actions from the dataset 'D instead of from #3- Meanwhile, we choose [(( (((s)())))) =
maxigien & (als) motivated by Ghosh et 2021
maxiess ( (/((s))) 2(a)
Theorem Given the distance KL #b((a())
we can derive the following
maXIGIEn #5 (als)
*g(als)
bound: KL 4) (a|s)]
Sa da (a(ss log maxicien (ale)
2(5)
The proof and the definition of (s) are presented in Appendix A.l diversity among behavior policies by optimizing the lower bound:
Then, we can enhance the
(alss)
Maximize:J(#5)) Blea))=logg] + (s,a log
(6)
maxigiEn (als))
where is papeeepar In the subsequent experimental section, empirically demonstrate that the policy ensemble can mitigate the mismatch issues between the estimated behavior policies and the offline dataset, especially for capturing the multi-modality, thereby substantially enhancing the performance of Uni-O4.
3.2 MULTI-STEP POLICY ENSEMBLE OPTIMIZATION
In this work, we propose a simple offline policy evaluation (OPE) method to achieve multi-step policy improvement, in contrast the frequent online evaluations required in BPPO (Zhuang al. 2023). In this way. Uni-O4 can better follow the setting of offline RL.
For each behavior policy, Uni-O4 performs policy gradient optimization similar Equation 4 but we aim to achieve multi-step policy optimization via querying OPE. The clipped surrogate objective for each policy is given follows. Since we add the iteration number as a subscript for behavior policies, we overload the notion of behavior policies as тे. rather than # i.c., mi := n...
Jk (T*) = Eg-po (a)r)) min (f(=') ) o (s, a). clip (r(*'), 1- 1.1---111 c.1+e) A (8, (s. (7)
4

---

Published as a conference paper at ICLR 2024
Supervised Learning
Offline Multi- Optimization
Online Fine-Tuning
AM-Q
Ny Ug -., Aty
Qr
Vxx
Rallouts
XE F++1
+x
D
Qr & V
A
V
wk+1
Tg++1
Fieed
Update
Replacement
005pl
Optimization
OPE
Figure 2: Uni-O4 employs supervised learning to learn the components for initializing the subse- quent phase. In offline multi-step optimization phase (middle), policies query AM-Q determine whether to replace the behavior policies after a certain number of training steps. For instance, AM-Q allows x² replace its behavior policy with its target policy but rejects the others. Subsequently, one policy is selected as the initialization for online fine-tuning. Specifically, O0Sp indicates out- of-support of dataset.
where superscript i specify the index of ensemble policies, and subscript is the iteration number, and r(*') x' (ala) denotes the probability ratio between the target policy æO and behavior policy (als)
T If set k i.e., fixing the estimated behavior policy by Equation for action sampling, this method will degenerate to one-step RL. However, this will lead to sub-optimal policies because the target policy constrained be close to the estimated behavior policy by the clip function and the performance is heavily dependent on the accuracy of the estimated behavior policy.
Offine policy evaluation for multi-step policy improvement. For achieving multi-step policy im- provement safely, we update the behavior policies by querying OPE. Specifically, our OPE method combines the approximate model (AM) and fitted evaluation (AM-Q).
Definition 1. Given the true transition model I and optimal value function Q', we define the AM-Q as: ( = ((x,))((T.x)
D 11- Q" (st. oi) where H is the horizon length.
In the offline setting, however, the agent does not have access to the true model and Q But Qr approaches Q" based on the dataset support constraints. We fit 7 and Q, by Equation and In this way, the practical AM-Q can be expressed as J+(TT) E (f.))~((.=)) the OPE bias is mainly coming from the transition model approximation.
E#-1 (+(s,, Thus,
Theorem Given the estimated T and Qr. we can derive the following bound: (J(x,T) where we assume Q7 is bounded by Quax-
J(x, T' & Qmax H(H-1),
The proof is presented in Appendix A.2 BPPO (Zhuang al.] 2023) have derived the offline monotonical improvement bound (their Theorem 3) between the target policy and behavior policy, i.e, T and gg. However, since the behavior policy is updated iteratively, this bound can result in cumulative errors that disrupt the desired monotonicity. Consequently, this bound cannot be directly utilized in the offline setting. This limitation is the reason why BPPO requires online evaluation to ensure performance improvement when replacing the behavior policy with the target policy. Given the OPE bound, we can replace the online evaluation with AM-Q guarantee monotonicity.
Consequently, AM-Q becomes a computationally efficient OPE method, as it only requires running all state-action pairs as input, resulting in an inference complexity of O(NH), where N is the trajectory number and no extra fitting requirements. In this way, the behavior policy Tİ is replaced by the target policy n' and the iteration number becomes k 1 OPE satisfies J, (TT) JT O. We will evaluate the accuracy of the AM-Q in the following experimental section.
We compute the advantage function by in online PPO due to the prohibition the interaction with environments. Intuitively, This alterna- rive and clip function can be naturally regarded as conservative terms for offine policy optimization. While multi-step policy optimization performs exploration to discover the optimal policies, the ad- vantage function and clip function constrain the policy improvement in the trust region. After the
rather than the GAE Schulman et al. 2015b used
5

---

Published as a conference paper at ICLR 2024
policy improvement phase, policy is chosen by querying OPE with minimal evaluation budgets in Appendix [A.13
Online PPO fine-tuning, The policy and value function obtained by offline training is directly used as initialization for a standard online PPO algorithm. The overall workflow of our algorithm listed in Algorithm| 2 of Appendix [A.13] There extra conservative regularization or replay buffer balance strategy in the whole offline and offline-to-online training. Benefitted from the preferable properties of the on-policy algorithms, the proposed algorithm very simple and efficient.
RELATED WORK
Offline-to-online RL. PEX (Zhang et al. 2023) introduce policy regularization through expansion and adopt Boltzmann action selection scheme. Lee et al. 2202 , Zhao al. (2023a) and /Zhao et al. (2023b) utilizes an ensemble of pessimistic value functions to address distributional shift, which can be seen as an implicit form of conservatism. [Nkkamoto
(2023) trains an additional value function to alleviate over-conservatism issues arising from the initialized value function in the
offline phase. Niu et (2022) introduces dynamics-aware policy evaluation scheme solve the dynamic gap between the source and target domains. Li et al. (223) propose policy regularization term for trust eegion-style updates. Yu & Zhang (2023) leverages standard actor-critic algorithm
for aligning offline-to-online learning. [Blll et al. (2023) and Zheng et al. (2023) take advantage of offline and online data for online learning. Guo et al. (223) propose an uncertainty-guie
method for efficient exploration during online fine-tuning, Also, the offline pertaining and online
fine-tuning paradigm has been proved helpful the visual RL (Ze et al. 2023a b Lin et al. 2023)
and transformer- based offline RL methods (Zheng et al.
2022
Hu ct al. 2023
EXPERIMENTS
We conduct numerous experiments on both simulators and real-world robots to answer the following questions: 1) Can the proposed algorithm achieve better offline RL performance and subsequently higher fine-tuning efficiency compared to previous SOTA methods? 2) Does the algorithm lead to better asymptotic performance? 3) Can the proposed offline policy evaluation (OPE) method provide an accurate estimation for multi-step policy improvement? 4) Do the ensemble behavior policies provide a more comprehensive state-action support over the offline dataset for policy improvement? 5) Can the algorithm work well some complex tasks of worldw robot learning?
Environment
Cdr.
TD34C
Chessop RI.
IQI.
COMBO
BPPO
ATAC
BC
Chrs
malfcheetal-meddans-e
44.0
48.3
48.4
47.4
54.2
44.0
54.3
42.1
52.642044
hopper medium-v2
58.5
59.3
29.6
66.3
97.2
93.9
102.8
52.8
164.4--066
waller 26-modiun--12
53.6
9.11:114
halfcheetab-maa meceetttddmmm--tepllllle
45.5
55.1
41.0
49.5
hopper-medium medium-replay
95.0
60.
97.3
94.7
89.5
92.5
102.8
25.7
1033-1:08
walker 2d-modiun--rppp
77.2
81.8
49.5
73.9
56.0
77.6
94.1
18.8
98.1
halfchectab--ng mediums--sseee expest
91.6
95.5
933--:13
hopper medium expert
105.4
111.1
112.8
52.6
walker 26-modiun-esper
108.8
110.1
113.0
109.6
103.3
113.1
116.3
107.7
118.1:2222
frcomotion total
698.5
677.4
684.6
692.4
738.3
751.0
818.9
463.5
816.4110.5
pen-human hammer human
37.5
71.5
117.8
79.3
65.8
1.4
2,0
0.2
1.4
9.6*
14.9
6.7
2.6
24.7:14.4
doce-human
9.9
5.24
25.9
8.7
4.3
27.11:1.3
relocate
0.2
pen cloneld
39,2
41.5
60.0
37.3
24.6*
110.8
73.9
60.7
101 19.33
hummer-clonae
2.1
0.8
2.0
2.1
3.34
8.9
2.3
0.4
7.011..9
doce cloned
0.4
0.4
0.27
6.2
8.2
0.9
10.2:1266
0.1
-0.2*
Adrait
93.6
155.2
84.2
297.4
180.2
135.0
kitchenn-oompktte
43.8
3.5*
91.5
2.0
68.3
93.61:255
kitchen-partial
49.8
57.0
32.5
kitchen--isxe mixed
31.0
25.0
28.0
51.0
1,49
62.5
1.0*
47.5
65.0-1-.66
kitchen rotal
111.5
47.5
63.5
159.8
6.1
2110
3.0
148.3
216.9:11077
Tonal
936.7
777.1
905.3
970.3
828.6
1253.r
10021
746.8
1322.0:::1.2
Table 1: Results on D4RL Gym locomotion, Adroit, and Kitchen tasks. We bold the best results and ours is caleulated by averaging mean returns over 10 evaluation trajectories and five random seeds. Most of the results are extracted from the original papers, and * indicates that the results are reproduced by running the provided source code.

---

Published as a conference paper at ICLR 2024
5.1
MAIN RESULTS
Baselines. We compare Uni-O4 with previous SOTA offline and offline-to-onie online algorithms. For offtine RL, we consider iterative methods like CQL. (Kumar et al. 2020). TD3+BC (Fujimoto & Gu
2021 and ATAC (Cheng et al.] (022), onestep methods such as Onestep RI. (Brandfonbrener et al.
2021 and IQI. (Kostrikov et al. 202T), model-based RI. approaches COMBÓ (Yu et al. 2021), and supervised learning methods (Chen et al.] (02T) and (Emmons et al. (021). For offline-to-online, we include direct methods: IQL (Kostrikov et al. 2021), CQL (Kumar et al., 2020), ODT (Zhngg | 2022), and AWAC (Nair et | 2022); regularization methods: PEX (Zhang et al. 2023), ) SPOT (Wu et ) and Cal-qi (Nakamoto et al., 2023); Q-ensemble based methods: OffZon (Lee et al. 2021]; and Scratch: training PPO from scratch. For real-world robot tasks, we consider using IQL and walk these ways (WTW) (Margolis & Agrawal 2023), a strong RL method for quadrupedal locomotion, as baselines for comparison. See Apeendix A.5 for more details about the experiment
et al.
settings and hyperparameter selection.
We first answer whether Uni-O4 can achieve competitive offline performance, which may serve as the initialization of online fine-tuning, We compare Uni-O4 with SOTA offline RL algorithms. As shown in Table [ Uni-O4 outperforms all algorithms on 14 out of the 20 tasks. Notably, Uni-O4 surpasses all one-step algorithms, including one-step RL and IQL, which constrain the policy to stay close to the behavior policy, In contrast, Uni-O4 achieves multi-step policy im- provement in trust region by querying OPE, and benefits from the natural constraints, i.e., clip function and one-step policy evaluation. Additionally, Uni-O4 performs better than most iterative and model-based algorithms. While ATAC achieves comparable performance to Uni- 04 on MuJocCo locomotion tasks, it lags behind on the more challenging Adroit and Kitchen tasks. BPPO demonstrates promising results on Adroit and Kitchen tasks. However, it heav- ily relies on online policy evaluation for policy improvement despite being an offline method.
Table 2: Results of on Antmaze tasks. In BC column, the symbol indicates that we use filter BC to recover the behavior policy. All results are extracted from the original paper except ours.
Env
CQL
TD3+BC
Onestop
IQL
Dr
ReS-R
RvS-G
BC
Ours
Umaze-v2
74.0
78.6
64.
87.5
65.6
64.4
65.4
54.6
93,7-1:322
Umuze diverse v2
84.0
71.4
60.7
62.2
51,2
70.1
50.9
48.2
83.5.1 11.1
Medium-plan play-v2
61.2
10.6
0.3
71,2
1.0
58.1
22.0*
75.2:14.4
Modiun-diverse-42
53.7
3.0
0.0
70.0
0.6
7.7
67.3
13.6
72.2::3.8
Large-play-v2
15.8
0.2
0.0
39.6
0.0
32.4
35.3
64.9-12.5
Large-divere--2
14.9
0.0
0.0
47,5
0.2
3.7
36.9
29.4*
58.7+3300
Totol
.03.6
163.8
61.0
378.0
118.6
153.9
321.0
236.7
447.9.128.1
To further evaluate Uni-O4, we con- ducted additional experiments on Antmaze tasks, which request the al- gorithm to deal with multi-task learn-
Mejota Lovam olier
Nirok
Nomallized Return
60
Normallaed
60
ing and sparse rewards. The results,
8
presented in Table - demonstrate that Uni-O4 outperforms the majority of one-step, iterative, and supervised learning approaches. Overall, Uni- 04 achieves an impressive 79.4% improvement over 26 tasks based on estimated behavior policies. These findings underscore the capability of
20
100
150
200
200
cnvironmeett Steps (x563)
Enviformant Steps (X5433
Figure 3: Aggregated learning curves of various approaches the MuloCo locomotion and Adroit manipulation tasks. It shares legend with Figure 4 for simplicity. Uni-O4 to serve as a satisfactory initialization for online fine-tuning.
Next, we conduct experiments to demonstrate the online fine-tuning performance of Uni-O4. We run 1M environment steps for all methods. The learning curves of normalized returns are presented in Figure [4] As observed, methods that inherit conservatism and policy constraints from the of- fline phase, namely CQL, IQL, and AWAC, exhibit relatively stable behavior but struggle to achieve high performance, only to converge towards the offline performance of Uni-O4 on certain tasks. The policy regularization method (PEX) displays instability and initially experiences a performance drop on most tasks, failing to attain high performance compared to conservative methods. Addi- tionally, Q Q-ensemble- based method (off2on) can be considered an implicitly conservative approach, outperforming other baselines on MuloCo tasks but not performing well the more challenging Adroit tasks. Moreover, these ensemble-based methods entail significant computational overhead
7

---

Published as a conference paper at ICLR 2024
making it unacceptable for real-world robot learning. Overall, Uni-O4 exhibits an integration of stability, consistency, and efficiency, eclipsing all bascline methods with its unified training scheme.
(18 hours vs. 30 minutes (Uni-O4), Figure 25(c) in Appendix A.12
Cal-ol
off2on
SPOT
Offilne
Baeehuman
140
300
L22
233
13
Morrakedd
5 5
Envizeemrt.ooem (4363]
Didigament Staps (+563]
Indrrnmmnn: Stspar
hutmmrettt med myau
8
ExRErSETAnTi Staps (+5
STVEORTTARE Suppa (45631
Envienmurt SSgg 1436d)
The learning curves of various methods on Adroit and MuJoCo locomotion tasks are presented across five different seeds. The solid lines indicate the mean performance, while the shaded regions represent the corresponding standard deviation.
Figure 4:
5.2 APPLICATIONS ON REAL-WORLD ROBOTS
Bridging the sim teal gap is a widely recognized challenge in robot learning, Previous studies (Tobin et al.] 2017: Yang et al.] 2021] Rudin et al.) 2022] Margolis & Agrawal| (203) tackled this issue by employing domain randomization, which involves training the agent in multiple random- ized environments simultaneously. However, this approach comes with computational overhead and poses challenges when applied to real-world environments that are difficult to model in simulators. To address this issue, we propose to leverage Uni-O4 in an online-offline-nline framework. The agent is initially pretrained in simulators (online), followed by fine-tuning on real-world robots (of- fline and online), as illustrated in Figure 5(a)
For more detailed information, see Appendix A.4
In our experiments, initiate with a policy pre-trained online in simulator, proficient in navigat- ing level terrains in the real world. However, when tested on a latex mattress, the policy struggled to generalize effectively complexity arises as the material properties of the mattress pose significant challenges to simulate accurately. To counteract this, we collect offline data prior to the overturning incidents and fine-tune the policy using our method, enabling successful navigation on the mattress (second row in Figure 5(a)). Subsequently, we enhance running speed by performing online fine-tuning of this adapted policy in the real world (third row in Figure 5(a). This methodology can be ubiquitously applied for deploying real- wallworldd solutions in environments tough to simulate, such as sandy or wetlands, show- casing the adaptability and effectiveness of our approach in handling challenging terrains, ultimately proving the superior versatility and efficacy of our method over existing solutions.
elastic terrains, resulting in overturns (first row in Figure 5(a)). The
After being fine-tuned using the collected offline dataset (180,000 environment steps), the robot is able to run on the latex mattress at a low speed. Comparing the results shown in Figure the poli- cies trained by WTW and IQL struggle to adapt to this challenging terrain, resulting in significantly lower average returns as they are unable to move forward smoothly. However, when switching to high-speed scenario, the policy fine-tuned by the offline dataset performs poorly, depicted in Figure [(c] resulting in crashes. Subsequently, after being further fine-tuned online (100,000 envi- ronment steps), the robot achieves speed of 1.62 m/s (as measured by RealSense sensors). Overall, the proposed tne-tuning method can achieve significant performance improvements in both offline
8

---

Published as a conference paper at ICLR 2024
OSNIC
Av Naturn
92-
Avcrage
gaimert
88
Prezain
SiP2heal Outs
Wethoos
Metrods
(a) Online-offline-oniin setting
(b) Low speed testing
(c) High speed testing
Figure 5: Real-world experiments: (a) the workflow of Uni-Q4 (b) Testing all methods with low-
speed commands. The reported results averaged five trials, with each trial having maxi- mum of 1000 time steps. (c) Testing all methods with high-speed commands, see Appendix A.4
and online phases, requiring only minimal interaction. In this way, offine fine-tuning ensures safety for weal-world robots, while online fine-tuning continues to enhance their performance.
5.3 ABLATION STUDY
OPE analysis. To assess the accuracy of the proposed OPE method, AM-Q. we perform experiments on all MuJoCo tasks. In parallel the execution of OPE, conduct the online evaluation and consider it as the ground truth for calculating the accuracy of OPE estimates. Furthermore, we also tested the accuracy within specific margin of error. As depicted in Figure [[a]] the OPE accuracy reaches approximately 80%. When allowing for 20% estimation error, the accuracy approaches 95%. This demonstrates the reliable evaluation of AM-Q for multi-step policy improvement.
analysis. We evaluated the disagreement penalty coefficient a and ensemble size in Figure [6(b) and [6(c) respectively. These two hyper-parametesn were determined through experiments conducted on all MoJoCo tasks. From the results, it is evident that minor disagree- ment penalty yields better results compared to having no penalty or using larger ones as behavior policy initialization. As for the ensemble size, we chose 4 as trade-off between performance and computational efficiency, which resulted only a small increase in training time. These findings demonstrate that the policies trained using Equation 6 offer better support over the offline dataset than a single policy. To further substantiate these phenomenons, we conduct several analyses on these two methods, shown in Figure I
B3
14
15
and of Appendix A.6 The results clearly
support the selection of key hyper-parameteen
2.2
2.0
MujoCo Locomation
0:86
Mommali Raltur g
MojoCo Locomation
Normattons Satumn
100
59
Normalized Scores
100
MojoCo Lecomnttio
0.75
96
80
-00
50
22
Scratch
0.5%
17%
2104
200
600
Porm ttel Bror
Cracient Steps (x500)
Policy Number
Environment Steps (xSe3)
(a) OPE ablation
(b) Alpha ablation
(c) Ensemble ablation
(d) Optimality analysis
Figure 6: Ablation study on on all MoJoCo tasks, refer to Figures 26
27
K
and 29 in Appendix
for all results. For the ablation study of key design choices of Um 04, see Appendix A.11
Optimality analysis. As an on-policy training baseline, we running PPO for three million steps in the MoJoCo Locomotion environments. As depicted in Figure o(d) once surpassing the offline performance, the improvement plateaus. In contrast, our fine-tuning method demonstrates rapid performance improvement, demonstrating its stable and efficient fine-tuning capabilities.
CONCLUSION
We introduce Uni-O4, which employs an on-policy algorithm to facilitate seamless transition between offline and online learning. In the offline learning stage, policy ensemble enjoys multi-step policy improvement by querying a proposed sample offline policy evaluation procedure. Leveraging the superior offline initialization, standard online policy gradient algorithm continuously enhances performance monotonically. The advantageous property of the on on-policy algorithm allows Uni-O4 scale effectively various offline and online fine-tuning scenarios across different tasks, making a promising candidate for various offline-to-online real robot tasks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We highly appreciate Jiacheng You for his valuable suggestions and discussions on Theorems and 2, and Zifeng Zhuang for the helpful discussions during the rebuttal round.
REFERENCES
PhilipJ. Ball, Laura Smith, llya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. Feb 2023.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruma. Offline without off- policy evaluation. Advances in neural information processing systems, 34:4933-4946 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 10111588445999 2021.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pp. 3852- 3878. PMLR, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In Proceedings the 2019 Conference of the North, Jan 2019. doi: 1118653/1/n119111
TRLhttp::// dx doi www.::o..........////1
v1/n19 1423
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:21 10751, 2021.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study ppo and trpo. In International conference on learning representations, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072/9, 2020. 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20144 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. arXiv: Learning,arXiv: Learning, Dec 2018.
Divyansh Garg, Jocy Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.02322. 2023.
Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P Adams, and Sergey Levine. Why generalization in rl is difficult: Epistemic pomdps and implicit partial observability. Advances in Neural Information Processing Systems, 34:25502-25515, 2021.
Siyuan Guo, Yanchao Sun, Jifeng Hu, Sili Huang, Hechang Chen, Haiyin Piao, Lichao Sun, and Yi Chang. A simple unified uncertainty-guidee framework for offline-to-onlin reinforcement learning. arXiv preprint arXiv:2.06.00 075441 2023.
Nico Gürtler, Sebastian Blaes, Pavel Kolev, Felix Widmaier, Manuel Wüthrich, Stefan Bauer, Bern- hard Schölkopf, and Georg Martius. Benchmarking offline reinforcement learning on real-robot hardware. arXiv preprint arXiv:2307.15690 2023.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, pp. 1861-1870. PMLR, 2018.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
10

---

Published as a conference paper at ICLR 2024
Joey Hong. Aviral Kumar, and Sergey Levine. Confidence-condittoned value functions for offline reinforcement learning. Dec 2022.
Zhang-War Wei Hong, Aviral Kumar, Sathwik Karnik, Abhishek Bhandwaldar, Akash Srivastava, Joni Pajarinen, Romain Laroche, Abhishek Gupta, and Pulkit Agrawal. Beyond uniform sampling: Offline reinforcement learning with imbalanced datasets, arXiv preprint arXiv:2310.04411 2023.
Kaizhe Hu, Ray Chen Zheng, Yang Gao, and Huazhe Xu. Decision transformer under random frame dropping. ArXiv, htss/3033.0339)) 2023.
Nan Jiang and Lihong Li. Doubly robust off policc value evaluation for reinforcement learning. In International Conference on Machine Learning, pp. 552-6612 PMLR, 2016.
Iyy Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint 1XXi::1100006665 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 13:1179-1191, 2020.
SeungHyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. reinforcement learning via balanced replay and pessimistic q-ensemble. 5th Annual Conference on Robot Learning,5th Annual Conference on Robot Learning, Jun 2021.
Jianxiong Li, Xiao Hu, Haoran Xu, Jingiin Liu, Xianyuan Zhan, and Ya-Qin Zhang. Proto: Iterative policy regularized dffline-to-onling reinforcement learning. arXiv preprint arXiv:2305.15669 2023.
Xingyu Lin, John So, Sashwat Mahalingam, Fangehen Liu, and Picter Abbeel. SpawnNet: Learning Generalizable Visuomotor Skills from Pre-trained Networks. arXiv e-prints, 2023.
Gabriel B Margolis and Pulkit Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In Conference on Robot Learning, pp. 22-31. PMLR, 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 13X2.533/ 2013.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint (20v::2006.09355) 2020.
Ashvin Nair, Brian Zhu, Gokul Narayanan, Eugen Solowjow, and Sergey Levine. Learning on the job: self-rewarding offline-to-online finetuning for industrial insertion of novel connectors from vision. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7154- 7161. IEEE, 2023.
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine- tuning, arXiv preprint arX4v.2303.05479 2023.
Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming HU, Xianyuan Zhan, et al. When trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning. Advances in Neural Information Processing Systems, 35:36599-36611 36612, 2022.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning, arXiv preprint arXiv:2007.0905: 2020.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pp. 91-100. PMLR, 2022.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015a.
11

---

Published as a conference paper at ICLR 2024
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint 05t1::1506002338, 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:170..06347, 2017.
Laura Smith, J Chase Kew, Xue Bin Peng, Sehoon Ha, Jie Tan, and Sergey Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. In 2022 International Conference on Robotics and Automation (ICRA), pp. 1593-1599. IEEE, 2022a.
Laura Smith, Ilya Kostrikov, and Sergey Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208. 07860, 2022b.
Laura Smith, Yunhao Cao, and Sergey Levine. Grow your limits: Continuous improvement with real-world rl for robotic locomotion. arXiv preprint arXiv:2310.176.. 2023.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-orienter deep offline reinforcement learning library. In 3rd Offline RL Work- shop: Offline RL as a "Launchpad", 2022. URL http::://oppereeieeeeeeee new. forum?in
SyAS49bBcv
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Cameron Voloshin, Hoang M Le, Nan Jiang, and Yisong Yue. Empirical study of off-policy policy evaluation for reinforcement learning. arXiv preprint arXiv: 19t:::991.068... 2019.
Jianren Wang. Sudeep Dasari, Mohan Kumar Srirama, Shubham Tulsiani, and Abhinav Gupta. Ma- nipulate by seeing: Creating manipulation controllers from pre-tedined representations. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3859-3868, 2023.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou, Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint 12Xiv:22088.6/990 2022.
Jialong Wu, Haixu Wa, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy opti- mization for offline reinforcement learning.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang. Victor Wai Kin Chan, and Xi- anyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303. 15810, 2023.
Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Ro- bust offline reinforcement learning via conservative smoothing. Advances in Neural Information Processing Systems, 35:23851-23866, 2022.
Ruihan Yang, Minghao Zhang, Nicklas Hansen, Huazhe Xu, and Xiaolong Wang. Learning vision-guided quadrupedal locomotion end-to-end with cross-modal transformers. arXiv preprint arXiv:2107.0399 07.90999.0999 2021.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-2888 229667, 2021.
Zishun Yu and Xinhua Zhang. Actor-critic alignment for offline-to-online reinforcement learning. 2023.
Yanjie Ze, Nicklas Hansen, Yinbo Chen, Mohit Jain, and Xiaolong Wang. Visual reinforcement learning with self-supervised 3d representations. 2023a.
Yanjie Ze, Yuyao Liu, Ruizhe Shi, Jiaxin Qin, Zhecheng Yuan, Jiashun Wang, and Huazhe Xu. H-index: Visual reinforcement learning with hand-informed representations for dexterous manip- ulation. NeurlPS, 2023b.
12

---

Published as a conference paper at ICLR 2024
Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging dffline-to-onling reinforce- ment learning, arXiv preprint arXiv:2302.00935, 2023.
Kai Zhao, Yi Ma, Jinyi Liu, HAO Jianye, Yan Zheng, and Zhaopeng Meng. Improving offline- to-online reinforcement learning with q-ensembles. In ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023a.
Kai Zhao, Yi Ma, Jinyi Liu, Yan Zheng, and Zhaopeng Meng. Ensemble- based offline-to-online reinforcement learning: From pessimistic learning to optimistic exploration. arXiv preprint arXiv:2306.06871, 2023b.
Han Zheng, Xufang Luo, Pengfei Wei, Xuan Song. Dongsheng Li, and Jing Jiang. Adaptive policy learning for offline-to-onee to-oline reinforcement learning. arXiv preprint arXiv:2303.0769; 2023.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In international conference on machine learning. pp. 27042-27059, PMLR, 2022.
Gaoyue Zhou, Liyiming Ke, Siddhartha Srinivasa, Abhinav Gupta, Aravind Rajeswaran, and Vikash Kumar. Real world offline reinforcement learning with realistic data source. In 2023 IEEE Inter- national Conference on Robotics and Automation (ICRA), pp. 7176-7183. IEEE, 2023.
Zifeng Zhuang, Kun Lei, Jinxin Liu, Donglin Wang, and Yilang Guo. Behavior proximal policy optimization. arXiv preprint arXhv:2302.11312, 2023.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 PROOF OF THEOREM 1
Proof. First let:
Z(s) - da max (a(s))
(8)
then we have:
[(s) /. da Σ
a "
Fy (als))
Σ/ / da (als)
Σ 5=1
1 n,
(9)
j=1
5=1 a
and
12 da
((s)
Ty (a|s)
(10)
71
Based on Equation and 10 we can derive the following bound:
KL TT)a|s)| max Ejsn Ty(als)
(a(a
da (als) log
(s)
maX15 (y(a))
2(s)
da (a(ss) log (a(ss)
maxiEján TTy (a(ss) + log Z(s)
da de(lss) log (a(as))
maxic/<n (y + log (a(s)))
da da log maxigjen Ty (a(s))
(11)
and
maxigyen Ty(als) ((s)
< da (a(s)) log (a(ss)
KI.
<=(a|s)]]
max + log
(12)
A.2
PROOF OF THEOREM 2
Proof. We first define AM-Q as:
H Σ
( = E (T.
Q² (st,
(13)
In this work, we use the optimal solution Q, to approximate Q" due to lim (+( = (s, a). However, just can get Q, by gradient- based optimization. And, the agent does not have access to the true model. Thus, the practical AM-Q is:
Ø-1 Σ 1=
[,( = E (f.))~(f.x))
(14)
which can be estimated by N trajectories with horizon H-steps.
Without loss of generality, let (X,o(X) be a measurable space and v oXX) Igo be non- negative measure. This choice allows us to consider any state distribution (X -S) or state-action distribution (X = S x A) denoted by μ. which is a probability measure (X, (XX). Furthermore, μ is absolutely continuous with respect to Vx, meaning that for any E oX),, if (E) 0, then (FF)) 0.
14

---

Published as a conference paper at ICLR 2024
By applying the Radon-Nikodym theorem, conclude that there exists unique (up to almost everywhere equivalence) Lebesgue vx -x-integrable function fp o - R such that:
m()) = / [ulo)dx(0)))EEO((
(15)
Furthermore, let's consider the set Mx consisting of all finite signed measures on (X, a (X)) that absolutely continuous with respect vx. can be easily verified M forms linear space over the field real numbers R.
Using the Radon-Nikodyn theorem once again, for any se Mx, there exists vx Px-integrable function g : (X) - IR such that
(bFF) = g0 (o())de
(16)
Furthermore, the function Do unique up to almost everywhere equivalence. Since Iv is ux- integrable, we can establish that Igel is also ux-integrable. As a result, we can define the norm of t) as the total variation norm.
[+(E) =1= / (x))
| = sup P
Σ EEP
(17)
Here, P C (X) represents arbitrary countable partition of X. Note: for all E (X), (BE) > 0, then guaranteed that (xX) = ||
By definition, for any probability measure 1a, we have |||||| = 1.
policy æ can be defined as linear operator from Ms to MSxA, where specifically p(s, a) (a)s))(ss)
The transition operator can be defined as linear operator from Msx a to Ms, where specifically p(s') = Σ s, aT [6'[s,,, a)),
Since T maps any probability measure to probability measure. For any μ (MSxA: as long as VE' E (S A), ((E)= > 0 and u(S x A) || 1, it follows that VE E a(S), (Tn)(E)
and (S) = |T =
This immediately implies |T||| > 1. To prove ||T|| < 1, let's assume there exists pa MMx p0 0, such that ||T > || without loss of generality, assume || =1
Define |HO| such that for any VE o(S x A), (Hol(E) max(0, /o (E), (P(())
SE (T)ddssa)))))
We have | ||| (sol(5) 1. Hence, (E) 0, E (S), (5) |T|
Assuming |Po| - Po = O, we have (E) > VE E aSS x A) and ||oo|| 1. However, this
contradicts the assumption that ||T > ||
Let's consider μ4
1 ni Mea woll
we have n4 (E) > 0, V E x A) and || =1
Similarly, we can take _ =
Hol T
satisfying μ (F) > 0, 0,40 E as x A) and ||w-||
Therefore, (B) 0, VE E (S). Consequently, (T(ppo F (B) > 0, Hence, we have + (T (F)) & (T (F)) VE E (S))
Since mi is -itttegrabl,,, is also measurable.
-1 We can define F.4 IT (00, +20) E (S)), F_ 9Ty ((-00, E (S()
15

---

Published as a conference paper at ICLR 2024
We have:
|o
19T 1o (x))
{ (tTwoor)dus(e - ffa STrO (c)d(s))
E+
(I+p))E+) (THo)(E-)
< {(THo))(E)) (T' (T)Mo))(E))
=TT U UE) = ( =
(18)
The contradiction arises from the inequality|[Tppu |ol 1 leading to the conclusion that |T] =.
Similarly, we can derive that || Consequently, the nomn of their composition |T] & 1.
Now let's consider a function R: S x A defined on S A. We define the performance metric [ax T) as follows:
( T) Eaao.. 1tt.:/...................00.....
Σ R(st, az)
(19)
Using the definition of expectation, let:
H Σ (TT)'
(20)
Then, have;
[(,,T) = { R(s, a(ssa)dAsssa) a)
(21)
Now let's consider different T1, T2 and their corresponding Λ₁, A2, as well as the corresponding @1. 92 associated with we have:
((TT:)'Tp-(77)) (TT2)'
(1 - TT2)mpl
TTmppl
([I(TT))='1++*........)))))))))) -|
<|/7T11[-14... T | -
t|]m(Timp [2
< +||>|||[T<T T2mpll
T T2
(22)
16

---

Published as a conference paper at ICLR 2024
Therefore, have:
H-1
H
A11 A2ll
(TT)) Tp
(TT2) ) TT
(xT22 1TP
TT
(TT) TT
(xT2) TP
[[(*Ti) - ( (pp
t||Timpp T2Tpll
H(H 1)
1TipP- -T2ppl
(23)
2
Suppose that: [[(s that:|R(s,,0) a)) & Rmax V(s,a) E E S
/ UxX R(s, ((s,a)d)((3,,a-
[ Ti) - JTT T2)1
-SXA R(s,a)d)2(ssa))
| 5xA
((s. R(s.a))((s.0dr((s.a)
SXA
SSya Sxx
[(ss,a)l) (2(s,, a))dr(s,a))
5xX
< Rmax
1 - (as,))-g(ss,a)dae a))s(s) a)
5x1
Rmax ||^ - A221
< Rmax H² [Tipp T2Tppl
(24)
According to Pinsker's inequality, we have: [[]p ToTpll & (2kxs(inmp/(2m)) Therefore, we have: [m( Ti) - [Tx, T2) & Rmax 2
H(H-1) (Tapl
A.3 RELATED WORK TO REAL-WORLL ROBOT LEARNING.
Real-world robot learning presents numerous challenges, including efficiency, safety, and autonomy. One possible approach to pretrain in simulator and then deploy real robots, but this approach hindered by the sim-to-real gap. To tackle these challenges, recent work suggests combining in-simulator pretraining with real-world fine-tuning Smith et al. (2022a) using deep reinforcement learning, or pure real-world learning (Smith et al. 2022b 2023). Additionally, Gürtler et al. (2023) introduces benchmark dataset for manipulation tasks, where real robot data is collected using policy trained in simulators and then used to train a new policy using SOTA offline RL algorithms.
A.4 DETAILED INFORMATION OF REAL-WORLD ROBOTS
Here, we present detailed information about the learning setting Uni-O4 on the real-world robot tasks, showcasing its ability to excel in real-world robot applications. Our approach introduces a novel process of online pretraining in simulators, followed by offline and online fine-tuning on real-world robots. In the offline fine-tuning phase, specifically, we deploy the pretrained policy on real-world robots to collect datasets in more challenging environments. As a result of fine- tuning the offline dataset, the policy becomes capable of ruining low speed in these demanding
17

---

Published as a conference paper at ICLR 2024
Online Pretrein (Simulator)
Offine Fine-ture (Real-worle)
Online Fine-tune (Real-wori))
Updane 7g++1
Mpre
Roff
Update T++1
Tyre
xE
cam Hoff
Rolloutsi
Rollouu (1)
Rollour eil
Dera oth
Figure 7: The workflow of our online-offline-nnine fine-tuning framework.
environments. Subsequently, we proceed with online fine-tuning to achieve further performance improvement. The whole workflow is presented in Figure 7 Overall, offline fine-tuning proves the safety of real-world robot learning, while online learning undergoes policy improvement. This paradigm showcases sample-efficient fine-tuning and safe robot learning.
In detail, the hardware utilized primarily includes the Unitree Gol quadruped, RealSense T265 sensor, and latex mattress. We deploy our algorithm the Unitree Gol quadruped and choose latex mattress as the deployment environment. The RealSense T265 sensor is employed to measure the robot's speed along the x, y, and yaw axes for reward calculation. Subsequently, we provide comprehensive definitions and outline the learning process in the subsequent sections.
A.4.1 STATE AND ACTION SPACES
In this subsection, we give the definition of the policy inputs and action space of both simulated and
quadruped environments in the learning process. The input to the policy is a 5-step history of observations o which includes the gravity vector in the body frame E S0(3), joint state St E R global timing reference f E IR and the timing reference variables of 4 legs the E R The policy outputs the target position of the leg joint a, R and the PD controller is then used to calculate the torque. The relative notion is listed in Table 4
joint 2 (joint position and velocity of each leg joint), last action a--1 R12, the command
A.4.2 REWARD FUNCTION
During pre-training, we use the same reward function as Margolis & Agrawal 2023). In the subse- quent offline policy optimization and online fine-tuning stages, we use the following reward func- tion, shown in Table We calculate the total reward Tpos exp (SppCCegT"neg 'neg) where ros is the sum of positive reward terms and Tneg is the sum of negative reward terms (we use Coeg = 0.02). Each term is summarized in Table 3 and the relative notion is listed in Table 4
Table 3: Reward terms for Offline and offline- toiiiee-tt-onnnne learning.
Term
Es Expression
Weight
xy velocity tracking
exp
Vay
/0xxx
0.02
velocity tracking
exp
(w=
w
/dws
0.01
vellcity
-4 10-1
joint torques
|r
-2 x 10
joint velocitics
4/2
-2 x 10-5
joint accelerations
-5 x 10-9
action smoothing
Jas-1 a.
-2 x 10
A.4.3 OFFLINE AND OFFLINE- TO-LLLINE LEARNING
In this section, we will provide a comprehensive explanation of the learning procedures involved in the suggested approach. As shown in Figure the difference between this framework and Algorithm iss the behavior policies are directly initialized from the simulator pretraining rather than estimated using BC. In this manner, the policy be trained to adapt to the challenging target environments
18

---

Published as a conference paper at ICLR 2024
Table Notation.
Parameter
Detinition
Units
Dimension
Robot
Joint Angles
rad
12
Joint Velocities
rad/s
Joint Accelerations
rad/s"
Joint Toqques
m
12
Vary
velacity
m/s
Angular velocity of mbot z axis.
rad/s
V:
The velocity of robot's z axis,
m/s
command velocity the Command angular velocity of robat The command velocity of the robot's
xy
m/s
rad/s
m/s
Control Policy
Policy Observation Policy Action
58
12
Simulator pretrained:
IQL offline trained:
Sim2Real:
offline fine-tuned:
Figure 8: Deployment on a latex mattress with low speed (1 m/s commands of the aixs) by various methods. The trails from top to bottom are tested by simulator pretrained, IQL offline trained, Sim2Real, and our offline fine-tuned policies, respectively.
19

---

Published as a conference paper at ICLR 2024
Sim2Real:
IQL online fine-tuneds
Ours offline fine-tuned:
Our online fine-tuned:
Figure 9: Deployment on latex mattress with low speed (2 m/s commands of the aix) by various methods. The trails from top to bottom are tested by Sim2Real, IQL online fine-tuned, our offline fine-tuned, and our online fine-tuned policies, respectively.
with minimal real-world data collection. In the offline fine-tuning phase, we also encourage the policy learn diverse behaviors. Thus, we revise Equation 7 with disagreement penalty
Ju (TT) = (7²) + a.E [ (f* (+ss]l f((**(-1s) 1)).
(25)
Z(s)
where the combined policy is the same as Equation 6
For the online simulator pretraining, initially trained a policy using the standard online PPO in IsaacGym, accumulating approximately seven million environment steps (training time is around 10 minutes using environment parallelism in IsaacGym). Notably, training flat ground without employing domain randomization. Subsequently, we deploy this policy on the real-world robot. As depicted in Figure ((b) and the performance on the latex mattress exhibited limitations, rendering it difficult to traverse distances greater than a few meters. Furthermore, the robot's feet were prone to sinking into the mattress, leading to instability and potential falls.
solely conducted
To address this challenge, we implemented offline fine-tuning by gathering approximately 180,000 step data frequency of 50 Hz. Each episode consisted of 1,000 steps, and we subsequently conducted offline policy optimization. The fine-tuned policy was then deployed onto the robot, enabling to navigate the latex mattress with increased freedom low speed of approximately 0.8 m/s. illustrated in Figure g
While the deployed policy, fine-tuned through offline methods, successfully runs this demanding terrain, it faces limitations in tracking higher-speed commands. To overcome this, we proceed with online finettniigg After 100,000 step interactions, the robot achieved the ability to traverse the mattress at higher speed (1.6 m/s), shown in Figure 9 In contrast, both the WTW and IQL methods exhibit limitations in adapting to the challenging environment at low or high speeds. See uni -04 gi thub. io for full videos.
20

---

Published as a conference paper at ICLR 2024
A.4.4 PSEUDO-CODE UNI-O4 FOR ONLINE-OFFLINE ONLINE SETTING
In this section, we present the pseudo-code of the nalin--ofline-to-nnlie fine-tuning on real-world robots, outlined in Algorithm 27
Algorithm On-policy policy optimization: Uni-O4
# Online training stage;
1: Initialize the policy and V-function;
for iteration j 1, ..... do
3: Run policy in the target environment for T timesteps
4: Compute advantage by GAE (Schulman et al. 2015b);
5: Update the policy by objective I
and value by MSE loss for multiple epochs:
end
Supervised learning stage:
7: Initialize the policy ensemble (en from online stage (set m3 = m));
8: Calculate value Q- and V/- by
9: Estimate the transition model T by
# Offline policy optimization stage:
10: for iteration = ... do
11:
Approximate advantage by Q V.
12:
Update each policy x² by maximizing objective then Perform OPE for each policy by AM-Q:
13:
if j%h
14:
15:
for policy id i 1...., m do
16:
if Jr > (TL. then
17:
Set m x'
19: end for
20: end if
21: end for
# Online fine-tuning stage:
22: Initialize the policy
V-function from offline stage;
23:
for iteration j 1, do
24:
Run policy in the target environment for 7' timesteps
25:
Compute advantage by GAE (Schulman al. 2015b):
26: Update the policy by objective and value by MSE for multiple epochs:
27:
end for
A.4.5 REAL-WORLL ROBOT BASELINE IMPLEMENTATION
We consider two baselines in the real-world robot fine-tuning setting. The first baseline is the sim- to-real work called "walk-theseewee wwy" [13]. This method focuses quadruped locomotion and demonstrates the ability to be deployed across various terrains such as grassland, slopes, and stairs, without the need for additional training in each specific environment. This remarkable generalization capability is achieved through extensive randomization of environments and a substantial amount of training data, totaling approximately billion environment steps. However, it should be noted that this method is highly data-inefficient and encounters challenges when attempting to model complex or challenging deployment environments accurately. Thus, include this baseline in our compar- ison to highlight the significance of real-world fine-tuning and emphasize the sample efficiency of our -off-on--oolineee paradigm. For WTW, we directly deploy the open- policy trained with environment steps in the simulator.
The second baseline we consider is IQL,, which is an offline- offliie-to-online method. We emphasize that IQL is regarded strong baseline for real-world robot fine-tining tasks (Zhou et al. (023). Several studies (Zhou et al. 20023 Wang et al., 2023 Gürtler et al. 2023 Nair et al. 2023) have utilized IQL for fine-tuning real-world robots in offline pretraining and online fine-tuning paradigm. In this work, we follow the offline-tur -oninee----nlie paradigm of IQL as a baseline, aligning with these previous begin by training IQL offline using collected dataset for
studies. For the IQL implementation, million steps. We save multiple checkpoints during this training phase for evaluation purposes. Hyper-parameter tuning is performed during the offline training process. Specifically, set = 0.7
21

---

Published as a conference paper at ICLR 2024
for value function training and 5 for policy extraction. Once the offline training is complete, we evaluate the checkpoints and select the one that exhibits the best performance for online fine- tuning, using the same set of syper-parameters, To ensure faimess, fine-tune the policy for an equal number environment steps with Uni-O4. Following the online fine-tuning, we deploy the fine-tuned policy for comparison.
A.5 EXPERIMENTAL SETUP DETAILS
A.5.1 ENVIRONMENT SETTINGS
Figure 10: The suite of tasks examined in this work is illustrated successively: MoJoCo Locomotion, Adroit, Kitchen, Antmaze, simulated and real-world quadruped robots.
This study involves the evaluation of Uni-O4 -0- using both simulated and real-world robot tasks, exam- ining its performance in both offline and offline-to-onliin fine-tuning scenarios. The visualization of these domains can be seen in Figure 10 For the simulated tasks, we utilized publicly available datasets from the D4RL benchmark suite (Fu al. 2020
Sim2Real quadruped robots. We employ quadruped robots to assess the effectiveness of the pro- posed fine-tuning framework, which comprises three primary stages: online pretraining (simulator), offline fine-tuning (real-world), and online fine-tuning (real-world). A comprehensive description of the experimental setup and training specifics can be found in Section
MoJoCo Locomotion Gym. We focus on three locomotion tasks using the MuJoCo physics simu- lator: HalfCheetah, Walker2d, and Hopper. The objective of each task is to achieve maximum for- ward movement while minimizing control costs. We consider three types of datasets. The medium datasets include rollouts from medium-level policies. The medium-replar datasets comprise all samples collected during the training of medium-level agent from scratch. Lastly, the medium- expert datasets consist of rollouts from both medium-level and expert-level policies, with an equal distribution from each.
Adroit. The dexterous manipulation tasks are highly challenging with sparse reward signals. The offline data used for these tasks is multi-modal, consisting of small set of human demonstrations and larger set of trajectories generated by a behavior-cloned policy trained on the human data. We the rigorous evaluation criteria in Kostrikov et al. (2021) to evaluate all methods. This evaluation criteria focuses on completion speed rather than success rate. This means that the efficiency and speed at which the tasks completed are prioritized over the mere achievement of the final goal.
22

---

Published as a conference paper at ICLR 2024
Antmaze. In these Antmaze navigation tasks, the reward is represented by binary variable that indicates whether the agent has successfully reached the goal or not. Once the agent reaches the goal, the episode terminates. To evaluate the performance, defined by Fu et al. 2020). Specifically, we conducted 50 trials to assess the agent's performance.
the normalized return, which is
Kitchen. This environment includes various common household items such microwave, kettle, overhead light, cabinets, and an oven. The primary objective of each task within this domain is to interact with these items achieve desired goal configuration. This domain serves benchmark for assessing the impact of multitask behavior in realistic non-navigation environment. Baseline
Table 5: Values of hyperparameters
Hyperparameters
Values
network
1024-1024
network
256-256-256
512-256-/22 for quadruped robots
Policy network
256-256-256 for others
200-200-200-200 for MuJoCo Locomotion tasks
Transition model network
400-400-400-400 for others
1 x 10 for MuJoCo Locomotion and quadruped robot tasks
Offiine policy improvement learning rale
1 x 10 for Adroit, Antmaze, and Kitchen tasks
Offiine clip ratio
0.25
Online clip ratio
0.1
3 x 10-5 for MuJoCo Locomotion and quadruped robot tasks
Online learning rate
8 x 10 for Adroit tasks
Gamma
0.99
Online Lamda
0.95
1000 steps for MoJoCo Locomotion
150 steps for Antmaze
Rollout steps H for OPE
steps Kitchen
20 steps for Quadruped rohots
implementation details. For CQL (Kumar al. 2020 IQL (Kostrkovv al. 2021 AWAC et al. 2020), Cal-qlooakamoto et al. mentation provided by Tarasov al. 2022) with default hyperparameters. For Off2on(Les
2023), and SAC (Haarnoja et al. 2018 we use the imple-
al.)
2021], COMBO(Yu Yu et al. (2021), ATACICheng et al.) [002)), and PFX et al. 2023). ) we use the authors' implementation with official hyperparameters.
A.5.2 HYPERPARAMETER DETAILS
Here, we provide the detailed hyperparameters used offline and offline-to-online fine-tuning phases, repetitively. We use Adam as an optimizer. For the network architectures of Q, V, म. and T listed in Table
Offline phase. As described in Algorithm 2 our offline learning algorithm contains two main phases: the supervised learning stage; 2) the multi-step policy improvement stage. In supervised stage, we train behavior policy for 4 x 10 gradient steps using learning rate 10-4, train Q and V networks for 2 1 gradient steps using learning rate 10- 4, Specifically,
update and V in the manner of IQL, thus we use the same value of coefficient in Kostrikov et al. (202T). i.e., T = 0.9 and haffcheetah-meddume 0.5 is best for these tasks. We train the dynamic model for
for Antmaze tasks and 0.7 for others, except for paapper-mdium--xppei expert. Because we found that x 10 gradient steps with learning rate 10-4, In the policy improvement stage, we conduct 10,000 gradient steps for each policy, in which OPE is queried per 100 steps. Learning rates listed in 5
Online phase. The policy and V value function are initialized from offline phases. Then, we update the policy and value for x 10 environment steps for the simulated tasks, and x 10 environment steps for real-world robot tasks. The values of key hyperparameters real-world robot tasks, the hyperparameters of online simulator-based pretraining are followed by Margolis & Agrawal 2023
listed in Table 5] For
23

---

Published as a conference paper at ICLR 2024
Low-return action
High-return action
Action Bkollhood
Action kollhood
(a) Standard BC
(b) Ensemble BC
Figure 11: Motivation example of the ensemble behavior policies. The gray line represents the action distribution of the dataset (TD) which demonstrates multi-modality. The main mode is primarily composed of low-return actions, represented by the yellow dots. Conversely, the subdominant mode consists of lew-iensity but high-return actions, denoted by the green dots. (a) Standard behavior cloning (BC) is susceptible to imitating the high-density but low-return actions, resulting in bias towards fitting the main mode. (b) Ensemble BC approach learns diverse behavior policies that are more likely to cover all modes present in the dataset.
A.6 POLICY ENSEMBLE ANALYSIS
In Section 5.3 we conducted analysis the number of ensembles and the penalty hyperparam- eter on MoJoCo tasks. The results demonstrated that our proposed policy ensemble-based method outperforms its counterparts in terms of offline performance. To further explore the reason why diverse policies are helpful to explore higher performance policies. We provide a simple motivation example in Figure [T to give clearer view. There is presence of multi-modality within diverse dataset. In such a scenario, standard behavior cloning (BC) is susceptible to imitating the high- density but low-return actions, resulting the offline multi-step optimization stage, the policy optimization is constrained by the clip function, making it difficult for the policy to escape this mode. Consequently, this can lead sub-optimal policy as it becomes unable to explore the high-return action region.
bias towards fitting the main mode. However, during
In contrast, our ensemble BC approach learns diverse behavior policies that are more likely to cover all modes present in the dataset. This facilitates exploration of the high-return region, enabling the discovery of the optimal policy. To validate our point of view, we conduct experiments to answer the following questions:
Does ensemble policies provide better support over the offline dataset? We visualized the re- lationship between state-action pairs obtained from single policy and the offline dataset, well as between those obtained from the combined policy ensemble used in Section 3.1 and the offline dataset. As depicted in Figure 12
observe that the state-action pairs supported by ensemble policies exhibit stronger correspondence with the ones projected from the offline dataset. In com- parison to Figure 12 a). where the points of the categories
scattered, the points of two colors in Figure (2]b) have greater overlap. This observation further confirms that the policy en- semble offers more comprehensive support over the offline dataset, ultimately leading to improved performance.
Does behavior cloning with disagreement penalty help in learning diverse behaviors? We conduct an analysis to determine if the behavior cloning loss with disagreement penalty can suc- cessfully learn diverse behavior policies. The results, depicted in Figure and the behavior policies learned using 0.1 exhibit significantly greater diversity compared to their counterparts.
and [4, demonstrate that
Does diverse policies help in exploring optimal policies? To answer this, we investigate whether the diverse policies help to explore the high-return actions region the dataset, which is a crucial factor for enhancing performance in offline learning (Hong et al. 2023). We visualize the action distribution of the policies learned during the offline multi-step optimization phase on two tasks. As depicted in Figure 15 and [6] the leamed policies effectively encompass the high-return actions region. This indicates that diverse policies have a greater potential for exploring optimal policies, thereby improving the overall learning process.
24

---

Published as a conference paper at ICLR 2024
Polies
Dataset
(a) Single policy
(b) Ensemble policy
Figure 12: We embed set of state-action pairs of offline dataset and different behavior policies into 2D space using I-SNE, We highlight the region of mismatch between the dataset and the policies.
Figure 13: The distribution comparison of learned behavior policies across various disagreement penalty coefficients on the MoJoCo Locomotion domain is illustrated, specifically for a = 0.0 (top) and 0.1 (bottom). For simplicity, only the first dimension of actions visualized.
1-
Figure 14: The distribution comparison of learned behavior policies across various disagreement penalty coefficients on the Adroit domain is illustrated, specifically for a = 0.0 (top) and a = 0.1 (bottom). For simplicity, only the first dimension of actions is visualized.
25

---

Published as a conference paper at ICLR 2024
(a) Dim
(b) Dim
Dim 3
00
Dim
(e) Dim 5
(1) Dim
Figure 15: The distribution of each action dimension in the policies learned during the offline multi- step optimization phase on the walker2d medium replay - x2 task. Specifically, we visualize the top 50 high-return actions from the offline dataset, highlighting the diversity of policies and their ability to explore and reach the region of high-return actions.
(a) Dim 1
(b) Dim 2
(c) Dim 3
Figure 16: The distribution of each action dimension in the policies learned during the offline multi- step optimization phase on the hopper medium replay x2 task. Specifically, we visualize the top high-return actions from the offline dataset, highlighting the diversity of policies and their ability to explore and reach the region of high-return actions.
26

---

Published as a conference paper at ICLR 2024
A.7 EXTRA OFFLINE COMPARISON
In this section, we have added the Diffusion QL (Wagg et al. 2022), RORL (Yang et al. 2022). XQL (Garg et al. 2023), SQL (Xu al. 2023) as strong baseline for comparison. Upon inspecting Table 6. it becomes evident that Uni-Q4 outperforms all other methods in terms of the total score across all tasks. Furthermore, Uni-O4 outperforms all other methods in 16 out of 26 individual tasks. While RORL surpasses Uni-O4 in the total score for MoJoCo Locomotion tasks, it performs worse than Uni-O4 in the other three domains and exhibits limited effectiveness the Kitchen domain.
Table 6: Extra comparison on D4RL tasks with other state-of the-----ttt baselines.
Environiennt
DiffusionQQL Wang
SORL Yang
XQL King
SQL
halfscheetub medrams-v2
hoppet-medium-v2
90.5
104.8
74.2
75.5
104.4
walker 2d-mediaan--
87.0
102.4
84.2
84.2
90.2
halfcheetab medrums replay
47.8
46.8
44.3
99.7
walker 2d mediamt ropley
55.5
90.4
82.2
81.2
98.4
halfcheetah medramt expert
96.8
107.8
94.2
98.0
93.8
happer-mai senper-edium-speer
walker 2w--edd--is----e exppert
118.1
locomotion
8J6.4
Umuzz- divers---ve-ee2
90.7
74.0
83.5
Medium play x2
16.5
76.3
76.0
80.2
75.2
Medium divene-22
78.6
69.3
73.6
79.1
72.2
Large-play-v2
Large diverse v2
56.6
41.0
49.0
52.3
$8.7
Autoauy
431.0
44R.2
pen-buman
hummer-haran
4.1
2.3
1.8
24.7
door-uution
6.9
3.8
11.5
7.2
27.1
1.7
pen-elonad
69.8
101.3
hummer-lonnd
1.7
4.3
2.1
7.0
docr-cloned
-0.1
4.8
10.2
relecate-ckneed
Adnilt
147.5
177.0
281.6
kilchen
84.0
76.4
93.6
purtial
kitchen-mixed
62.6
67.4
65.0
total
216.3
216.9
1763.1
A.8 EXTRA OFFLINE-TO-ONLINE RESULTS ON D4RL TASKS
In Figure 4 the results demonstrate that Uni-O4 achieves effective initialization based on subopti- mal datasets for online fine-tuning. Leveraging the favorable properties of on-policy RL, Uni-O4 consistently enhances performance without any drop in performance. In this section, we conduct experiments to explore the performance of Uni-O4 on random datasets. As depicted in Figure I7 Uni-O4 exhibits rapid performance improvement with modest initialization.
Furthermore, Figure 18 and 19 present a comparison on more challenging tasks, such as the multi- task kitchen with a long horizon and adroit hand with sparse rewards. Uni-O4 not only achieves better initialization performance but also demonstrates further performance improvements. Uni-O4 outperforms all baselines significantly, showcasing its superiority in tackling these demanding tasks.
A.9 THE COMPARISON BETWEEN @, AND Q. UNI-O4
In this section, conduct experiments to investigate why we chose @ instead of QR, in this work. In OPE and the computation of the advantage function, it is common to fit QTz during policy improvement. However, in the offline setting, both policy optimization and evaluation depend on the fitted Q-function. This can lead to overestimation due to off-policy estimation and the distribution shift present in offline RL. As depicted in Figure 20 increasing the steps of off-police estimation
27

---

Published as a conference paper at ICLR 2024
hospet-randam-22
halfcheet arrchhmtal--nnddmme?
ODT
Nommalzen Return
1.00
Normalized Rteur
100
off2onn
SPOT
62 Durs
Gurs
82
Noomalized
40
50
70
Environment
Environmant
Environment
(b)
(c)
Figure 17: The experimental results on D4RL random dataset. The dotted line indicates offline
initialization performance.
kitchen mixed
kitchen partial-vO
Nomalzze Roturn
80
Normaltzed Return
Normalized Returm
600
MEX
60
Cal
Calqgi
40
offzun
ofizenn
SPOT
SPOT
100
20
10
Entiioonment: Steps (x1e4]
Environment Steps (x(664)
Environmmnt Steps (X104)
(b)
(c)
Figure 18: The experimental results on D4RL Kitchen dataset.
peh v2
doon-lonodivv
har men clonedivi
1.0
1.00
Normalized Retum
201
Callel
80
PEX
Cal
sttoon
75
affin
30
SPOT
Ours
Normaltzed
60
SPOT
40
Durs
10
50
200
150
50
100
150
150 Emvironment Steps (x5e3)
50
100
Envirommen Steps [x5e3]
Environment Steps (xbe3)
(b)
(c)
Figure 19: The experimental results on D4RL adroit cloned tasks.
28

---

Published as a conference paper at ICLR 2024
results in worse performance. Conversely, using smaller step, even relying solely on Qxa. can lead to suboptimal outcomes.
On the other hand, 2 offers favorable choice as it approximates the optimal Q" while considering the constraints imposed by the dataset support. As demonstrated, the selection of Q, significantly outperforms the iterative updating of the Q-function.
hopper medium replay
walker2d medium replay V2
100
100
Normwlized fetuum
Normalized Return
600
600
Nomakkzd
40
20
Uni-44
2tcp-20
35.0
Uni 04
step=20
alep-200
stap-
step-:200
2000
20
stepm 2000
2000
2tc5-25000
stcp- 20000
30.0
stop-20009
Gra dint Steps (x500)
Gradient (x:500)
Gradlent Steps (x500)
(a) Dim
(b) Dim
(c) Dim
Figure 20: The comparison between Q and Qa in Uni-O4. Uni-O4 represents that Q is used as a component AM-Q and to compute the advantage function. Additionally, the steps 20, 200, 2000, and 20000 represent the number of iterations required to fit my following the replacement of the behavior policy.
A. THE DESIGN CHOICES OF ENSEMBLE BEHAVIOR POLICIES FOR MULTI-STEP POLICY OPTIMIATION
A.11 DESIGN CHOICES ABLATION STUDY
In our evaluation, we apply the well-known 'code-level optimization' (Engstrom et al. 2019) tech- niques of PPO to enhance the performance of Uni-O4. Followed by the implementation of online PPO and BPPO (Zhuang et al. 2023). we use learning rate and clip ration decay, orthogonal ini- tialization, state normalization, reward scaling. Tanh activation function, and mini-bace mini-batch advantage normalization in Uni-O4. In this study, specifically analyze the design choices follows.
Reward scaling: Instead directly using rewards from the environment in the objective, the PPO implementation employs scaling scheme based on discounting. In this scheme, the rewards divided by the standard deviation of a rolling discounted sum of the rewards, without subtracting and re-adding the mean. For more details, please refer to [Egstrrm et al. (2019).
State Normalization: Similarly to the treatment of rewards, the raw states are not directly fed into the policies. Instead, the states normalized by the mean and variance calculated from the offline dataset instead of the initial mean-zero and variance-one vectors in standard PPO.
Tanh activations: The Tanh activation function are used between layers in the policy.
Value function clipping: For value network training, we use the PPO-like objective:
L max (Vo. - Ving)² (clip (Vo., Vo._, - Vo._, + +) Viarg) }]
where Vo is clipped around the previous value estimates (and is fixed to the same value as the value used to clip probability ratios in the PPO loss function.
As illustrated in Figures 21 22 and 23 these design choices consistently exhibit enhanced perfor- mance compared to their alternatives on various tasks, including walker2d-medium, medium-replay. and medium-ent medium-expert. Notably, the design choices of state normalization, reward scaling, and Tanh activation function show particularly significant benefits on the medium-replay tasks. Furthermore, the performance does not significantly vary with the design choice of value function clip.
A. 12 RUNNING TIME ANALYSIS
In this section, we analyze the runtime for both the offline and online phases.
29

---

Published as a conference paper at ICLR 2024
walker26 medium
walkerdd medium raplay
walker2d medium expert v2
130
Noraated Reour
200
Normalzed Resurn
Normalze Return
-20
200
60
300
10
80
20
Withoue
Without
Without
Steps
Steps,
Figure 21: Ablation study state normalization during online fine-tuning.
walker2d medium-v22
wdlker2d wwdimmemmmmmmmee raplayav2
walker2d medium expert
3.40
120
Norra lied Retur
Normaliaed Raturn
Normal (oad Rocum
200
80
60
Demaad
Rewand
Reward
Without
Without reward
50
30
30
Environment
Figure 22: Ablation study reward scaling during online fine-tuning.
walker2d medium-22
walker2d meduum---dduuumm-raaaa.....
walker26 medium expertv22
Nommatzee Bestum
Nomalzzd Return
3.0
Normalaad
s0
200
60
80
Figure 23: Ablation study activation function during online fine-tuning.
walkor2d modium v2
walker2d medium replay
walkordd modium expert
320
Nommllzee Barume
Nomalbod Rezurn
3,00
1.20
Narmaliedd
cli
80
<lp
without
40
withoue die
without
Sceps
Figure 24: Ablation study value function clip during online fine-tuning.
30

---

Published as a conference paper at ICLR 2024
In the offline pretraining phase, illustrated in Figure 25(a) the running time during offline training shows a minor increase with the number of policies. This slight increment in time is deemed ac- ceptable, given the significant performance improvement achieved. We also measure the pretraining time for all methods in Figure 25(b) RORI. Diffusion-QQ. COI, and off2on methods require more than hours, while the remaining methods complete within hours.
Moving on to the online fine-tuning phase, we measure the fine-tuning time for methods. As illustrated in Figure(25(c) the Q-ensemble-based method (off2on) takes over 1000 minutes, whereas Uni-O4 only requires 30 minutes to complete the fine-tuning phase. The other baselines range from approximately 200 400 minutes, all significantly slower than our method. These results highlight the simplicity and efficiency of Uni-O4 in terms of runtime.
All experiments are conducted on the workstation with eight NVIDIA A40 GPUs and four AMD EPYC 7542 32-Core CPUs.
PBC
Figure 25: Running time analysis. Left: the comparison of running time over ensemble size. Middle: running time of various methods during offline pretraining. Right running time of various methods during online fine-tuning.
A.13 PSEUDO-CODE UNI-04
In this section, we present the pseudo-code for the offline-to-online stages, outlined in Algorithm 2
To perform online fine-tuning, we select single policy from the ensemble of policies by querying
AM-Q based on its proven accurate evaluation performance, as demonstrated in Section 5.3 For instance, when allowing a 25% permitted error, the evaluation accuracy reaches approximately 95%.
Alternatively, we can choose the top-k policies with the highest evaluation scores. These selected policies are then evaluated through interaction with the respective environments. Specifically, we choose & 2 for Mujoco locomotion and & 3 for the Adroit and Kitchen domains. Based on the OPE scores, the interactions with the environments can be conducted more safely, while keeping the budgets minimal.
31

---

Published as a conference paper at ICLR 2024
Algorithm 2 On-policy policy optimization: Uni-O4
# Supervised learning stage:
1: Estimate policy ensemble (m) [
mb mb):
2: Calculate value Q- and V- by
3: Estimate the transition model T by
Offline policy optimization stage:
4: for iteration j = 1,2, ... do
Approximate advantage by - V.
6:
Update each policy x by maximizing objective if 1%AC== = 0 then
7:
8:
Perform OPE for cach policy by AM-Q:
for policy id E I,...,71 do
10:
if (=x)) (mL) then
Set x + x & its 11 end 13: end for
11:
12:
end if
15: end for
# Online fine-tuning stage:
16:
Initialize the policy and V-function from offline stage;
17: for iteration, j 2, ... do
Run policy the target environment for T timesteps
19:
Compute advantage by GAE (Schulman et al. 2015b);
20: Update the policy by [bjective[[ann value by MSE loss for multiple epochs:
21: end for
A. FULL RESULTS OF ABLATION STUDY
Here, we provide the full results of Section 5.3 curves of ablation OPE accuracy, hyperparameter a, the number of ensemble policies, and the optimality on MuJoCo tasks.
Figure 26
27 28 and 29 present the learning
32

---

Published as a conference paper at ICLR 2024
walkke224-medlum-22
walker72-meeduun--rppaawwa
wwllerr2e-meelum-eepprre
1.0
0.8
OPE Accuracy
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
hopper-medlum-22
hoppe--meduu--rpplayyy222
hosper-medium-expetee
1.0
0.8
OPE Accuracy
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.0
0.0%
5%
10%
20%
0.0
0.0%
5%
10%
20%
0.0
0.0%
5%
10%
20%
Permitted Error
Permitted Error
Permitted Error
halfcheetah-medium-
halichetta--medlum--elaaaa
haffcherthh-medium-eeae
0.8
OPE Accurory
0.0
OPE Accuracy
ORE Accuraxy
0.4
0.2
0.2
0.0
0.0%
5%
10%
20%
0.0
0.0%
3%
10%
20%
0.0
0.0%
5%
10%
20%
Permitted Error
Permitted Emrr
Permitted Error
Figure 26: Full results of offline policy evaluation accuracy ablation study.
woller20 mediumiv2
Narmalzed busum
11.5
80
Narmatzed
Nomatteed
310
40
100
20
dphi-0.3
95
apram05
apra-
0
10
20
15
20
10
15
(x500)
Gradient Steps (x500)
Geuclent Steps (x500)
hopper-maok.m-4g
hoppar-medium expert-22
100
3.00
Nomakaad Ruturn
Normakiad and Retuan
ao
20
60
Normltzee
60
62
40
3iph2=0:1
40
aptcm0
31ph2=035
alph=
20
15
15
Groolent Stops (x500)
Grodieen Steps (x500)
Gradient Staps (x500)
halfcheetah medium v2
haffchestah medium replay
halrchoctah medium expertv2
Normatord Retum
Commabou Rocur
20
Hood Retum
85
68
33
50
66
30
12
5
10
20
10
20
10
20
Gradient Steps (x500)
Geadient Steps (x500)
Gradient Steps (x500)
Figure 27: Full results of alpha ablation study.
33

---

Published as a conference paper at ICLR 2024
walker2d-medlum-v2
walker2d medium expert-v2
100
Return
80
Return
Normalized Return
60
75
40
25
Policy Number
Number
Policy Number
hopper-men 2opper-medium-22
hoppe-medium-replayy
seaper-meiimm-expetee
Nomalzze Return
100
Normalized Return
100
1000
80
80
(80
60
60
40
40
Policy Number
Policy Number
Policy Number
halfcheetah-medlum-22
hallcchetta-medduu--eelayyy
halfcheetah-medium-e expert-v2
50
Nomraized Retur
Normaizee Return
Normalzad Return
30
60
20-
20
40-
10
0-
Policy Number
Policy Number
Policy Number
Figure 28: Full results of ensemble ablation study.
walker2d--edium-22
me::ii2ddeedummmmomm replay
walker2d medium expert-v2
125
nomlliized Scores
125
Normalized Scores
100
100
100
Normaiizd
75
50
50
50
25
Ours
25
Ours
Ours
Scratch
Scratch
Scratch
200
400
200
400
200
400
Environment Steps (x5e3)
Environment Steps (x5e3)
Environment Steps (x5e3)
hopper-medlum--2
hogpe-mmeduum-rellvve um-replay-22
hogeermeedum-expprtwe
Normalized Scares
Nomalized Scores
Scores
75
75
75
50
50
50
Ours
25
Ours
25
Ours
Scratch
Scratch
Scratch
200
400
200
400
200
400
Enviranment Steps (x5e3)
Environment Steps (x5e3)
Environment Steps (x5e3)
halfcheetah salcchetth--medium--> mmedium-v222222222
halfcheetah medium replay
helfcheetah-med expeetr
Nomalized Scores
Normalized Scores
Normalizze Scores
100
so
Ours
Durs
Scratch
Scratch
Scratch
200
400
200
400
200
400
Environment Steps (x3)
Environment Steps (x5e3)
Environment Steps (X5e3)
Figure 29: Full results of optimality analysis.
34


Reviews:
[{"review_id": "3hRDRbWl1s", "reviewer": "Reviewer_CZhd", "summary": "The authors propose a new approach, Uni-O4, to combine offline and online reinforcement learning, which is an important and challenging problem in the field. Uni-O4 can effectively address the mismatch issues between the estimated behavior policy and the offline dataset,  and it can achieve better offline initialization than other methods and be more stable for the later online fine-tuning phase. The experimental results on several benchmark tasks show that Uni-O4 outperforms existing state-of-the-art methods in terms of stability, final performance, and the capability for real-world transferring.", "questions": "Suggest to add legends for Fig. 3 or bringing the legend in Fig. 4 forward.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Uni-O4 can seamlessly transfer between offline and online learning, enhancing the flexibility of the learning paradigm.\n\n2. The experiments are sufficient and persuasive. The experiments on real-world robots showed very good performance in the provided videos.", "weaknesses": "1. Notions are confusing in this paper, especially after the overloading in Equ. (8).\n\n2. In Fig.2, It is hard to capture the Offline Multi-Step Optimization process, i.e. the sequence relationship of each step.\n\n3. In Sec 3.1:  \"BPPO leads to a mismatch ... due to the presence of diverse behavior policies in the dataset D\",  could authors explain further why the diversity is blamed for the mismatch?\n\n4. Lack of theoretical analysis (to support the motivation of technique details), but it has sufficient experiments thus this point is acceptable I think.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mGbBYnhTsi", "reviewer": "Reviewer_rY3g", "summary": "This paper proposes a new algorithm called Uni-O4 that unifies offline and online reinforcement learning using an on-policy optimization approach. The key ideas are:\n- Using an on-policy PPO objective for both offline and online learning to align the objectives.\n- In the offline phase, using an ensemble of policies and offline policy evaluation to safely achieve multi-step policy improvement.\n- Seamlessly transferring between offline pretraining and online fine-tuning without extra regularization or constraints.\n- Evaluating Uni-O4 on both simulated tasks like Mujoco and real-world quadruped robots.", "questions": "- What are the computational overheads associated with the ensemble policies, and how do they impact the method's scalability?\n- Why don't use the ensemble approach to mitigate mismatches instead of other methods for handling the diverse behaviors in the datasets? For example, Diffusion-QL [1] demonstrates that Diffusion model can be used to learn multimodal policy.\n\n[1] Wang, Zhendong, Jonathan J. Hunt, and Mingyuan Zhou. \"Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning.\" In The Eleventh International Conference on Learning Representations. 2022.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Simple and unified design without needing extra regularization or constraints for stability. Avoids issues like conservatism or instability in prior offline-to-online methods.\n- Impressive results surpassing SOTA on offline RL and offline-to-online tasks. Significantly boosts offline performance and enables rapid, stable online fine-tuning.\n- Policy ensemble provides good coverage over offline data distribution. Offline policy evaluation enables safe multi-step improvement.\n- Excellent results on real-world robots - pretraining, offline adaptation, online finetuning. Showcases efficiency and versatility.", "weaknesses": "- The complexity of the method, especially regarding the ensemble behavior cloning and disagreement-based regularization, may present a steep learning curve for practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i3HHx9c0Ls", "reviewer": "Reviewer_bAiM", "summary": "The article introduces Uni-O4, a new method for combining offline and online reinforcement learning. It eliminates redundancy and enhances flexibility by using an on-policy objective for both phases. Uni-O4 employs ensemble policies and a straightforward offline policy evaluation approach in the offline phase to address mismatches between behavior policy and data. The approach leads to better offline initialization and efficient online fine-tuning for real-world robot tasks and achieves state-of-the-art results in various simulated benchmarks.", "questions": "1) Can you provide a detailed reconstruction method for policy $\\hat{\\pi}_{\\beta}$, whether to select any one from the policy set $\\Pi_n$ or integrate it using the f function to obtain a policy?\n\n2) Is there a way to evaluate the quality of behavior cloning? Can you compare your proposed method of behavior cloning with previous methods?\n\n3) Can we analyze the errors in the approximate part? You can cite the results of previous work to prove it. For this article, you do not need to prove the size of the approximation error. You only need to quantify the approximation error to a certain extent, analyze the potential impact, and find ways to avoid negative effects.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) Despite some minor flaws, this paper is written in a standardized and organized manner, allowing people to quickly capture the core innovative points and ideas of the paper.\n\n2) The Uni-O4 framework proposed in the article unifies the learning objectives of online and offline learning, making the transition from offline learning to online learning smoother.\n\n3) This method has shown excellent performance in various experiments and has also achieved good results in real-world machine experiments.", "weaknesses": "1）The behavior cloning method proposed in section 3.1 requires training multiple policy networks, which incurs significant computational overhead. At the same time, it does not mention how to get $\\hat{pi}_{\\beta}$ from a policy set.\n\n2）Definition error, the definition of f used in formulas 6 and 7 is incorrect. Taking the maximum value of multiple distributions cannot guarantee a single distribution (the sum cannot be guaranteed to be 1), and analysis based on this definition is also meaningless. If the code is truly implemented based on this definition, I am skeptical about the final performance of the algorithm.\n\n3) The proposed offline strategy evaluation method relies on the accuracy of the probability transfer model T, and using the transfer model for evaluation will introduce more errors.\n\n4) The entire method has made too many approximations to the problem and lacks corresponding error analysis.\n\n5） The legend in Figure 3 is missing to know the correspondence between curves and algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pxbXeYg87y", "reviewer": "Reviewer_jjSi", "summary": "This paper focuses on unifying offline and online RL to achieve efficient and safe learning. Specifically, this paper proposes Uni-O4, which utilizes an on-policy RL objective for both offline and online learning. For offline learning, this paper combines the advantages of both BPPO and OPE to achieve the desired performance. For online learning, this paper directly utilizes the standard PPO for finetuning. Experiments under offline RL and offline-to-online RL setting demonstrate the effectiveness of Uni-O4. Furthermore, this paper extends the offline-to-online setting to address a practical robotic scenario, transforming it into an online-to-offline-to-online setting. Empirical results highlight the seamless integration across these three stages in Uni-O4.", "questions": "See Weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper investigates an interesting research problem: offline-to-online setting, and online(simulator)-to-offline(real-world)-to-online(real-world) setting in robotic scenarios.\n- This paper performs extensive experiments to derive empirical findings.", "weaknesses": "Overall, this is a descent paper. However, in the current manuscript, I think the following concerns should be addressed.\n\n== Major concern ==\n\n- Unclear empirical motivation in Figure 1. What does these variants (Conservatism, Constraint, Off-policy) mean in (a)? How does Q value compare with V value in (b)? Moreover, from (b), it seems that CQL->SAC shows faster improvement that On-policy (V). How this conclude that Q values of SAC exhibit slow improvement? Furthermore, CQL->CQL and CQL->SAC are naïve solutions for offline-to-online RL. What about advanced offline-to-online RL algorithms, such as off2on?\n- The technique seems incremental by just combining BPPO with OPE.\n- I think there exhibits slight overclaiming of the experimental results in Introduction without sufficient comparison of SOTA algorithms.\n> Experimental results show that Uni-O4 outperforms both SOTA offline and offline-to-online RL algorithms.\n    - Insufficient comparison of offline RL, including but not limited to:\n\n    [1] RORL: Robust Offline Reinforcement Learning via Conservative Smoothing.\n\n    [2] Extreme Q-Learning: MaxEnt RL Without Entropy.\n\n    [3] Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization.\n\n    - Insufficient comparison (including PROTO, ODT, E2O, SPOT, etc.) or at least discussion of related works on offline-to-online RL. Particularly, the baselines include AWAC, CQL, IQL, which are naive solutions for offline-to-online RL. PEX presents weak sample-efficiency for above-medium datasets. Cal-ql is not empirically designed for MuJoCo tasks. There is only one relatively strong baseline, i.e., off2on.\n\n    [1] Adaptive policy learning for offline-to-online reinforcement learning\n\n    [2] Actor-Critic Alignment for Offline-to-Online Reinforcement Learning\n\n    [3] A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning\n\n    [4] Efficient online reinforcement learning with offline data\n\n- Minor improvement on MuJoCo tasks in Figure 4. As shown in the figure, off2on significantly outperforms Uni-O4 by a large margin in halfcheetah-medium and halfcheetah-medium-replay. Besides, I also want to point out that 100 D4RL score already achieves expert-level performance in D4RL benchmark. Thus, further improvement on other settings over 100 is not necessary. Thus, I also wonder why this work does not consider random dataset, which presents a significant challenge for online finetuning to achieve expert performance.\n\n- Comparison in Section 5.2 seems not fair enough. Firstly, I want to know which is the claimed baseline WTW in Figure 5? Additionally, given that IQL is not designed specifically for a real-world robotic scenarios, is the comparison between IQL and Uni-O4 fair? (Uni-O4 is revised to adapt to robotic scenarios as stated in the appendix) Maybe a strong baseline can be considered to verify the superiority of Uni-O4.\n\n- I feel a little struggling to follow Section 5.2. Maybe a pseudo-code like A.6 can be provided to make the readers understand the online-offline-online setting more clearly.\n\n- The experimental results in A.3 make me confusing. I cannot identify obvious differences between Figure 11 (a) and (b).\n\n== Minor Concerns ==\n\n- Figure 2 is not that intuitive. Maybe more explanations can make it clearer.\n\n- How many seeds and evaluation trajectories for AntMaze tasks in offline RL setting? Why offline-to-online RL setting does not consider Kitchen, AntMaze and Adroit-cloned and -relocate tasks?\n\n- Why 18 hours training time is **unacceptable** for real-world robot learning?\n\n- Lack of reproducibility statement.\n\n- Maybe more details on baseline implementation for real-world robot tasks can be provided.\n\n- Why this paper does not provide offline training time for comparison?\n\n== Typos ==\n\n- Page 4, above Equation 7: dataset $D$ -> $\\mathcal{D}$\n\n- Page 9, Hyper-parameter analysis, loss 7 -> Equation 7 is an optimization objective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)