Offline Reinforcement Learning at Multiple Frequencies
Authors: Kaylee Burns
Tianhe Yu
Chelsea Finn
Karol Hausman
Venue: CoRL 2022 Poster (2022)
Keywords: offline reinforcement learning
robotics

Abstract: To leverage many sources of offline robot data, robots must grapple with the heterogeneity of such data. In this paper, we focus on one particular aspect of this challenge: learning from offline data collected at different control frequencies. Across labs, the discretization of controllers, sampling rates of sensors, and demands of a task of interest may differ, giving rise to a mixture of frequencies in an aggregated dataset. We study how well offline reinforcement learning (RL) algorithms can accommodate data with a mixture of frequencies during training. We observe that the $Q$-value propagates at different rates for different discretizations, leading to a number of learning challenges for off-the-shelf offline RL algorithms. We present a simple yet effective solution that enforces consistency in the rate of  $Q$-value updates to stabilize learning. By scaling the value of $N$ in $N$-step returns with the discretization size, we effectively balance $Q$-value propagation, leading to more stable convergence. On three simulated robotic control problems, we empirically find that this simple approach significantly outperforms na\"ive mixing both terms of absolute performance and training stability, while also improving over using only the data from a single control frequency.

Offline Reinforcement Learning at Multiple Frequencies
Kaylee Burns', Tianhe Yu Chelsea Finn 1.2 Karol Hausman 1.2
Stanford University Google Research
kayburns@cs stanford. boybburs@es.staaford.eee edu
Project Website: https://sitee google com/stant ord. edt//alaapiiee-stee--rtrraa
Leveraging many sources of offline robot data requires grappling with the heterogeneity of such data. In this paper, we focus on one particular aspect of heterogeneity; learning from offline data collected at different control frequen- cies. Across labs, the discretization of controllers, sampling rates of sensors, and demands of task of interest may differ, giving rise to a mixture of frequencies in an aggregated dataset. We study how well offline reinforcement learning (RL) al- gorithms can accommodate data with a mixture of frrqueeciess during training. We observe that the Q-value propagates at different rates for different discretizations, leading to a number of learning challenges for off-the-shelf offline RI.. We present a simple yet effective solution that enforces consistency in the rate of Q-value up- dates to stabilize learning. By scaling the value of N in N-step returns with the discretization size, we effectively balance Q-value propagation, leading to more stable convergence. On three simulated robotic control problems, we empirically find that this simple approach outperforms naive mixing by 50% average.
Abstract:
Keywords: offline reinforcement learning, robotics
Introduction
Given the cost of robotic data collec- tion, we would like robots to learn from all possible sources of data. However, the robot data that is avail- able may be heterogeneous, which can present challenges to offline rein- forcement learning (RL) algorithms. One particular form of variability that we focus on in this work the con- trol frequency. Data collected inde- pendently may have been collected at different frequencies due to different sensor sampling rates or how the data was collected. For example, data col- lected through teleoperation may have higher frequency to accommodate more reactive and intuitive user interface, while data collected via online RL may use lower frequencies for more sta- ble learning [1]. In light of the benefits from learning from large and diverse datasets [2, 3], our goal in this paper is to study whether offline RL algorithms can effectively learn from data with multiple underlying frequencies, and improve over learning separate policies for different frequencies.
propigtter
06641
98 yf 87
Human Hz
Robot
Figure 1: Training offline RL with diverse frequencies chal- lenging because the discrete MDP depends on the size the timestep between observations. This discretization affects the rate of value propagation. with smaller discretizations requiring more training updates to propagate the values.
While prior work studies how to stabilize RL algorithms at a single high frequency [1], effectively learning from data with multiple frequencies has been paid less attention. Work on time-series analysis has studied how to handle irregularly sampled data [4, 5], but not in the context of the ap- proximate dynamic programming updates that arise in value-based reinforcement learning settings. As we will find in Section 4, running offline RL algorithms on data with multiple control frequencies leads to differing Q-values and diminished performance, a problem that, to our knowledge, has not been analyzed or addressed in previous research.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

A key insight in this work is that Q-learning from different data discretizations leads to different rates of value propagation: value propagates more quickly for coarser time discretizations. These different rates creates inconsistent regression targets, which can lead to high variance updates and training instability during optimization. Fortunately, we can use knowledge of the time discretization to normalize the rate of value propagation across different discretizations: our updates can look ahead to farther Q-values for data with a smaller discretization. This intuition leads us to a simple adjustment to N-step returns [6. 7] to align the discretizations, where we adaptively select the value of N based on the time discretization of the particular trajectory.
The main contribution of this paper is to analyze and provide a solution to the problem of offline reinforcement learning from data with multiple time discretizations. First, we show that naïvely mixing data of different discretizations leads to diminished performance, and posit that this issue caused by varying rates of value propagation for different discretizations. Then, we provide a simple technique that addresses this challenge via Adaptive N-Step returns. On three simulated robotic control problems, including two manipulation problems on simulated Sawyer and Panda robot arms, we find that our Adaptive N N-Step update rule leads to stabler training and significantly improved performance, compared to naive mixing or only using learning policies individually for each discretization.
2 Related Work
Supervised learning of irregular time series. Datasets with varying frequencies can arise when- ever observations are pulled inconsistently from a continuous time system. Some approaches focus on consuming irregularly sampled data [8, 4, 5] by modeling the latent state of the data-consuming process as a differential equation. Unlike past work that integrates continuous time models into control tasks [9, 10], we are focused on the setting where the discretization remains constant within each trajectory, but varies within batch updates.
Continuous time RI. We focus on control tasks learned with offline RI., so in addition to learning a policy that can operate at multiple time scales, we also need to learn value function multiple time scales. Past work [11, 12, 13] uses the Hamilton-Jacobi Bellman equation to derive algorithms for estimating the value function. These formalisms have been used to create algorithms that can handle settings where the environment evolves concurrently with planning [14] and to learn to budget coarse and fine-graine time scales during training [15]. In addition to modeling challenges, continuous time introduces challenges during training as well. Tallec et al. [1] shows that deep Q-learning can fail in near- continuots time (i.e., fine- grainedrdd)) settings and uses the formalisms from continuous RL to make Q-learning more reliable for small discretizations by using an advantage update.
Offline RL. Offline RL [16, 17, 18, 19] has emerged promising direction in robotic learning with the goal of learning policy from a static dataset without interacting with the environment [20, 21, 22, 23]. Prior offline RL approaches focus on mitigating the distributional shift between the learned policy and the data-collecting policy [24] via either explicit or implicit policy regularization [25, 26, 27, 28, 29, 30, 31J, penalizing value backup errors [32], uncertainty quantification [26, 33, 34], and model-based methods [35, 36. 37, 38. 39]. None of these works consider the practical problem of learning from offline data with different control frequencies, which the focus of this work.
3 Preliminaries
A discrete-time MDP is a process that approximates an environment that is fundamentally con- tinuous. For example, robot may be moving continuously but receiving observations and sending actions at a fixed sampling rate, 1/5t, where öt is the time between the observations. A discrete-ting time MDP that takes into account 8t can be derived by discretizing continuous-time MDP [12, 1].
Given a time-discretization őt, we arrive at the discrete MDP [1] Mat = (8, A, To,, 7), where Tac(s')s, a) denotes the transition function over the timestep 8t, S and A denote the state and action spaces, r the reward function, and 7 E (0, the discount factor. The cumulative discounted return of this MDP can be written as: Rst : Σi 7 kot r(kk, a4) )8t; and the 5t-dependent Q-function can be written as: Q3 = r(s, a a)ôt + 5 ETTTTTTTT 1 7 p ag (6) 8g dependent rewards and Q Q-functions to analyze the case of mixing data with different discretizations, i.e. with different values of 5t, in offline RI. These definitions are similar to those of a standard MDscrete-MDP. but with appropriate scalings for the given discretization, öt.
We will use the 81-

---

Offline RL with conservative Q-learning, Offline RL tackles the problem of learning con- trol policies from offline datasets where the main challenge is the distribution shift between the learned policy and the behavior policy that was used to collect the data. A common of- fline RL approach that mitigates this issue focuses on an additional pessimism loss that encour- ages the Q-value update to stay close to the actions that it has seen in the data. One pop- ular instantiation of this idea is conservative Q-learning (CQL) [32], which optimizes the fol- lowing objective: Qt+1
arg arggmig) aa (E,wD, (anp(()))) [Q(s, a)] EgwD, (an_D..a(((sss(@@))) (Q(s, a)l) + 2 s' (Q(s,a)) r(s,a) a) >1E, (als) (s, a)])] where D is the offline dataset of states, actions, and rewards, ja is a wide action distribution close to the uniform distribution, and f is the behavior policy that collects the offline data. We will show how we can use modified version of CQL to incorporate data from different discretizations in offline RL.
4 Mixed Discretizations Can Destabilize Offline RL
In this section, we introduce the problem of simultaneously learning from several data sources with different frequencies. We show how differing rates of value propagation can create instability during training, which causes naively mixing frequencies to fail, and verify this intuition
simple task.
Problem setup. Our objective is learn a policy or set of policies that maximizes the expected return over dataset that contains mixture of different observation frequencies. Concretely, our goal is to learn T²" (als, 6t) over a distribution, A, of discretizations: (aa(a = x² (als, öt) arg max- 1 [R,,T, [Rae]]. To accomplish this goal, we utilize offline RI. and focus our analysis on the behavior of Q-values in the presence of mixed discretizations. Specifically, we are interested in studying the objective Est-a (fq), where Lo is any loss based on the Bellman equation.
We use CQL as our offline RL algorithm of choice, but our analysis is applicable to other value-based offline RL methods. We modify the CQL objective described in Sec. 3 to incorporate a distribution of discretizations, öt as follows:
Q = arg min Estas a
8)
Q
1 BanssDDa [(Qne(s,a) (()n((s))--((ssd) r(8, 1 2
The Q-targets of different di may receive value updates at different rates because it takes longer for value to propagate when the state space is divided by more fine-grained steps. We study the challenges this creates for Q-learning next. We refer to this unmodified objective as Naïve Mixing.
Value propagates in algorithmic time, not physical time. key challenge in mixing data of different discretizations together is that algorithmic time, k, scales inversely with the size of the discretization: se This implies that value propagates along the state space a rate that is pro- portional to the discretization. This phenomenon is illustrated for simple gridworld environment in Figure 2. The agent starts on the right side and receives a reward of 10 for moving forward to the left. In the top row, the agent can take steps of size and in the bottom row the agent can take steps of up to size 2, which simulates policies that operate at different frequencies. At every step of the algorithm, k. we perform the update rule (s) in this toy example, we are performing tabular updates, we can see how the nature of the Bellman update generates a distribution of targets that is difficult for function approximator, like neural network, to match. For example, at step A = 3, a network would receive the following targets for the first three states from the start: 0 for öt = 1 and 5.3 for öt 2 for the first and second state, and 0 for öt = and 6.6 for öt = 2 for the third state. We posit that the rate at which the targets are updated for different discretizations and the resulting difference in the target values leads to training instability and subpar performance. In the next section, we verify this intuition empirically.
max, r(s, a) + TV,"" (s'). Although
4.1 Analyzing Naïve Mixing
We study how offline RL behaves when naïvely mixing discretizations in a pendulum environment and posit that differing rates of Q-function updates lead to training instability that hurts performance.
Setup. To test our hypothesis, we use the previously-used simple pendulum environment from the OpenAl Gym [1]. We modify the pendulum environment by changing the underlying time
3

---

k=1
k=2
k=3
8t-1, y
10
0
0
8.1
0
10
9
8.1 7.3
0
Mt-2, y
10
8.1
8.1
0 0
1D
8.1
8.1 6.6 6.6
10 8.1 8.1 6.6
6.6
5.3
5.3
Figure 2: We visualize how value propagates for different öt a simple gridworld. In the top row, agent takes actions of length and in the bottom maximum length 2. reward 10 is attained at the goal state in the most square of the grid. For each update step k, the value propagates twice much in the second setting, creating regression targets that difficult to match. For example, step 2, the Q-value needs to regress targets 6.6 and 0 from the same states. Boxes color-coded by update step.
Pendulum
at
Pondulum (cb=0.32)
10000
Individual nairiigg
Bvg Reuant
0000
4000
40000
Narve Nuong. at 0.005 Nixing.
2000
20000
Update
(b)
Figure 3: We analyze training with mixed discretizations for the pendulum swing up task. (a) Over the course of training, the average Q-value scales with őt. As result, the Q-value of more fine-grained discretizations propagates more slowly than for coarser discretizations. (b) Compared to training individually, the final perfor- mance of policy trained for öt 0.02 is corroded by adding more data different discrctizations.
discretization of the simulator and collect data with discretizations E (0.02, 0.01, 0.005) seconds. We make the reward function for this task sparse by creating window of angles and velocities around O where the algorithm receives a constant reward. This increases the difficulty of the task and gives us a more precise understanding of the comparisons between the Q-values. Without this modification naive mixing converges in under 10 gradient steps. We first collect data by running Deep Advantage Updating (DAU) [1] for each discretization individually and then store buffers of 500k observations per discretization. The data collected is comparable to an expert-replay dataset. We combine the data coming from the three different discretizations together into single offline dataset.
Value propagation. We look the average Q-value for different discretizations over the course of training. The average cumulative reward in the dataset for each 5t is similar, so we expect the average Q-value of each discretization be the same. However, because more fine-grained data propagates the sparse reward at slower rate than for coarser data as shown in the previous subsection, we see in Figure 3 (a) that the coarser discretization of ôt 0.02 reaches higher average Q-value more quickly than a fine-grained discratization of öt = 0.01, which in turn reaches a higher Q-value more quickly than the most fine-grained discretization öt 0.005. By the end of training the average Q- values across different 8t converge. This confirms the hypothesis that the Q-values grow at different rates because of the nature of the Bellman update and not because the true value different.
Optimization challenges. Next, we want to understand if the different rates the Q-value update for different discretizations cause optimization challenges during offline RL. In Figure 3 (b) plot performance over the course of training with and without adding data. Ideally our algorithms would be aided by the addition of more data, but after introducing data of different frequencies, we the average return across training drop for öt = 2.
To summarize, these results suggest that, even within simple control task, naïvely mixing data with different discretizations hinders the performance by slowing value propagation and introducing optimization challenges. Next, present method that aims to avoid these issues that additional data can be leveraged effectively.
5 Adaptive N-Step Returns
The goal of our method is to "align" the Bellman update rate for different discretizations so that they can be updated at a similar pace, creating more consistent value targets. Intuitively, we aim to mimic

---

k=1
k=2
k=3
8t-1, y-.9
y
10
0
0
8.1
7.3
6.6
10
9
8.1
7.3
5.9
5.3
Mt-2, you
10
8.1
8.1
0 0
1D
8.1
8.1
6.6
6.6
10 8.1 8.1 6.6
6.6
5.3
5.3
Figure 4:
We modify value iteration on a simple gridworld environment, In the bottom row, agent can take actions that are double the size the other agents actions. However, because the agent in the top row uses an N N-step update rule with N = 2, the value propagates along the state space at the same rate for each agent.
the "update stride" of coarser discretizations when training on more fine- theeggraied discretizations to bring consistency to the Q-value updates.
The core idea behind our approach is to utilize N-step returns as a tool that accelerates the propaga- tion of Q-values. In particular, we calculate Q-targets with Adaptive N-Step returns, where scale N by the value of öt. To better understand the idea behind this approach, we revisit the same intuition as presented in Figure but with the updated N-step modifications in Figure 4. In this case, use
the following value iteration update rule: Vi((s,) -r== 7 r(st+er,(++)))
a+ter) + y V
This update rule is equivalent N-step returns for value iteration, but where the number of backup steps, N, is scaled by 5t. Figure shows an example with N = 2, where the top row uses 61 1 and the bottom row uses ôt When we adjust the number of backup steps by the frequency òt, to N = { 2 and N = = 1 in the top and bottom respectively, we see that the differences in the st targets available Note that the slight differences that remain are due to the precision allowed by the discretization.
function approximator
significantly smaller magnitude than Figure 2.
Our normalization scheme be implemented as Q-value update rule that can be integrated into any offline RL algorithm that learns Q-values. At every update step, we calculate the target Q at öt as:
68 Lurget
r(stuv actv) + + Q5 a
=
In all of our experiments, we select N be the largest öt present in the dataset (i.e., the most coarse discretization). With this choice of N, the more fine-grained discretizations exactly match the update stride of the coarsest discretization under standard offline RI..
This update rule applies to any Q-learning algorithm. In particular, we use CQL in our experiments. This requires adapting both the standard Bellman loss well as the pessimism and optimism terms, which are lEswD. LssDaax-pals)) (s, a)] and (ool:o) ( a)] respectively. We replace the Bellman operator optimism terms are applied to the state and action at Se. where the target Q-function is evaluated.
Equation with N-step returns scaled by discretization. The pessimism and N
This results in the complete objective:
LCoL= max CB, Da..-DaBBBBalllaaaao.. [Ba>n(als)] ($ N get x: Ba (50 Gct
)ll R(a)
CN Step Estta feor
1 - B [(0
Qurges) 21
If N is not integer, sample the number of steps from the nearest digits (i.e., rounding up or 3r
down) with Bernoulli distribution where the success rate parameter p the fractional part of N 5t"
Experiments
In Section 4, we showed how naively mixing data with different discretizations updates the Q- function different rates leading instability in training. In our experiments, we test this trend on popular benchmarks and validate that our method proposed in Section 4 can remedy these chal- lenges. In particular, we seek answer the following questions: (1) Does Adaptive N-Step returns fix the performance challenges introduced by mixing different discretizations? (2) What impact does Adaptive N-Step returns have on the learned Q-value during training? (3) What the influence of Adaptive N-Step on training stability? (4) How does leveraging multiple discretizaiton data sources compare to learning only from single discretization?
(5) How much of the improvement can be attributed to general benefits from N -tep returns compared to the synchronization of N?
5

---

Env Name
86
Naive mixing
Adaptive N-step (ours)
pendulum
.02
64.8 8.4
91.1 +: 3.7
.01
57.5 1 10.5
81.5 1 2.4
.005
37.5 3.5
62.0 + 6.3
Avg
53.3 7.5
78.2 +:: 4.2
door-open (max success)
10
100.0 + 0.0
89.5 + 8.1
95.0 3.9
92.9 + 5.7
100.0 0.0
100.0 0.0
1
5.7 = 4.2
87.5 + 7.5
Avg
75.2
92.5 +:
kitchen-complete-v0
40
20.2 5.7
34.6 + 8.7
30
9.3 + 4.4
19.9 + 7.2
Avg
14.7 5.0
27.3 +
Table I: Performance of Naïve Mixing and Adaptive N-Step. Within the sparse reward environments of
pendulum and kitchen, Adaptive N-Step outperforms Naïve Mixing when evaluated on any öt. For the dense reward door-open task, Adaptive N-Step enables strong performance on öt 1 while maintaining strong performance coarser discretizations.
6.1 Experimental Setup
Tasks and datasets. We evaluate Adaptive N-Step on three different simulation environments of varying degrees of difficulty described below.
Pendulum. Similarly to Section 4, we used a modified Open AI Gym [40] pendulum environment. We modify by changing the underlying discretization of the simulator and collect data with disc- tretizations of 0.02, 0.01, 0.005 by training DAU [1] on each discretization independently and storing the replay buffer, which contains 500k observations. We use a sparse reward of 100 scaled by št that is attained in range of angles and velocities around O. In all mixing baselines, we condition Q-value and policy on the discretization to train single RL algorithm on all discretizations.
Wertd. We modify the door environment from Meta-Worle [41] to support different discretiza- tions by adjusting the frame-skip. We experiment with frame skips of 2,2,5, 5, and 10. We focus on door-open because it is a straightforward manipulation task. In this environment, we find not con- ditioning on òt give stronger performance for our method as well as the baselines. To evaluate performance, we look at the maximum success score, which measures whether or not the environ- ment was in the goal configuration at any point during the trajectory.
FrankaKitchen. FrankaKitchen [42] is a kitchen environment with 9-DoF Franka robot that can in- teract with household objects. We study the long-horizon task sequence of opening the microwave, moving the kettle, turning on light, and opening a drawer with a sparse reward. Upon completion of any of the four tasks, the agent receives reward of We mix data with frame skips of 30 and 40. The data at 40, which is the default frame skip, comes from the D4RL dataset [43]. Specifically, use the expert data called kitchen-compp cope--ooooleeeeeeee To generate data at frame skip of 30, we run trained policy at that frame skip. The quality of this policy is lower, making the data at 30 most similar medium-replay dataset.
Implementation and training details. We build on top of the CQL implementation from Geng [44], and use the following CQL hyperparameters: all of our models
CQL alpha term of 5, a policy learning rate of 3e 5, and a Q-function learning rate of 3e - 4. For Pendulum we use a discount de
scaled with the size of 86, specifically .99 maratca #7 For Meta-Workee we found constant discount factor of .99 to work best across the experiments. Each task has maximum trajectory length of 500 regardless of the discretization size. We implement Q as an optionally òt-conditioned neural network by adding normalized öt feature to the state.
6.2 Results
Final performance with and without N-step. To answer question (1) (Does ôl-scaled N-step returns fix the performance challenges introduced by mixing different discretizations)) we com- pare the normalized performance the end of training with and without N-step modification in

---

Pondulum Values Across
volues
Kitchan Valuucs Across
0003
1903
Hanva Hicho
Herve Hidht 21 30
6003
Larg O100
4003
hete Monng 0.01
hake Miving d1 6.00
-900
2003
Adaptiss femml 2.0L
atayriee su-ti (oard) 19mh
01
-1500
302
400
700 820
202
302 000
Updaar
Figure 5: Adaptive N-step results in more consistent q-values across discretizations and tasks. In all figures, blue denotes naive mixing and red denotes Adaptive N-step returns.
Table 1. Results are averaged and reported with 95% confidence intervals over runs from 5 seeds with 5 evaluations from each run. For the sparse reward tasks pendulum and kitchen - 5t scaled N-step significantly improves final training performance across discretizations, resulting in al- most double average return in the case of kitchen, and more than 50% improvement in pendulum. In the dense reward tasks, i.e. door-open, N-step provides advantage for more fine-grained dis- cretizations without compromising the high performance naive mixing already achieves on larger discretizations. These results confirm that our method improves mixing discretization performance across the benchmark tasks. Next, we aim to understand its impact on important quantities such as Q-values.
Q-values across training. In Section 4, we observed that the Q-value learned with the naive mixing strategy is updated differently for different discretizations. We aim to confirm this hypothesis on other, more complex environments as well as analyze whether our adaptive N-step method can remedy this issue and lead to more consistent Q-value updates. To accomplish this goal, present Fig. 5, where we visualize the average Q-value for all discretizations with and without adaptive N-step method. When we compare the Q-values after using adaptive N-step, we observe that the Q-values become more consistent across discretizations. In the top left of Figure 5, we see that the Q-values obtained for pendulum are nearly identical for different discretizations across the course of training, resolving the issue pointed out in Sec. 4. Interestingly, we also note that the Q-values of naive mixing eventually converge to the same value as N-step, but the performance of the policy learned with these values is still diminished as previously described in Table 1. This suggests that the consistency of the updates across training and not just the absolute value of the Q-function are important in learning good policy. In the top right and bottom left plots in Fig. 5, we see that, in Meta- World, Adaptive N-Step returns has two effects. It makes the average Q higher for all discretizations and minimizes the difference across discretizations similarly to the pendulum experiment. Within Kitchen, N-step has the same effect of pulling up the Q-values and has the added effect of minimizing the standard error of the Q-value across seeds. With naïve mixing, on the other hand, the Q-values are inconsistent throughout training and, contrary to pendulum, they end up exploding by the end of training. These experiments indicate positive, stabilizing impact of Adaptive N Steep on the learned Q-values during training, answering question (2).
across training. To answer question (3) (What is the influence of our method on training stability?), we plot the evaluation performance over the course of training averaged over all discretizations in each environment. The results for Pendulum, For Kitchen and Pendulum, reward corresponds to task success
Performance
Worad Door, and Kitchen
tasks are visualized in Figure so we plot the average reward. For the Door task in Meta-World, to better reflect the task, we plot the maximum success attained in the evaluation trajectory. We observe a significant difference in performance and training stability in Pendulum and Kitchen when comparing Adaptive N- and naive mixing. In particular, the Kitchen task indicates that naive mixing experiences substantial instability in training, where around 200 steps into the training the performance goes down. This showcases that the performance is unstable even when a reasonable policy can be learned at some point during training. The performance plot for the Door task shows comparable performance when all discretizations are averaged together, however, we refer back to Table 1 to show that certain discretizations (i.e. 5, 1) perform significantly worse than others compared to the Adaptive N- Step strategy.
Comparison to individual training. Following our motivation for this work, we investigate ques- tion (4), whether mixing multiple discretizations with our method outperforms training on a single discretization. In mixing diverse data sources together we hope to leverage the data from each source
7

---

Door
Olthhen
00500
Vix 5EE3
Adagttro H. trane
60
1.25
1,02
Miac W-Sopp
CAt the w-ttep Durrd
40003
Novi six rg
0.79
Indissdual Thainnn
0001
100
213
300
S20
100
2003
100
4001
500
100
XCD
100
0
220
200
300
400
500
000
700
500
Updatt
Figure 6: Average performance over the course of training. Without Adaptive N- N-etep returns, naive mixing can suffer from instability. Even though Max N-Step uses a larger value of N for most discretizations, Adaptive N- N-Step still converges to high average reward more quickly than Max N on Pendulum and attains a higher final average performance on Kitchen.
to learn a model that performs better than a set independently trained models. We train separate policies the dataset for each discretization and present the averaged performance in Fig. 6 as Individual Training. The results clearly indicate that training individually on a single discretization underperforms relative to training with Adaptive N N-Step returns across discretizations.
Comparison to max-N step. We study question (5) to understand the importance of synchro- nization of the update stride of the Q-values. Another explanation of the performance gain from Adaptive N- N-Step is simply that N N-Step returns a good design choice for reducing variance across discretizations. To explore this possibility, create additional baseline, Max N-Step Returns, where we use the largest adapted for all discretizations present our dataset. We modify the Q-target calculation follows:
N
Nmax
minste 5l
N Niaa
Que
Vurget
r' acte) + Nuuuxx ( ONaus)
In Fig. 6, see that Adaptive N-Step returns converges high average reward more quickly than Max N-Step in the Pendulum environment. This is surprising because Max N uses larger value of N for öt .101 and 81 .005 and therefore could propagate reward more quickly than Adaptive N. In the Door Open task, Max N performs similarly to Adaptive N across discretizations. For FrankaKitchen, Adaptive N-Step returns exceeds the final average performance of Max N-Step returns.
7 Limitations
Although our adaptive N-step method works reliably for discretizations where the step sizes be aligned, it is unclear how well adaptive N could work when the frequencies do not have small common multiple. In the kitchen setting, we saw that the frequeneies do not need to be multiples of one another to reap the benefits of adaptive N-step returns, but this strategy could become less viable the least common multiple between frequencies grows. Other strategies, such as sampling different N with probability proportional to the fractional difference of the two frequencies may need to be adopted. Another limitation of this work is that the discretization of each trajectory is assumed to be fixed and given. If the frequency of observations varies within trajectory, the value of may need to be predicted from observations.
Conclusion
In this work, we studied the problem of mixing data from different discretizations in offline RL and presented simple and effective approach for this problem that improves final performance across three simulated control tasks. We found that our Adaptive N-Step returns method counteracts varying rates value propagation present when mixing data naively. While our approach provides a step towards offline RL methods that can consume data coming from diverse sources, more future work is needed address training offline RI. agents on other sources of variation that are commonly present in robotics data.

---

Acknowledgments
We thank Google for their support and anonymous reviewers for their feedback. We're grateful to Evan Liu and Kyle Hsu for reviewing drafts of our submission and Annie Xie for providing references to implementations. Kaylee Burns is supported by an NSF fellowship. Chelsea Finn is CIFAR fellow. This work was supported by ONR grant N00014-22-1-2621.
References
[1] C. Tallec, L. Blier, and Y. Ollivier. Making deep q-learning methods robust to time discretiza- tion. In International Conference Machine Learning, pages 6096-6104 PMLR, 2019.
[2] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.048033 2018.
[4] P. Kidger, J. Morrill, J. Foster, and T. Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696-6707, 2020.
[5] A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in Neural Information Processing Systems, 33:1474-1487, 2020.
[6] C. Watkins. Learning from delayed rewards. 01 1989.
[7] J. Peng and R. Williams, Incremental multi-step q-learming. Machine Learning, 22, 09 1998.
[8] Y. Rubanova, R. T. Chen, and D. K. Duvenaud. Latent ordinary differential equations for imaeel time series. Advances in neural information processing systems, 32, 2019.
[9] Singh, M. Ramirez, Varley, A. Zeng, and V. Sindhwani. Multiscale sensor fusion and continuous control with neural cdes. arXiv preprint arXiv:2203.087155 2022.
[10] S. Bahl, M. Mukadam, A. Gupta, and D. Pathak. Neural dynamic policies for end-to-end sensorimotor learning. In NeurlPS, 2020.
[11] R. Munos and P. Bourgine. Reinforcement learning for continuous stochastic control problems. Advances in neural information processing systems, 10, 1997.
[12] K. Doya. Reinforcement learning in continuous time and space. Neural computation, 12(1): 219 119-245, 2000.
[13] J. Kim and Yang. Hamiton--jachi--ellnn equations for q-learning in continuous time. In Learning for Dynamics and Control, pages 739-748. PMLR, 2020.
[14] Xiao, F. Jang, D. Kalashnikov, S. Levine, J. Ibarz, K. Hausman, and Herzog. Thinking while moving: Deep reinforcement learning with concurrent control. International Conference on Learning Representations, 2020.
[15] Т. Ni and E. Jang. Continuous control on time. In ICLR 2022 Workshop on Generaliz- able Policy Learning in Physical World, 2022. URL https httppp////eeerrreeeeeeee/ftror id-BtbG3NT4y-c.
[16] D. Ernst, P. Geurts, and I.. Wehenkel. Tree-hased batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
[17] M. Riedmiller. Neural fitted q iteration-fisst experiences with data efficient neural rein- forcement learning method. European Conference on Machine Learning, pages 317-328. Springer, 2005.

---

[18] S. Langee T. Gabel, and M. A. Riedmiller. Batch reinforcement learning. In Reinforcement Learning, volume 12. Springer. 2012.
[19] S. Levine, A. Kumar, G. Tucker, and Fu. Offline reinforcement learning: Tutorial, review and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[20] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pages 651-673. PMLR, 2018.
[21] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn. Offline reinforcement learning from images with latent space models. Learning for Decision Making and Control (L4DC), 2021.
[22] A. Singh, A. Yu, J. Yang, J. Zhang, A. Kumar, and Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint arXiv:2010.14500, 2020.
[23] A. Kumar, A. Singh, S. Tian, C. Finn, and S. Levine. A workflow for offline model-free robotic reinforcement learning. arXiv preprint arXiv:2109. 10813, 2021.
[24] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. arXiv preprint arXiv: htff::/8//.229000 2018.
[25] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offiin reinforcement learning. arXiv preprint 1rXiv:1911.1133 2019.
[26] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11761-11771, 2019.
[27] W. Zhou, S. Bajracharya, and D. Held. Plas: Latent action space for offline reinforcement learning. arXiv preprint arXiv:2011.072.1 2020.
[28] S. K. S. Ghasemipour; D. Schuurmans, and S. Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pages 3682-3691. PMLR, 2021.
[29] Fujimoto and S. S. Gu. minimalist approach to offline reinforcement learning. arXiv preprint 11X.:22006.06800, 2021.
[30] I. Kostrikov, Fergus, J. Tompson, and O. Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774-5783. PMLR, 2021.
[31] W. Goo and S. Niekum. You only evaluate once: simple baseline algorithm for offline rl. In Conference on Robot Learning, pages 1543-1553. PMLR, 2022.
[32] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning. Advances in Neural Information Processing Systems, 33:1 11991, 2020.
[33] R. Agarwal, D. Schuurmans, and M. Norouzi. An optimistic perspective on offline reinforce- ment learning. In International Conference on Machine Learning, pages 104-114. PMIR, 2020.
[34] Y. Wu. S. Zhai, N. Srivastava, Susskind, J. Zhang, R. Salakhutdinov, and H. Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140, 2021.
[35] Yu, G. Thomas, L. Yu, S. Ermon, Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. arXiv preprint 1xXiv:2005.132399 2020.
[36] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline rein- forcement learning. arXiv preprint htt.::200000095550.. 2020.
[37] Swazinna, S. Udluft, and Runkler. Overcoming model bias for robust offline deep rein- forcement learning, arXiv preprint arXiv:2008.0055: 2020.
10

---

[38] B.-J. Lee, J. Lee, and K.-E. Kim. Representation balancing offline model-based reinforcement learning. In International Conference on Learning Representations, 2021. URL https://
Spenreview.net/Eourme heet/forrmmmiddQlllSt.....
[39] T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn. Combo: Conservative offline model-based policy optimization. arXiv preprint arXiv:2102.08363, 2021.
[40] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint 1606.010 2016.
[41] Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094-1100, PMLR, 2020.
[42] A. Gupta, Kumar, C. Lynch, Levine, and K. Hausman. Relay policy learning: Solving long horizon tasks via imitation and reinforcement learning. Conference on Robot Learning (CoRL), 2019.
[43] Fu, A. Kumar, O. Nachum, G. Tucker, and Levine D4rl: Datasets for deep data-driven reinforcement learning, 2020.
[44] X. Geng. Cql. https:// com/young gerroungggeng/ooe 2022.


Reviews:
[{"review_id": "ZFmJSyN4Jgir", "reviewer": "Reviewer_Z97e", "summary": "This work investigates the problem of offline learning with data collected by different frequencies of control.\nThe major contributions are:\n1. Identifying that directly learning on the mixed data with different control frequencies can downgrade the performance.\n2. The authors propose the adaptive N-step update, which adjust the N in N-step TD as well as the discounting.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "***Strengths***:\n1. Offline RL at different frequencies is a relatively overlooked but important area of research. This work confirms the existence of learning challenge under this regime.\n2. The proposed method is reasonable, by adapting the N in the N-step TD update, data with different frequencies align better.\n\n***Weakness***:\nIn general, I think the empirical evaluation is not comprehensive enough to convince me that the proposed problem is universal. Adding more experiments can significantly improve the paper's credibility. Other than this:\n1. From the empirical result, it seems the proposed method only works for coarse discretization. So if the data are collected with the same frequency, the proposed method might perform worse than standard offline RL?\n2. From my understanding, the proposed method essentially adapts the more on data with more fine-grained discretization. But this usually introduce higher variance as well. I wonder if this is the main problem.\n3. As the authors mentioned, the proposed method relies on having a least common multiple for all frequencies. But this assumption is too specific and unlikely holds in practice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think the proposed problem is overlooked but practical. The proposed solution, though simple, is reasonable from my opinion.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Please clarify and address my concerns in the weakness section.\n\nIt is said that the authors use door-open from meta-world because it is a straightforward manipulation task. I do not think this is convincing because almost all tasks in meta-world are not considered very hard or long horizon. Personally I find the proposed problem and solution interesting. But the empirical evaluation is the weakest part of this paper at its current form. I think adding experiments on more environments are largely improve its credibility.\n\n"}}, {"review_id": "Q2zBCxluyJu", "reviewer": "Reviewer_Avb9", "summary": "The paper presents a method for offline reinforcement learning (RL) from heterogeneous datasets, where the trajectories are collected at different control frequencies. The paper identifies a key issue that the rate of value propagation via dynamic programming for data collected with different control frequencies can vary significantly, causing off-the-shelf offline RL algorithms to be unstable in practice. The authors propose a method to balance the rate of Q-value propagation by scaling the N-step returns by the discretization timestep. Experiments on three simulated control tasks demonstrate that this simple modification can stabilize offline reinforcement learning and lead to superior performance than naively using data from different discretizations or using only a subset of data with a fixed control frequency.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is well motivated and the writing is clear and concise. The proposed algorithm is easy to understand, and the toy experiments on grid world and simple pendulum clearly illustrate the issues with Q-value propagation at different time frequencies. The experimental results demonstrate how the proposed method can make Q-function updates more consistent across different discretizations as well as lead to superior performance and stable learning as compared to naive mixing or using data only from a single discretization.\n\n**Weaknesses**\n\nThe major concerns center around the lack of exhaustive ablations and strong baselines in the experimental section. The method is tested on a small set of hand selected discretizations and an exhaustive sweep over the control timesteps is missing. Since the method is only tested on discretizations that were present in the training data, it is unclear if the learned Q-function will generalize to time discretizations that were not seen during training, which is likely to occur in real-world robotics. The current experiments are limited to only 3 simulated tasks which also leaves the question unanswered if this would transfer well to real-world robots. \nFurthermore, the baselines considered do not include other methods that attempt to make Q-learning robust to the time discretization [1]. Even in the current results, naive mixing outperforms the proposed method on one of the tasks, and without proper ablations, it is difficult to judge the true impact of adaptive N-step updates. \n\nOn a more technical level, the use of N-step returns would make the algorithm on-policy which could break Q-learning based algorithms. It would be helpful if the authors can comment on why this is not the case in their implementation. \n\n[1] Tallec et. al; 2019, Making Deep Q-learning Methods Robust to Time Discretization\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The reviewer’s recommendation for a weak reject is based on insufficient experimental evidence for the efficacy of the approach as well as unclear impact on real-world robotics. While the authors have identified an important sub-problem in offline reinforcement learning from heterogeneous datasets, in the current form the paper is not fit for publication. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "I will consider changing the recommendation if the authors can address the following issues:\n\n1. Provide results of exhaustive ablations over the time discretizations and empirical evidence of generalization to discretizations not seen during training.\n \n2. Compare against stronger baselines such as [1].\n\n3. Although this might be hard to achieve given the limited time, demonstration of the method on a real-world robotics task will be convincing for a strong accept.  \n"}}, {"review_id": "Eqwx0ytOnZM", "reviewer": "Reviewer_1JmA", "summary": "This paper studies the challenges in RL that come from using offline data collected at different control frequencies. It shows that simplifying running existing algorithms (CQL) on such data (called “naive mixing”) leads to diminished performance. It hypothesizes that the issue is from the inconsistent Q-value targets caused by the mixed control frequencies. They illustrated the hypothesis in a toy example and proposed an adaptive N-step return method to address the issue. The hypothesis and the efficacy of this proposed method are verified in benchmark environments. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The hypothesis is clearly stated and verified in the experiments. \n2. The method seems simple and effective.\n3. The topic of study (offline RL with data at mixed control freq)  is new to my best knowledge \n\nWeaknesses;\n1. It used a modified Pendulum with sparse rewards. It would be nice to see results in the original env especially since it seems from door-open task that dense reward may be “harder” for the method.\n2. No theoretical analysis\n\nMinor:  \n1. I’m not sure if I understood the state aliasing in Fig. 2. \nFor instance, in the bottom row k=1, why would the unvisited state (second to the left) have the same value as the adjacent state rather than just 0? It’s perhaps a choice for better illustration but I wanted to double-check.\n2. Typos\n  - line 283: Fig. 6.2 -> 7\n  - line 177,178: the top row should have 2 transitions and bottom row should be 1. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "It is a good paper overall. The problem setting and the hypothesis are clearly stated. The algorithm design and the experiments are nicely set up to verify the hypothesis.\nIt would make it a stronger paper to include theoretical analysis.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "1. Could the authors add results for the original pendulum?\n2. Could the authors clarify the state aliasing?\n", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "Cyc9U7eLdYB", "reviewer": "Reviewer_W1tF", "summary": "The paper considers a novel setting of offline reinforcement learning from datasets collected at different control frequencies. This setting presents challenges for approximate dynamic programming since the value functions are different for each control frequency. A naïve mixing of datasets with different control frequencies is shown to cause unstable training in the pendulum control task. The key idea in the paper is to use adaptive N-step value targets where N is determined based on the control frequency of the sample. This is shown to stabilize training in three different tasks (pendulum, door, and kitchen), making effective use of the heterogeneous datasets of different control frequencies.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n* The paper explores a novel setting of offline RL from trajectories collected at different control frequencies. This is a realistic setting in robot learning and an effective algorithm could improve the possibilities of offline RL. \n\n* The proposed approach is simple to implement and understand. It significantly outperforms the naïve mixing strategy in the tested benchmark tasks. \n \n\nWeaknesses: \n\n* I believe the key problem in the considered setting of RL at different frequencies is that the optimal value function is different for policies acting at different frequencies. For example, a policy acting at a higher frequency could perform much better in a task that requires such quickly changing actions. It is not clear how the proposed approach solves this problem.  \n\n* The proposed solution consists of the adaptive N-step returns and also the conditioning of the actor-critic networks on the $\\delta t$ value. The influence of the latter on the agent performance is not clear. If the problem in the offline RL at multiple frequencies setting is that value propagation occurs at different rates, isn’t it theoretically possible that this doesn’t cause any training instabilities since the critic network could predict the right value target given a particular control frequency? For example, in Figure 2, it is argued that the Q network needs to regress to different targets from the same states but if the Q network is conditioned on $\\delta t$ then it is correctly trained to predict different targets based on the control frequency. An ablation study with naïve mixing and $\\delta t$-conditioned actor-critic networks would be interesting. \n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The problem setting considered in the paper is novel and the proposed solution is shown to be effective. However, the overall influence of the proposed adaptive N-steps is not clear, and more ablation studies have to be performed to better support the contributions of the paper. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "See weaknesses listed above. \n\n* Experiments on tasks with different optimal values for policies with different control frequencies would provide more intuition about the proposed approach. \n\n* Tasks with very small $\\delta t$ are known to cause problems for RL [1] and using DAU instead of SAC as the basis of CQL could help improve the performance in these settings.\n\n* Results on more tasks such as the D4RL benchmark would further support the contributions of the paper. \n\n* Figure 6.2 should be Figure 7 in L283."}}]

Author rebuttal:

Decision: Accept (poster)