Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning
Authors: Zihan Ding
Chi Jin
Venue: ICLR 2024 poster (2024)
Keywords: Generative Model
Expressiveness
Deep Reinforcement Learning
Primary area: reinforcement learning

Abstract: Score-based generative models like the diffusion model have been testified to be effective in modeling multi-modal data from image generation to reinforcement learning (RL). However, the inference process of diffusion model can be slow, which hinders its usage in RL with iterative sampling. We propose to apply the consistency model as an efficient yet expressive policy representation, namely consistency policy, with an actor-critic style algorithm for three typical RL settings: offline, offline-to-online and online. For offline RL, we demonstrate the expressiveness of generative models as policies from multi-modal data. For offline-to-online RL, the consistency policy is shown to be more computational efficient than diffusion policy, with a comparable performance. For online RL, the consistency policy demonstrates significant speedup and even higher average performances than the diffusion policy.

Published as a conference paper at ICLR 2024
CONSISTENCY MODELS AS A RICH AND EFFICIENT POLICY CLASS FOR REINFORCEMENT LEARNING
Zihan Ding, Chi Jin Department of Electrical and Computer Engineering Princeton University {zihand, @pr inceton.eee edu
ABSTRACT
Score-based generative models like the diffusion model have been testified be effective in modeling multi-modal data from image generation to reinforcement learning (RL). However, the inference process of diffusion model can be slow, which hinders its usage in RL with iterative sampling. We propose to apply the consistency model as an efficient yet expressive policy representation, namely consistency policy, with an actor-critic style algorithm for three typical RL settings: offline, offliee-to-online and online. For offline RL, we demonstrate the expressiveness generative models as policies from multi-modal data. For offline-to-online RL, the consistency policy is shown to be more computational efficient than diffusson policy, with comparable performance. For online RL, the consistency policy demonstrates significant speedup and even higher average performances than the diffusion policy.
INTRODUCTION
Parameterized policy representation is important component for policy- deep reinforcement learning (DRL) (Sutton & Barto, 2018; Arulkumaran et al., 2017; Dong et al., 2020). Prior works have developed a variety of policy parameterization methods. For discrete action space, existing policy pa- rameterization includes Softmax action preferences (Sutton & Barto, 2018), Gumbel-Softmax for cate- gorical distributions (Jang al., 2016), decision trees (Frosst & Hinton, 2017: Ding et al., 2020), etc. For continuous action space, the most typical choice is unimodal Gaussian distribution. However, in practice the demonstration dataset often encompasses samples from mixture of behavior policies. To capture the multi multi-modality in data distribution, Gaussian mixture model (GMM) (Jacobs et al., 1991; Ren et al., 2021), variational auto-encoders ( (VAE) (Kingma & Welling, 2013; Kumar et al., 2019), denoising diffu- sion probabilistic model (DDPM) (Ho et al., 2020; Song et al., 2020; Wang et al., 2022; Chi et al., 2023; Hansen-Estruce et al., 2023; Venkatraman et al., 2023) are adopted as policy representation.
The desiderata for policy representation in DRL includes: 1. The strong expressiveness of the function class is found to be critical for modeling multi-modal data distribution offline RL (Wang et al., 2022) or imitation learning (IL) (Chi et al., 2023): 2. Differentiability of the model is usually required for ease of optimization with stochastic gradient descent; 3. Computational and time efficiency for sampling can be essential for RL agents learning from interactions with environments. Previous works with action diffusion models (i.e., diffusion policy) testify the expressiveness of diffusion models for multi-modal action distributions (Wang et al., 2022: Chi et al., 2023; Hansen- -attunhEstruch et al., 2023; Janner et al., 2022; Ajay et al., 2022). Although GMM and VAE also capture multi-modalityy diffusion models with large sampling steps are found be more expressive for IL and offline RL scenarios (Wang et al., 2022; Chi et al., 2023). However, it is known that the diffusion model with progressive denoising over large number of steps can lead to slow sampling speed. The action inference can be a critical bottleneck for online RI heavily depends on sampling from environments. direct usage of diffusion policies for online settings with policy gradient for optimization requires backpropagating through the diffusion networks for the number of sampling steps, which is not scalable for its large time consumption and memory occupancy. Consistency models (Song et al., 2023) based on probability flow ordinary differential equation (ODE) is proposed as rescue with comparable performances as diffusion models but much less computational time, which allows few-step generation process thus significantly reduce the time consumption at inference stage.
This paper takes the first step adapting the consistency model-as an expressive yet efficient generative model-as policy representation for DRI. The consistency policy is embedded in both behavioral

---

Published as a conference paper at ICLR 2024
cloning (BC) method and an actor-critic (AC) algorithm, namely Consistency- and Consistency-AAe Experimental evaluation includes three typical RL settings: offline, model and consistency model-are thoroughly compared in all three settings on D4RI. dataset (Fu et al., 2020). For offline RI., we propose a new loss scaling for stabilizing the training process of consistency policy with policy regularization, and demonstrate the expressiveness of two generative policy models. This is illustrated by showing BC with an expressive model like diffusion or consistency provides fairly good policies outperforming some previous offline RL methods. The performances are further improved by leveraging the actor-critic style algorithm with necessary policy regularization to avoid generating process of the consistency policy not only helps to reduce the training time, e.g. by 43% for offline BC, but more importantly, improves the time efficiency for online interaction in the environments by accelerating action inference. For dataset and online setting with learning from scratch, the consistency policy shows comparable or even higher performances than the diffusion policy in some tasks, using significantly shorter wall-clock time for training and inference. The source code is available'.
and online.
Policies with two generative models-diffusse
-itto---istibuutionn actions. The fast sampling
setting with initialized models trained on offline
RELATED WORKS
Offline and Offline-to-nniin RL. The offline RL is the problem of policy optimization with a fixed dataset. is well known for suffering from the value overestimation problem for -fi-t-i-iitiittioon states and actions from the dataset. Existing methods for solving this issue fall into categories of (1) explicitly constraining the learning policy with offline data using batch constraining, behavior cloning (BC) divergence constraints (e.g., Kullback-Leibler maximum mean discrepancy). including algorithms Caath-Coostrainedd deep Q-learning (BCQ) (Fujimoto et al., 2019), TD3+BC (Fujimoto & Gu, 2021), Onestep RL (Brandfonbrener et al., 2021), Advantage Weighted Actor-Critic (AWAC) (Nair et al., 2020), Bootstrapping Error Accumulation Reduction (BEAR) (Kumar et al., 2019), BRAC (Wu et al., 2019), Diffusion Q-learning (Diffusion QL) (Wang al., 2022), Extreme Q-learning (X-QL) (Garg et al.. 2023) and Actor-Restricted Q-learning (ARQ) (Goo & Niekum, 2022), or (2) implicit regularization with pessimistic value estimation, like Conservative Q-learning (CQL) (Kumar et al., 2020), Random Ensemble Mixture (REM) (Agarwal et al., 2020). Implicit Q-learning (IQL) (Kostrikov al., 2021), Implicit Diffusion Q-learning (IDQL) Flansen-Estruce et al. (2023), Model-based Offline Policy Optimization (MOPO) (Yu et al., 2020), etc. MoRel (Kidambi et al., 2020) is a model- offline RL. algorithm constructing pessimistic MDP for learning conservative policies, which does not clearly fall into above two categories. Offline-to-oniing RL usually suffers from a catastrophic degraded performance at initial online training stage, due to the distribution shift of training samples. Previous research has studied online fine-tuning with offline data or pre- mre-traine policies, including Hybrid Q learning (Song et al., 2022), RLPD (Ball et al., 2023), Cal-QL (Nakamoto et al., 2023), Action-free Guide (Zhu et al., 2023), Actor-Critic Alignment (ACA) (Yu& & Zhang, 2023) and Lee et al. (2022).
Score-based Generative Model for RL. For policy representation in RI., recent work also uses Denoising Diffusion Probabilistic Models (DDPM) (Ho et al., 2020; Song et al., 2020), which we loosely refer to as the diffusion model (original diffusion model traces back to Sohl-Dickstein et al. (2015)) in this paper, to capture the multi- modal distributions in offline dataset. Diffusion QL (Wang et al., 2022) uses diffusion model for policy representation in the Q-learning+BC approach. Implicit Diffusion Q-learning (IDQL) (Hansen- -sttrnch et al., 2023) is a variant of IQL using diffusion policy. Diffusion policies (Chi et al., 2023) applies diffusion models for policy representation under imitation learning settings in robotics domain. Diffuser (Janner et al., 2022) and Decision Diffuser (Ajay et al., 2022) combines decision transformer architecture with diffusion models for model-based reinforcement learning from offline dataset. Diffusion policies are also used for goal-conditioned imitation learning (Reuss et al., 2023) and human behavior imitation (Pearce al., 2023). Q-guided Policy Optimization (QGPO) (Lu et al., 2023) proposes new formulation for intermediate guidance in diffusion sampling process. Latent
Q-Learning (LDCQ) (Venkatraman et al., 2023) proposes apply latent diffusion model with a batch-constrainee Q value to handle the stitching issue and the extrapolation errors for offline dataset.
http:///gghha..oommuummmmeccoeee Model For Reinforcement Learning
2

---

Published as a conference paper at ICLR 2024
3 PRELIMINARIES
3.1 OFFLINE AND ONLINE RI.
For RL, we define Markov decision process (S, .Po.7). where S is the state space, the action space, R(s.a): S A - R is the reward function, T(s' (ss)ssa)):
:..a):Sx : A Pr((S) is the stochastic transition function, Po (5p) SS - Pr(S) is the initial state distribution, and E [0.1] is the discount factor for value estimation. A stochastic policy x (a]s)):
Pr(AA) determines the action a A for the agent to take given its current state E E, and the optimization objective for the policy is its discounted cumula- tive reward: E,, |o (5r((s. For offline RL, there exist dataset D = {ss,,,r, ,done)] collected with some behavior policies T², and the current policy iT is set be optimized with D. For online RL, the agent is allowed collect samples through interacting with the environment to compose an online training dataset D for optimizing its policy. We consider parameterized policy representation as Tg.
3.2 CONSISTENCY MODEL
The diffusion model (Ho et el., 2020; Song et al., 2020) solves the multi modal distribution matching problem with a stochastic differential equation (SDE), while the consistency model (Song et al., 2023) solves an equivalent probability flow ordinary differential equation (ODE): dx_. = -VVlogp; (x) with dr
(x) = Puam (x) & (8(00,711 for time period 7 E [0,7], where Paata (x) is the data distribution. The reverse process along the solution trajectory ($t)re(<,T] this ODE is the data generation process from initial random samples X > (N(,, with e as a small constant close to for handling numerical problem at the boundary. For speeding up the sampling process from a diffusion model, consistency model shrinks the required number of sampling steps to a much smaller value than the diffusion model, without hurting the model generation performance much. Specifically, it approximates parameterized consistency function fo: (T) Xx,, which is defined as a map from the noisy sample x, at step back to the original sample x,, instead of applying seep-by-step denoising function po (x |xx) as the reverse diffusion process in diffusion model. The training and inference details of consistency model refer to Appendix B. For modeling the conditional distribution with condition variable c, the consistency function is changed to be fo (c.,x,,)) ,,),, which is sightly different from original consistency model.
4 CONSISTENCY MODEL AS RL POLICY
The consistency model as policy representation in RL can be formulated in the following way. To map the consistency model to policy in MDP, set:
c s.
x x a, Pesta _ pp)ss),
me(s) è
(I)
where PD (als) (s) is the action-state conditional distribution from offline dataset D.
Consistency Action Inference. By setting the condition variable c as state and generated variable action a, the consistency function fe can be used for generating actions from states following the conditional distribution of the dataset, i.e., a behavior RI. policy. The parameterized policy my
is defined implicitly in terms of fo, with which action conditioned on state s can be generated following the Cons i stteecceenneeeeee Satrcy__nffeneeeeee as Alg. 1 with predetermined (T, In E [N]) sequence. During the inference process, trained consistency model fe fe(s,ā,,,,n, .n.) iteratively predicts denoised samples from the noisy inputs å,, = a / 72 ²22 along the probability flow ODE trajectory at
step E [N]. with Gaussian noise : ~ (O.I)) [T E [M] is a sub-sequence of time points certain time period [c, with 71 = = T. For inference, the sub-sequence is linspace of [c,7] with (N - 1) sub-intervals. A single-step version of
www.eittteee.eeeeeeeeeeeeeeeeeee Infference can be achieved by just set (T,, ( = [e. Notice that 7 here is the time horizon for denoising process in the consistency model instead of the episode length of the sample trajectory.
Consistency Behavior Cloning. With the offline dataset D, the conditional consistency model as policy can be trained with loss by adapting the original (Song et al., 2023):
(0)= = -0))=EEn,(,,----- (GG
(.
(s,ar.::n))
(2)
where X(-) is step-dependent weight function, dr,, a+ T and d(-,-) is the distance metric. for is exponential moving average of fe for stabilizing the target estimation in training. In classical actor-critic algorithm, there exists the same delayed update of the policy network कØr (i.e. for ) for estimating target Q- Q-values, which is set to coincide with the target in estimating the consistency loss. The setting for Tn is detailed in Appendix B. Pseudo-code of Consistency BC refers to Alg. 2.
3

---

Published as a conference paper at ICLR 2024
Algorithm 1 Consistency Action Inference
Algorithm 3 Offline Consistency Actor- Critic
Input 6, (TT,
Input offline dataset 'D
Initial a f frr
Initialize consistency policy network Tg. critic
No==N-1to000 do
networks 1
âr, a 2,,
Initialize target network parameters: (T t 0,
at (o
of f- [-01.0] f-Ø2
end for return a
for policy training iterations 1: L.K,., do
Sample minibatch B ((s,,,,,,,
% Q-value Update
Algorithm 2 Consistency Behavior Cloning
Update Q00 with Eq. 3;
Input offline dataset !
% Policy Update
Initialize consistency policy Te. target (T + for iterations k = I....KKK do
Update policy ma (with model fa) via Eq. 4; % Target Update
Update policy ze (with model fo) using loss
Update target: OT t- aBT + + - (1-a)0,0] +
(0) as Eq. 2:
oΦ +(1 (1-0)) end for return ne. Qo.
Update target: 0T + a00 a1--a0 end for
Consistency Actor-Critic. As as estimation of the state-action value current policy, the parameterized (s(aa) function can be learned with the double Q-learning loss (Fujimoto et al., 2018) with batched data CD:
C ( = (s... ~ not (160) ( min Qsz(s',a)) Qa, -q, (s.aa) [ |
(3)
(1,2)2)
with QΦt as a delayed update of Qa .E (1,2) for stabilizing training.
The regularized policy πy on offline dataset learned with a combination of policy gradient through maximizing the expected (s,aa function and behavior cloning regularization with consistency
loss (0):
( [())=L((0)++<(c()))
(4)
where - [Q,(s,
(5)
where a a To is action inference from the consistency policy as Alg. 1. It can be noticed that the actions generated with N denoising steps will produce the policy gradients through the (s(s,)) a) in above equation, thus it also backpropagates through fo for N times in the gradient descent procedure, which can lead to additional time consumption apart from the multi-step model inference. Therefore, reducing the denoising steps N can be critical for the speed of this type of models as RL policies. The consistency actor-critic (Consistency- algorithm is provided in pseudo-code Alg. 3.
Loss Scaling. The consisteney loss Eq. 2 matches the denoised predictions from two consecutive timesteps 7n and T Due to the usage of (A) schedule (detailed in Appendix B), their difference [T++1 Tx decreases the training iteration i increases (thus N N(k) also increases), which allows the consistency model to have a coarse-to-
matching process across different time scales. This also leads decreasing loss value [(0:k) as k increases from
K since the predictions from smaller time intervals are easier to match. Actually, the loss (0) changes drastically across several magnitudes within an epoch, which leads to severe imbalance with the second loss term (,) in Eq. The coefficient 7) is constant hyperparameter independent of k, so it cannot help alleviate this issue. Although original consistency model applies A(Tx)) = 1 image generation, we empirically find that in offline RL this imbalance of two loss terms can hurt the effect of policy regularization in some tasks, as evidenced by ablation studies in Sec. 5.2. To solve this issue, we propose k-dependent weighting mechanism balance the values of two loss terms. This is found to improve the performances of this policy regularization method with consistency model on offline RL. Specifically, X(-) in Eq. (k)) where & is set according to tasks (or absorbed in r)). The denominator captures the loss scale at iteration & conveniently.
is chosen to be: X(Tn,Ta+13kh
5 EXPERIMENTAL EVALUATION
To evaluate the expressiveness and computational efficiency of the proposed consistency policy and cor- responding algorithms, conduct experiments on four task suites (Gym, AntMaze, Adroit, Kitchen) in
4

---

Published as a conference paper at ICLR 2024
D4RL benchmarks under three canonical RL settings: offline (Sec. 5.1, Sec. 5.2), offline-to-online and online (Sec. 5.3). is known that the D4RL offline dataset can exhibit monlti- since the samples may be collected with mixture of polices or along various sub-optimal trajectories, which makes the expressiveness of policy representation critical (Fu et al., 2020; Wang et al., 2022). For offline RI., the Diffusion-BC) generative models as policies are evaluated with both behavior cloning (Consistency-Bi and actor-critic type (Consistency-AC. Diffusion -L) algorithms, in terms of both performances and computational time. The Diffusion- QL is also an actor-critic algorithm though with name QL. Variants of Consistency- ACC are compared as ablation studies and the best performances are reported. For offline- to-online and online RL settings, the learning curves and final results are compared for different methods. For evaluation, each model is evaluated over 10 episodes for Gym tasks and 100 episodes for other tasks, following the settings in previous work (Wang et al., 2022). By default, the consistency policy applies the number of denoising steps N 2 with a saturated performances on most of D4RL tasks, while diffusion policy uses N 5 (Wang et al., 2022). Effects of different choices of N are discussed in Sec. 5.2.
5.1 OFFLINE RL: BEHAVIOR CLONING WITH EXPRESSIVE POLICY REPRESENTATION
Empirical finding 1:
By behavior cloning alone (without any RL component), using an expressive policy representation with multii-modality like the consistency or diffusion model achieves performances comparable to many existing popular offline RL methods. Learning consistency policy requires much less computation than learning diffusion policy.
The proposed method Consistency-BC follows Alg. 2 with consistency policy for behavior cloning, and Diffusion-BC is by replacing the policy representation with a diffusion model and replace the policy loss with the diffusion model training loss La(O) as specified in paper (Wang et al., 2022). Results for classic BC with Gaussian policies and previous offline RL baselines. including AWAC (Nair et al., 2020), Diffuser (Janner et al., 2022). MoRel (Kidambi et al., 2020), Onestep RL (Brandfonbrener et al., 2021). TD3+BC (Fujimoto BCQ (Fujimoto et al., 2019), BEAR (Kumar et al. 2019), BRAC (Wu et al., 2019) and REM (Agarwal et al., 2020), are adopted from previous paper (Wang et al., 2022). SAC (Haarnoja et al., 2018) is the
Gu, 2021), Decision Transformer (DT) (Chen et al., 2021),
Table 1: The average scores of vanilla BC (with Gaussian), Consistency- BC, Diffusion BC and several offline RL baselines on D4RL Gym, AntMaze, Adroit, and Kitchen tasks are shown. For Consistency- BC and Diffusion-Bel BC, each cell has two values: one for offline model selection and another (in brackets) for online model selection. Each result is averaged over five random seeds with standard deviations reported. The bold values are the highest among each row.
Gym Tusks
CB
Diffasion
AWAC
Diffuser
MoRel
Onestep RL
TD3-BC
hallcheetall-m
42.6
41:0204466622000
6.4418(66331))
43.5
44.2
42.1
48.4
48 3
42.6
hopper-m
52.9
71.7180(88342..)
65.1458771115.55
57.0
58.5
95,4
59.6
59.3
67.6
walker&d-m
75.3
(6.11..3)
6122:17 (81.31.))
72.4
79.7
77,8
81.8
83.7
74.0
halEcheetab mr
36.6
(44.5.3(45.4400))
40.5
42,2
40.2
38.1
44.6
36.6
hoşper-mr
18.1
(1000.5((000111..0))
67.9122.1 (99.1
37.2
96.8
93.6
97.5
60.9
82.7
walker2d-mr
26.0
02.3+6.7
77.5+4477 (80.8++-3))
27.0
61.2
49.8
49.5
81.8
6666
halkcheetll-
55.2
12.7211 (3.66113.
90.81 (93.5.10.4
42.8
79.8
53.3
93.4
90.7
hoyper-me
52.5
107.6 107.644.33111740..3
55.8
107.2
108.7
103.3
98.0
107.6
walker2l-me
107.5
10.4:07111664077
108 (11002206610055
74.5
108.4
95.6
113,9
110.1
108.1
Average
51.9
(7.7()5.9)
(62.4)
50.1
75.3
72.9
76.1
75.3
74.7
AutMaze Tasks
BC
Consistenc-BB
Diffusion-BB BC
AWAC
BCQ
BEAR
Onestep RL
TD3-BC
DT
antmaaze-a
54.6
71..8..2788.....9
56.7
78.9
73.0
64.3
78.6
59.2
antrazo-sdd
45.6
49.3
55.0
61.9
60.7
71.4
53.0
antmaze-mn mp
0.0
56.8130.1 (71.300171611133 (71661145)
43.4137.8
(38.8.4.)
0.0
0.0
0.0
0.3
10.6
0.0
antmaae-md
31.6::22 (66.0.666)
29.8:36.33
(60.4112.3)
0.0
0.0
antmaze-Ip
(0.22-66(150433.))
14.6=11222 52.84+55.8)
6.7
0.0
0.0
antmaze-ld
(2.8+82(10:844..))
26.6 10.7 (33.0+-822)
1.0
2.2
0.0
0.0
0.0
Average
16.7
44.1 (57.0)
(5.2()3.3)
18.0
23.8
23.7
20.9
27.3
18.7
Adroit Tasks
BC
Consisteney-BC
Diffusion-Be
SAC
BCQ
BEAR
BRAC-P
BRAC-v
REM
pen-human--vl
25.8
52.4: (63.777.))
61.125.5 (66.7 (66.74:4 1.99
4.3
68.9
-1.0
8.1
0.6
3.4
pen-cloned-vl
38.3
33.4+6.00 (51.0)001111)))) (51.9++6.6))
9.. (62.7++6.1)
-0.8
44.0
26,5
1.6
-2.5
-1.0
Average
3
(57.8)
59.4 (64.7)
1.8
$6.5
12.8
4.9
-1.0
2.2
Kitchen Tasks
BC
Consistency-BC
Diffusion-Be BC
SAC
BCQ
BEAR
BRAC-p
BRAC-v
AWR
kitchen-e
16.2255.000001111
76.5 (87 (8763)
15.0
8.1
0.0
0.0
kitchco-p
33.8
22.62.3. (23.812.8)
30.3
18.9
0.0
15.4
kitchen-m
47.5
23.5+1.8
(24.3+
$6.5 (64.7 7+4.6)
2,5
8.1
47.2
0.0
10.6
Average
38.4
(33.0)
61.1
5.8
11.7
20.1
0.0
0.0
8.7
5

---

Published as a conference paper at ICLR 2024
algorithm used for collecting data in D4RL Gym tasks. Detailed hyperparameters for Consistency- and Diffusion-Bi BCC refer to Appendix C.I.
Results from Tab. show the advantage of us- ing multi-modal policy representation for of- fline RL even only with the BC method. For
Training Time opach)
Time per epoch)
reference purpsse, the values in the brackets allow for online evaluation to achieve the best model selection from the set of trained models, which serve as the maximal possible values for the standard offline selection without leverag- ing online evaluation. Compared with vanilla BC using the Gaussian distribution for policies, Consistency- Be with multi colliiimodlltyy outper- forms it on 14/20 tasks, and Diffusion-Bi has better or equivalent performance as BC for 20/20 tasks. Through leveraging multi-modal representation in BC, the improvement of normalized scores averaged over tasks significant, and this is mainly caused by the multi-modality within the offline dataset by mixing over policies. Moreover, compared with previous offline RL baselines, which do not just apply BC, the Consistency-Be and Diffusion- BC show comparable performances, and even superior performances for tasks like mredium-replay-vi walker2d-ne miilkerrd-mediun--ee medium-experr--22 -etpmm---xpp222222 v2 in Gym tasks, most of AntMaze, Adroit and Kitchen tasks. The consistency policy is slightly less expressive than the diffusion policy, which within our expectation due to its heavy reduction on the sampling steps. However, the consistency policy shows higher computational efficiency than diffusion policy as compared Fig. 1, with an average reduction of 42.97% computational time across 20 tasks. Detailed computational time for each task is provided in Appendix C.2 Tab.
28
Consistancyy
Gym lasks
ArtNaa
Tusls
Adralt lasks
Kitchan Tacks
Task Group
Figure I: Average training time (seconds per epoch) for Consistency- BC and Diffusion- BB across tasks.
v2,
walker2d-
5.2 OFFLINE RL: CONSISTENCY ACTOR-CRITIC
Table The performance of Consistency-AC and SOTA baselines on D4RL Gym, AntMaze, Adroit and Kitchen tasks for offline RL setting. For Consistency-AC and Diffusion-QL, each cell has two values: one offline model selection and another (in brackets) for online model selection. The bold values are the highest among each row.
Tasks
CQL
IQL
X-QL
ARQ
IDQL-A
Diffusion-QU QL
Consistency-AC
halfcheetah-m
44.0
47.4
48.3
a: 0.3
51.0
51.1 051.5 1. 0.3)
69.1 0.. 0.8))
hopper-m
58.5
66.3
74.2
61
65.4
90.5 4.6 3.4)
80.7 10.5 (99.7 12.3)
walker2d-m
72.5
78.3
84.2
81 0.7
82.5
87.0 09 (87.3 0.5)
83.1 10.3 (84.1 20.3)
halfchectah-mr
45.5
44.2
45.2
42 0.3
45.9
47.8 = 0.3 (48.3 0.2)
58.7 #3.9 (62.7 =0.6)
hopper-mr
95.0
94.7
100.7
24.2
92.1
101.3 = 0.6 0.))
99.7 (100.4
walker2d-mr
77.2
73.9
82.2
85.1
95.5 (98.0
79.5 (8. 1.5)
halfchectah-me
91.6
86.7
94.2
91 1 0.7
95.9
96.8 0.3 (97.2 1.04)
84.3 4.1 (89.2 3.3)
hopper-me
105.4
91.5
111.2
110
108.6
111.1 1.3 (312.
100.4 (106.0
walker2d-me
108.8
109.6
112.7
109:2...5
112.7
0.3 0.31.2
11..4
(11.6666
Average
77.6
77.0
83.7
76.2
82.1
87.9 (89.3)
85.1 (89.8)
antmaze-u
74.0
87.5
93.8
97 2: 0.8
94.0
93.4 3.4 (96.0 3.3)
75.8 1: 1.6 (85.6
antmaze-ud
84.0
62.2
82.0
62 12.1
80.2
66.2 8.6 (84.0 10.1)
77.6 6.3 (82.4 3.4)
antmaze-mp
61.2
84.5
(79.8 8.7)
56.8 (71.6 6 14.5)
Average
73.1
73.6
83.9
79.7
86.2
78.7 (86.6)
70.1 (79.9)
pen-human-v1
35.2
71.5
45 5.2 (v0)
72.8 9.6 (75.7
63.44 7.7 (67.9 5.3)
1
27.2
37.3
50 + 7.1 (v()
57.3 + 11.9 (60.8 11.88
50.1 2.3
Average
31.2
54.4
47.5
65.1 (68.3)
56.8 (60.8)
kitehen-e
43.8
62.5
82.4
37 14.2
84.0 7.4 (84.5 6.).
51.9 6.7 2.7)
kitchen-p
49.8
46.3
73.7
50
60.5 6.9 (63.7 5.2)
38.2 (39.8
kitchen-m
51.0
51.0
62.5
39 9.4
62.6 = (66.6 3.3)
45.8 2: 1.5 (46.7 0.9)
Average
48.2
53.3
72.9
42.0
69.0 (7166)
45.3 (51.4)
Total Average
66.2
67.4
80.3
72.1 (77.9)

---

Published as a conference paper at ICLR 2024
Empirical finding 2: Replacing diffusion model with consistency model in TD3-BC type algorithum for offline RL will lead to speed up of model training and inference, with slightly worse performances while still outperforming some other baselines.
For offline RL., the proposed method Consistency-AC follows Alg. 3 with consistency model for policy representation, and the consistency policy is embedded in an actor-critic algorithm with BC policy regularization to avoid generating out-of-distribution actions. Results for previous baselines, including CQL (Kumar et al., 2020), IQL (Kostrikov et 2021), I), X X--Q (Garg et al., 2023), ARQ (Goo & Niekum, 2022), IDQL-A (Hansen Haansen-Estruch Estruch et al., 2023) and Diffusion-QL (Wang et al., 2022) are adopted from results reported in corresponding papers. Detailed hyperparameters for Consistency-AS and Diffusion- QL refer to Appendix C.I.
Results from Tab. 2 show the average normalized scores of different methods across five random seeds, with standard deviations reported for Diffusion-QL. and Consistency-AC. The results Tab. are directly comparable with the results in Tab. since they follow the same offline RL setting.
Tab. 2 shows that although Consisteney-ACC achieves a slightly lower average score (72.1) than Diffusion-QL (80.3), it outperforms the other baselines in most of the tasks, like Gym and Adroit. The AntMaze tasks are found to be hard for the Consistency- AC method, we conjecture that this potentially caused by the sparse reward signals (as evidenced by Appendix A Fig. 5) in dataset, which makes the difficulty of -Q-eeanning become more of a bottleneck than modeling the multi-moda distributions with behavior cloning. The conservativeness of the Q-value estimation might be important but orthogonal to the proposed consistency policy. Considering the reduction of denoising steps in the training and inference stages of Consistency-ACe it can be regarded as a trade-off between the computational efficiency and the approximation accuracy of multi-modal distribution, which will be discussed as following.
Method
N
Training Time (s per epoch)
Inference Time (ms sample)
Avg. Norm Score
50
206.44:1 16.
30.65+2.11
20
108.65 2.85
13.0410.90
109 (1LL1119
Diffusion-QU QL
10
76.34=10.74
6.87 +0.55
108.6:0.66 (1)2.5+0.2)
37.06= 19.16
3.76:10.29
108.2 (112331.))
2
31.59= 10.32
1.90+00110
53.6 (103.6
30.23+88775
+.37
2.8 1.3.11111112
50
150.841131.02
06.50411.9
20
76.22.19.99
11.1210.77
101.316..
10
54.044144.33
5.95 5.944
Consistency-Ao AC
10.7911279
3..0
1014
31.9441..50
1.84+00.2 0.22
102.4 (106.22
28.51 1.78
1.23++0.11
6.2+5.4 (19.1 (19.1
Table 3: Comparison of computational time for two methods with different denoising steps N on the task hoopper-eediun--xxperr--- The gray lines apply default N values for two models.
Computational Time. To evaluate the computa- tional efficiency of -onsistency-AC and Diffusion- QL with different denoising steps N, we conduct experiments for evaluating the training and infer- ence time for N E (1,2,5, 22,,,00,200 10,20,50) on the hopper- mediimm-expert-v22 environment. As generative models based on probability flow, both the con- sistency model and the diffusion model require the computational time directly dependent on the number of denoising steps N, and consistency model (Song et al., 2023) by design requires a smaller number of steps for achieving similar gen- erative performances as the diffusion model. The results are summarized in Tab. for both the train- ing time (seconds per epoch) and the inference time (milliseconds per sample) with the model training using different denoising steps N, as well as the average normalized scores for models trained after 2000 epochs with each N. Each cell contains the mean and standard deviation over five random seeds. Consistency-AC saturates its performance with only N N=2 while Diffusion-QL. saturates at N 55 which consumes about 1.786x more training time
Average Score and Training Time Step Number N
100
200
Traning Time per spocin)
Ava. Score
time
150
Sosurates
60
100
40
50
Ofhiates Sing Revort
MMatt Dhanlig Time
0 10a
10' Scale)
Figure 2: The average normalized scores and training time versus N for two models on hopper- medium-expert.
7

---

Published as a conference paper at ICLR 2024
while yielding slightly better performance (1.057x). The "-" the table with N= indicates missing value of the average score due to exceeding the limited time (72 hours) for the job. Moreover, as shown in Fig. 2, Consistency-AG has better scaling laws than Diffusion-QU QL for both training and inference in time consumption with increasing N, which is further testified by the linear fitting results in Appendix. C.2 Fig. 6.
Ablation Studies. Hansen Hansen-Estruch et al. (2023) proposes to use residual networks with layer nor- malization for network parameterization in diffusion policy, namely LN-Resnet, which is also tested for consistency policy in our experiments. As an ablation study, we compare different variants of Consistency-ACe for offline RL setting, including (1) Consistency- by setting v) 0 and without using loss scaling (A(T,) =1 in Eq. 2); (2) only without loss scaling; (3) the standard setting with multi-layer perceptrons (MLP) networks for the parameterization of fe: (4) the LN-Resnet parameterization of fo. These variants can be regarded as various hyperparameters
training settings for the proposed Consistency- AC algorithm, and the reported results in Tab. are the best choices among these variants. The comparison results for four variants across four task domains are summarized in Fig. 3. Detailed results for this ablation study are shown in Appendix C.3. We find that LN- Les-eetet does not consistently improve over MLP across tasks for the consistency model but benefits mainly for the Adroit tasks. Without loss scaling, the performance degrades significantly (by 37.8% on average) for most tasks, although for some specific tasks (e.g.. AntMaze) it may improve the performance a bit without loss scaling. For most tasks except for AntMaze, Consistency- BBC cannot achieve the best performances and the Q-learning loss (0) (as Eq. 5) with proper scaling helps to further improve the scores.
Average Performance Consistency-AC Variants Different Tasks
80
Condetency-l
Condenwcy-AC Condsnency-CC
70
Condimanncy loes
Gym
ArtMaae
Adroit
Kitchen
Task Group
Figure 3: Comparison of variants of Consistency-AC across tasks in offline RL setting.
5.3 OPFLINE-TO-ONLINE AND ONLINE RL
Empirical finding 3: Consistency policy has close but slightly worse performance than diffusion policyfor dffline-to-onling RL, but significant improvement of computational efficiency.
For online RL, consider both online learning from scratch and the offline-tooonline setting with the model trained on offline dataset as an initialization for online fine-tuning. As discussed in previous Sec. 5.2, the offline model can be selected in either an online or offline manner, respectively by model evaluation with or without online experience. Both types of models are used for initializing the policy and value models at the beginning of online fine-tumine -inintunng. For online fine-tuning, it follows the standard actor-critic algorithm, that the Q value is updated with Eq. 3 using online data, and the policy is updated with the Q-learning loss (0) only as Eq. The algorithms use r-greedy for exploration with a decaying schedule. Pseudo-codes for dilline-to-onling and online Co are provided in Appendix D. Hyperparameters for training refer to Appendix D.2.
Table 4: Comparison of normalized scores (last epoch) for methods in affline-to-onling and online RL.
Offine-to-Onlinn Diffusion-QL
Online
Gym Tasks
SAC
AWAC
ACA
Consistency-AC
Diffusion-
Consistency-AC
balfchhctab-m
75.2
50.5
66.6
99.6
3 (90.8991111 8::1
98.711. (1771.999...8 2.9.
47.3+2.9
56.1#7.0
hopper-m
73.4
97.5
96.3
77.212256
01118)
60.531..661 (1:26.6)
82.62.0.6
86.4228.
walkerlddmm m
79.6
1.9
74.7
118.3
108.
1.0113000000000 110.911111100 10.5)
17.0..25.7
69.4 38.9
halfeheetah-mr
68.9
46.8
59.0
96.3+3.900
80.7
43.5+5..7
56.6+
hopper-mi
74.0
96.0
85.
68 14:20 (90 6421
71.625.11
(63
1.34=100 16.77
91.0=122
76.8=26.8
walker 2r
85.4
80.8
85.2
96.7+1888
(106.b 13.7)
11.
87.8
09.0=42.3
belfcheetal--me
82.2
68.8
93.7
(1022
199.1 99.64
39.743.6
56.7#5.8
hopper-me
65.4
73.1
98.0
71.71:111 (67. 9118.6)
65.4155.7551.7222
62.5
78.6 14.6
walker2d
87.2
45.2
110.5
117.0163 10.6)
101.8
13.3 (8022 16.2)
14.6
86.2 27.8
Average
7688
62.3
86.5
94.2
88.0
G7.7
70.4
Tab. 4 summarizes the quantitative results for average scores achieved with Consistency-AC and Diffusion- QL across five random seeds for two settings over 9 Gym tasks, as well offline-to-onie tonoine baseline methods SAC, AWAC and ACA (Yu & Zhang, 2023). Both the Consistency-AA and Diffusion- QL are pre-trained the offline dataset for 2000 epochs. Each model is trained for one million steps
8

---

Published as a conference paper at ICLR 2024
for online fine-tuning learning from scratch. The results for SAC, AWAC and ACA are adopted from the paper (Yu & Zhang, 2023) with each model fine-tuned for 100k online steps. Each cell has two values: one for offline model selection initialization and another (in brackets) for online model selection as initialization. The normalized scores are slightly lower for Consistency-AR compared with Diffusion- in offlnne-oo-onliee settings, but higher in online RI. from scratch. On average, the two methods achieve lower values in online setting than the offline-to-onling setting, which testifies the improvement of learning efficiency by initializing with pre-trained generative policy models. However, since the training is set to have a fixed overall timesteps and using the same learning rate 1 x 10-5, the purely online models do not converge its optimal performances yet. The learning rate is chosen for the fine-tuning setting, and the purpose is not to show online RL can achieve scores higher than 100 with sufficient training but to compare with the offlin-tt--nnlig setting, for demonstrating the effectiveness of initialized models with offlin pre-training.
Empirical finding 4: Consistency policy could outperform diffusion policy for online RL. setting mostly in computational efficiency and sometimes in sample efficiency, especially for hard tasks.
happerreeSuntteee expertt
we wnkerziinedummeoeee..
holthherth-meddang experneeeeeettett
Colocze heverz
Learning curves of Diffusion -L and Consistency- for online RL and offline-to-onlin RL with offline model selection in time axis (all trained with one million environment steps). Each curve is smoothed and averaged over five random seeds, and shaded regions show the 95% confidence interval.
Figure 4:
Fig. 4 shows the learning curves of Consistency-AC and Diffusion-QL for both oflline-to-online and online RL settings with one million online training steps on three example tasks (full results in Appendix D.3 Fig. 8). Different methods consume different time to finish the entire training. The diagrams are plotted with x-axis being the wall-clock time, therefore the curves exhibit different lengths. The diagrams with x-axis being the training steps are shown in Appendix D.4 Fig. 9. The results for offline-to to-online setting with online model selection from the offline pre-trained models are provided in Appendix D.4 but with similar performances. For most tasks, the consistency policy has comparable performances with the diffusion policy and significantly shorter time to finish the entire online training. The fffline-to-onling methods are usually more sample efficient than the online methods except for three hopper tasks, which relatively easy to learn a good policy. For the online setting, the consistency policies demonstrate significantly more efficient learning than the diffusion policies, especially for more complex tasks like halfcheetah. Consistency policies show a sharper score-increasing slope for 8/9 tasks than the diffusion policies. Our conjecture is that the expressiveness of model is more essential in offline setting than online setting. For deterministic optimal policy in MDP, overly expressive policy models like diffusion may hinder the convergence in online setting by being too explorative. For fffline-to-oningg setting, this advantage is less obvious presumably due to the lower initial performances of consistency policies from the offline pre-training. We refer to Appendix D.3 Tab. 9 and Fig. 7 for more analysis of the training time for two methods.
CONCLUSION
The consistency model as a RL policy strikes balance between the computational efficiency and the modeling accuracy of multi-modal distribution on offline RL dataset, and achieves comparable performances with the diffusion policies but significant speedup in three typical RL. settings. The proposed Consistency-AC algorithm leverages a novel policy representation with policy regularization for offline RL, orthogonal to other offline RL techniques. Future directions include combining the consistency policy with other techniques like conservative Q-value estimation for offline RL, better alignment and initialization from offline to online fine-tuming, advanced exploration methods for online RL, etc. Scaling up the task complexity, where more sampling steps required for the generative models, will reveal a greater potential for consistency policy to show its benefits in retaining expressiveness while reducing the computational cost.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by National Science Foundation Grant Qinqing Zheng for insightful discussions, and anonymous reviewers for feedback on an early version of this paper.
The authors thank
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pp. 104-114. PMLR, 2020.
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for deeisionnmmaiing" arXiv preprint arXiv:2211.156677 2022.
Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep rein- forcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26-38, 26.2, 2017.
PhilipJ J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data, arXiv preprint arXiv:2302.02948, 2023.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in neural information processing systems, 15:4933-49444 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 1084 2021.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137 2023.
Zihan Ding, Pablo Hernandez-Leat Gavin Weiguang Ding, Changjian Li, and Ruitong Huang. Cdt: Cascading decision trees for explainable reinforcement learning, arXiv preprint arXiv:2011.07553, 2020.
Hao Dong, Hao Dong, Zihan Ding, Shanghang Zhang, and Chang. Deep Reinforcement Learning. Springer, 2020.
Nicholas Frosst and Geoffrey Hinton. Distilling a neural network into soft decision tree. arXiv preprint arXiv: 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint wrXiv:2004.00219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems,
2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning. pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference machine learning, pp. 2052-2062. PMLR, 2019.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent
without entropy. arXiv preprint 2iXX::2000.02328 2023.
Wonjoon Goo and Scott Nickum. Know your boundaries: The necessity of explicit behavioral cloning in offline rl. arXiv preprint arXiv: 00695, 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pp. 1861-1870. PMLR, 2018.
10

---

Published as a conference paper at ICLR 2024
Philippe Jlansen-Estruce Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint S.Mi::2304.105720 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:68400366 6851, 2020.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-8)) 797 1991.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax arXiv preprint www. 000 11.001114 2016.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:220S. 09991, 2022.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
Rahul Kidambi, Aravind Rajeswaran, Pranceth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances neural information processing systems, 33: 21810-21822 2020.
Diederik P Kingma and Max Welling. arXiv:13/2.6114. 2013.
Auto-encoding variational bayes.
arXiv preprint
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:21 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. Advances Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learming for offline reinforcement learning. Advances Neural Information Processing Systems, 13:1179-119,, 2020.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference Robot Learning, pp. 1712- PMLR, 2022.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guidd guiig diffusion sampling in offline reinforcement learning. arXiv preprint arXiv:2304. 12824, 2023.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint MxXiv:2006.09359, 2020.
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline pre- pre-training for efficient online fin-tuning. arXiv preprint arXiv:2303.054799 2023.
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Ser- gio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301. 10677 2023.
Jie Ren, Yewen Li, Zihan Ding, Wei Pan, and Hao Dong. Probabilistic mixture-of-expeer for efficient deep reinforcement learning. arXiv preprint axXi=:2104.09//2,,200
Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies. arXiv preprint arXiv:2304.02532. 2023.
Jascha Siibl-DDckstein... Eric Weiss. Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015.
11

---

Published as a conference paper at ICLR 2024
Yang Song, Jascha Sohl-Dickstein Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.
Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient, arXiv preprint arXiv:2210.067/8. 2022.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Siddarth Venkatraman, Shivesh Khaitan, Ravi Tej Akella, Dolan, Jeff Schneider, and Glen Berseth. Reasoning with latent diffusion in offline reinforcement learning. arXiv preprint arXiv:2309.06599, 2023.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.066 06133, 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1361, 2019.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142. 2020.
Zishun Yu and Xinhua Zhang. Actor-critic alignment for offline-to-online reinforcement learning. 2023.
Deyao Zhu, Yuhui Wang, Jürgen Schmidhuber, and Mohamed Elhoseiny, Guiding online reinforcement learning with action-free offline pretraining. arXiv preprint arXiv:2301.12876, 2023.
12

---

Published as a conference paper at ICLR 2024
A
D4RL DATASET VISUALIZATION
T-SNE Plot
Figure 5: Visualization of t-SNE plots for 10000 (3000 for pen-human-vl and kitchen-complete-v))
randomly selected (s,0) samples in D4RL, dataset, colored by normalized reward (range [-1,1]).
13

---

Published as a conference paper at ICLR 2024
B CONSISTENCY MODEL TRAINING AND INFERENCE DETAILS
Training. The consistency model fe for modeling data distribution Pdam has the loss func- tion (Song et al., 2023):
(o(0)=Ennu(1.)) p. +
Tn 11 (x
(6)
where d(.:)) is the distance metric and we use la distance d(x. y) = |x y12.. For training, the sub-sequence {Tn E [N]] is different from inference, and it follows the Karras bound- ary (Karras et al., 2022) schedule: Tn = (e - c//))" The schedule function (k) / A ((s)+1)2 (12--))) + 1] + with as the current training iteration of total K iterations within one epoch".
Inference. After training, the consistency model fo can be used for generating samples given initial noisy input Xy (0, (T), following either single-step sampling x fo(X:, T), or multistep sampling by iteratively calculating x = So :x) with &,, = x + / <az following given time sequence {Tn [N]]. For inference. the time sequence is linspace of [e, with (N - 1) sub-intervals as: Tn
+ [N].
C OFFLINE RL EXPERIMENT DETAILS
C.I HYPERPARAMETERS
The offline training of Consistency-BBl and Consistency-AC uses batch size of 256 for training 1000 epochs (500 for pen-cloned-vl, 1500 for Kitchen tasks, 2000 for Gym tasks) on D4RL tasks, cosine annealing decaying schedule for learning rates, with other hyperparameters listed in Tab. 5. & = 100.0 (in X(-)) in our experiments for loss scaling in Consistency--Ae Max Q backup (Kumar al., 2020) is optional. norm indicates the normalization of subtracting the mean and dividing the standard deviation for the target Q values in stabilized training. Gradient norm is to clip the -n-ormm of the gradients. The Diffusion-BC and Diffusion QL training follows the hyperparameters of original paper (Wang et al., 2022).
Table 5: The hyperparameters for Consistency-AC in offline (including BC) training on D4RL Gym, AntMaze, Adroit and Kitchen tasks.
Hyperparameters
Tasks
learning rate
17
Q norm max Q backup
gradient norm
hallchectab-mediim-yy
3 10
1.0
False
False
9.0
hopper-medium--2
3x 10
0.1
False
False
9.0
walker2d-medium--
3x
1.0
True
False
L.0
halfcheetah-mmedimm replay-v2
3x 10
1.0
False
False
2.0
hopper-medium -epplaay-mmeela---222
3x 3x10
0.1
False
False
4.0
3x
0.1
False
False
4.0
3x10-4
1.0
False
False
7.0
medium -expeitueeeeee-v2222
3x10
1.0
False
False
5.0
walker2d walker2d-medium-eee e
3x10
1.0
True
False
5.0
antmaze-une umzzzzev-uaaeoo
3x10
0.01
True
False
2.0
aulmaze--maaze-diverss...
3x10-4
0.01
True
True
3,0
antmaze-mecdium-play--w
1x10-3
0.01
False
True
2.0
pen-human-vl
3x10-5
0.01
True
False
pen-cloned-vl
3x10-5
0.01
True
False
8.0
kitchen-conee kitchen-complete-v0
3x10
0.5
True
False
2.0
kitchen-parrial-v0
3x
0.5
True
False
2.0
kitchen-min -o0
3x10
0.5
True
False
2.0
Our experiments
=000027788;p 2, 150 following Song et al. (2023)
14

---

Published as a conference paper at ICLR 2024
C.2 COMPUTATIONAL TIME
Overall Training Time. Tab. shows the comparison of Diffusion-BC and Consistency-BB in terms of the computational time during training for D4RL Gym, AntMaze, Adroit and Kitchen tasks. Each result is averaged environments are trained for various numbers of total epochs, the comparison is based on per-epoch time consumption. The two methods use the same batch size and number of iterations within each epoch, as well as the same network architecture.
five random seeds with standard deviations reported. Since different
Table 6: The training time (seconds per epoch) for two BC methods on D4RL Gym, AntMaze, Adroit and Kitchen tasks.
Tasks
Diffusion-BC
Consistency-Be
halfcheetah-m
67. 07.00
43.07+0.61
hopper-m
61.00
38.00+0.49
walker2d-m
6..17
43.58
halfcheetah-mr
67.07+22.07
42.75++0.83
hopper-mr
64.89-33.29
38.03++0447
walker2d-mr
66.1111.669
42.70
halfcheetah-me
6..73+1..78
43.60+H0.86
hopper-me
63.04+4..25
38.56+00.56
walker2d-me
69.10+22.83
43.88+0072
Average
66.12:42..3
11.37
antmaze-u
97.13+44.21
47.88
antmaze-ud
104.8311..0
47 2.220
antmaze-mp
109.6611..82
57.92.44.44
antmaze-md
112.25
51.80 2.29
antmaze-Ip
113.15+11.11
56.52+33.11
antmaze-Id
118.6243.42
$3.89.1..11
Average
109.27#3.44
52.54
pen-human-vl
92.20+3.11
46.94
pen-cloned-vl
94.64:16.16
50.3311.22
Average
93.42+
48.6442.58
kitehen-c
96.77=2.74
66.7114.64
kitchen-p
94.25442.27
61.85
kitchen-m
93.02++3.49
66.60+2..50
Average
94.88
65.05
Total Average
86.0813.166
49.09
Scaling Law. Fig. 6 further shows the scaling laws of training time and inference time with increasing N for Diffusion-QL-ann and Consistency-AChn in offline setting, based on results in Tab. 3 for environment hopper-mediune -xperttt-.xxper-2... Notice that the coefficients of Consistency-AAe are smaller than Diffusion- QL in both training (2.47 vs. 3.54) and inference (0.515 vs. 0.598), which indicates smaller time consumption with increasing N.
C.3 ABLATION STUDIES
Four variants of Consistency-AA AC are compared for offline RL setting, including (1) Consistency- by setting = and without using loss scaling (A(T,) = in Eq. 2): (2) only without loss scaling; (3) the standard setting with MLP networks for the parameterization of fo: (4) the LN-Resnet parameterization of fo. These variants can be regarded various hyperparameters training settings for the proposed
15

---

Published as a conference paper at ICLR 2024
Training Time vs. N with Linear fitting
Inference Time with Linnar Fitting
Oitaiir-O. Dara
Cinesterryy Octo
Dinae
Inforence Time (ms 22n1
One stancr4C
Thiining
0525 5.91
27.50
Figure 6: The training time (left) and inference time (right) versus denoising steps N for Diffusion- QL and Consistency-AC in offline RL, evaluated hopperr-edium-eeppreee environment.
Consistency-AC algorithm. The average scores for five random seeds over D4RL Gym, AntMaze, Adroit and Kitchen are shown in Tab.
Table 7: The performance Consistency-AC variants on D4RL Gym, AntMaze, Adroit and Kitchen tasks for offline RL setting. Each cell has two values: one for offline model selection and another (in brackets) for online model selection. Each result is averaged over five random seeds with standard deviations reported.
Gym Tasks
Consisteney-Be BBC (1-
Consisteney-AC (no less scale)
Consistency-AC (MI.))
Consistency-AC AC (L.N-Resmet)
halfcheetah-m
31.0+0.4 (46.2+0.4)
60.1+007 (71.9-+0)))
(5..40-4))0.444..)
hopper-m
(1.7+8.0(883+-.)))
(09.7+2.3)
(96.4-4.))
76.146.7633.74855
walker2d-m
(8.1
3.5:1177 5.5:11..(11.22111 (2122116)
(7.0:5.2(750021..)
(812:5.2(755::11:
balfchoetall-mr
(4..25.3355.110.7))
58.713.9 (62.7:0.6)
(7.3310.2277888333
(18.20.3
hopper-mr
(1.7-0.5()00
(103.4++11)
(1000-44(00000-..
(1.7429010064433)
walker24-mr
73.3++5.7 +80.8++2.4)
72.3 12.3=15.4 (105
(8.14
(53.++3.663300
balfchoetal-me
(2..:21.2
12.6=10. (56.2=11.6)
84.3114.1 (89.2=3.33)
(68.:11..6
bopper-me
10.1:16.210:::55:1
100.4:1:.5 (108.0114)
43.115.7(66..511.2)
walker2d-me
110.4+0.7 (11
2.4.4.3114
(9777333)
84.1+5.1 (97.5++1.0
Average
(75.9)
44.7 (60.5)
76.7 (82.2)
(73.6)
AutMaze Tasks
Consistency-Be (n=0)
Consistency-AC loss
Consistency-A
Consistency-ACC (LN-Resne)))
unimaze- u
75.8 (8.7
75.4 (32.6.13.8)
6..811.6
antmazo-ud
77. 77.6266. (33.0)
75.2+6.6 (80.222.8)
(3..
72.1+3.561..21..991
antmaze-mp
56 36. (71.6114.5)
(5.22226.99733222...)
(7..4.2.
10.0222469..122...
Averuge
70.1 (80.3)
65.3 (78.7)
63.2
$2.7(7554)
Adroit Tasks
Consistency- BC ()=0)
Consistency-ACC (no loss scale)
Consistency-AC (MLP))
Consistency-Ae -C (LN-Resnet)
len-human-vi
13.7 (63.747.4)
8.44-24. (22.1 +20.5)
10.
63.447.77 (67.9+5.3)
pen-cloned-vi
33.128.0651..666
48 18.2=10 10.88 (58.2112.6)
36.8533.
30.1 (58.743.4)
Average
42.9 (57.8)
28.3 (40.2)
(53.6)
56.8 (60.8)
Kitchen Tasks
Consistency-Be (1)
Consistency--Ke C loss scule)
Consistency-AC
Consistency (L.N-Remee)
kitehen-e
15.245.0 (50.933..
10.0=20.1 (25.6=21.)
51.926.0 6+2.7)
36.93.2 0
kitchen-p
22.623.88 (23, $+288
(174166991700111
38.2+1.8 (39.8+1.66)
25.85..5 (28.6:2.7)
kitchen-m
13.5.11..00 (24.3
(.1.2..335.....2..
(5.811.5
26.0.13..0 (28.8.111)
Average
30.4
(.1
(513
23.6
Total Average
50.7 (67.0)
(54.1)
(72.5
56.8 (65.0)
16

---

Published as a conference paper at ICLR 2024
D OFFLINETTO-ONLINE AND ONLINE RL DETAILS
D.1 ALGORITHMS
Algorithm 4 Offline-to--Oiin Consistency Actor-
Algorithm 5 Online Consisteney Actor-Critic
Critic
Initialize
policy T and critic networks
Input offline pretrained policy Te and critic net-
Qa. -0
works @ 1Qoz
Initialize online dataset D = target network parameters: AT + f, of + $1.01 + 02
Initialize online dataset D = 0. target network parameters: 8T 8, o 2-1100 x-
for
for episode i MM do
Reset the environment and observe for 1o...., do
Reset the environment and observe 81.
for 1ort=1..... do
% Collect Samples
Collect Samples
Infer action a based on s, with consis-
Infer action based 8t with consis- tency policy πe by Alg. 1. Execute actions at, observe reward rt.
tency policy by Alg.
Execute actions a,, observe reward rt.
next state 814-1.
next state 8t+1-
Store data sample (5,,01,7(,,5411) into
Store data sample (st,0t,Tt;S++11 into
D.
D.
Sample minibatch ((s,a,r,s") S
Sample minibatch B = {(s,,,,,s'}}
D;
D; Q-value Update
% Q-value Update
Update Q Qaz Eit 3; % Policy Update
Update Qo, QQ2 with Eq. 3; % Policy Update
Update policy my (with model fa) via
Update policy no (with model fo) via
loss f.y (0) as 5q
loss C as Eq. 5;
% Target Update
% Target Update
Update target: OT atT (1
Update target: OT t aBt + (1
00,00 o + aΦp +(1-a)o,,i (1,2};
(1,2):
0 <ao + (E
end for end for
end for
end for
D.2 HYPERPARAMETERS
Table The hyperparameters for Consistency-AC offline-to-onlie
Aonsisteny-ACoofffli---oooiii and online training on Gym tasks.
Hyperparameter
Value
learning rate batch size g-redd schedule
1x 10-3
256
linear
eo Coo exploration fraction
1.0
0.01
0.1
discount
0.99
buffer size
1x 10
D.3 COMPUTATIONAL TIME
The overall training time for one million environment steps using Diffusion-QLL and Consistency- in oflline-t--oling and online RL settings is shown in Tab. 9, with the average training time for each setting summarized in Fig. 7. The reduction of computational time in online setting is less significant than the offline setting (as Fig. 1) because there is large portion of time consumed by the environment simulation steps following the agent's ' action inference. The improvement model inference and update will not affect the environment simulation time.
17

---

Published as a conference paper at ICLR 2024
Table 9: The overall training time (hours) for offline-to-online and online settings Gym tasks.
Offline-to-Online
Online
Gym Tasks
Diffusion-QL
Consistency-Ae
Diffusion QL
Consistency-AC
halfcheetah-m
11.55=4.08
9.30
9.09
0.77
hopper-m
8.97
0.79
8.06
6.99
walker2d-m
9.17::0.29
8.19
8.23
6.98+0.89
halfcheetah-m
9.18+t022
7.72=#0.99
8.72
7.76=1.02
hopper-mr
8.22+0.20
7.26+1.69
8.12#0.81
6.92
walker2d-mr
8.85+:0.22
7.32
8.07+1.01
1.05
halfcheetah-me
9.24++0.21
8.46
8.54
1.65
hopper-me
8.28+0.20
7.47
8.02
7.01
walker2d-me
9.93:11.27
9.41
8.35
8.29
Average
9.27
8.04
8.36
7.99
Training Time Comparison
Average Training Time (hours)
9.27
8.56
8.
7.49
Diffusion-QL QL
Consistency-AC
Offline-to-Omline
Online
RL Setting
Figure 7: The average training time (hours) for offline-to-online and online training with Diffusion-QL. and Consistency-ACe on 9 Gym tasks.
D.4 MORE RESULTS
Table 10: Comparison of scores (unnormalized, maximum over epochs) for methods in offline- tone-tttoonneeee and online RI. settings.
Ofline20nline
Online
Task hallcheetah m
Diffusion-O QL
Consistency-AC
Dilfusion-UL
Consistency- AC
12445.9 1315.8 (12428. 222.9)
12280.3
1241 (12273.4 206.3)
5745.9 388.5
6725.2 944.44
bopper-m
91236.4)
1421366
367..511477
1589.71100
walker2d-m
57748=2177 (6561 260.2)
5336.
1-300 3 (5662.0-4 114.3)
6316.24612.11
3750. 9=# 1677.5
halfcheetah-mr mr
12060. 1 1: 265.7 (16.5.7(111555 (12198.6
168.9)
9941.1
1343.2
(10274.
1312.7)
5218.5 726.2
6890.3
196331
hopper-me
3657 (3955554:4444
3262.44709.
3663. 99.7
3418.0-00134
walkerldd me
5240.66 682.8 (5584.5 5.15584..111.. 347.0)
5082.3 408.5 (53304.4
(.08.3))
4675.4
3918.7
1673.3
hallchoeiah-me
12916.742122 (126766 (17881)
12480
#339 (11916.92 115677)
1725.61 407.9
6889
65.65.55
hopper-me
3513
190.8 (3027, 2429772)
3036.9
(00200
33688 8337.
37:1.0449922
walker2d.me
5630.5 209.0 (5720.7 377.9)
144770-43007. (5568.1 (553)))
3887. 1 1701-5
5040.0 1-6600 166.0
Average
7206.
6797.0
4397.2
48849
18

---

Published as a conference paper at ICLR 2024
kppermediumm?
Epsoock Rowend
Eprick
Trme 10
Time
waller2d mediumine x2
walle'dd medi r2
walle 24 medium expertv2
Esteck Rowdd
Cslect- Rowand
15am 10
T1am
12
medium
Ephost
difiaken offinnantihe
dffaakm--eiie comaatenyycniin
Time:
Figure Learning curves of Diffusion-QI. and Consistency-Aee for online RL and offline-to-online RL with offline model selection in time axis (all trained with one million environment steps). Each curve smoothed and averaged over five random seeds, and the shaded regions show the 95% confidence interval.
Weber teweeeeeeemmeeeeeeeee
Coroedt Tamere
holfcheeta" mediumv2
teticreetah medium rep 42
boftheeth--mduumecee espeee
offickn dffnezwiiee
2upp
200e
Steps
Figure 9: Learning curves of Diffusion QL and Consistency-AC for online RL and offline-to-onling RI. with offline model selection in step axis. Each curve is smoothed and averaged over five random seeds, and the shaded regions show the 95% confidence interval.
19

---

Published as a conference paper at ICLR 2024
kppermedi-m-?
Epecook Rowend
Time 10
Time
walker2d mediumis x2
wallerdd medi r2
wale 24 medium expert
Estcck Rowdd
10
Tlme 10
replat..2
diftaken offinnzntiie dffaakm--riie comsatency-cciin
Tiner
Figure 10: Learning curves of Diffusion-QL. and Consistency-ACC for online and offline-to-online RL with online model selection in time axis (all trained with one million environment steps). Each curve smoothed and averaged over five random seeds, and the shaded regions show the 95% confidence interval.
weber temeeeeeemmmeeeeeeee
Cotendo Ramere
Cplodd
holfcheeta" me
teticreetah medium k2
boftherth-meduumeeeo expeeet
offickn diieewwiie
2000
Stpps
Figure 11: Learning curves of Diffusion-Ol -LL and Consistency-AC for online RL and offline-to-onling RL with online model selection in step axis. Each curve is smoothed and averaged over five random seeds, and the shaded regions show the 95% confidence interval.
20


Reviews:
[{"review_id": "IQpXvZI8Tp", "reviewer": "Reviewer_259J", "summary": "This paper proposes to use the recent \"consistency models\" as a way of parameterizing the policy for deep RL.  This is explored in offline RL, offline-to-online RL, and online RL with an actor-critic setup.  The performance is competitive with diffusion-BC despite requiring far fewer sampling steps.  This seems like a strong empirical advance to me, because the speed of sampling is essential for online reinforcement learning.  While the approach here is unsurprising, combining consistency models with RL, this still seems like an important contribution.  \n\nnotes: \n  -Diffusion inference is slow, so there could be value in using consistency models to define the policy, particularly for an actor-critic style algorithm.  \n  -This paper considers both online, offline-to-online, and offline setups.  \n  -The policy class is important for RL, especially that it be multi-modal.  \n  -The paper lays out policy regularization to prevent out-of-distribution actions.", "questions": "-Have you thought about also using the consistency model as the \"model\" in the RL sense, i.e. to learn p(s' | s,a)?  If so, do you see any interesting challenges or opportunities there?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-I think that this project is very critical for the success of RL, as insufficiently rich policy classes are a major limitation.  Additionally, in RL it is important to draw as many samples as possible (either in a model or in the environment) so evaluating the policy quickly is critical.  So I think this work will have a lot of impact.   \n  -I think it's also particularly impressive that the paper shows success in online RL, because in online RL it is important that the policy perform well even when it's imperfect (i.e. early in the training process).  Whereas in offline-RL, we could imagine that the policy only needs to perform well near the end of the training process.  We indeed see that the consistency model outperforms the diffusion model in the purely online setting (Figure 4).", "weaknesses": "-The idea of using consistency models as RL policies is fairly intuitive and not terribly surprising.  \n  -On the harder tasks like Kitchen and Adroit, there is a significant gap with Diffusion-BC baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LNY2K5oLYw", "reviewer": "Reviewer_gzrQ", "summary": "This work proposes utilizing the consistency model to learn policies in modeling multi-modal data from image generation, which perform more efficiently than diffusion models. The authors evaluate their models on three typical RL settings: offline, offline-to-online, and online, and experiments show that the consistency policy can reach comparable performances than the diffusion policies while reducing half computation costs.", "questions": "(1) The authors claim that \"By behavior cloning alone (without any RL component), using an expressive policy representation with multi-modality like the consistency or diffusion model achieves performances comparable to many existing popular offline RL methods.\", I'm wondering where is the multi-modality. Is the consistency policy trained for all tasks across all suites? Or does the offline dataset have different successful behavior policies? The author should make it more clear. \n\n(2) In the specific tasks in RL, the scenarios are not abundant and the trajectories are quite similar. If more expressive policy representation can result in better performances, what if using some large pre-trained representation models, can this problem be solved? (E.g. R3M)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, and the experiments are sufficient, which include three different RL settings and four task suites with BC and RL baselines.\n- The motivation behind this paper is natural and valuable.", "weaknesses": "The author claims the consistency policy is much more efficient than diffusion policies while keeping comparable results. However,\n- the results in Tab. 1 and 2 can not support the conclusion in a way, which shows there are some significant drops in some tasks (such as halfcheetah-me, kitchen-xxx) between the Diffusion-BC and Consistency-BC or Diffusion-QL and Consistency-AC. \n- Moreover, in Tab. 3, when N = 5, the performance between Diffusion-QL and Consistency-AC is comparable while the time cost is also similar. This indicates that when the denoising step is small, the absolute scores of both methods are good enough. In this case, the consistency model has few advantages, which makes the improvement much more limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZcvHdeMTha", "reviewer": "Reviewer_JCUm", "summary": "It is known that the inference process of the diffusion model can be slow. In the context of RL, the diffusion model has been introduced and widely adopted recently. The authors focus on addressing the slow inference speed issue of diffusion in this paper. Their solution is to replace the diffusion model with the recently proposed consistency model. The authors make some minor adaptations of consistency models to make it fit the RL setting. The authors also test their method across offline RL setting, online RL setting, and offline2online finetuning setting, by building a consistency model on top of behavior cloning and actor-critic structure.", "questions": "- why do you use different baselines on different domains? It is somewhat confusing that you use some quite weak baselines on domains like antmaze, adroit, and kitchen. As an example, why do you use AWR, BRAC, and REM on adroit tasks and kitchen tasks in Table 1? It seems to me that the advantages of consistency-BC and consistency-AC are illustrated by carefully picking the baselines.\n\n- how important is the gradient norm to the consistency-BC and consistency-AC algorithms?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "# Strengths\n\nThis paper pinpoints an interesting question that the current diffusion-based offline RL algorithms (e.g., Diffusion-QL) suffer from slow inference speed. The authors then propose to address this issue by using an existing method, the consistency model, and adapting it to the offline RL setting. This, as far as the reviewer can tell, is the first work that introduces the consistency model into the offline RL setting, online RL setting and offline2online setting. This paper is generally quite well-written, and I enjoy reading this work. The structure of this paper is clear and easy to follow. It is easy for the readers to capture the core points/conclusions in the experiment section. The authors conduct several experiments in D4RL domains like MuJoCo, AntMaze, etc., to show that their method can reduce the training time and inference time while maintaining comparable performance. The authors also do a good job in the related work part.", "weaknesses": "# Weaknesses\n\nDespite the aforementioned strengths of this paper, I think this paper is below the acceptance bar of this venue. Please refer to the following comments\n\n- (major) The novelty of this paper is limited. The authors simply borrow an existing method from the vision community and apply it to the RL tasks, with some minor modifications. I do not see much novelty in doing so. This paper seems more like a technical report or experimental report, in that the authors conduct several experiments and summarize the conclusions. One serious flaw of this paper is that the author merely reports some experimental phenomenons while the corresponding explanations and discussion are unfortunately missing. From an ICLR paper of this kind, I would expect to understand why such a phenomenon occurs. This paper leaves me with more questions than answers. For example, why on many offline and offline2online tasks, the consistency model underperform the diffusion model, while on some online hard tasks, the consistency model seems to be better?\n\n- (major) This paper does not consider statistical significance. Written statements and the presentation of the results as tables (often without standard deviations) obscure this flaw. In fact, ALL tables in this paper do not include any signal of statistical significance for baseline methods, e.g., std, IQM [1]. We have reached a point of maturity in the field where claims need to be made in reference to actual statistical evidence, which seems to be lacking in the current presentation.\n\n[1] Deep reinforcement learning at the edge of the statistical precipice. NeurIPS\n\n- (major) The proposed consistency-BC or consistency-AC do not show much improvement over the diffusion-based counterparts. If one looks at the experiments in this paper, it is clear that the consistency-BC and consistency-AC cannot beat diffusion-BC or diffusion-QL on most of the tasks. The authors say that consistency-AC or consistency-BC can achieve less training time. Well, I do not see this as an appealing advantage over the diffusion-based methods, since training cost is somewhat unimportant, while the most critical part, from my perspective, is the inference speed. Let us then take a look at the inference cost. Based on Table 3 in the main text, it is clear that the inference speed of diffusion-QL is quite similar to that of the consistency-AC. For instance, when setting $N=5$, the inference speed of diffusion-QL gives 3.76ms and consistency-AC gives 3.39ms, while their performance differs (diffusion-QL has an average score of 108.2, while consistency-AC only has 101.4). I actually do not see many advantages of utilizing consistency-AC or consistency-BC in practice\n\n- (major) Even worse, it seems that the performance of consistency-AC and consistency-BC are acquired by carefully tuning the hyperparameters. As a piece of evidence, one can see the hyperparameter setup in Appendix C (Table 5). This indicates that the generality and effectiveness of the proposed method are limited\n\n- (minor) On page 3, Eq 3 is the objective using clipped double Q-learning instead of vanilla double Q-learning. The authors ought to cite the TD3 paper here instead of the double Q-learning paper.\n\n- (minor) No codes are provided in this paper, and the authors do not include a reproducibility statement section in the main text", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)