Value-aligned Behavior Cloning for Offline Reinforcement Learning via Bi-level Optimization
Authors: Xingyu Jiang
Ning Gao
Xiuhui Zhang
Hongkun Dou
Yue Deng
Venue: ICLR 2025 Poster (2025)
Keywords: offline reinforcement learning;bi-level optimization;value alignment
Primary area: reinforcement learning

Abstract: Offline reinforcement learning (RL) aims to optimize policies under pre-collected data, without requiring any further interactions with the environment. Derived from imitation learning, Behavior cloning (BC) is extensively utilized in offline RL for its simplicity and effectiveness. Although BC inherently avoids out-of-distribution deviations, it lacks the ability to discern between high and low-quality data, potentially leading to sub-optimal performance when facing with poor-quality data. Current offline RL algorithms attempt to enhance BC by incorporating value estimation, yet often struggle to effectively balance these two critical components, specifically the alignment between the behavior policy and the pre-trained value estimations under in-sample offline data. To address this challenge, we propose the Value-aligned Behavior Cloning via Bi-level Optimization (VACO), a novel bi-level framework that seamlessly integrates an inner loop for weighted supervised behavior cloning (BC) with an outer loop dedicated to value alignment. In this framework, the inner loop employs a meta-scoring network to evaluate and appropriately weight each training sample, while the outer loop maximizes value estimation for alignment with controlled noise to facilitate limited exploration. This bi-level structure allows VACO to identify the optimal weighted BC policy, ultimately maximizing the expected estimated return conditioned on the learned value function. We conduct a comprehensive evaluation of VACO across a variety of continuous control benchmarks in offline RL, where it consistently achieves superior performance compared to existing state-of-the-art methods.

Published as a conference paper at ICLR 2025
VALUE-ALIGNED BEHAVIOR CLONING FOR OFFLINE REINFORCEMENT LEARNING VIA BI-LEVEL OPTIMIZA- TION
Xingyu Jiang', Ning Gao', Xiuhui Zhang', Hongkun Dou', Yue Dengl.2 2:2 Beihang University, Beijing Zhongguancun Academy
(i (ixy33zrhd, gaoning ag_ aaangiiuuuii douhk, ydeng) @buaa. edu. cn
ABSTRACT
Offline reinforcement learning (RL) aims to optimize policies under pre-collected data, without requiring any further interactions with the environment. Derived from imitation learning, Behavior cloning (BC) is extensively utilized in offline RL for its simplicity and effectiveness. Although BC inherently avoids out-of- distribution deviations, it lacks the ability to discern between high and low-quality data, potentially leading to sub-optimal performance when facing with poor-quality data. Current offline RL algorithms attempt enhance BC by incorporating value estimation, yet often struggle to effectively balance these two critical components, specifically the alignment between the behavior policy and the pre-trained value estimations under in- in-sample offline data. To address this challenge, we propose the Value- aallee Behavior Cloning via Bi-level Optimization (VACO), a novel bi-level framework that seamlessly integrates an inner loop for weighted super- vised behavior cloning (BC) with an outer loop dedicated to value alignment. this framework, the inner loop employs meta-scoring network to evaluate and appropriately weight each training sample, while the outer loop maximizes value estimation for alignment with controlled noise to facilitate limited exploration. This bi-level structure allows VACO to identify the optimal weighted BC policy, ultimately maximizing the expected estimated return conditioned on the learned value function. We conduct a comprehensive evaluation of VACO across a variety of continuous control benchmarks in offline RI., where it consistently achieves superior performance compared to existing state-of-the-al methods.
INTRODUCTION
Over the past decade, reinforcement learning (RL) has made remarkable advancements and exerted significant influence in domains such robotics control (33), dynamic programming (49), and rec- ommendation systems (6). However, in real-world scenarios where interactions are either extremely challenging or prohibitively costly, traditional online RL faces considerable limitations and may often prove impractical. As result, offline RL has emerged as a promising alternative, attracting growing attention and active research. Unlike online RL, which actively collects trajectory samples through direct interaction with the environment, offline RL is restricted to deriving policies from a static, pre-collected dataset (30), without any further interactions with the environment. Interestingly, the primary advantage of offline RL, its independence from environment interactions, also poses its most significant challenge. Due to the limited and potentially sub-optimal nature offline data samples, offline RL confront two severe challenges: I) out-of-distribution (OOD) issues and 2) value aligmment issues. Both of these issues fundamentally stem from inadequate sampling of the state and action spaces in offline datasets.
As illustrated in Fig M we provide a schematic diagram to intuitively depict the two critical challenges in offline reinforcement learning. [ a) presents the learned value-action curve for given state, where the blue dots represent actions collected from the offline dataset, which we refer to as the * *in-sample d The adjacent red area, representing actions that have been collected, is
"Corresponding author

---

Published as a conference paper at ICLR 2025
Q(s.a)
optimal
In-Sample
action
Sub-optimal
Sub-optimal
(a) OOD issues in offline RL
(b) alignment issues in offline RL
Figure I: Illustration of the two challenges offline RL: (a) value overestimation in OOD issues and (b) sub-optimal policy extraction in value alignment issues.
termed the coutto--distribution (OOD) domain Empirically, the values of actions in the OOD domain tend to be overestimated, while the action values within the in-sample domain are more confident and closer to their true values. ()g further refines the in-sample domain. During the behavior policy learning process, the inadequacy of action space often causes the behavior policy to converge on sub-optimal actions within the in-samlee domain (e.g.. A A-), which actually do not align with the optimal actions suggested by the value estimations (e.g., A*). This misalignment can lead to sub-optimal policy performance.
To avoid above challenge, behavior cloning (BC (37) is initially employed in offline RL settings. BC utilizes supervised learning to directly learn from the in-sample data, effectively avoiding OOD challenge. However, BC lacks the capability discern between high-quality and low-quality data, which typically results in competitive performance only with expert offline datasets and leads to failure within poor/sub-optimal offline datasets. To circumvent this limitation, recent offline RL approaches have begun incorporating certain auxiliary information (e.g., value function) into BC, guiding it towards differentiated learning. These methods can be well categorized into following primary types: (1) explicit regularization constraints (7),,,, which involve heuristic regularization terms like KL divergence or expectile regression directly constrain the behavior policy; (2) implicit regularization constraints
(22) 15), which leverage generative models such as VAEs (9 29) and
diffusion models (19 44 enforce constraints on the behavior policy within latent action space; and
(3) return-conditioned supervised learning (5 23), which utilizes decision transformer to directly model dynamic programming process a conditioned supervised learning manner. Although the aforementioned methods have achieved notable performance, they often struggle to balance the OOD challenge and value alignment issue concurrently.
Our contribution. In this paper, inspired by recent advancements in meta-learning (34 21 52
we introduce the Value-aligned Behavior Cloning via Bi-level Optimization (VACO), novel bi-level offline RL framework balance the OOD and alignment challenges. This framework integrates a simple multi-layer perception (meta-scoring network) to assign differential importance weights to various state-action sample pairs. In this way, the conventional BC loss function is transformed into a weighted summation of individual sample losses, enabling differentiated learning for varying quality samples. Such learning manner is intuitive and closely mirrors human behavior learning processes, In detail, the internal loop of VACO executes weighted supervised behavior cloming with the assigned scores of in-sample data; and the external loop of VACO maximizes value estimation for value alignment and introduces controlled noise to enable limited exploration. Such bi-level configuration allows VACO to identify the optimal weighted BC policy, ultimately maximizing the expected estimated return conditioned on the learned value function without numerous hyper- parameters or complicated secondary components, such as generative models and transformer. is worth mentioning that the meta-scoring network utilized in VACO is operational solely during the training phase and can be deactivated during testing, ensuring that the inference speed of the algorithm remains unaffected.
We evaluated the proposed VACO on the D4RL benchmark dataset (1) for continuous control tasks. The results indicate that model achieves state-of ofee--of-hheart (SOTA) performance, surpassing the above three categories of offline reinforcement learning algorithmss, through a simple integration of BC (37) with DPG (42 algorithm in bi-level manner. Such achievement provides new insights
2

---

Published as a conference paper at ICLR 2025
on the potential of bi-level optimization in offline RL, and further ablation studies underscore the effectiveness of our proposed meta meta-scoring network.
2 RELATED WORK
In recent years, offline RI. has emerged as a pivotal solution for real-world scenarios, particularly in domains such as autonomous driving (43), dynamic programming (49), and robotics control (33) Offline RI. is inherently limited by insufficient data and the lack of interaction with the environment, leading to two major challenges: 1) the OOD problem and 2) the value alignment issue. Traditional offline methods (e.g., BC (37) and TD3 (44) often focus on one specific aspect. More recent strategies (15 [54 24
47 56 seek to integrate these approaches for performance boosting. Depending on the integration manner, these approaches can generally be classified into three categories:
Explicit Regularization directly incorporates BC as a regularization within value estimation. For instance, TD3+BC (13 directly combines the loss function of BC and TD3, with hyperparameters fine-tuning the balance between them. BRAC+ (53) integrates BC into the value and policy update processes by KL divergence (18) to provide regularization and constraint, IQL (27) employs expected regression to perform in-sample value estimation, subsequently guiding BC across various state-action pairs using advantage-weighted regression (AWR) (36). This category of methods typically designs a divergence measure to ensure that the learned policies closely with the dataset's sampling policy to some extent.
Implicit Regularization posits that constraints should be moderately stringent, often executed within latent space to subtly constrain policies. Key contributions in this area include MOPO (5I), which introduces model-based concept into offline settings by incorporating penalty rewards to limit the learning process; PLAS (55), which uses CVAE (2) to model policies within the action latent space, thus implicitly restricting learning to the support range of the in iammle dataset; and EDP (22), which utilizes diffusion models for implicit action estimation. These methods (2)) typically employ generative models to encode state and action into latent space, thereby constraining learned policies.
Return-conditioned Supervised Learning represents novel paradigm in offline RL that conditions the policy not only on the current state but also on expected future returns. This can be viewed as variant of conditional BC. For instance, DT (5) introduced the use of decision transformers and returntto-o to achieve supervised modeling of offline trajectories:DS4 (8) innovates by replacing the time-invariant state-space layers with transformer, thereby facilitating efficient dynamic modeling. DC emphasizes the importance of local attention in Markov processes and implements an innovative approach by substituting attention mechanisms with convolution.
Bi-level Optimization in Offline RL. Bi-level optimization is committed to optimizing another set of parameters other than target network parameters, which describes higher-level elements related to training neural networks (52 21). In offline RL domain, the most relevant work to us is (56) and we differ motivation and upper-lower functions as follows: (i)iotivation: we introduce the bi-level optimization framework, aiming to balance the OOD and value alignment problems, while (56) mainly focuses on the distributional shift issue. (2)Upper-lower functions: Different motivations lead to different upper and lower functions. We first introduce meta-scoring network in the upper function to assign adaptive weights and mainly focus on the behavior policy updating in the lower function. Differently, (56) adopts the behavior policy updating in the upper function and mainly focuses on the Q value approximation in the lower function.
In our work, explore the integration of BC and value estimation through bi-level optimization framework, introducing a novel training methodology for offline RL to balance OOD and value alignment challenges concurrently. Although our proposed framework, VACO, is grounded in BC and DPG value estimation loss, it should better be regarded as flexible framework capable of bridging supervised behavior policies with value estimation loss. With tailored modifications, this framework can be properly applied to the aforementioned three kinds of methods.
3 PRELIMINARIES
Behavior cloning. BC is an approach within imitation learning that trains policies to emulate expert behaviors by directly mapping observed states to corresponding actions. Typically, BC (37
3

---

Published as a conference paper at ICLR 2025
employs supervised learning models to approximate the policy demonstrated by the expert, effectively replicating the expert's 's decision-make making process. Recently, behavioral cloning has gained popularity in offline RL due to its simplicity and straightforward application; however, its efficacy is heavily dependent on the quality and comprehensiveness of the demonstration data.
Offline reinforcement learning setting. In RL setting, the dynamic system is described in terms of a Markov decision process (MDP) setting, which is represented as tuple M = {S, r, P. 7), where S and A are the state and action space, r(s, a) scalar reward function, dynamics, p is the initial state distribution, and 7 E (0, 1) is a discount factor. The objective of RL is learn policy s[aas) with parameters @ by maximizing the expected cumulative discounted return E, 1>== ("t( az)], which is typically approximated by a value function Q a) with parameters For actor-critic based methods (26) (7) in a continuous action space, the parameters @ typically updated by minimizing the squared Bellman error with an experience replay dataset 'D and target function:
the transition
(0) = -10 (s)))
-
where Q3 denotes a target Q-function, which is a delayed copy of the current Q-function Qp. Then, the policy ge can be updated following the deterministic policy gradient (DPG (42) theorem:
(x) = EE- (s, (o(s))
In offline RL setting, the learning policy is constrained on pre-collected dataset ((s, a, s', r) without further interaction with the environment during the learning process. Meanwhile, the dataset D is generated by a unknown behavior policy πg. Directly applying standard RL methods in the offline setting suffers from severe OOD problem in value and alignment issue in policy. To avoid above challenges, widely used offline RL framework (48) (7) adopt the following behavior regularization scheme which regularizes the divergence between the learned policy T and the unknown behavior policy T of the dataset DD;
mig Bn-l- (5, (T(s)) + (TT
(3)
where D(-II-) is some divergence measures, which can have either an explicit or implicit form, to constrain how "closeness" of the learned policy Th and the unknown behavior policy Tg.
4 METHOD: VALUE-ALIGNED BEHAVIOR CLONING WITH BI-LEVEL OPTIMIZATION
In this section, we first discuss the motivation for balancing OOD and alignment problem. Subse- quently, we introduce how to transform the behavior cloning supervised loss function into a weighted behavior cloning supervised loss function. Finally, we describe how to correctly guide the proposed weighted supervised loss through bi-level optimization manner, thereby effectively balance the OOD issue and the alignment problem.
4.1 MOTIVATION FOR BALANCING OOD AND ALIGNMENT PROBLEM
In offline reinforcement learning (RL) settings, two classic sertoacchs--ehavvie cloning (e.g., BC (37)) and value estimation (c.g., TD3 (14) are commonly employed address the OOD and value alignment challenges, respectively. The objective function of behavior cloning (see Eq effectively mitigates OOD issues but suffers significantly from value misalignment.
contrast, the objective function of value estimation (see [q adeptly resolves alignment issues but is highly susceptible to OOD challenges. These phenomena are evident in our experimental results, as shown in Fig 2 behavior cloning consistently demonstrates stable performance across different datasets but tends to under-perform, while value estimation achieves superior results in specific datasets but displays extreme instability, sometimes resulting in "zero" performance in others. In offline RL, these dual challenges- OOD and value mnisaignment--simmffiiannnn significantly impact model performance, and is crucial to strike appropriate balance between them.
4

---

Published as a conference paper at ICLR 2025
112.6
97.2
101.0
142,3
97.4
60.9
Normalized results in Hopper -medium
Normalized results in Hopper-medium--eg
Normalized results in Hopper-medfum-eppee
114.5
107.5
DLA
94.9
71.5
84.5
0.2
Normalized results in Wallter2D medium
Normalized recults Walker2D medium-replay
Normalizad results in SallkerDD-medine
Figure 2: Normalized result comparisons on various Hopper and Walker2D datasets across TD3, BC, TD3+BC, EDP and VACO. Our method achieves superior performance.
Current offline RI. algorithms (3:553 533) attempt to balance the OOD issue and the alignment problem
by using mixed policy learning objective function, as represented in Eq with notable success. However, the straightforward combination of behavior cloning (26) and deterministic policy gradient
(DPG) (42) objective functions (e.g.. TD3+BC (33) can still lead to sub-optimal performance in some certain scenarios, illustrated in Fig 2 To address this gap. propose novel bi-level optimization framework. In this framework, a weighted behavior cloning algorithm is employed in the inner loop for policy extraction, while value estimation algorithm is adopted in the outer loop for value alignment. Additionally, we introduce a meta-scoring network to facilitate an indirect fusion between behavior cloning and value estimation. the following sections, we will describe the bi-level framework in detail.
4.2 WEIGHTED BEHAVIOR CLONING
We start with behavior cloning. In conventional behavior cloning training, a batch of state-action pairs (s, a) sampled from the dataset D and fed the policy network Tg. The policy network T employs a supervised learning approach, utilizing simple L.2 loss function to optimize the learning process. The objective function JBC can be expressed follows:
JBC(Ø) = ((T()))((TT())))) - =]²
Above additive loss JBC implies that different state-action pairs are treated equally in training. although they actually worse action choices. For more reasonable loss design, the weights of different training pairs should be properly evaluated and exploited. Therefore, we introduce a more intuitive behavior cloning loss Ji by meta-scoring network w(s, a, Qo(s, with parameters a:
(Ø) (wa.) a, Qo(s, a)) (TT(() - a}3)
We emphasize here that evaluating individual state-action pairs is extremely challenging, particularly because most sample trajectories in offline reinforcement learning are sub-optimal. This complexity arises from the inherent nature of the collected data, which does not always represent the optimal policy but rather mixture of various policy executions, often leading less than ideal decisions being captured the dataset. Accordingly, rather than a heuristic manner, we opt to adopt the parameterized meta-scoring mechanism that automatically assess the importance of each training pair through a learnable neural network w. with (8, a, Qo a)) as input.
4.3 BI-LEVEL OPTIMIZATION FRAMEWORK
While the aforementioned parameterized weighting concept is simple, it yields a highly under- determined and non-convex objective function coupled with two unknown neural networks. Without

---

Published as a conference paper at ICLR 2025
extra constraints, the direct minimization of Jec can easily lead to trival solution. In this case, the meta-scoring neural network Wa may intend to assign (near) zero weights to all sample pairs and hence totally mute the functions of the policy network (see the multiplications between the weight term and behavior cloning loss term in Eq 5). To avoid such trivial solution, extra constraints or guiding information should be imposed restrict the feasibility of the learned policy. In this work, aiming to balance the OOD problem and value alignment issue concurrently, we consider enforcing the feasibility of the learned policy by maximizing the value estimation function in Eq thereby meanwhile achieving value alignment. With this objective, the whole learning process is subject to the following bi-level optimization with controllable Gaussian noise N N(0. o)) for limited exploration:
min (+) =- E.- DEs-n(-QQ(s, To (10
(6)
s.t. (a) arg mingminJee Bc (0) = (( (8, a, Qo(s,)) (F)(s) a}²}
The above bi-level optimization is composed of the internal loop in the constraint and the external loop in the objective function. The internal loop minimizes the empirical squared error of the offline samsa dataset under guidance of the different weights provided by the external loop. The external loop maximizes the value estimation of the learned policy to tune the parameter space of the meta-scoring network for better value alignment. Through such a bi-level optimization framework, achieve policy extraction from weighted behavior cloning in the internal loop avoid out-of- distribution challenges and maintain value alignment from value estimation maximization in the external loop, respectively.
In practice, we opt not to incorporate the learning of the value network Qe within our bi-level optimization framework, as detailed Eq Consistent with practices in IQL (27). in offline RL, the learning of the value network often remains uncorrelated with the iterative updates of the policy network Tg. As a result, value estimation can be independently executed prior to policy extraction. As delineated in Algorithm[] to maintain the stability of training for the meta-scoring network, we structured the training process into two distinct phases: the initial phase focuses on value network evaluation following the TD learning of (27), and the subsequent phase engages in bi-level optimization for achieving value-aligned behavior cloning. Although integrating value network learning into the bi-level optimization is feasible, further details on this integration provided in the Appendix[] [T] Additionally, we incorporated controlled, progressively decreasing noise in the outer loop to facilitate limited exploration during the early stages of training.
To implement our VACO framework, we alternate between the internal loop and external loop optimization. The parameter is only involved in the internal loop and can be easily updated with typical gradient descending approaches, where the denotes the learning rate of policy model:
Φi + 0: 1/1 JoJ
The major difficulty of the VACO optimization stems from the external loop to learn parameter a for the meta-scoring neural network. As witnessed in Eq derived by applying the chain rule and assuming
6 coupled into & and its gradient can be
Da
V (+
(d+
det
Det
Ze
0J₂(0)
a( 1 VoJgolór
Do:
da
(8)
~ (J+(#)
8 (JHc((r-1))
0b:
dá.-1da
8J+
0) (3)
a/²
Owa Q0.(8...005..0)
0b:
86:-1
8a
The above equation is an approximate solution for external loop updating. Thus, we get the update rule of o,
Mooe details can be found in Appendix

---

Published as a conference paper at ICLR 2025
(J)(6)
JBc
at - a-- + 172
0Φ.
dd
According the updating rules based Eg and Eq as in Algorithm
alternately optimize two sets of parameters
Algorithm 1: Value-aligned Behavior Cloning via Bi-level Optimization (VACO)
Input:
Fixed offline dataset D, value network Qo. policy network meta-scoring network
update steps for value phase K1, update steps for bi-level phase K₂
// Value Training Phase
2 for update step k = 1...K do
Sample minibatch sample pairs (s. a, a.r,")))))))).. a') from D
Update value 9 according to IQL's(7)) TD learning
end
// Bi-level Optimization Phase
for update step k = 1...Kg do
Sample minibatch sample pairs (s, r, a') from D
Fix meta-scoring a and update policy & according to Eq
10
Fix policy & and update meta-scoring according to Eq
end
EXPERIMENTS
5.1 SETTING
D4RL. We utilize the MuJoCo and AntMaze domain tasks from the D4RL 11 benchmark for evaluation. Mujoco domain includes a variety of continuous locomotion tasks with dense rewards. Within this domain, walker2d. For each environment, we investigate four different v2 datasets, each representing distinct data quality level: medium, medium-replay, medium-expert, and expert. Consequently, the MuJoCo domain provides an excellent platform for assessing the effects of diverse datasets derived from policies varying proficiency levels.
perform experiments in three environments: halfcheetah, hopper, and
Baselines. We consider baselines including four different domain to provide thorough compar- ison. (1) Classic methods: (C(77), TD3 and CQI 28 (2) Explicit regularization methods:
TD3+BC(1), IQL (27), PRDC (4). T2) and (2) (3) Implicit regularization methods:
M1PO(51), (5)) and 22); (4) Return-conditioned methods: DT(5), DS4 and DC(3)::
Other methods: SAC-RND(35).
Setup. We implement our VACO framework by combining the official implementation (3) and (5). Specifically, value network, policy network and meta-scoring network are all 3-layer MLP. All hidden dimensions of the network are set to 256. Relu (I) activation is performed after each hidden layer. As for training, network and with a learning rate of 3e-4 for value and policy network. All experiments are conducted on single NVIDIA RTX 3090 GPU.
adopt Adam optimizer (25) with learning rate of 3e-5 for meta-scoring
More detailed experimental setting of D4RL dataset, baselines and hyperparameter setting, are all available in Appendix A
5.2 PERFORMANCE ON MUJOCO DOMAIN
Tab presents the performance outcomes for various algorithms, including other baselines and our VACO model, offline settings using the D4RL-Mujoco datasets. All scores are normalized, with a value of 100 indicating the performance level of expert policy, described by (11).
As shown, we observe the following: (1) Our VACO surpasses the various methods across almost all tasks and varying quality datasets. (2) Some methods exhibit performance deficiencies in certain
7

---

Published as a conference paper at ICLR 2025
Table 1: Averaged normalized scores on MuJoCo locomotion tasks for D4RL dataset. We evaluate 10 times averaged over 5 random seeds, + standard deviation. The dataset names are abbreviated as follows: 'medium' to 'm', 'medium-replay" to 'm-r', 'medium-expert to 'm-e', 'expent' 'e'. The first segment of the table contains classic offline methods, the second segment for explicit regularization methods, the third segment for implicit regularization methods and the fourth segment for return-conditooeed decision transformer methods. Our model outperforms various offline RL algorithms on almost all tasks. We mark the best results in bold and the second best with an underline for easy comparison. denotes averaged scores without 'expert' dataset.
Dataset:
Hopper
HaltCheetah
Walker2D
average
average
Method
m-c
TD3+BC
59.3
60.9
98.0
109.6
48.3
44.6
90.7
93.4
83.7
81.8
110.1
110.0
82.53
75.27
IQL
66.2
94.7
91.5
108.8
47.4
86.7
95.0
78.3
73.8
109.6
109.4
83.78
76.93
CQL
61.9
86.3
96.9
106.5
46.9
45.3
95.0
97.3
79.5
76.8
109.1
109.3
84,20
77.52
explicit regularization
methods:
PRDC
100.3
100.1
109.2
63.5
55.0
94.5
85.2
92.0
111.2
90.11
TD7
76.1
91.1
108.2
58.0
53.8
104.6
91.1
89.7
111.8
87.16
A2PO
107.4
47.1
44.7
95.6
84.9
112.1
implicit regularization manner methods:
MOPO
26.5
40.2
57.9
14.0
42.7
55.0
37.68
PLAS
32.9
27.9
111.0
39.3
43.9
96.6
44.6
30.2
89.6
57.33
EDP
81.9
101.0
97.4
52.1
49.4
95.5
6.9
94.9
110.2
85.48
relum-conditionss manner methods:
DT
67.6
82.7
107.6
106.3
42.6
36.6
86.8
92.4
74.0
66.6
108.1
107.6
81.58
74.73
DS4
87.7
110.5
109.3
47.3
43.8
94.8
89.1
81.4
80.3
109.6
105.7
87.42
82.77
DC
92.5
94.2
110.4
110.5
43.0
41.3
93.0
87.5
79.2
76.6
109.6
107.8
87.13
82.2
97.2
102.3
112.6
114.0
60.2
51.4
98.3
100.6
88.8
92.4
114.5
112.9
95.43
90.86
VACO
-4.2
17.1
42.2
14.5
-2.4
11.
17.2
11.5
118
14.9
10.3
-0.6
tasks or at specific data levels, whereas VACO consistently achieves competitive performance across the board. For example, MOPO(5I) fails on each expert datasets and EDP(22) gets large decrease on Walker2D medium dataset
5.3 PERFORMANCE ON ANTMAZE DOMAIN
Tab2preesent the performance outcomes of various algorithms on the AntMaze tasks, which primarily test the model's trajectory stitching capability. As shown, VACO algorithm achieved the highest average score and demonstrated competitive performance across tasks of varying difficulty levels. In contrast, TD3+BC(L1 (C3 and PLAS(55) failed to perform well the larger-scale tasks, while SAC-RND(5) and 22) showed significant drops in performance compared to our method on the medium-play and large-play datasets, respectively.
Table 2: Averaged normalized scores on AntMaze tasks. We evaluate 100 times averaged over random seeds, + standard deviation. Our model outperforms various methods on the average score of all tasks. We mark the best results in bold and the second best with an underline for easy comparison.
Tusk uutmaze-umant
BC
TD3+BC
IQL
CQL
PLAS
SAC-RND
EDP
VACO(ours)
65.0
66.3
83.3
74.0
70.7
97.0
94.2
96.4 +0.66
mammaze-umaze diverse
55.0
53.8
70.6
84,0
45.3
66.0
79.0
83.4 49.4
antmaze- medtau---mdiiumpppa
0.0
26.5
64.6
61.2
16.0
38.5
81.8
82.8 +3.8
nntaace-mediun-dieee
0.0
25.9
61.7
0.7
74.7
antmaze-large-play
0.0
0.0
42.5
15.8
0.7
43.9
42.3
63.8
114.
untmuze-large laruu---arge-iieeeee
0.0
0.0
27.6
14.9
0.3
45.7
60.6
53.2 117.4
Average
20.0
28.7
22.3
76.7
8

---

Published as a conference paper at ICLR 2025
BC without any weight
BC with reciprocal form weight
BC with advantage form weight
Percent Difference
Percent Difference
Percent Difference
Figure 3: Percent difference of the performance of comparison among our approach and other heuristic weighting strategy. HC=HalfCheetah, Hop=Hopper, W=Walker, m medium, mr = medium- replay, me mediim-experr, expert. As expected, our model obtains superior results.
5.4 COMPARISON WITH OTHER HEURISTIC WEIGHTING STRATEGY
In our work, we introduce a meta-scoring neural network to generate meta weights for weighted behavior learning. One may naturally wonder. can we adopt heuristic weighting strategies, assigning greater weights to samples with larger value estimation, as more efficient alternative? To answer above question, we compare VACO with two heuristic weighting strategies: 1) using the reciprocal of value as weight (13) and 2) using advantage-weit weeght regression (36) weight. As shown in 3ig[] the two hcuristic weighting strategies can boost performance of BC in some situations, but the improvement is far less than VACO almost all datasets. The reason behind this may be that the heuristic approaches are difficult to achieve alignment between the learned policy and value.
lel metescoring thout value nput
[bi metessoring se whouut value and astion input
(c) VACO withou: rahs
Percent Difference
Percent Difierence
Purcent Differnnce
-30
Figure 4: Percent difference of the performance of the ablations of our approach, compared to the full algorithm. HC=HalfCChetah, Hop=Hopper, W=Walker, m medium, mr medium-replay, me medium-expert, = expert.
5.5 ABLATION STUDY ABOUT THE NOISE AND DIFFERENT INPUTS OF META-SCORING NETWORK
To verify the effectiveness of proposed component, conduct ablation studies on the inputs of meta-scoring network and the controllable noise. In practice, we sequentially remove the value and state from the inputs, and compare the performance with original VACO configuration. As shown in Fa) and (b), removing the two parts of input results severe performance drop. especially medium and medium--eyy replayy datasets. This demonstrates the significance of the information within current input for sample weight determination. As shown in Fig c), the gradual reduced noise can boost model performance some extent, especially in halfcheetah environment, in where limited exploration is encouraged.
6 CONCLUSION
We present VACO, novel bi-level framework to balance OOD problem and value alignment issue concurrently for offline reinforcement learning. The framework comprises two workflow loop: the internal loop for weighted behavior cloning and the external loop for policy-value alignment. In detail, VACO additionally introduces a meta-scoring network for assigning different importance weights in-sample offline data. Extensive experiments demonstrate that proposed VACO achieves

---

Published as a conference paper at ICLR 2025
of- the-art performance variety of continuous control tasks the D4RL benchmark and the related ablation studies also verify the effectiveness of our method.
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China under Grant 62325101 and Grant 6203 1001, National Key Laboratory of Unmanned Aerial Vehicle Technology in NPU (Grant No. WR202403), and the Fundamental Research Funds for the Central Universities.
REFERENCES
[1] Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08377. 2018.
[2] Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. Cvae-gan fine- fine-grained image generation through asymmetric training. In Proceedings of the IEEE international conference on computer vision, pages 2745-2754, 2017.
[3] Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. arXiv preprint 1033:177033478.. 2017.
[4] Jérôme Bolte, Edouard Pauwels, and Samuel Vaiter. One-step differentiation of iterative algorithms. Advances in Neural Information Processing Systems, 36, 2024.
[5] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 1584411977,
2021.
[6] Xiaocong Chen, Siyu Wang, Julian McAuley, Dietmar Jannach, and Lina Yao. On the op- portunities and challenges of offline reinforcement learning for recommender systems. ACM Trunsactions on Information Systems, 2023.
[7] Peng Cheng, Xianyuan Zhan, Wenjia Zhang, Youfang Lin, Han Wang, Li Jiang, et al. Look beneath the surface: Exploiting fundamental symmetry for sample-efficient offline rl. Advances in Neural Information Processing Systems, 36, 2024.
[8] Shmuel Bar David, Itamar Zimerman, Eliya Nachmani, and Lior Wolf. Decision s4: Efficient sequence-based rl via state spaces layers. In The Eleventh International Conference on Learning Representations, 2022.
[9] Carl Doersch. Tutorial on variational autoencoders. arXiv preprint 1rXiv:1606.005000 2016
[10] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.1073 2021.
[11] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
[12] Scott Fujimoto, Wei-Di Chang, Edward Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[13] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20143 2021.
[14] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[15] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforeement learning without exploration. In International conference on machine learning, pages 2062-2062. PMLR, 2019.
10

---

Published as a conference paper at ICLR 2025
[16] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving Long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXfv:1910.11956 2019.
[17] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
[18] John R Hershey and Peder A Olsen. Approximating the kullback leibler divergence between gaussian mixture models. 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSPP00 volume 4, pages IV-317. IEEE, 2007.
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances neural information processing systems, 33:6840-6851, 2020.
[20] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-basee policy optimization. Advances in neural information processing systems, 32, 2019.
[21] Xingyu Jiang, Hongkun Dou, Chengwei Fu, Bingquan Dai, Tianrun Xu, and Yue Deng. Boosting supervised dehazing methods via bi-level patch reweighting. In European Conference on Computer Vision, pages 57-73. Springer, 2022.
[22] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[23] Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung. Decision convformer: Local filtering in metaformer is sufficient for decision making. arXiv preprint 10.0v:2310.03022, 2023.
[24] Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung. Value-aided conditional supervised learning for offline rl. arXiv preprint arXiv:2402.02017, 2024.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 112iv:1412.6980, 2014.
[26] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
[27] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-leaming, arXiv preprini arXiv:2110.06169. 2021.
[28] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179.
1 1191, 2020.
[29] Matt Kusner, Brooks Paige, and José Miguel Hernández-Lobato. Grammar variational autoencoder. In International conference on machine learning, pages 1945-1954. PMLR, 2017.
[30] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the-ar,, pages 45-73. Springer, 2012.
[31] Risheng Liu, Pan Mu, Xiaoming Yuan, Shangzhi Zeng, and Jin Zhang. A generic first-order algorithmic framework for bi-level programming beyond lower-level singleton. In International conference on machine learning, pages 6305-6315. PMLR, 2020.
[32] Cong Lu, Philip J Ball, Tim GJ Rudner, Jack Parker Holder, Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. arXiv preprint arXiv:2206.04777 2022.
[33] Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Dieter Fox. Iris: Implicit reinforcement without interaction scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4414 IEFF, 2020.
11

---

Published as a conference paper at ICLR 2025
[34] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leaming algorithms. arXiv preprint 1803. 2018.
[35] Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti- exploration by random network distillation. In International Conference on Machine Learning, pages 26228-2822 26244. PMLR, 2023.
[36] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. ewvantage-weightee regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 1rXiv:1910.00177, 2019.
[37] Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural compulation, 3(1):88-97, 1991.
[38] Yunpeng Qing, Jingyuan Cong, Kaixuan Chen, Yihe Zhou, Mingli Song, et al. Advantage-awand policy optimization for offline reinforcement learning. arXiv preprini arXiv:2403.07262, 2024.
[39] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:/709 10087, 2017.
[40] Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. Policy regularization with dataset constraint for offline reinforcement learning. In International Conference on Machine Learning, pages 28701-28717. PMLR, 2023.
[41] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta- weight-net: Learning an explicit mapping for sample weighting. Advances in neural information processing systems, 32, 2019.
[42] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387-395. Pmir, 2014.
[43] Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4rl: Surprisingly simple self-supervision for offline reinforcement learning robotics. In Conference on Robot Learning, pages 907-917. PMLR, 2022.
[44] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
[45] Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 36:1 1692-11620,, 2023.
[46] Xinyi Wang, Hieu Pham, Paul Michel, Antonios Anastasopoulos, Jaime Carbonell, and Graham Neubig. Optimizing data usage via differentiable rewards. In International Conference on Machine Learning, pages 9983-9995. PMLR, 2020.
[47] Yuanfu Wang, Chao Yang, Ying Wen, Yu Liu, and Yu Qiao. Critic-guided decision trans- former for offline reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 15706-15714, 2024.
[48] Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. Advances Neural Information Processing Systems, 35:31278-312222 2022.
[49] Taku Yamagata, Ahmed Khalil, and Raul Santos-Roude RodriggR Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. In Interna- tional Conference on Machine Learning, pages
PMLR, 2023.
[50] Rui Yang. Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Ro- bust offline reinforcement learning via conservative smoothing. Advances neural information processing systems, 35:23851-23866 2022.
12

---

Published as a conference paper at ICLR 2025
[51] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-1444 14142, 2020.
[52] Huining Yuan, Hongkun Dou, Xingyu Jiang, and Yue Deng. Task-aware world model learning with meta weighting via bi-level optimization. Advances in Neural Information Processing Systems, 36, 2024.
[53] Chi Zhang, Sanmukh Kuppannagari, and prasanna Viktor, Brac+: Improved behavior regular- ized actor critic for offline reinforcement learning. In Asian Conference on Machine Learning, pages 204-219, PMLR, 2021.
[54] Jing Zhang, Chi Zhang, Wenjia Wang. and Bingyi Jing. Constrained policy optimization with explicit behavior density for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[55] Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning, pages 1719-1735. PMLR, 2021.
[56] Wenzhuo Zhou. Bi-level offline policy optimization with limited exploration. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
A ADDITIONAL EXPERIMENTAL SETTINGS
A.1 DETAILS OF D4RL
In our study, we utilize two types of environments in D4RI 2 experiments.
Gym Mujoco and Antmaze to conduct
A.1.1 GYM MUJOCO
We focus on three primary environments Gym Mujoco: Hopper, Half-Cheetah, and Walker2d, shown in Fig 5 These environments well-suited for testing the robustness and efficacy of reinforcement learning algorithms across spectrum of data quality levels, each derived from different policies. We adopt the v2 datasets ,each reflecting four distinct types: "medium", "medium-replay"" "expert", and "medium-expert"" reflecting varying degrees of policy optimization and performance.
(a)
(b)
(c)
Figure 5: Visulization of (a) Hopper, (b) Walker2d and (c) Half-Cheetah.
Medium. Each "medium" dataset is generated by training policy using the Soft Actor-Crite algo- rithm. The training process is halted one-third the performance of an expert policy. This early stopping ensures that the policy is only partially trained, simulating suboptimal policy performance.
once the policy reaches approximately
Medium-replay, The "medium-replay" datasets consist of all samples stored in the replay buffer during the training phase of the medium policies. This dataset captures a wide array of experiences, both successful and suboptimal, encountered by the policy as it progressed towards its medium-level performance cap.
Expert. The "expert" datasets comprised solely of data derived from fully trained policy, where the performance is considered to be optimal or optimal. These samples provide a benchmark of high-performance behavior each environment.
Medium-expert. The "medium-expent" datasets are constructed by mixing equal parts of samples from expert demonstrations with those from the medium datasets. This mixture offers unique blend of high-quality and average-quality samples, useful for assessing the algorithm's capability to generalize across different levels of policy proficiency.
A.1.2 ANTMAZE
The AntMaze domain is a navigation environment that replaces the simple 2D ball from Maze2I) with complex 8-DoF "Ant" quadruped robot. Data is collected by training goal-reaching policy, combined with the same high-level waypoint generator from Maze2I.. This generator provides subgoals that guide the Ant robot towards the final goal. The terms "umaze." "medium," and "large" refer to different map sizes, as shown in Fig
Specifically, 1) in the "antmaze-umaze" setting, the Ant tasked with reaching specific goal from fixed start location. 2n the "diverse" datasets, the Ant is directed to random goal from a random start location. 3) In the "play" datasets, the Ant guided to specific, hand-picked locations within the
D4RL dataset available at http:://gthhh..cmm//htteeeelllll
14

---

Published as a conference paper at ICLR 2025
Figure 6: Visualization of the Antmaze environment, with "umaze," "medium," and "large" from left to right.
maze (which not necessarily the evaluation goal), starting from different set of hand- hand-picked start locations.
A.1.3 FRANKA KITCHEN
The Franka Kitchen (16) domain involves controlling 9-DoF Franka robot in kitchen environ- ment, as shown in Fig 7 with common household items like microwave, kettle, overhead light, cabinets, and an oven. The goal of each task to interact with these items to achieve a desired goal configuration. The "complete" dataset consists of the robot successfully performing all tasks in sequence, making it straightforward for imitation learning methods. contrast, the "partial" and "mixed" datasets contain undirected data, where the robot performs subtasks unrelated to the goal configuration. In the "partial" dataset, portion of the data guarantees successful task completion, allowing imitation learning agents learn from the right subsets selectively. The "mixed" dataset contains no full task completions, requiring RL agents to piece together relevant sub-trajectories to solve the tasks.
Figure 7: Visualization of the Franka Kitchen environment
1.4 ADROIT
The Adroit domain (39), as shown in Fig 8 involves controlling a 24-DoF Shadow Hand robot to perform four tasks: hammering a nail, opening a door, twirling a pen, and picking ball. In the D4RL benchmark, the Adroit domain includes three types of data: human (25 trajectories), expert (large RL policy data), and cloned (data from an imitation policy mixed with demonstrations), The human dataset designed to test the impact small amounts of human demonstration data, the expert dataset evaluates the performance with large amounts of expert data, and the cloned dataset simulates a scenario where a small amount of additional data is collected by an imitation policy trained on demonstrations. The domain is challenging due to sparse rewards, high nigh-dimmnsiinal tasks, and narrow expert data distribution, making it a benchmark for evaluating offline RL approaches in realistic, manipulation settings.
15

---

Published as a conference paper at ICLR 2025
(d)
(d)
(e)
Figure 8: Visualization of the Adroit environment: (a) picking up a ball given place; (b) twirling
a pen; (c) opening door; (d) hammering a nail; (e) 24 degree of freedom ADROIT hand.
A.1.5 V-D4RL
V-D4RL (32) is an offline reinforcement learning dataset based on the D4RL format, designed for vision-based high-dimensional observations. As shown in Fig 9 it includes three tasks from the DMControl Suite: walker-walk, where a planar walker aims to stay upright and maintain a target velocity: cheetah-run, where biped agent is rewarded for its forward velocity; and humanoid- walk, a challenging task with a 21 1-jjointed humanoid aiming for a target velocity. For each task, we use two levels of data: medium, collected from medium- -erftumm-erfoomacce policy, and expert, collected from high-pes segh-performacc expert policy.
K
Figure 9: Visualization of the V-D4RL environment.
A.2 BASELINE DETAILS
We consider baselines including following different domain to provide a thorough comparison.
(1) Classic methods: Behavior Cloning (3). TD3 (4). and CQL (28); (2) Explicit regularization
methods: TD3+BC (33), IQL (7)), 40) T2) and A2PO 38 (3) Implicit regularization
methods: MOPO (1), PLAS (55). and EDP (2); (4) Return-conditioned methods: DT (5), DS4 (8),
and DC (23). We obtain baseline performance scores for CQL, TD3+BC and IQL from (27). for DT,
DS4 and DC from (23), for BRAC+ from (53). for MOPO from (5I) for PLAS from (55). for EDP
from (22). For TD3, BC and methods lacking results in expert dataset, we reproduce the results using the code provided by the respective authors.
As for the additional Adroit tasks, we add RORL (50) as new baseline. As for Kitchen tasks, we add as new baseline.
(2) enhances TD3(14) for offline RL with the State-Action Learned Embeddings (SALE) mech- anism, capturing atate-action dynamics for better predictions and generalization. It also incorporates a behavior cloning term to maintain policy alignment with the dataset distribution, ensuring stable offline training. SAC-RND(35) extends Soft Actor-Critic (SAC) with Random Network Distillation (RND) to pemalize -uitt--distributinn actions, reducing reliance on ensemble-based methods for efficiency. It uses Feature-wise Linear Modulation (FiLM) to enhance conservatism, ensuring stable offline learning without direct environment interactions. A2PO(38) resolves constraint conflicts in offline RL by using Conditional Variational Auto-Encoder (CVAE) to disentangle behavior policies based on advantage values. This approach enables prioritization of high-advantage actions while preserving policy diversity, leading improved advantage estimation and performance
16

---

Published as a conference paper at ICLR 2025
diverse datasets. PRDC 40)
addresses value overestimation in offline RL by regularizing policies to align with the nearest state-action pair in the dataset. This softer point-to-set constraint balances generalization and conservatism, reducing overestimation and improving performance across tasks.
A.3 IMPLEMENTATION DETAILS OF VACO
We implement VACO by combining the official DT code transformer), TD3+BC code httpss//iihub..co//sjjjmi BC) http:://sttbbb.coo//sssrrroo///p .
and IQL
code
B RUNNING TIME COMPARED WITH OTHER METHODS
3 presents comparison of the total runtime of our method against other approaches. We evaluate thest methods using the PyTorch library on a single NVIDIA RTX 3090 GPU for le6 update steps. Compared to (2)(2)) our method is 1.7 times slower; while compared to DT(5), our method is 2.6 times faster. It is worth mentioning that the meta-scoring network introduced in our framework can be removed during the testing phase, which has impact on the forward inference time of the policy model.
Table 3: Total running time compared with other methods.
Methods
BC
TD3+BC
IQL
DT
DC
VACO
total running time
1.3h
3.2h
3.4h
14.9h
12.6h
5.8h
scores on locomotion tasks
64.57
82.53
83.78
81.58
87.13
95.43
C CLARIFICATION ABOUT
0 0
Da
Following recent methods 46 21 in meta-learning, make the Markov assumption, under which
we assume 0b O. This approximation widely used in the field of bi-level optimization(4] 31 da and is referred to as the "one-step differentiation. Under this assumption, at step t, given 6;-1 we do not consider about the specific values of from previous steps that led Φ1-1- Intuitively, this assumption suggests that a;-1 has already been updated with respect to Φ1-1, so the effect of a on Φ likely minimal. (46)
Of course, more accurate gradients can be obtained by iteratively applying the update formula for à (Eq 7 over multiple steps, known as "K-step differentiation" li-le optimization domain. However, is evident that achieving more precise gradients comes at the cost of significantly increased complexity, both in terms implementation difficulty and computational/memoryy overhead. More
0d.-1 RJ 0) agrees to recent (or has demonstrated that the one-step differentiation (i.e., assuming dia
an upper bound the between the one-step differentiation gradient and the theoretical gradient. For more details, see Corollary 4 in (4). Additionally, in extensive experiments, (4) showed that the one-step differentiation achieves an error of order 10 in logistic regression problem.
Building on the theoretical and practical foundations established by these prior works, we believe that Ddh-1 Rs 0 is valid. This assumption allows for significant simplification and acceleration of computation, within an acceptable range gradient error. As stated in (46), *"Our use of the Markov assumption is based its and empirical success in previous work on bi-level optimization, such as Hyper Gradient (3 and many others. Of course, this a simplifying assumption, but we believe that our empirical results show that the proposed method is useful nonetheless. Relaxing this assumption would be an interesting avenue for future work. However at the same time how to do so without resulting in large increases in complexity is challenge that would require additional methodological advantages beyond the scope of the current work."s
Da
17

---

Published as a conference paper at ICLR 2025
D ADDITIONAL EXPERIMENTS ON ADROIT AND KITCHEN DOMAIN
We further evaluated the proposed VACO method on the Adroit and Kitchen(16) environments of the D4RL benchmark. and 1 the experimental results the Adroit and Kitchen datasets, respectively (partial results adopted from (45), thanks for their awesome work). While our method does not achieve the best performance on certain individual datasets, it consistently demonstrates the best average performance across all datasets compared to other methods, which highlights the competitiveness of our approach.
Table 4: Averaged normalized scores Adroit tasks. We evaluate 10 times averaged over 3 random seeds, standard deviation. Our model outperforms various methods on the average score of all tasks. We mark the best results in bold and the second best with an underline for easy comparison.
Task Name
BC
BEAR
TD3+BC
IQI.
CQL.
SAC-RND
RORL
VACO(ours)
Pen-human
34,4
-1.0
81.8 114.9
81.5
1175
37.5
5.6 +5.8
33.7 17.6
106.0 111.
Pen-cloned
56.9
26.5
61.4 119.3
71.2 117.
39.2
2.5 16.1
35.7 13.1
98.3 15.8
Pen-expert
85.1
105.9
146.0 +7.3
133.6 16.0
107.0
45.4 +22.9
130.3 +4.2
144.8 +4.8
Door-human
-0.3
-0.1 10.0
#2.0
0.0 40.1
3.78 +0.7
6.1 #3.8
Door-cloned
-0.1
-0.1
0.1 10.6
0.8 =1.0
0.4
0.2 40.8
-0.1 #0.1
1.6 115
Door-expert
34.9
103.4
84.6 +-4.55
105.3 +2.8
101.5
73.6
+26.7
104.9 +0,9
104.7 +1.0
Hammer-human
0.3
0.4 10.4
215
-0.1 =0.1
=19
1.8 1009
Hammer-cloned
0.8
0,3
0.8 10.7
1.1 =0.5
2.1
0.1 +0.4
1.7 =0.5
2.5 1.9
Hammer-expert
125.6
127.3
117.0 :30.9
129.6 10.5
86.7
24.8 +39.4
132.2 10.7
127.0 10.4
celocate-human
-0.3
-0,2 10.0
0.1 =0.1
0.2
0.0 10.0
=0.0
0.6 10.4
Relocate-cloned
-0.1
-0.3
-0.1 20.1
0.2 #0.4
-0.1
0.0 40.0
0.0 =0.0
0.1 10.1
Relocate-oxpert
101.3
98.6
107.3 11.6
106.5 12.5
95.0
3.4 14.5
47.8 133.5
107.3 +3.0
Average
38.4
53.4
12.9
58.4
Table 5: Averaged normalized scores Kitchen tasks. second best with an underline for easy comparison.
We mark the best results in bold and the
Task Name
BC
BCQ
BEAR
CQL
IQL
RyS-G
PLAS
VACO(ours)
Kitchen-complete
65.0
9.1
43.8
62.5
50.2
38.1
60.0
Kitchen-partial
38.0
17.6
0.0
49,8
46.3
60.3
27.0
52.5
Kitchen-mixed
51.5
11.5
51.0
51.0
51.4
30.0
57.5
Average
51.5
12.7
0.0
48.2
53.3
$4.0
31.7
56.7
E ADDITIONAL EXPERIMENTS ON V-D4RL DOMAIN
In addition to the D4RL dataset, we also conducted experiments on the V-D4RL dataset. Similar to [I), V-D4RL provides analogous problem settings and uses the same data collection process. The key difference is that in V-D4RL, the agent's state observations are replaced with image-based representations of the environment. Following the protocol of the original V-D4RL work we present the corresponding experimental results in Tab Compared to other methods, proposed performance on most tasks.
approach achieves state-of-the- the-art or of
Table 6: Averaged normalized scores on V-D4RL tasks. We evaluate 10 times averaged over random seeds, + standard deviation. We mark the best results in bold and the second best with an underline for easy comparison.
Environment
Datuset
Offline DV2
DrQ+BC
CQL
BC
LOMPO
VACO(ours)
medium
34.1 119.7
46.8 42.3
14.8 =16.1
40.9 13.1
43.4 111.111
53.8 +1.8
walker-walk
expert
10.6
68.4 17.5
19.6
91.5
+3.9
5.3 17.7
92.4 4:3.7
medium
17.2
$3,0
40.9 +5.1
51.6
16.4
53.7 +0.6
cheetah-run
expert
10.9 +3.2
34.5 +8.3
61.5 4:433
67.4 #6.8
14.0
75.8 +6.5
medium
0.2 +0.1
6.2 4224
0.1 =0.0
13.5 #4.1
0.1 10.0
15.3 +2.2
humanoid-walk
expert
0.2 10.1
2.7 40.9
-05
6.1 13.7
0.1 10.0
4.3 10.8
Average
11.2
35.3
34.8
45.2
13.2
49,2
18

---

Published as a conference paper at ICLR 2025
F ANALYSIS ABOUT THE RELATIONSHIP BETWEEN META WEIGHT AND Q-VALUB
To examine the relationship between the learned meta-weights and Q-values, conducted exper- iments on the Hopper medium-expert dataset, which comprises a combination of expert-level and suboptimal trajectories. As depicted in Fig 14, the meta-weights produced by the meta-scoring network are presented alongside the corresponding Q-value estimates for each state-action pair, visualized through scatter plot.
For statistical assessment, we employed the Pearson correlation coefficient to quantify the linear association between Q-values and meta-weights across all state-action pairs in the Hopper medium- expert dataset. The computed correlation coefficient of 0.24 suggests weak positive correlation, indicating that the meta-scoring network exhibits tendency to assign higher weights to actions with greater Q-values, while lower weights are generally assigned to actions with smaller Q- Q-values. Nonetheless, the relatively low magnitude the correlation coefficient reveals that this relationship is not strictly proportional, with some instances deviating from this trend. This also indicates that meta-score not only affected by Q-value, but also related to state and action.
type
State-action pairs on Hopper medium- dataset
Q-value
...-1.01 -..01-0.63-00 0011 -0.1.01-...3...11-..500 0.14 0.21 0.32 0.76 1.23
weight
0.29 0.36 0.95 0.48 0.84 2.55 1.05 2.11 1.50
Meta weight (Y-axis)
2-
-2.0
-0.5
0.5
1.5
Q-Value (X-axis)
The Pearson correlation coefficient of weight and Q-value: 0.24
Figure 10: Visualization of the learned meta weights and corresponding Q-value estimation on Hopper medium-exppe exepert-expert dataset. The Pearson correlation coefficient indicates that there is a weak positive correlation between the weights and Q-value.
19

---

Published as a conference paper at ICLR 2025
G ANALYSIS ABOUT THE RELATIONSHIP BETWEEN META WEIGHT AND TRAJECTORY REWARDS
Meen and Variance of meta weight
4.5-
4.7-
Averge WOOgH
4.00
2.50
125
390
300
beow anong segments according
aboae
The total reward trajectory
(a)
(b)
The Pearson correlation coefficient of meta weight and trajectory total rewards: 0.954
Figure 11: Visualization of the learned meta weights and corresponding trajectory total rewards on Hopper medium-cxpert dataset. (a) A scatter plot showing the total trajectory reward versus the average meta-weight of state-action pairs across the entire trajectory. The trajectories are divided into three groups based the total reward: 1) total reward below 500, 2) total reward between 500 and 3500, and 3) total reward above 3500, representing trajectories generated by "random," "medium," and "expert" policies, respectively. (b) A visualization of the mean and variance distributions of meta-weights corresponding to the three trajectory groups categorized by total reward. The Pearson correlation coefficient indicates that there is strong positive correlation between the average meta weights and trajectory total rewards.
Appendix F illustrates a weak positive correlation between meta-weights and Q-values, indicating that higher Q-values do not necessarily correspond to higher meta-weights. To further explore the intrinsic nature of the learned meta-weights, we conducted additional experiments on the Hopper medium-expert dataset, focusing the relationship between the average meta- weight of entire trajectories and their corresponding total rewards.
Specifically, the Hopper medium-exper dataset consists of 3213 trajectories generated by various policies, encompassing a total of 1,999,400 state-action pairs. The total trajectory rewards span the range [315.87, $759.08]. We categorized the trajectories into three groups based on their total rewards: trajectories with rewards below 500 are considered to originate from "random" policies, those between 500 and 3500 from "medium" policies, and those above 3500 from "expert" policies.
Fig (a) presents a scatter plot of the average meta-weight for each trajectory against its total reward. In this visualization, blue points represent trajectories generated by random policies, orange points correspond to medium policies, and red points indicate expert policies. To statistically quantify the linear relationship between average meta-weights and total rewards, we computed the Pearson correlation coefficient, which yielded a value of 0.954. This result demonstrates strong positive correlation between weights and total rewards.
Fig visualizes the mean and variance of the average meta-weights for trajectories across the three policy groups. The results show a clear separation of meta-weights among trajectories generated by different policy groups.
These findings suggest that weig-weights can effectively reflect the quality the data: higher meta- weights are associated with state-action pairs more likely generated by expert policies, whereas lower weights are linked to those generated by random policies. This is an exciting discovery, as provides novel insights into the functionality and interpretability of the meta-wel weights learned by the meta-scoring network. This observation also aligns with the original design intent of the meta-scoring network for distinguishing among various data quality.
20

---

Published as a conference paper at ICLR 2025
H ANALYSIS ABOUT THE TRAINING CURVES OF THE INNER AND OUTER LOOP
As shown in Fig 12 we present the training curves of the inner and outer loops on the Hopper expert dataset. During training, the weighted actor loss in the inner loop decreases continuously, while the Q maximization in the outer loop increases until it stabilizes. Notably, when the Q maximization in the outer loop stabilizes, the weighted actor loss in the inner loop still tends decrease.
This phenomenon aligns with the goal of our designed bi-level optimization framework, where we aim find an optimal weighted behavioral cloning policy. This policy not only aligns well with the value function (i.e., maximizes Q) but also minimizes the distance to in-sample state-action pairs. Therefore, the relationship between the inner and outer loops is more complementary rather than adversarial.
Hopper-expert
Hopper-expert
Inte kopewagght scue 1900
20
0.4
1000
Grofient stenslle3)
Gradient stepsle3)
Figure 12: Visualization of the inner loop and outer loop training curves. (a) illustrates the curve of the weighted actor loss function in the inner loop with the number of updates; (b) depicts the curve of Q maximization in the outer loop with the number of updates.
21

---

Published as a conference paper at ICLR 2025
1
ToY EXAMPLE FOR THE ALIGNMENT OF VACO
(a)
(b)
height
Velocity=10m/s
reward
Action:
State: h
state
distance
Reward:
(c)
action
(d)
Methods
Distance
BC
13.396
IQL
14.012
VACO(ours)
14.143
Figure 13: (a) A schematic diagram of the toy example environment: (b) Visualization of the learned Q-value function; (c) Alignment between different methods and the value function when the state (height) is fixed 2m; (d) Experimental results of various methods in the toy example environment.
To intuitively validate the performance improvements brought by alignment, we designed the toy example environment shown Fig [3]] a). In this environment, ball is projected with an initial speed of 10 m/s from an initial height à and angle 0. To simplify the modeling, we only consider the effect of gravity. The initial height serves the state input, the projection angle the action, and the final distance traveled by the ball when its height drops to zero as the reward.
For offline reinforcement learning training, randomly sampled 500 points within the state height range of [O, 10]. For each state, we further randomly sampled 4 points within the action angle range of [O, n/21, resulting in total of 2000 offline samples as the offline dataset. We trained BC, IQL, and VACO on this dataset and tested these three methods at 100 equally spaced points within the state height range of [O. 10]. [3 d) presents the experimental results of BC, IQL, and VACO. Comparatively, our method achieves superior performance.
In the experiments, IQL and VACO share the same value function, and Fig 13 b) visualizes the learned value function. For intuitive comparison, we plotted the cross-section when the state height fixed at 2m in Fig (3 The horizontal axis represents the action, while the vertical axis represents the value function. The blue curve illustrates the relationship between actions and the true rewards, and the orange curve represents the relationship between actions and the learned value function. The green, red, and blue vertical lines represent the action outputs of IQL, VACO, and BC at a state height 2m, respectively. It can observed that the action output by our method is closer to the optimal point of the value curve, resulting in the highest corresponding reward.
22

---

Published as a conference paper at ICLR 2025
J MORE DISCUSSION OF VACO
min (()
(10)
s.t. o (a), =
=arg argmin min Bc( + Jo())
6,0
We can place the updates of the value network within the internal loop, allowing simultaneous optimization of both the value and policy networks, as demonstrated in Eq I0 As mentioned the manuscript, since there is no intersection between the value network and either the policy network the meta-scoring network, introducing value network does not affect the optimization processes of the latter two. The pseudo-code incorporating value updates is presented in Algorithm 2
As illustrated in Eq T0, VACO can be considered a novel training framework for offline reinforcement be substituted with appropriate value estimation loss function,
learning. In this framework, Jo (f) such as adopted in IQL (27) or CQL (28), and Jg (6) can be replaced with appropriate behavior cloning-based policy extraction loss function, such AWR (36). These substitutions indicate the potential applicability of VACO in the field of offline reinforcement learning.
Algorithm Value-aligned Behavior Cloning via Bi-level Optimization (VACO)
Input: Fixed offline dataset D, value network Qo, policy network meta-scoring network Wa. update steps K
for updale step k = 1...K do
Sample minibatch sample pairs (8, a, r, a') from D
Update value 9 according to IQL's (27) TD learning
Fix meta-scoring a and update policy @ according to Eq
Fix policy & and update meta-scoring according to Eq
end
23

---

Published as a conference paper at ICLR 2025
3000
2500
1000
1000
500
200
400
500
800
3000
200
400
800
3003
Gracient staps(3el1
Gradient stepsi1e23
(a) Hopper expert
(b) Hogper medium--xpert
[a] Haffcheetah expert
(b) Halfcheetah medium expert
5000
2000
2000
3000
2000
2000
1000
3000
Cradient staps/1e3) expert
Graiilent stepsile3)
Walker2d medium--eupe expper
Figure 14: Learning curves for MuJoCo tasks.
24

---

Published as a conference paper at ICLR 2025
Module
Name
Value
Value network
Batch size
256
Learning rate
3e-4
Number of layers
Update steps
1e6
Hidden dim
256
Policy network
Batch size
256
Leaminng rate
3e-4
Number of layers
3
Update steps
le6
Hidden dim
256
Meta-scoring network
Learning rate
3e-5
Number of layers
3
Hidden dim
256
Common
Discount
0.99
Activation function
ReLU
Optimizer
Adam
Adam epsilon
le-5
Weight decay
le-6
of noise
1-> 0
Table Main hyperparameters of VACO. All experiments were run single Nvidia RTX 3090 GPU with Python 3.7 and Torch 1.12.0. Refer to our sample code for full implementation details.
25


Reviews:
[{"review_id": "fY511EOcLy", "reviewer": "Reviewer_BNkz", "summary": "In offline RL, behavior constraints (BC objectives) and value estimation (Q-values) are commonly used strategies to train policies. However, when both strategies are applied together, they tend to interfere with each other, often resulting in sub-optimal performance. To address this issue, the paper proposes VACO, which first learns the value network (Q-function) and then employs a bi-level learning process: training a meta-score network in the outer loop, and updating the policy using a weighted BC objective in the inner loop. The proposed VACO method is compared with several baselines on tasks from the D4RL dataset. Experimental results demonstrate that VACO achieves superior performance across tasks and various data resource settings.", "questions": "1. Could you please clarify how the assumption on line 261 is derived? I believe the Markov property may not be an appropriate basis for this assumption.\n2. Does the proposed VACO demonstrate sufficient generalization ability? Can it still achieve superior performance on more challenging tasks, such as those in the Adroit or Robomimic datasets?\n3. Can the learned meta-score network assign low weights to state-action pairs collected by policies that exhibit poor performance? Any visualizations or discussions would be helpful.\n4. Please provide more details about the compared baselines so we can better assess the superiority of the proposed method.\n5. From line 748 to line 753, the paper states that any value estimation loss function and any behavior cloning-based policy extraction loss function can be used. Is this claim supported by any experiments?\n6. Why is Gaussian noise used in Eq. 6? Additionally, why is the noise injected into the state rather than into the intermediate latent variables or the output logits?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Paper Writing:** The paper has a coherent and concise narrative. The challenge is clearly described, the proposed method is easy to understand, and the figures and tables are well-designed, allowing readers to quickly grasp the key points.\n\n- **Method Originality:** Although applying learnable models/parameters to dynamically adjust the weight of each training sample is not a new idea, the insight to develop bi-level optimization in the context of behavior constraints in offline RL is underexplored, lending sufficient originality to the method.\n\n- **Reproducibility:** The inclusion of source code and detailed experimental information enhances the paper’s reproducibility.", "weaknesses": "**Major Concerns**\n- **Theroem Correctness:** While the proposed method seems to have enough originality, I have doubts about its supporting theorem, particularly the assumption on line 261, i.e. $\\frac{\\partial\\phi_{t-1}}{\\partial\\alpha} \\approx 0$. The paper indicates that this assumption is supported by the Markov property. However, if I understand correctly, the Markov property primarily concerns state transitions in a stochastic process, where the future state depends only on the current state (and action, in the case of an MDP) and not on past states. In this scenario, however, it involves dependencies between functions across iterations, not probabilistic state transitions. I believe the divergence of $\\phi_t$ and $\\phi_{t-1}$ should be evaluated to better determine whether this assumption is valid.\n- **Evaluation Tasks:** The proposed method and baselines are only examined on simple tasks from one benchmark, i.e., D4RL, making it unclear whether the comparison results hold sufficient generalization. I recommend conducting the evaluation on Adroit tasks (also from D4RL) at least, or on more complex benchmarks such as Robomimic [R1], which also provides collected trajectories with different levels of expertise.\n- **Ablation Studies:** The paper lacks in-depth ablations. While conducting the study on removing inputs of the meta-score network is a nice start, it is more basic. More comprehensive ablations, such as visualizing the relationship between assigned weights and the state-action pairs collected by different policies, could better demonstrate the effectiveness of the proposed method.\n- **Baseline Details:** The paper lacks descriptions of the selected baselines. Although the paper categorizes baselines into four groups, there is no explanation of each baseline's key insights or features, raising concerns about whether the compared baselines are robust enough.\n\n---\n\n**Minor Concerns**\n- **Task Setting:** While I acknowledge that offline RL is a critical area, especially considering safety and training efficiency, the setting studied in the paper is somewhat dated; the policy can only learn from a single collected dataset. Recent offline RL works have explored more diverse directions, such as auxiliary unlabeled data [R2, R3], primitive discovery [R4], or offline-to-online settings [R5], which may have the potential to create a greater impact.\n\n- **Experimental Results:** As mentioned in the appendix, some results from previous works are copied. I suspect this is also the reason for some missing fields in Table 1. I suggest conducting experiments for all methods under the same conditions (same devices and random seeds) to better demonstrate the fairness of the comparison.\n\n- **Typos:** While the paper is generally well-written, there are some typos that should be corrected. For instance:\n\n  line 094, line 478: \"In details\" -> \"In detail\"\n\n  line 227: \"Gauss noise\" -> \"Gaussian noise\"\n\n  line 236: \"square error\" -> \"squared error\"\n\n  line 239: \"Through such bi-level optimization framework\" -> \"Through such a bi-level optimization framework\"\n  \n  line 307: (1) classical methods: (1) Explicit regularization methods: -> This line needs clarification or correction.\n\nNote: Addressing these concerns is highly encouraged, but they did not substantially affect my evaluation.\n\n---\n\n[R1] Ajay Mandlekar et al., \"What Matters in Learning from Offline Human Demonstrations for Robot Manipulation,\" CoRL'21.\n\n[R2] Konrad Zolna et al., \"Offline Learning from Demonstrations and Unlabeled Experience,\" NeurIPS'20.\n\n[R3] Tianhe Yu et al., \"How to Leverage Unlabeled Data in Offline Reinforcement Learning,\" ICML'22.\n\n[R4] Anurag Ajay et al., \"OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning,\" ICLR'21.\n\n[R5] Shenzhi Wang et al., \"Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning,\" NeurIPS'23.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TdkyX64eUD", "reviewer": "Reviewer_8RhL", "summary": "Summary:\n\nThis paper introduces a novel bi-level offline DRL framework called value-aligned behavior Closing via bi-level optimization (VACO).\nThe authors identify critical challenges in offline RL, particularly out-of-distribution (OOD) and value alignment issues.\nTo address these issues, the VACO framework employs a bi-level optimization strategy integrating a meta-scoring neural network to generate adaptive weights for behavior learning.\nIn the various tasks in the D4RL benchmark, VACO achieves state-of-the-art performance compared to other SOTA methods.", "questions": "Please check the weakness section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Strengths\n\n1. This paper's writing is excellent. It utilizes illustrative figures to clearly demonstrate the out-of-distribution (OOD) and value alignment issues in offline DRL and effectively explains the motivation behind the method. The methodology and experimental setup are clearly presented, with an appendix and code provided for additional details.\n\n2. This approach combines the maximization of the value estimation function with weighted behavioral cloning from a novel perspective, proposing a bi-level learning framework that simultaneously mitigates OOD issues and value alignment issues.\n\n3. The method was tested across multiple tasks on the D4RL benchmark. The results are impressive, as the method, using a simple MLP, surpasses most other methods, including diffusion networks, and achieves the highest average performance.", "weaknesses": "Weaknesses\n\n1. The name of the method may not be entirely appropriate. Although it includes \"bi-level optimization,\" in lines 9 and 10 of the algorithm, the two networks are alternately trained without forming an outer and inner loop. This approach resembles adversarial training, similar to the generator and discriminator in GANs. A more suitable name might be \"Adversarial/Complementary Value-Aligned Behavior Cloning.\"\n\n2. Could the authors further explore the relationship between maximizing the value estimation function and weighted behavioral cloning in Equation 6? For instance, do they exhibit an adversarial relationship, or are they complementary? It would be helpful to provide training loss curves or other visualizations of the two losses to clarify this relationship.\n\n3. Could the authors provide additional results for tasks in the adroit and kitchen environments in D4RL, as well as learning curves for MuJoCo tasks?\n\n4. I am very curious about the relationship between the meta-score and Q-value for a given state-action pair. Could you provide some statistical analysis? Do they exhibit a proportional relationship, such that a larger Q-value corresponds to a higher meta-score? If so, could the Q-value be used as a substitute for the meta-score?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0jGrltXJ14", "reviewer": "Reviewer_M9oj", "summary": "This work proposes a novel algorithm, VACO, that combines two offline RL approaches: behavior cloning and value estimation, in a meta-learning framework. In particular, the outer loop uses value estimation to train the weights of behavioral cloning data samples. The approach is designed to balance common issues in offline reinforcement learning: namely, the over-estimation of the value of out-of-distribution state-action pairs, and convergence on sub-optimal actions in-distribution.", "questions": "For the experiments, is the same value network used across all approaches in each environment? Or does each approach have a separately trained value network?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The motivation is clear: the issues with current offline RL methods are clearly established.\n\nThe approach is original: it combines two well-established methods in a novel way that produces better results than either method alone.\n\nSOTA performance on an established benchmark.", "weaknesses": "The related works section (Section 5) is placed after the experiments (Section 4), despite describing the methods compared in the experiments. This made the experiments section much harder to parse through on the first pass. Placing this information somewhere before Section 3 could greatly improve the paper’s clarity on the first readthrough.\n\nThe writing is hard to parse in a few places. In the abstract (lines 25-26), the description of inner and outer loops appears to suggest that the outer loop only adds noise, rather than training the weighting, which it does according to the description in 3.3.\n\nThere is no analysis on training costs (number of iterations required or wall time). Given the use of meta-learning, I would assume that the approach is generally more costly than most of the alternatives. Because of the competitive performance of the algorithm, it doesn’t need to be faster than other methods, but cost would be an important concern in determining when to use this approach versus a simpler one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sfp5qdo10v", "reviewer": "Reviewer_vQjh", "summary": "The paper proposes to address the issue of balancing the value estimation and behaviour cloning in face of data of a mixed quality. The proposed idea is to turn the policy learning and behaviour regression losses to a bi-level optimization. The core idea is to learn a score network to assign a weight to each datapoint in the inner loop such that the learning signal from low-quality data is downplayed. The score network is tuned in the outer loop with a form of policy learning loss with a regularization of adding Gaussian noise to the predicted action. The inner loop is expanded as a one-step update rule so the gradient can be derived. The approach is tested on Hopper, HalfCheetah, Walker2D and AntMaze environments, reporting improvements over a range of baselines with different behavior cloning regularizations.", "questions": "* Can the method be demonstrated to work well on more comprehensive benchmarks including more diverse skills and sensory input?\n\n* Can the observed improvement be linked to a better alignment that was identified as the root cause of inferior performance of previous approaches?\n\n* Can the score network be used to evaluate data snippets to show they are indeed reflecting the optimality of the data?\n\n* Eq. (6) indicates a noise perturbation to the state. Typo?\n\n* Eq. (8) avoids second-order derivatives and it seems due to the assumed $\\frac{\\partial \\phi_{t-1}}{\\partial \\alpha} \\approx 0$. Why this can be valid and what is defined as the Markov process here?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The idea is well motivated with the challenges of aligning trained behaviour and the one used for value estimation well elaborated. \n\n* The core idea of enforcing a weighted objectives through bi-level optimization is intuitive and easy to understand.\n\n* The selected baselines seem relevant to the comparison and assessment.", "weaknesses": "* Insufficient evaluation on more complicated task environments. The selected tasks look simple and could be expanded to cover tasks beyond locomotion. Test results on tasks relying on visual input and manipulation tasks such as Fetch could improve the soundness of the results.\n\n* The results only contain ultimate performance on a limited set of tasks. There is no direct evidence or analysis that how the proposed learning scheme mitigate the alignment issue. A study that can explicitly attributes the improvement to a better alignment could lead to a stronger argument.\n\n* The performance improvement seems diminished on more complicated AntMaze task. This raises the question whether the method can scale up to more complicated scenarios as suggested above.\n\n* Part of the writing on the inner loop derivative is a bit unclear. See the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)