Efficient Offline Reinforcement Learning via Peer-Influenced Constraint
Authors: Yujia Zhang
Lin Li
Wei Wei
Jianguo Wu
Yi Ma
Jiye Liang
Venue: ICLR 2026 Poster (2026)
Keywords: Offline Reinforcement Learning
Distributional Shift
Peer-Influenced Constraint
Generalization
Uncertainty Estimation
Primary area: reinforcement learning

Abstract: Offline reinforcement learning (RL) seeks to learn an optimal policy from a fixed dataset, but distributional shift between the dataset and the learned policy often leads to suboptimal real-world performance. Existing methods typically use behavior policy regularization to constrain the learned policy, but these conservative approaches can limit performance and generalization, especially when the behavior policy is suboptimal. We propose a Peer-Influenced Constraint (PIC) framework with a ``peer review" mechanism. Specifically, we construct a set of similar states and use the corresponding actions as candidates, from which we select the optimal action to constrain the policy. This method helps the policy escape local optima while approximately ensuring the staying within the in-distribution space, boosting both performance and generalization. We also introduce an improved version, Ensemble Peer-Influenced Constraint (EPIC), which combines ensemble methods to achieve strong performance while maintaining high efficiency. Additionally, we uncover the Coupling Effect between PIC and uncertainty estimation, providing valuable insights for offline RL. We evaluate our methods on classic continuous control tasks from the D4RL benchmark, with both PIC and EPIC achieving competitive performance compared to state-of-the-art approaches.

Published as a conference paper at ICLR 2026
EFFICIENT OFFLINE REINFORCEMENT LEARNING VIA PEER-INFLUENCED CONSTRAINT
Yujia Zhang', Lin Li?, Wei Wei Jianguo Wu?, Yi Ma², Jiye Liang² 1 School of Computer Science and Technology, North University of China, Taiyuan, 030051, Shanxi, China
2 Key Laboratory of Computational Intelligence and Chinese Information Processing of Ministry of Education, School of Computer and Information Technology. Shanxi University, Taiyuan, 030006, Shanxi, China
com, zhangy nucu.. /ww.euuuuuueceeeeeeeeeeee edu. en (Yujia Zhang); wei we i @ sxu edu. en (Wei Wei, corresponding author)
E-mails: 3425645350Q0
ABSTRACT
Offline reinforcement learning (RI.) seeks to learn an optimal policy from a fixed dataset, but distributional shift between the dataset and the learned policy often leads to suboptimal real-world performance. Existing methods typically use be- havior policy regularization to constrain the learned policy, but these conservative approaches can limit performance and generalization, especially when the behav- ior policy suboptimal. We propose a Peer-Influenced Constraint (PIC) frame- work with a "peer review" mechanism. Specifically, we construct set of similar states and use the corresponding actions as candidates, from which we select the optimal action to constrain the policy. This method helps the policy escape local optima while approximately ensuring the staying within the in-distribution space, boosting both performance and generalization. We also introduce improved version, Ensemble Peer-Influenced Constraint (EPIC), which combines ensemble methods to achieve strong performance while maintaining high efficiency. Addi- tionally, we uncover the Coupling Effect between PIC and uncertainty estimation, providing valuable insights for offline RL. We evaluate our methods on classic continuous control tasks from the D4RL benchmark, with both PIC and EPIC achieving competitive performance compared to sfte-offthhe-at approaches.
INTRODUCTION
Reinforcement learning (RL) focuses optimizing sequential decision-making in dynamic envi- ronments (Sutton & Barto 2018). By leveraging neural networks, online RL has achieved success in gaming and robotic manipulation (Mnih al. 2013 2015 Zhang et al. 2024a Shi et al. 2024). However, its reliance on continuous interaction with the environment limits its application in sce- narios where data collection is expensive, time-consuming, dangerous (Cao al. 2025 Yu al., 2021a). Offline RI. addresses this by learning from a fixed, pre-collected dataset, eliminating the need for further interactions during training (Lange et al. 2012] Levine et al.) 2020). Although promising, traditional online RL algorithms often struggle in offline settings due to the lack of on- line interaction for error correction, leading instability and potential learning failures when erross propagate (Fujimoto et al.) 2019: Tarasov et al., 2023a; Zhang et al. 2024b 2025a). This difficulty accurately evaluating out-of distrrbbuutin (OOD) actions further challenges the value-based policy learning process.
To address distributional shifts between the dataset and the learned policy offline RI., regular- ization is often applied during training to encourage policies to stay close to dataset actions during evaluation and improvement. Existing methods can be broadly categorized into value regularization and policy regularization methods. Value regularization methods penalize the Q-values of OOD
actions to discourage them from being selected (Kumar et al. 2020 Yu et al. 2021b Wu et al,
2021 Lyu et al. 2022, Nakamoto et al. 2024). Methods such SAC-N and EDAC (An et al.
2021 use ensembles to estimate value distributions and apply pessimistic penalties. However, these
methods often require large ensembles, leading to high computational costs and slower convergence

---

Published as a conference paper at ICLR 2026
TD3-BC
Our
TD3-BC
Our
(a) State visitation
(b) Learned Q table
Figure The illustrations and results of the motivation experiment. (a) State visitation heatmap (darker higher probability): PIC escapes local optima and reaches the goal, while TD3+BC, con-
strained by its one-to-one ptte-actioo mapping, stalls. (b) Q-table visualization (darker arrows higher Q): PIC drives the agent toward the goal; TD3+BC remains pessimistic.
speed. Additionally, many uncertainty estimators overlook critical information, such as inter-sample distances, which are essential for distinguishing in-distribution and OOD actions in RL.
Policy regularization methods constrain the policy by focusing on actions within the dataset, thereby avoiding OOD actions without relying on ensemble modeling (Kostrikov et al. 2022a Nair et al.) 2020 Fujimoto Gu. 2021, Ma et al. 2024 Luo et al.) 2024p. While these methods are computa- tionally efficient, their generalization is limited by the restricted set of available actions, which may not always optimal. For example, TD3+BC (Fujimoto & Gu 202I) shows limited generalization in 2D grid world task (Sutton & Barto 2018 Xu et al. (202b)) when the dataset contains missing actions, such as moving right (illustrated by the green triangle in Figure [ see Appendix [E]]] for details). This raises critical question: How
more effectively utilize dataset information to enhance generalization while preserving computational efficiency?
To answer the above question, we propose Peer-Influenced Constraint (PIC), plug-in regularizer that exploits relationships within the dataset via a "peer review" mechanism. For a query state, PIC reuses actions from similar peer states and steers the policy toward high-value candidate. This expands the effective in da-datset action set rersenign actions observed elsewhere rather than re- lying only on the behavior action-whild remaining compute-lige -ight and ensemble-agnostic, When combined with ensembles, we observe a Coupling Effect between PIC strength and uncertainty es- timation, enabling effective OOD penalties with substantially smaller ensembles. Building on this, we develop Ensemble Iefreneflee Constraint (EPIC) to further improve data-coverage general- ization at competitive training cost.
The contributions of this paper are as follows:
We introduce PIC, drop-in policy regularization module that reuses cross-state informa- tion in the dataset to guide the policy toward high-value, in-distribution actions, without training additional generative models or relying on large ensembles.
We identify a Coupling Effect between PIC strength and uncertainty estimation that enables effective OOD penalties with smaller ensembles: building this, we introduce EPIC, PIC-enhanced ensemble with strong performance and high efficiency.
We instantiate PIC in TD3, SAC, and IQL, demonstrating competitive performance and computational efficiency on offline RL benchmarks, as well as in offline-to-ollie and controlled generalization evaluations.
2 PRELIMINARIES
In this section, we briefly introduce the common RL settings, notations, and details offline RL methods.
2

---

Published as a conference paper at ICLR 2026
2.1 REINFORCEMENT LEARNING
In RL, the environment is typically modeled as an infinite-horizon Markov Decision Process (MDP), represented by the tuple (S, A, P, Po, r, 7). This includes a state space S, an action space A. tran- sition probabilities P S x A x S - [0, 1], an initial state distribution po- a reward function T:S X A
R. and a discount factor y E [0.
Given the MDP and the agent's policy T6 with parameter é. at time t, the agent observes the state St and selects an action a F (st). The environment then provides reward r(s,, az) and transitions to the next state $1+1- Let R Ei=o >'r(st, at) denote the expected discounted reward of T starting from time t, accounting for the randommess in the initial state distribution Po and the transi- tion function P. The Q-value Q" S x A + I represents the expected cumulative return obtained by taking action a at state 8 and subsequently following the policy TΦ- quantifies the long-term value of state-action pair under the given policy:
(= E
r'r(s,, ai) 8, 8,00
The goal in RL is to find the optimal policy x r²(s) argmax, Q a), maximizing Re.
2.2 OFFLINE REINFORCEMENT LEARNING
Classic RL requires environment interaction during training, which impractical in tasks with high safety cost constraints. Offline RL addresses this by learning optimal policy from pre- existing dataset D = (s, a, r, r')') without environment access. However, adapting existing algorithms often leads suboptimal policies due overestimation bias for unseen state-action pairs, which is corrected in online RL through exploration and feedback but requires additional mechanisms in offline RL stabilize training.
To counteract overestimation bias, common approach is to address the root cause: the Q-value estimates. Uncertainty estimation helps by highlighting that Q-values for OOD actions are more un- certain than those for in-sample actions. Both SAC-N and EDAC (An et al., 2021) use an ensemble of N Q-functions to approximate the Q-value distribution. They update the i-th critic parameter 0, by minimizing the squared temporal difference loss:
L(O.) = E( (V - (s,
(2)
where y = r + 7 min Q0 (s', Ts (s')), and ø' represent the parameters of the target networks for the critic and actor, respectively, B is a minibatch from D.
i=......
The policy can be improved by:
Lí(0) EB-B min Qo, (s, (T( iil....,N
(3)
Using the minimum Q-value penalizes OOD actions and avoids overestimation with large N, but methods such as SAC-N and EDAC incur high cost from maintaining N Q-networks.
In addition to value regularization methods, policy constraint methods directly constrain the learned policy to stay close to the behavior policy in the dataset, thereby avoiding OOD actions. For exam- ple, TD3+BC (Fujimoto & Gu 2021) extends the Equation (3) by incorporating behavior cloning loss term into the policy update:
LBC (s,)~1 [lla- T (s)
TD3+BC achieves competitive performance with lower computational costs compared to ensemble- based methods. However, often struggles datasets with limited action diversity, which restricts policy improvement and leads to suboptimal generalization.
3 METHOD
In this section, we introduce our method. We first define PIC and establish performance gap bounds, before integrating PIC into offline RI. algorithm. We then extend PIC to ensemble-based RI.
3

---

Published as a conference paper at ICLR 2026
(5, a. r.s)
ag Gg
4g
Offline Dataset
Peer States
Candidate Actious
NCrities
Stage Identification
Stage 2: Optimal Aciins Ideutification
PIC (52)
BL, + ö%E n =ldb" (s.aa)))aa))
Stage
EPIC Loss Calculation
R. Actor Loss
T
Given
Distance
Priperation
Mimibotch Sampling
Figure 2: Actor training workflow of EPIC. It consists of three stages: identifying peer states, lecting optimal actions, and computing the PIC loss. First, peer states are retrieved for given state. Next, the corresponding optimal actions are determined. Finally, the PIC distance is calculated, and the actor is updated by jointly optimizing the Q-value loss L1 and the PIC loss.
methods, where we uncover Coupling Effect between PIC and uncertainty quantification. Building on these insights, we propose EPIC, which unifies PIC and the Coupling Effect to enable efficient and generalized policy learning. The gradient propagation of actor training in EPIC is illustrated in Figure
3.1 INFLUINNLUENCED CONSTRAINT
Inspired by representation learning- based RL methods (Fujimoto et al. 2023 Ni et al. 2024) and
dataset-constrained approaches (Ran al. 2023). which capture structural information in the vironment by mapping similar states or state-action pairs to similar embeddings, we observe that offline RL datasets often contain rich but underutilized structural information about states. Specif- ically, we propose to utilize the inherent structure of the dataset, where states with similar charac- teristics (e.g., proximity
shared features) often contain valuable information for guiding policy learning. We first introduce PIC, novel approach that leverages peer states to enhance policy learn- ing offline RL. We then describe an efficient peer states search method for fast and effective state matching. Finally, we present practical algorithm that integrates PIC into TD3 to enhance offline RL performance.
Definition (PIC Distance). Let be state from the dataset DD and let TT(s) be parameterized policy. The PIC distance is defined as:
dPC (1= - -all,
(5)
where a" is the optimal action selected from a candidate action set A'.
The optimal action a" determined
a arg max min Qo, (8, a),
(6)
aEA' 1=1
where Qo, and Qo, are the Q-values estimated by the two critics. To address potential overestima- tion issues critics, we use two critics to provide more reliable candidate action value estimates Fujimoto & Gu 2021 Zhang et al. 2025b).
The candidate set includes: (1) Actions of the given state 8; (2) Actions corresponding to the K peer states. Each peer state, 8y. is determined as: To ensure that peer states are distinct from we explicitly exclude s during nearest- neiiestt retrieval. Formally, the j-th peer state 8, is defined as:
3
arg
min (D)
||s $||,
j= 1, K,
where Dj-1 1 is the set of already selected j peer states of s. Since one state in D could correspond multiple actions, the size of A' may be greater than K 1.
4

---

Published as a conference paper at ICLR 2026
PIC is built upon the assumption that, in continuous control domains with local smoothness, nearby states tend to share similar optimal actions, which makes the peer-based action constraint both feasi- ble and effective. This approach yields more optimal behavior constraint by accounting for actions of the given state, potential peer-state actions, effectively escaping local optima for more generalized policy learning. Furthermore, we provide analysis of the performance gap between the learned policy T using PIC constraint and the optimal policy π².
Theorem 1 (Performance Gap of PIC). Assume that μ. and P(s'(s,a) are Lipschitz continuous. For any (8, a) and its peer pair (8,, az), suppose ||š - a|l Es. ||m(3;) T(>)]l S Ear and max,es TT" (s) --(s) E.. conditions that can be satisfied by PIC. Then, the following inequality holds:
CT.p-mmax (<a Lacs C.), 1-y
where C is a positive constant, Lp and L. represent the Lipschitz constant, Rmax is the max value of reward.
For a detailed proof, please see Appendix B Observed from Theorem T the performance gap is influenced by Ea. Cz. and e... reCuces e, by ensuring that peer states 8, are close to the given state s. thereby minimizing state deviations. PIC reduces E. by selecting actions a" from the candidate set that close both the behavior policy yu(s) and the optimal policy *" (s), thus minimizing the action discrepancy. Finally, PIC reduces c, by aligning the learned policy with the optimal policy, narrowing the performance gap and improving generalization.
PIC is similar to TD3+BC (Fujimoto & Gu 2021). but with significant differences. While TD3+BC relies on strict state-action correspondences, limiting exploration and adaptability, our approach avoids these constraints. Unlike methods that minimize the distance between (s, (x)) and its nearest state-action pair in D (Ran et al.) 2023). which can result in incorrect constraints due to state mismatches, our method considers actions from both the given state and similar states. This enables optimal action selection without state mismatches and eliminates the need for extensive tuning, ensuring better performance, avoiding the OOD issues.
3.1.1 EFFICIENT PEER STATES SEARCH
PIC identifies, for each state E D, its K nearest peer states in the state space. To accelerate retrieval, we adopt a precomputed KD-Tree (Bentley 1975), a standard index for efficient nearest- neighbor search (Ran et al.] 2023 Zhang al. 2023). which significantly reduces training -ime computation by avoiding redundant searches. Concretely, we build the KD-Tree over all dataset states once before training. and query it during training to fetch the K nearest states in O(Is] log (D]) time per query (Ram & Sinha 2019). where |s] denotes the state dimensionality. Unlike dataset-
constraint methods (Ran et al. 2023) that search jointly over state and action spaces, our approach compares states only, lowering computational cost while maintaining accuracy.
3.1.2 PRACTICAL ALGORITHM
PIC is versatile and can be integrated as a component into any Actor-Critic methods. In this paper, we combine it with the straightforward and easy-to- to--mplemmeett TD3 method, introducing a practical algorithm called PIC-TD3. The actor update for PIC-TD3 is:
(pr(Φ) = Eg- [- BQ0,, (s, + SE,-B [dP[(s)] (s)]
(9)
where a = (s). 3 Σ En a.). and the a in / is hyper-parameter. is the PIC strength parameter. The setting of B is similar to that in TD3+BC (Fujimoto & Gul 2021). designed to
a|B]
mitigate sensitivity to Q-value estimates. The pseudocode of PIC-TD3 provided in Appendix C
Note that PIC is designed as a versatile plugin that can be seamlessly integrated into different RL algorithms without requiring major architectural modifications. In Appendix F33] we incorporate PIC into IQL (Kostrikov et al. 2021b) and SAC (Haamoja et al.) 2018), showing that consistently improves performance and scales effectively.
5

---

Published as a conference paper at ICLR 2026
beggue
Senaled
75
NC
5
10
15
-20
300
50
60
0.2
64
0.6
0.5
1o
0.
0.2
0.6
0.
1.
MC
(a) Effect N and 6
(b) Effect on N and 6
Effect on Quia
(d) Effect Qui
Figure 3: The impact of ensemble size N and PIC strength on performance and uncertainty.
3.2 ENSEMBLE PEER-INFLUENCE CONSTRAINT
In this section, we first assess how PIC augments ensemble-hased methods and reveal a Coupling Effect between PIC strength and uncertainty estimation. We then introduce EPIC, which attains state-of-the-art performance with strong generalization and improved computational efficiency. Fi- nally, we discuss why EPIC works.
3.2.1 INTEGRATING PIC WITH ENSEMBLE METHODS
Policy constraint methods are typically more efficient and stable, but ensemble-based value con- straint methods, like EDAC (An et al. 2021). can perform well in certain environments. However, these methods often require large networks and extensive resources, especially some expert-level datasets. We explore whether PIC can bring benefits to such ensemble methods by integrating with EDAC and testing its effectiveness. As shown in Figure 3 (a) and (b), decent performance is achieved with N = (the original PIC-TD3 version), and performance improves with increas- ing N. However, when N too large, such as in hopper-medius (N 50) or halfcheetah-expper (N = 20), the policy converges more slowly, and the convergence performance also degrades. Addi- tionally, we find that higher PIC strength reduces the number of crities needed to achieve comparable performance.
We further measure and analyze the relationship between various uncertainty metrics, ensemble size N, and PIC strength á on the hopper-medium-experr dataset. We sample 10,000 state-action pairs and compute the uncertainty metrics Qmin Qual and Qclip = Quean Qmin the same time step under different values of N and 5. We then calculate the average values and present the heatmaps in Figure
(c) and (d); results for Qclip are reported Appendix F.I
Our findings show that, for fixed N, increasing 5 raises uncertainty, allowing smaller ensembles to maintain sufficient penalties for OOD actions. Conversely, for fixed ö, increasing results in higher uncertainty well. We analyze that PIC strength increases, the policy increasingly selects in-distribution actions while assigning more conservative values to potential OOD actions. As consequence, Qmin becomes ensemble disagreement Qua tends to increase on OOD candidates, and Qclip also tends to increase. These changes reflect stronger penalization of high-risk, potentially OOD actions, similar the increased OOD penalty produced by using a larger ensemble size.
pessimistic, thereby further reducing overestimation, and
These observations suggest that without PIC, Q-value estimates for OOD actions often exhibit low uncertainty, leading to the overestimation of their values. This in turn requires the use of large en- sembles to mitigate the risk inaccurate Q-value predictions and ensure reliable performance. By incorporating PIC, we introduce a "Coupling Effect" that not only improves policy performance with smaller ensembles but also significantly reduces computational overhead. Furthermore, we measure the fraction of policy actions that lie in in-distribution regions under different training steps, PIC strengths 8, and ensemble sizes N. The results, reported in Appendix F. show that introducing PIC significantly increases the in-distribution ratio compared to the no-PiC baseline. Moreover, this ratio consistently grows as 5, the training horizon, and N increase, indicating that the policy is pro- gressively concentrated the dataset support. From the perspective of the policy distribution, this provides additional evidence for the coupling effect: stronger PIC constraints and larger ensembles jointly help keep the learned policy close in-distribution regions.

---

Published as a conference paper at ICLR 2026
Algorithm Ensemble Peer-Influenced Constraint Algorithm (EPIC)
Input: Observation state S, offline dataset D Output: Learned policy π
actor network Td. target networks (Qo; 1oo d', and peer Initialize: Critic networks {Qo: L.V
state number
for / 1 do
Sample mini-batch B as,, from D
Find peer states in of sample state by 8; = arg mingep || Construct candidate action set .4'
Select the optimal action a" using a* arg m m (8,
Calculate PIC distance of sample state by d (s) = [ a Update 9g by minimizing (0,) E, ).) [(y Qa. (8, a) ES
Update á using gradient descent with LEPIC = BL, (0) + oT BTE (8)
0 < T0, + (1 T)0,, o' + TO + (1 - T) end for
Building on this Coupling Effect, apply PIC EDAC and introduce the EPIC, which produces state-of-the-art, generalized policies with high computational efficiency. The actor update for EPIC as follows:
LEFIC(O) (L) - 8EgB [d (s)]
(10)
where Lı is defined in Equation 3 and the optimal action a" in d(((()) (s) is
arg max
min Qa, as)
(11)
aEA' 1=1,2,- .N
The critic update method in EPIC is consistent with EDAC, with the addition of an ensemble simi- larity (ES) term to Equation ②
LEPIC ((s,,,)))))) [10- Qa. a))² ES].
(12)
where ES N EiSEAIe (v.Qo, (s, a), VaQo, (s,a)), is hyperparameter and N is the en- semble size. The ES term encourages diversity among critics, but to better evaluate the effectiveness of PIC, we set 77 = in most environments. EPIC's pseudocode provided in Algorithm
3.2.2 WHY EPIC WORKS
EPIC combines the strengths of PIC and ensemble methods to enhance policy learning by leveraging in-sample optimal action selection and the Coupling Effect. It constrains the policy through PIC by constructing a candidate action set from similar states, avoiding unreliable OOD actions and local optima. By utilizing the diversity of multiple critics, EPIC stabilizes value estimates and uses maxmin-like approach to select the optimal action a", effectively reducing overestimation and improving action selection.
Unlike traditional policy constraint methods (e.g. TD3+BC), which often struggle to surpass the performance of the behavior policy, or value constraint methods, which may inadequately penalize OOD actions, EPIC achieves better balance between generalization and conservatism. The Cou- pling Effect between PIC and uncertainty quantification ensures an optimal trade-off among gener- alization, conservatism, and computational efficiency, resulting in stronger overall performance.
EXPERIMENTS
In this section, we conduct several experiments to evaluate the empirical performance of the pro- posed method. Our goal is to answer five key questions: (1) Does method outperform previous approaches on standard offline benchmarks? (2) Is our method efficient?
(3) Can our method learn optimal policies from peer states enhance generalization? (4) How do different ensemble sizes N and PIC strengths 8 affect EPIC's performance? (5) Is our method still effective in affline-to-onlln fine-tuning?
7

---

Published as a conference paper at ICLR 2026
Table 1: Average normalized score over final evaluations across five seeds on Gym-MuJoCo tasks.
Task Name
TD3+BC
IQL
CQL
SAC-RND
PRDC
SAC-N
EDAC
PIC-TD3 (ours)
EPIC (ours)
balfcheetah random
halfchectah-medte meeeiim
54.7
50.0
46.9
66.4
63.5
67.5
65.9
68.4 + 2.4
68.9
2.7
halfcheetah expert
93.4
95.5
97.3
102.6
105.2
106.8
104.2
1: 3.2
107.9
6.4
balfcheetah-meede meeetaa--edim---- expert
106,3
balfcheetah medium replay
45.0
42.1
45.3
51.2
55.0
63.9
61.3
63.1 1: 0.9
64.8 1: 1.5
balfchectah-foll- -ellpyy
75.0
75.0
76.9
81.2
84.5
84.6
83.7
87.3 1.0
hopper-nundom
8.5
10.1
5.3
19.6
26.8
31.3
25.3
25.4 3.2
27.9 4.2
hopper-mnedim
60.9
63.2
61.9
91.1
100.3
101.6
100.7 = 1.5
102.2
hopper-expert
109.6
108.8
106.5
109.8
110.3
110.1
111.8
2.1
112.2 1.5
bopper medium expert
87.8
85.5
96.9
109.8
109.2
110.1
110.7
105.6 1: 1.1
112.3 0.9
repllay
55.1
86.3
100.5
Supper-full-ree
97.9
104.4
101.9
107.4
102.9
105.4
105.8
0.8
107.6
walker2d-random
2.0
11.3
5.1
18.7
5.0
21.7
16.6
14.6
22.4 1.6
walker2d-medium
77.7
80.7
79.5
92.7
85.2
87.9
92.5
94.9
2.2
95,9 3.2
2alker2d-xxpert
110.0
96.9
109.3
104.5
107.4
115.1
114.6
1.5
117.7 2.1
walker2d medium-exper
104.6
111.2
114.4
malker2d-medium-ane
68.0
75.4
76.8
89.4
92.0
78.7
87.1
95.5 + 2.4
95.2 2.2
walker2d full -eeply
90.3
97.5
94.2
105.3
94.6
99.8
103.0 = 1.9
107.4
2.2
Average
70.4
72.9
73.7
82.6
84.4
85.2
85.1
87.8
Table 2: Average normalized score over the final evaluation and five seeds on the AntMaze tasks.
Task Name
TD3+BC
IQL.
CQL
SAC-RND
PRDC
MSG
SAC-BC-N
PIC-TD3 (ours)
EPIC (ours)
antmazo-umaze
66.3
83.3
74,0
97.0
98.8
97.9
98.6
96.4
3,5
98.6 + 0.5
ammazz-umaz--divese
70.6
66.0
90.0
79.3
91.2
91.1
2.3
94.3 + 4,2
mediuz---mdiuu-play
26.5
64.6
61.2
38.5
85.9
85.8
79.3
5.4
88.1 3.3
aummz--mciiumddiivee
25.9
61.7
53.7
74.7
78.8
84.6
73.8
77.9
6.8
83.8 4.0
lamaze-targe-play
antmaz-largo-divvrre
0.0
27.6
14.9
45.7
50.0
71.3
75.8
53.2 - 4.7
66.9 + 5.3
Average
58.4
61.0
80.6
4.1 MAIN RESULTS ON BENCHMARKS
We evaluate the proposed method on three suites of D4RL tasks (Fu et al.
2020 Gym-MuJoCo,
AntMaze, and Adroit. For each suite, we consider all available datasets.
We compare our results
to several ensemble-free baselines, including TD3+BC (Fujimoto & Gu
2021). IQL (Kostrikov
et al. 2021b)) CQL (Kumar et al., (200)) SAC-RND (Nikulin et al.) 2023). and PRDC (Ran al.]
2023 Given the effectiveness and efficiency of our approach, we also compare to ensemble- based baselines, including SAC-N, EDAC (An al. (022), MSG (Ghasemipour al. 2022 and
SAC-BC-N (Beeson & Montana 2024). Detailed implementation and bascline result sources are provided in Appendix
We train PIC-TD3 and EPIC for one million steps on each task within the Gym-MuJoCo, AntMaze, and Adroit benchmarks. The test results are presented in Tables 2 and 3 In these tables, the mean-wise best results among algorithms are highlighted in bold, the second- best performance is underlined, and the symbol represents the standard deviation across the seeds. Our method, EPIC, outperforms all baselines in the vast majority of tasks, demonstrating the effectiveness of the PIC and Coupling Effect. Additionally, PIC-TD3 surpasses ensemble-free methods and achieves performance comparable ensemble-based methods.
Appendix F.2 reports the minimum Q-ensembles for the results in Tables 2 and 3 showing that our method achieves competitive performance with substantially fewer critics, thus highlighting its efficiency and robustness. Appendix F.8 provides additional training curves, which highlight the stability, convergence speed, and overall learning dynamics of our method. Appendix F. extends the PIC framework to other RL algorithms, where consistent gains further confirm its effectiveness in enhancing policy generalization and robustness.
8

---

Published as a conference paper at ICLR 2026
Table 3: Average normalized score over the final evaluation and five seeds on the Adroit tasks.
Tisk Name
BC
TD3+BC
IQL
COL
SACARND
EDAC
BORL
PIC-TD3 (ours)
EPIC (ours)
pen-human
34,4
81.8
81.5
37.5
5.6
51.2
33.7
84.7 2.4
111.7 5.5
pen-cloned
$6.9
61.4
77.2
39.2
2.5
68.2
35.7
80.2 + 1.1
94.6 :: 8.6
pea-expert
85.1
146.0
133.6
107.0
45.4
122.8
130.3
133.7 =
150.9
doer-human
0.5
-0.1
3.1
9.9
0.0
10.7
3.7
3.7 1.3
12.9 =: 6.2
clonedloned
0.1
0.8
doo--expert
34,9
84.6
105.3
101.5
73.6
-0.3
104.9
105.6 = 4.1
108.4 144
humnun
0.4
2.5
-0.1
0.8
2.3
2.1
6.5
hammer-cloned
0.8
0.8
1.1
2.1
0.1
0.3
1.7
2.4 1.2
12.7 3.3
expert
125.6
117.0
129.6
86.7
24.8
0.2
132.2
121.7
3.1
130.4 3.8
relocate human
0.0
-0.2
0.1
0.2
0.0
0.1
0.0
0.5 4:5::00 0.7
2.7 1: 1.2
selocate-cloned
-0.1
-0.1
0.2
-0.1
0.0
0.0
0.0
0.3 +:3++0 0.2
1.1 0.5
relocate expert
101.3
107.3
106.5
95.0
3.4
00.3
47.8
105.7 2.1
108.4 1.4
Average
53.5
13.0
4.2 COMPUTATIONAL EFFICIENCY COMPARISON
Time complexity plays a critical role in offline RL. We evaluate PIC-TD3, EPIC, and several baseline methods on the hopper-medium dataset. Appendix F.4 shows training time (relative to TD3+BC) and normalized scores. Ensemble methods are accurate but costly, BC-based methods efficient but weaker, while EPIC and PIC-TD3 achieve better balance of efficiency and performance.
4.3 GENERALIZATION
The advantage of PIC lies in its ability leverage peer states and corresponding actions from the dataset, allowing the learning of potentially optimal actions not present in the current state. This gives PIC better generalization compared traditional policy constraint methods. Inspired by (Ran (t.33)2022) and (Kumar, Aviral [2019), we create lineworld environment. The environment details lineworld- by increasing difficulty. We train TD3+BC (Fujimoto & Gu 202T), BEAR (Kumar et al., 2019), PRDC (Ran et al. (023), and PIC-TD3 on the datasets, and the results are presented in Appendix F.5.I In easy and random datasets, all methods reach the opti- mal policy. On medium and hard datasets, reward sparsity and conservative regularization hinder TD3+BC and BEAR, while PRDC overlooks potential optimal actions. PIC generalizes by leverag- ing peer states' optimal actions, even unseen ones, to escape local optima.
are the Appendix F.2 We generate four datasett-linewword--eusy lineworld-rande
medium, and lineworld-hand-ordeer
Further, systematically evaluate the impact of different state-space distance metrics EPIC. We test multiple EPIC variants on both MuloCo tasks and the high-dimensional quadruped benchmark Walk These Ways (WTW) Margolis & Agrawal (2023). The details of the WTW environment, its data collection procedure, and the complete results are provided in Appendix F.5.2 EPIC remains robust across distance metries in MuJoCo and continues achieve competitive performance even in the high-dimensional WTW setting, demonstrating that its effectiveness stems from the peer-induced constraint mechanism rather than reliance on specific state-space metric.
4.4 PARAMETER STUDY
In this section, we conduct a parameter study on EPIC, focusing on the number of peer states K (which affects the candidate action set), PIC strength 6, and ensemble size N. We evaluate their sensitivity across various Gym-MuJoCo tasks, running for one million steps with five random sceds.
Number of Peer States K: The number of peer states K is key hyperparameter for PIC-TD3 and EPIC, influencing action diversity. As shown in Figure (a) and (b), larger improves performance and accelerates convergence, but when K > 20 the gains plateau; PIC-TD3 slightly declines due to value estimation errors, while EPIC remains robust through ensemble regularization.
PIC Strength 8 The PIC strength 8 controls the intensity of the peer constraint. Larger 8 enforces closeness to samsa actions, limiting exploration, while smaller 6 prioritizes Q-value fitting, risk- ing deviation from the behavior policy. As shown in Figure (c), performance hopper-medium

---

Published as a conference paper at ICLR 2026
106
Nomeltes Kitum
K-10
K-28
50
K-20
28
K-58
K-
K-191
K-100
100
010
6.22
6.30
0.35
100
5.0
0.34
6.50
0.75
1.00
61
2
Steps (100)
Times
(a) Effect (PIC-TD3) Effect (EPIC)
(c) Effect
(d) Effect N
Figure 4: The impact of peer states K, PIC strength ó, and ensemble size N on performance.
degrades when < 1 or d 4, with the best results in the moderate range E [1,3]. Similar trends appear in AntMaze and Adroit (See Appendix F.6.I. indicating that proper tuming of 6 ensures robust performance across tasks.
Ensemble Size N: Ensemble size N immroves OOD penalization but reduces efficiency. As shown improves with when O, but declines for N 2 50 with small and decreases further N grows under larger 6. Similar trends observed in additional experiments, reported in Appendix F.622
in Figure 4 (d), performance on
These results highlight the Coupling Effect: moderate enhances generalization, but excessive ò amplifies uncertainty and weakens ensembles. A balanced choice (5 E [1,3], N E [5, 20) achieves robust performance.
4.5 PERFORMANCE ON OFFLINE-TO- OOLLINE D4RL
Evaluating offline-to-onlne performance is critical aspect of offline RL algorithms, especially in light recent advancements. Based on this, we conduct additional tests on EPIC, as it demonstrates remarkable effectiivneess in offline pretraining through PIC and the Coupling Effect. As shown in Appendix F.7 EPIC achieves competitive results, outperforming in three of six AntMaze datasets and reaching state-of-the-art final scores Adroit tasks.
CONCLUSION, LIMITATIONS, AND FUTURE WORK
To address the limitations of existing offline RL methods in generalization and efficiency, we propose the Peer-Ine Refr-Inffueeeed Constraint (PIC) and its enhanced version, Ensemble Beer-Influenced Constraint (EPIC). PIC improves policy generalization by identifying potentially better actions from peer states. We also discover direct relationship between PIC and uncertainty estimation, termed the Coupling Effect. Leveraging PIC and the Coupling Effect, EPIC achieves state-of-the-e performance with smaller ensemble sizes, reducing computational costs in offline RL. Extensive comparisons with strong baselines show that our method outperforms others across various tasks. We also validate its effectiveness during offline-to-onllnn transitions.
We hope this work inspires further exploration of the interaction between constraint methods and uncertainty estimation, offering new insights to the community. A limitation of our approach is the inefficiency of peer state search large-scale, high-dimensional datasets. We aim to develop more efficient offline RI methods for such datasets in future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers for their very constructive comments. This work was supported by the National Natural Science Foundation of China (62276160), and the Natural Science Foundation of Shanxi Province, China (202203021211994)
REFERENCES
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-baser offline re- inforcement learning with diversified q-ensemble. In Proceedings of the Advances in Neural Information Processing Systems, pp. 7436-7447, 2021.
Chenjia Bai, Lingxiao Wang. Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhao- Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning, In Proceedings of the International Conference on Learning Representations, 2022.
Chenjia Bai, Lingxiao Wang, Jianye Hao, Zhuoran Yang, Bin Zhao, Zhen Wang, and Xuelong Li. Pessimistic value iteration for multi-task data sharing in offline reinforcement learning. Artificial Intelligence, 32::102::0 104048, 2024.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence, volume 15, pp. 103-129, 1995.
Alex Beeson and Giovanni Montana. Balancing policy constraint and ensemble size in uncertainty- based offline reinforcement learning. Machine Learning. 113(1):443-48 2024.
Jon Louis Bentley. Multidimensional binary search trees used for associative searching. Communi- cations of the ACM, 18(9)500--517, 1975.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 2016.
Hongye Cao. Fan Feng. Jing Huo, and Yang Gao. Causal action empowerment for efficient inforcement learning in embodied agents. Science China Information Sciences, 68(5): 15020,, 2025.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforeement learning. arXiv preprint arXiv:2004.07211
2020. 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems, pp. 20132-20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. Proceedings of the International Conference on Machine Learning, pp. 2052- 2019.
2019.
Scott Fujimoto, Wei-Di Chang, Edward J Smith, Shixiang Shane Gu. Doina Precup, and David Meger. For sale; state-action representation learning for deep reinforcement learning. In Pro- ceedings of the Advances in Neural Information Processing Systems, pp. 11573-61624, 2023.
Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating un- certainties for offline through ensembles, and why their independence matters. In Proceedings of the Advances in Neural Information Processing Systems, pp. 18267-18281, 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the International Conference on Machine Learning, pp. 1861-1870, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 131iv:1312.61 14, 2013.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In Proceedings of the International Conference on Machine Learning, pp. 5774-5783, 2021a.

---

Published as a conference paper at ICLR 2026
Iyy Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2110.06169, 2021b.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Proceedings of the Advances in Neural Information Processing Systems, pp. 11844-17994 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for of- fline reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems, pp. 1179-1191, 2020.
Kumar, Aviral Data-driven deep reinforcement learning. httpp:://bii..eeeeeleee b olog/ 201 20199/1//05/bbrr/ 2019. Accessed: 2019-12-05.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning: State-of-the-ar, pp. 45-73. 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0164) 2020.
Zeyuan Liu, Kai Yang, and Xiu Li. Cdsa: Conservative denoising score- based algorithm for offline reinforcement learning. arXiv preprint Mxxi:22406.7755 2024.
Fan-Ming Luo. Tian Xu, Hang Lai, Xiong-Hui Chen, Weinan Zhang, and Yang Yu. A survey basel reinforcement learning. Science China Information Sciences, 67(2): 101101, 2024.
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline rein- forcement learning. In Proceedings of the Advances in Neural Information Processing Systems, pp. 1711-1724, 2022.
Yi Ma, HAO Jianye, Xiaohan Hu, Yan Zheng, and Chenjun Xiao. Iteratively refined behavior regularization for offline reinforcement learning. In Proceedings of the Advances in Neural Infor- mation Processing Systems, pp. 56215-56243, 2024.
Liyun Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-dice: In- sample diffusion guidance for offline reinforcement learning, arXiv preprint arXiv:2407.20109, 2024.
Gabriel B Margolis and Pulkit Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In Proceedings of the Conference on Robot Learning. pp. 22-31, 2023.
Volodymyr Mnih, Koray Kavukcuoglu. David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning. Nature, 518(7540):5295533 2015.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint 201i:22006.0339,, 2020.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine- tuning. In Proceedings of the Advances in Neural Information Processing Systems, pp. 62244- 62269, 2024.
Tianwei Ni, Benjamin Hysenbach, Erfan SeyedSalehi, Michel Ma, Clement Gehring, Aditya Ma- hajan, and Pierre-Luc Bacon. Bridging state and history representations: understanding self- predictive rl. In Proceedings the the International Conference on Learning Representations, 2024.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. In Proceedings of the International Conference on Machine Learning, pp.
2023.
12

---

Published as a conference paper at ICLR 2026
Parikshit Ram and Kaushik Sinha. Revisiting kd-tree for nearest neighbor search. In Proceedings of the Sigkdd International Conference on Knowledge Discovery & Data Mining, pp. 1378 1388, 2019.
Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. Policy regularization with dataset constraint for offline reinforcement learning. In Proceedings of the International Conference on Machine Learning, pp. 28701-28717, 2023.
Naman Saxena, Subhojyoti Khastagir, Shishir Kolathaya, and Shalabh Bhatnagar. Off-policy av- erage reward actor-critic with deterministic policy search. In Proceedings of the International Conference on Machine Learning, pp. 30130-30203, 2023.
Jiyuan Shi, Chenjia Bai, Haoran He, Lei Han, Dong Wang, Bin Zhao, Mingguo Zhao, Xiu Li, and Xuelong Li. Robust quadrupedal locomotion via risk-averse policy learning. In Proceedings of the IEEE International Conference on Robotics and Automation, pp. 11459- 1466, 2024.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems, pp. 11592-11620, 2023a.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library. In Proceedings of the Ad- vances in Neural Information Processing Systems, pp. 30997-31020, 2023b.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Wei Wei, Yujia Zhang, Jiye Liang, Lin Li, and Yyuze Li. Controlling underestimation bias in reinforcement learning via quasi-median operation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8621-8628, 2022.
Jialong Wu, Haixu Wa, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy opti- mization for offline reinforcement learning. In Proceedings of the Advances Neural Information Processing Systems, pp. 31278-3129) 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1911111111.11311, 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M Susskind, Jian Zhang, Ruslan Salakhutdi- nov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In Proceedings of the International Conference on Machine Learning, pp. 11319-11328, 2021.
Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The in-sample softmax for offline reinforcement learning. In Proceedings of the International Conference on Learning Representations, 2023.
Huaqing Xiong, Tengyu Xu, Lin Zhao, Yingbin Liang, and Wei Zhang, Deterministic policy gra- dient: Convergence analysis. In Proceedings of the Uncertainty in Artificial Intelligence, pp. 2159-2166 2022.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8753-8760, 2022.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xi- anyuan Zhan. Offline rl with no ood actions: ia-sae learning via implicit value regularization. In Proceedings of the International Conference Learning Representations, 2023a.
Kang Xu. Chenjia Bai, Xiaoteng Ma, Dong Wang, Bin Zhao, Zhen Wang, Xuelong Li, and Wei Li. Cross- domain policy adaptation via value-guided data filtering. In Proceedings of the Advances in Neural Information Processing Systems, pp. 73395-73421, 2023b.
13

---

Published as a conference paper at ICLR 2026
Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. In Proceedings of the Advances in Neural Information Processing Systems, pp. 23851-23866 2022.
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey, ACM Compuling Surveys (CSUR), 55(1):1-36, 2021a.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. In Proceedings of the Advances in Neural Information Processing Systems, pp. 28954-28967, 2021b.
Xinyi Yuan, Zhiwei Shang, Zifan Wang, Chenkai Wang, Zhao Shan, Meixin Zhu, Chenjia Bai, Xue- long Li, Weiwei Wan, and Kensuke Harada. Preference aligned diffusion planner for quadrupedal locomotion control. arXiv preprint 2rXiv:2410.1133 13586, 2024.
Hongchang Zhang, Jianzhun Shao, Shuncheng He, Yuhang Jiang, and Xiangyang Ji. Darl: distance- aware uncertainty estimation for offline reinforcement learning. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 37, pp. 11210-11218 2023.
Junjie Zhang, Chenjia Bai, Haoran He, Zhigang Wang, Bin Zhao, Xiu Li, and Xuelong Li. Sam- Leveraging visual foundation model with sequence imitation for embodied manipulation. In Proceedings of the International Conference on Machine Learning, pp. 58579-58598, 2024a.
Lijun Zhang, Lin Li, Wei Wei, Huizhong Song, Yaodong Yang, and Jiye Liang. Scalable constrained policy optimization for safe multi-agent reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems, 138698-138730, 2024b.
Lijun Zhang, Lin Li, Yajie Qi, Huizhong Song, Yaodong Yang, Jun Wang, and Wei Wei. Risk- aware direct preference optimization under nested risk measure, In Proceedings of the Advances in Neural Information Processing Systems, 2025a.
Yujia Zhang, Lin Li, Wei Wei, Yumpeng Lv. and Jiye Liang. A unified framework to control estima- tion error in reinforcement learning. Neural Networks, 178: 106483, 2024c.
Yujia Zhang, Lin Li, Wei Wei, Jianguo Wu, and Jiye Liang. Pseudo-distribution elite critics: Enhanc- ing accuracy in reinforcement learning value estimation. Neural Networks, pp. 108018, 2025b.
14

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
Behavior Cloning (BC) (Bain Sammut 1995) is a common approach in policy-constrained of-
fline RL. BCQ (Fujimoto et al. 2019) uses Variational Autoencoder (Kingma & Welling, 2013)
to model the behavior policy but is prone to fitting errors. TD3+BC (Fujimoto & Gu 202 inte- grates BC directly into policy loss, achieving competitive results without explicit behavior policy modeling. However, the distribution constraint of BCQ and TD3+BC often struggles to distinguish between optimal and suboptimal actions. BEAR (Kumar et al. 2019) and BRAC (Wu et al. 2019) constrain policy divergence via Kullback-Leibler or Wasserstein metrics. IQL (Kostrikov et al, 2021b) employs expectile regression and advantage-weighted BC without evaluating OOD actions. While these methods are efficient with limited networks, they often underperform ensemble-base approaches and are overly conservative. PRDC (Ran al. 2023) enforces distance constraints using KD-Tree (Bentley 1975) for neighbor search but faces challenges in balancing state-action priori- ties, manual parameter tuning, and large search spaces. Diffusion-DICF (Mao al. 2024) trans- forms the behavior distribution into the optimal policy distribution using score decomposition and a guide-then-selec strategy to avoid value exploitation errors. InAC (Xiao et al. (023). SQL. (Xu et al. 2023a), CDSA (Liu et al. 2024), and CPQ (Xu et al. 2022) incorporate in-sample constraints to avoid OOD actions, enhancing policy stability and generalization. These insights motivate our exploration of richer dataset utilization for precise behavior policy modeling and generalized policy learning.
In online RL, ensembles are used to build robust value estimates and enhance exploration (Wei et al. 2022 Zhang et al. 2024c). In offline RI., SAC-N (An et al., (021) penalizes OOD actions via min- imum ensemble values but requires up 500 networks and 3M training steps. EDAC (An al.) 2021 reduces uncertainty and ensemble size by minimizing pairwise cosine similarity, promoting diversity among networks. RORL (Yang et al. 2022) smooths OOD action Q-values by learning un- certainties for different state-action pairs. MSG (Ghasemipour et al. 2022) strengthens pessimistic
estimates by replacing shared targets with independent ones. PBRL (Bai et al. 2022) and MTDS (Bai et al., (2024) penalize value estimates based on action deviations from the mean. Though effec- tive in uncertainty quantification, these methods often need large ensembles. SAC-RND (Nikulin) et al. 2023) employs random network distillation with two Q-networks yet still incurs heavy training costs. Our method leverages the Coupling Effect to penalize OOD actions effectively with a smaller ensemble size, preserving data relationships and enabling generalized, efficient policy learning.
B THEORETICAL PROOFS
Proof. Since P(s'|s, a) and μ are Lipschitz continuous, we have the following equations:
[P(('ls, P(s'1s, (2){s,;a))l a2)ll < Lp|la: a2ll:
(13)
[lı(sı) - m(s2)] s L, -sll.
(14)
for all E S,a E A. Lp and L, represent the Lipschitz constants. The above equations have
received significant attention in theoretical RL research (Ran et al. 2023 Saxena et al. 2023 Then, we have:
- J(T') =|)))= - J(1) + ((1) ((T) s [J(*') - J(z) + ((z) - ((H)). (15)
Firstly, considering [J(z)) J(+)), define the occupancy measure:
dx)('') (1 -1- E
Σ
(o(())((s ( t=
(16)
where p(s + 8', [, m) represents the probability density of reaching state s' from state s after steps under policy æ. Consequently, Equation (T6) can be rewritten as:
1 B. Bandy s (s) [r(s))
J(x)= -
(17)
15

---

Published as a conference paper at ICLR 2026
Then, have
1
[J(T) ( J(H]
E (o
(8) ((s)] Band ( (r))
1-
1
(d (s) -s)(s) r(s) ds
1 1-7
1
dd (8) d,((s) In(s) ds
1
R Rmax
|da (8) -dp(s)] ds
1
CLpRmax
max (T)) - p(ss]]
1-7
SE8
CLpRmaxx
max [s()) (6BB) + x(و,) - p(s)]l
1-7
SES
CLp
max |T(() |ل33مم| + max |m((5)) - p(s)
1-Y
SES
SEB
CLp 1-7
(Ea + I.gts).
(18)
Here, (1) is due to Js ldr, (s) (s) ds < Clp max,es [|[(s) ((()]l the proof of this in- equality can be found in the appendix of (Xiong et al. 2022
Then, considering [J(T)) -(4)), have
- J(4]
E8 Eand (1) [r(]
1- E Bend,
1 (s) -du(s) r(s) ds
1
1 ds)(s) (s) -d)(s) (r(s)] ds
R drz (s) (s) ds
CLpRaas
max | (s) - mkss)|
8ES
CLp Rmmax 7
(19)
Finally, combining Equation
(8)and and Equation
19 we have
[J( [==) = [J(TT) ((m) + [pp - [(T]
< [J(T') J(n) [J(T) - (())
CLpRmaxx (Ea + Lises E..).
(20)
L
The proof of Theorem iss finished. From Theorem I the performance gap is influenced by far ca and e., EPIC reduces ég by selecting peer states 5, that are close to the given state s, ensuring that the state space of the learned policy is well-aligned with the in-sample states. This minimizes the risk of state distribution shift. EPIC reduces 6, by selecting actions that are close to both the behavior policy uu(s) and the optimal policy x- (s),, using combination of in-sample actions and the optimal policy. This ensures that the learned policy adheres to both the behavior and optimal policies, preventing large deviations in action selection. Finally, EPIC reduces 6. by leveraging the structure of the dataset and using peer-influenced constraints to enforce the learned policy's s alignment with the optimal policy. By effectively using peer states and action selections, EPIC narrows the performance gap and significantly enhances policy generalization, avoiding overfitting to the behavior policy and ensuring robust performance.
16

---

Published as a conference paper at ICLR 2026
C PSEUDOCODE OF PIC-TD3
Algorithm 2 TD3 Algorithm with Peer-Influenced Constraint (PIC-TD3)
Input: Observation state S, offline dataset D
Output: Learned policy π
Initialize: Critic networks Qo. Qb2 actor network To, target networks 0f 01.022 + 02 o Ø, and peer number K
for 1 = 1 T do
Sample a mini-batch = {(*, a,s.,, from D Find peer states in 'D of sample state by 8j arg mingeD) $11
Construct candidate action set A'
Find optimal action from A' by a" = arg max min Q0, (8,
Calculate PIC distance of sample state by d (s) = (s) - =a
Update 0g by minimizing L(0;) = (s,,,,,,)) [(V) Qo, (s, a)) if l mod d then
Update á using gradient descent with (o) = EswB 1-3Qo, (8, a)] + 8E (sb((s)
0 <- - (1 - -) ø' + TO + (1-1)) T)
end if end for
D IMPLEMENTATION DETAILS
PIC-TD3 implemented using PyTorch based on the TD3 implementation and EPIC is imple- mented based on the author-provided implementation of EDAC
The v2 version of D4RL benchmark datasets is utilized in Gym-MuloCo and AntMaze tasks. For comparison results for CQL, SAC-N, and EDAC
Adroit tasks, we use v1 version. The are sourced from (An al.) (022), while the scores for other methods are taken from Tarasovv al., 2023a). The AntMaze comparison results for CQL are taken from (Ghasemipour et al. 2022), while the scores for other methods are taken from (Tarasov et al. 2023a). The Adroit comparison results for BC, CQL, EDAC and RORI. are taken from (Yng et al. 2022). while the scores for other methods are taken from Tarasov et al. 2023a).
D.1 SOFTWARE
We use the following hardware:
Python 3.8
D4RL 1.1 (Fu al. 2020)
MuJoCo 2.1.0 (Todorov et al.
2012)
Gym 0.23.1 (Brockman et al. 2016
MuJoCo-py 2.1.2.14 Todorov et al.
2012)
PyTorch 2.1.2
CUDA 11.8
D.2 HARDWARE
NVIDIA RTX 4090
14th Gen Intel(R) Core(TM) 19-14900K
httras//qitthbb..om//fffii//
http::///github..cmmm /RDAC
17

---

Published as a conference paper at ICLR 2026
D.3 HYPERPARAMETERS
The hyperparameters of the PIC and EPIC are detailed in Table and Table 5
Table 4: General hyperparameters of EPIC and PIC-TD3.
Parameter
Value
Shared
Optimizer
Adam
Learning rate
3 x 10
Discount
0.99
Batch size
256
Size of hidden layers in critic and actor
256
Target network update rate
x 10
Update-To-Data ratio
Nonlinearity
ReLU
2.5
PIC-TD3
Target network update frequency
Action noise
(O 0.1) clip(AV 0.2). - 0.5,
Target policy smooth e
Critic number M
Table 5: Special EPIC hyperparameters on D4RL benchmark.
Task Name
Task Name
N
K
!]
5
halfcheetab-rrmddm--22
2
10
1
antmaze-umaze--2
10
20
2
1
mrmmazz-umaze-diverrs
10
20
2
halfchectah-expprti
2
10
1
antmaze-medium-play:
10
20
1
2
sinicheetah-mediiu--xxpeeee
I
autmmaze-mediu--ddierss
10
2
selicheetah-medium--eee
5
10
1
autmaze-large-plaay-22
10
20
1
2
halfchectta--full-relaaaay
antmazz-large-diveree--V
10
20
2
hopper-random-v2 v2
10
10
1
pen-human-v/
10
10
200
1
hopper-medium--2
10
10
1
pen-cloned-vl
10
10
200
1
hopper-expert-v2
10
10
1
pen-expert-vl
10
10
200
1
hopper-nmedfuum-experr--2
10
10
I
door-human-vl
10
10
200
3
hopper-mcddun--colaaywz
10
10
I
door-cloncd-vl
10
10
200
3
hopper-full-replay-v2
10
door-expert-vl
10
10
200
3
walker2d-random--22
hammer-human-vl
10
10
200
3
walker2d-medium--22
5
10
1
hammer-clomed-vi
10
10
200
3
walker2d-expeent-2
hammer-expert-vl
10
10
200
3
walker2d-mediume
-ediiuuee-p------2222222 v2
10
1
gelocate-human-
10
10
200
3
10
1
relocate-cloned-
10
10
200
3
walker2d-ffll--eepaa--2
10
1
relocate-expert-vl
10
10
200
3
DETAILED ENVIRONMENT SETTING
E.1 GRIDWORLD
In the gridworld environment, the agent's state is represented by its X-Y coordinates, and at each time step. it selects one of four possible actions: Up. Down, Left, or Right, with equal initial prob- abilities for each action. The agent only receives reward of 1.0 when it successfully reaches the goal. Each episode terminates when the goal is reached or when the maximum episode length of 256 steps is reached. The layout of the grid world is shown in Figure
18

---

Published as a conference paper at ICLR 2026
The agent interacts with the environment for total of 3 10 time steps, from which the last 2 x 10 transitions are used to form the offline dataset. Notably, we remove any experiences where the agent attempts to move right at the green triangle location. This exclusion is intentional, allowing to test whether different methods can escape local optima and identify potentially better actions beyond those seen in the initial data.
Figure 5: Visualization of gridworld.
E.2 LINEWORLD
As shown in Figure 6 our lineworld environment consists of four task variations:
lineworddeeasy: In the dataset, the ratio of - to + actions for each state is 1 9.
lineworld-random: In the dataset, the ratio of to +1 actions for each state is 1. innwooid-meedum: In the dataset, the ratio of -1
actions for each state 1. lineworld-harde the dataset, except for state 99, the ratio of -1 to +1 actions for each state is 99 1. In state 99, only the action to move left is available.
1:9
1:9
1:9
1:9
6
5
Easy
S
2
3
97
98
99
G
L
9:1
91
9:1
9:1
1:1
11
E1
1:1
1:1
Actions: 4-
Random
S
2
3
97
98
99
r(s, =
1:1
1:1
1:1
1:1
1:1
1:1
r(s,
9:1
9:1
9:1
9:1
9:1
No 100
Medium
S
2
3
97
98
Initial state: S
Goal State:
1:9
1:9
1:9
1:9
1:9
99:1
99:1
90:1
100+0
Hard
S
2
3
97
98
99
G
1:99
1:99
1:90
1:99
1:99
Lineworld
Figure 6: In the lineworld environment, the agent starts position and aims reach the goal G. The action space consists of [-1, +1], where -1 represents a move to the left, and +1 represents move the right. Reaching the goal ends the episode with reward of +100, while all other actions result in reward of 0.
19

---

Published as a conference paper at ICLR 2026
F MORE RESULTS
F.1 COUPLING EFFECT
We measure and analyze the relationship between various uncertainty metrics, ensemble size N, and PIC strength 8 on the hopper-medium-expe compute the uncertainty metrics Q Qua. and Qciip = Qmean min at the same time step under different values
dataset. We sample 10,000 state-action pairs and
and 6. We then calculate their average values and generate heatmaps, as shown in Figure 7 We find that, with fixed N, increasing leads higher uncertainty. Similarly, when ó is fixed, increasing N also results in greater uncertainty. Considering that an excessively large N can reduce computational efficiency, we can achieve effective uncertainty estimation with moderate level of PIC strength.
275
20
Expomble Size (N)
10 5
-80
250
100
225
20
175
100
(8)
(8)
(a) Effect Q
(b) Effect Qua
(c) Effect Qelip
Figure 7: The impact of ensemble size N and PIC strength on uncertainty metrics.
To further examine the influence of PIC strength uncertainty modeling, measure the evolution of Quin. Qad. and Qclip across different training steps (10k-IM) and various values of 5. The results summarized in Figure We observe that 5 and training progress increase, Qmin becomes in- creasingly pessimistic, while Qui and Qetip tend to rise on potential OOD action candidates, indicat- ing stronger penalization on high-risk actions. These dynamics corroborate the proposed Coupling Effect: larger 5 amplifies OOD penalties and yields more robust value estimation without requiring large ensemble sizes.
40
15.9
-00
12.5
Tinerepp (10kg)
5
-80
220
150
140
2.5
100
5
10
20
160
10
20
0.0
5
10
20
PIC (8)
P1C
PIC
(a) Effeect on Q (N = 2)
(b) Effect on @N = 2)
(c) Effect on Qeip V 2)
-60
12.5
Timesoons (L0k)
5
-80
10.
220
10
100
75
200
20
120
160
100
160
160
00
P1C(6)
P(C.
(d) Effect Qmin (N 10)
(e) Effect on Qua 10)
(f) Effeett Qaip N
10)
Figure 8: Uncertainty metric dynamics over training steps and PIC strengths for different ensemble sizes N.
20

---

Published as a conference paper at ICLR 2026
In continuous action spaces, a policy action almost never matches dataset action exactly. We there- fore adopt simple threshold-basee criterion. We first estimate a global action-distance threshold Ta from the offline dataset: for each dataset state, retrieve its nearest neighbor states, collect their behavior actions, and record the minimum action difference between the current action and these neighbor actions. The 90th percentile of these local minimum differences is taken Ta- rep- resenting the typical scale of action variation in the dataset. During evaluation, we randomly sample 10,000 states from the policy's 's state distribution. For each sampled state, we directly compute the minimum distance between the policy action and all dataset actions. If this distance is below Ta the action is classified as in-distribution; otherwise it treated OOD. The in-distribution ratio defined as the fraction of sampled states satisfying this condition.
Using this criterion, we observe from Figure 9 that PIC significantly increases the in-distribution ratio compared to the no-PIC baseline, and this ratio continues to rise with larger PIC strength 6, longer training horizons, and larger ensemble sizes N. This demonstrates that PIC progressively steers the learned policy back toward the dataset support, providing an additional perspective on the Coupling Effect discussed in the main text.
1.0
Tineetops (10kg
0.9
0.9
50 20 10
0.8
0.8
0.8
9.6
066
0.6
PIC (6)
(IC
PIC (6)
N
(b)
10
(c) N 50
Figure 9: In-distribution action ratio over training steps and PIC strengths d for different ensemble sizes N.
F.2 MINIMUM NUMBER OF Q-ENSEMBLES
The minimum number of Q-ensembles for the results in Tables I 2 and 3 are in Figure 10 As 5 increases, the required number of Q-ensembles decreases further. Notably, our method achieves competitive results with significantly fewer critics. demonstrating its efficiency and robustness. In Gym-MuJoCo tasks, m-c = medium-expert m-r = medium-replay, and f-r = full-replay. In Adroit tasks, h human, cloned, expert, ham hammer, rel relocate.
To further assess the robustness and fairness of our method, we additionally evaluate EPIC under single shared hyperparameter configuration across all Gym-MuJoCo tasks. Specifically, we test two unified settings: EPIC-I with N 10, K 10, 8 = and EPIC-II with N = 10, K = 10, shown Table 6 EPIC under these unified hyperparameters still outperforms traditional baselines most tasks, and its performance remains very close to the results reported in the main table, with only minimal differences. These results demonstrate that EPIC maintains stable and consistent performance even without task-specific adjustments, further supporting the robustness and practical usability of the proposed method.
F.3 EXTENSION OF PIC TO IQL AND SAC
In addition to its application in TD3, further extend the PIC framework to other RL algorithms, specifically IQL (Kostrikov et al. (222b) and SAC (Haarnoja et al. 2018). This allows us to evaluate the versatility and scalability of PIC.
The results, shown in T demonstrate that PIC maintains its effectiveness in enhancing the gen- eralization and robustness of the leamed policies across variety of environments. These findings highlight the scalability of the PIC framework, confirming its utility beyond TD3 and suggesting that it can be effectively applied wide range of RI. algorithms.
21

---

Published as a conference paper at ICLR 2026
hallcheethh
walkerdd
SAC-N
100
20
50
resken Invckarr
1-
andden medium
radem moliam
14
pet & door
Jummer relocate
100
10
P03- 200 260-
$26-2
J0c14
38353
10/c
Figure 10: With 8 = 1, the minimum number of Qennembbees required attain comparable perfor- mance in Tables and
Table 6: Average normalized score over final evaluations across five seeds on Gym- MymmMuJoCCo tasks.
Task Name
SAC-RND
PRDC
SAC-N
EDAC
PIC-TD3 (ours)
EPIC (ours)
EPIC-1
EPIC-11
balfcheetah- ramdom
27.6
26.9
28.0
28.4
25.3 2.1
28.9
1.4
27.6
25.0
4.1
halfcheetah-medee mediii
66.4
63.5
67.5
65.9
68.4 + 244
68.9
- 2.7
67.7
: 2.1
65.2
24
balfcheetab-cxper
102.6
105.2
106.8
104.2
107.9
107.7
3.1
108.5
balfcheetab- mederal-mmd--eeeeeeet
108.1
94.5
107.1
106.3
99.8
103.
388
1024
1: 3.2
104.6
4.5
halfcheetal-meduue -eddl---mu---paay
51.2
$5.0
63.9
61.3
63.1 + 0.9
64.8 =
62.9
+ 2.4
63.3
= 2.1
balfcheetah full replay
81.2
84.5
84.6
83.7 1.1
87.3
1.0
87.5
85.5 1 2.3
hopper-random
19.6
26.8
31.3
25.3
3.2
27.9
27.9 11
28.3
3.1
hopper-medium
91.1
100.3
100.3
101.6
100.7
+
102.2
102.2 - 0.5
101.3
1.2
bopper expert
109.8
110.3
110.1
111.8
2.1.1
112.2
1.5
108.7
1: 3.2
112.2
1.5
cooper-medium-eee
109.8
109.2
110.1
110.7
105.6 1
112.3
0.9
113.7
0.8
112.3
0.9
hopper medium replay
97.2
hopper-fil full-llal
107.4
102.9
105.4
105.8
1 0.8
107.6 - 1.6
107.6 = 1.6
106.8
2.0
walker2d-rande
18.7
5.0
21.7
16.6
14.6
+ 4.1
22.4
15.4 + 3.8
23.7 = 2.1
walker2d medium
92.7
85.2
87.9
92.5
94.9
2.2
95.9
3.2
97.7
1. 2.7
96.7
4.3
walker2d.ers expextert
115.3
1
104.6
111.2
116.7
114.7
114.4
0.9
116.4
+ 1.7
117.3
+ 1.4
116.8
2.2
walker2d med-im--eeppllee
89.4
92.0
78.7
87.1
95.5
2.4
95.2
2.2
93.1
2.6
95.5
3.3
walker2d-full-ree repllyy
105.3
107.4
105.5
2.2
Average
82.6
F.4 PERFORMANCE AND EFFICIENCY COMPARISON
Time complexity plays critical role in offline RL. We evaluate PIC-TD33, EPIC, and several baseline methods on the hopper-medium dataset. Figure T1 shows both the average training time (propor- tional TD3+BC) and the corresponding normalized scores. Ensemble-based methods demonstrate strong performance but incur substantial computational overhead. In contrast, non-ensemble base- lines such TD3+BC, IQL, and CQL offer superior computational efficiency. though often the cost of lower performance. EPIC successfully balances this trade-off. achieving both efficiency and robustness in policy learning. PIC-TD3, with its straightforward implementation and efficient search process, achieves performance comparable to ensemble methods while requiring significantly less training time.
F.5 GENERALIZATION RESULTS
F.5.1 LINEWORLD
We generate four finnssts-iiewvord-eeaas lineworld-random lineworld-medimm and lineworld- hard-orderee by increasing difficulty. In lineworld-eate the ratio of -1 to +1 actions is 1 99. In dndeworld-randonee the ratio is balanced at 1. In finewrrldmmedine the ratio skews to 9 1 for -1 actions. Finally, lineworld-hard has 99 ratio of -1 to +1 actions, except for state 99, where
22

---

Published as a conference paper at ICLR 2026
Table 7: Average normalized score over the final evaluation and five seeds on the AntMaze tasks.
Task Name
CQL SAC-RND
PRIC
MSG
SAC-BC-N
PIC-SAC (auns)
PIC-IQL (ours)
PIC-TD3 (ouns)
EPIC (ours)
antmaze-unaze
74,0
97.0
98.8
98.6
95,4=22
97.9=1.5
96.4 =
98.6
antmazs-umaze-diie
umazze-diveree
84.0
66.0
90.0
79.3
912
93.2=2.1
87.6#3.6
91.1 - 2.3
94.3
= 4.2
medium
61.2
38.5
52.8
85.9
85.8
82.4=3.7
84.5=28
79.3
88.1
3.3
antmazs-mediumn
53.7
74.7
78.8
75.5=49
80.2433
77.9
antmazs-large-plan
15.8
43.9
54,8
62.3-39
62.3+211
55.7
antmaze- Caanar--lrrg--iee large-diverse
14.9
45.7
50.0
71.3
75.8
47.6-4.2
59.6+3.1
53.2 4.7
66.9 5.3
Average
50.6
61.0
75,9
80.6
81.8
76.1
78.7
75.6
82,9
110
EPIC-10
EDAC-50
100
EPIC-5
EDAC-10
Vormalised Score
PIC-TDS
SAC-RND
IQL
SAC-N-100
TD3-BC
2.5
Training time Relative to TD3+BC
Figure 11: Comparison of performance and efficiency across different methods on hopper-medium. Here, method names such as EPIC-5, EPIC-10, EDAC-10, and EDAC-50 indicate the number of critics used in each ensemble.
only the -1 action is available. These datasets vary in difficulty based the action distributions, offering a diverse set challenges for the agent.
We train TD3+BC, BEAR, PRDC, and PIC-TD3 the datasets, shown in Figure 12 random datasets, all methods learn the optimal policy. However, with medium and hard datasets, reward sparsity and conservative policy regularization prevent TD3+BC and BEAR from achieving the optimal policy. While PRDC leverages dataset state-action pairs, ignores potential optimal actions. PIC uniquely generalizes by exploiting optimal actions peer states, even those unseen in current states, escaping local optima.
In easy and
Easy
Random
(al methods
(b) methods
Medium
s
G
[c) TD3+BC and BEAR
(d) PRDC and PIC-TD3
S
(e) TD3+BC, BEAR R and PRDC
(i PIC-TD3
Figure 12: Visualization of the learned policy of different methods on four lineworld datasets.
F.5.2 ANALYSIS STATE-SPACE DISTANCE METRICS
To comprehensively examine the influence of state-space distance metrics and representation meth- ods on EPIC, we conduct extensive experiments on six MuloCo tasks thopper-mediug hopper- expert, hopper-mediim--exee expert, walker2d-mediun walker2d-expent walker2d-medium-eet on the high-ding diseeniinnsional quadruped locomotion benchmark Walk These Ways (WTW) (Margolis & Agrawal 2023
and
23

---

Published as a conference paper at ICLR 2026
WTW is realistic, high-dimensional, and strongly multimodal quadruped control platform built upon the Multiplicity of Behavior framework. Following the reference implementation we set the state dimension to 70 to balance realism and computational tractabilityy. WTW provides behavior- conditioned interface that enables a single controller to generate diverse locomotion behaviors, in- cluding pacing, trotting, bounding, pronking, and crouching.
We construct a velocity-tracking task using three representative gaits: pacing, trotting, and bound- ing. For each gait, we collect 4096 episodes of 250 time steps each, forming a large-scale, high- dimensional, and multimodal offline dataset Yuan et al. (2024). Performance is evaluated using the following metrics:
Stability: fraction of episodes without falling within 250 steps;
Average x-axis speed: average forward velocity.
To analyze the effect of distance metrics and state representations on nearest-neighhor retrieval, we evaluate several EPIC variants:
CQL Kumaar al. 2020) and EDAC (An et al. 2021 as strong value-regularization base- lines;
EPIC-Raw: KD-tree built on raw states using Euclidean distance;
EPIC-Norm: per-dimension min-max normalization before KD>tree construction;
EPIC-PCA: PCA applied standardized states, retaining principal components that plain at least 99% of the cumulative variance;
E.PIC-Embed: two-layer MLP autoencoder trained to map states into 64-dimensional
latent space, where the KD-tree built (all other EPIC components remain unchanged).
As shown in Tables and EPIC achieves highly consistent performance across Raw, Norm, PCA, and Embed variants MuJoCo tasks. In these medium-dimensionall and well-structured state spaces, PCA and normalization introduce only mild transformations, and the autoencoder typi- cally learns almost linear embedding. As result, the neighborhood structure is largely preserved, making EPIC insensitive to the particular choice of distance metric.
In the high-dimensional and multimodal WTT environment, EPIC's overall performance decreases because raw Euclidean distance can be dominated by few high-variance dimensions, reducing neighbor quality. Nonetheless, EPIC still outperforms CQL and remains comparable EDAC (3M steps). Among the variants, EPIC-Norm provides limited improvements through scale alignment; EPIC-PCA produces more reliable neighborhoods by capturing dominant variation directions; and EPIC-Embed performs best, as its learned latent space clusters states with similar gait and forward- velocity characteristics.
Overall, the benefits of EPIC arise primarily from the peer-induced constraint mechanism rather than dependence on specific metric. While raw Fuclidean distance already effective on standard MuJoCo benchmarks, introducing mild representation learning further improve neighbor quality and policy performance in ligh--imensional or strongly multimodal environments.
Table 8: Effect
of state similarity metrics EPIC across MuJoCo tasks (five seeds).
Task Name
CQL
EDAC
EPIC-Raw
EPIC-Norm
EPIC-PCA
EPIC-Embed
hopper-medium
61.9
101.6
102.2 + 0.5
101.4 0.8
101.5
103.1 +: 1.6
hopper-expert
106.5
110.1
1
113.1
113,4
111.0 + 1.4
hopper-medium-exppe
96.9
110.7
112.3
112.5
3.1
111.8
108.5 1: 2.8
walker2d-medinn
79.5
92.5
3.2
96.5 - 2.7
96.8 2.0
93.2 9 1.9
walker2d-expert
109.3
115.1
117.7+22
116.9 3.1
114.9
116.3
+ 1.8
walker2d-mmedium-xxperr
109.1
114.7
116.411.
117.7 1.4
115.1 1 1.6
117.2 1 0.9
Average
93.9
107.5
109.5
109.7
108.9
108.2
24

---

Published as a conference paper at ICLR 2026
Table 9: Average speed (m/s) and stability (%) comparison of different Hlte-similarity metrics for EPIC in quadruped locomotion velocity-tracking tracking (WTW) tasks (three seeds).
CQL
EDAC
EPIC-Raw
EPIC-Norn
EPIC-PCA
EPIC-Embed Speed Stablitty
Speed Stubiiity
Speed / Stability
Speed Stability
Speed / Stability
Speed Stubility
Pacing (0.5 m/s)
0.31 / 37.21
0.38 41.52
0.33 400.88
0.41 64.35
0.36/ 61.6/61.62
0.37/ 67.35
Trotting (0.5 ma/)
0.33/ 35.53
0.30/ 39.41
0.36 37.44
0.32 50.37
0.35 57.35
0.39/ 58,97
Bounding (0.5 m/s)
0.24 21.66
0.36/
0.32 30.57
0.44/ 40.12
0.39/53.60
0.45 52.45
Pacing (1 m/s)
44.9/4
0.44/51.67
0.53 54.68
0.52/61.45
0.55 / 71.61
0.56/71.51
Trotting (1.0 in/s)
0.33 /4..37
0.31 440.34
0.41 50.76
0.47 46.66
0.43 0.43/54.63 54.63
0.46/ 55.54
Bounding (1.0 m/s)
0.28 / 32.45
0.51/45.52
0.40/ 34,68
0.42/ 43.69
0.47 / 63.48
0.55/ 0.55/65.79
F.6 PARAMETER STUDY
F.6.1 PIC STRENGTH 5
The PIC strength d directly controls the intensity of the peer constraint. A larger 8 places more emphasis on keeping the learned policy close to the in-sample optimal action, potentially reducing exploration. Conversely, smaller focuses more on fitting Q-values, which may lead to greater deviations from the behavior policy. As shown in Figure 13 in the Gym-MuJoCo environments the algorithm's 's performance degrades markedly when the peer-influenced constraint strength 6 is set too low (< 1) or too high (> 4). In contrast, the best results are consistently observed within a moderate range (8 3)). A similar trend holds for the AntMaze and Adroit benchmarks, indicating that appropriately tuning 5 yields robust performance across diverse task types.
100
90
80
80
Soeds
70
70
60
0.1
0.1
(a) hopper-medium
(b) antmaze-umaze
(c) pen-human
Figure 13: Impact PIC strength on PIC-TD3 performance.
F.6.2 ENSEMBLE SIZE N
Ensemble size N controls uncertainty estimation strength. Larger N helps penalize OOD actions in ensemble methods, but too large N reduces efficiency. As shown in Figure [4] without PIC
(ō 0), performance improves with N. but for small 6, performance starts declining when N 2 50. For larger 8, performance decreases as N increases.
To further assess the sensitivity of EPIC to different action-selection rules, we conduct ad- ditional study on hopper-medium, comparing the standard min-over-critics, Random Double Q (sampling two critics from N 10), and Mean-Q aggregation under varying ensemble sizes N E 21, 2, 10, 50}. The results (Figure 15) show that once N > 2, all three variants exhibit similar performance with only minor deviations, suggesting that EPIC largely insensitive the specific critic aggregation rule. When N becomes very large (e.g.. N 50), learning slows down due to overly pessimistic value estimates, consistent across all variants. When N the perfor- mance degrades some extent. However, under the peer-influenced constraint, the policy remains supported within the dataset distribution and can still learn a moderately good policy. In contrast, other single-critic networks typically fail in this setting Wei et al. (2022)
F.7 PERFORMANCE ON OFFLINE-TO- ONLINE D4RL.
Evaluating fifline-to-onling performance a critical aspect of offline RL algorithms, especially in light of recent advancements. Based this, conduct additional tests EPIC, as it demonstrates
25

---

Published as a conference paper at ICLR 2026
100
Eisariil 040
90
75
10
80
80
30
8
100
100
100
0.1
0.1
(a) hopper- menrer--ediu--eplaa
(b) antmaze-ume -uzzze
(c) humman
Figure 14: Impact ensemble size N on EPIC performance.
hopper-
100
Nottaalzzd Return
80
83
60
40-
20
0.0
0.2
0.6 Times Steps (1e6)
0.4
1.0
Figure 15: Performance of different action-selection strategies under varying ensemble sizes N.
remarkable effectiveness in offline pretraining through PIC and the Coupling Effect. We follow the methodology of Tarasov al. 2023b) to assess EPIC's performance in offline-to-online transitions. We compare EPIC's performance with TD3+BC (Fujimoto & Gu 2021). IQL (Kostrikov et al., 2021b), and SPOT (Wu et al., 22022 during the transition from offline online learning. Baselines scores are taken form (TTarsovv et al. 2023a).
The results, including scores from the offline phase and after online fine-tuning, are presented in Table 1O, EPIC demonstrates competitive performance, outperforming in three out of six AntMaze datasets and achieving state-of-the-art results the final scores for the Adroit tasks.
Table 10: Normalized performance after offline pretraining and online fine-tuning on D4RL.
Task Name
TD3 + BC
IQL
SPOT
EPIC (ours)
antmaze-umaze
66.8 + 91.4
77.0 - 96.5
91.0
99.5
98.6 99.5
59.1
59,5
36.3 95.0
94.3 98.3
mntmaze-mediim--e play
59.2 + 49.4
71.8 89.8
71.8 - 97.5
88.1 95.4
meimaze-medium-diinn
62.6 + 49.4
64.3 92.3
73.8 + 94.5
83,8 93.1
antmazo-large-play
21.5
38.5
31.8 97.3
65,9 89.3
luimaze-largeddivrrs
9.5 - 0.4
26.8 64.3
17.5 - 81.0
66.9 74,8
AntMaze Average
46.5 + 40.0 (-6.5)
56.3 + 78.5 (+22.2)
53.7 + 94.1 (+40.4)
82.9 - 91.7 (+8.8)
pen-cloned door-cloned
86. - 110.3
84.2 -> 102.0
6.2 > 43.6
94.6 > 125.4
1.2 20.3
2,0
7.1 21.2
hammer-cloned
2.4 - 11.6
1.4 + 57.3
4,0 + 3.7
12.7 - 64.3
relocate-cloned
-0.1 + 0.1
-0.2 + -0.2
-0.2 + -0.2
1.1 2.0
Adroit Average
22.1 + 31.4 (+9.3)
21.7 (+23.2)
(+10.2)
28.9 (+24.4)
Total Average
36.7 36.5 (-0.2)
42.5 (422.))
33.4 61.8 (+28.4)
63.4 77.7 (+14.3)
F.8 TRAINING CURVES
We present the training curves of the algorithms in experiments. The solid lines in the figure represent the average performance across five random seeds, while the shaded areas indicate the standard deviation, providing measure of variability in the results.
26

---

Published as a conference paper at ICLR 2026
In Figure [6 display the training curves of EPIC and EDAC (An et al. 2021 over three million steps across 18 tasks in MuJoCo. EPIC is implemented based on the author-provided implementa- tion of EDAC In Figure 17 we further compare the Interquartile Mean (IQM) values of EPIC and EDAC. IQM is a more reliable metric than simple averages, as it takes the distribution of outcomes into account and better reflects the overall performance, especially in terms of risk and outlier han- dling. Additionally, we plot the training curves for the AntMaze and Adroit environments, where compare EPIC against IQL (Kostrikov et al. 2021b), stable baseline known for its robust performance, as shown in Figures 18 and 19
IQL implemented based on the CORL codebase
http::///4tthbbb..ommmmmmmmm///
27

---

Published as a conference paper at ICLR 2026
expert
Nomallzed Retun
Noralized Rotum
100
Normalized
50
EDAC
RDAC
0.0
0.5
1,0
1.5
2.0
2.5
3,0
Times Strpp: (le6)
Times Steps
Times Steps (1e5)
halfcheetahh medium expert
halscheetah medium explay
halfcherual: full replay
Nomalized Retum
100
Nomalized Retum
50
Nomalizzd Retum
%
50-
50
20
25
EPIC
EDAC
0
HDAC
EDAC
Times Stepo (1 06)
Times Steps (166)
Times Stops (1e6)
honper-random
hopper-medium
hogper-cxpert
Nomallnn Keum
%
Norallad Rotmm
Normalized Betum
100
75
50
10
25
IPIC
BPIC EDAC
REXAC
BDAC
0
Times Steps (1o6)
Times Steps (1ef)
Times Stops (Lo6)
Ropper Beeper-mmeliun---qpee
hopper-tte hopeo-meeiimmeeply
hompe-ffll--eplly
100
Normalazd Ketam
Normalzze Retum
100
Nomallzzd
50
50
EPIC
HPIL'
BPIC EDAC
EDAC
EDAC
Times Stego 06)
Steps (166)
Stops
walker2D rundems
-nIlor2Dmmeduum
walker2D-oxpet
150-
Nomoltte Rotum
100
Nommalaadd Retum
100
Neumallaed
so
10-
BNC
EDRU
BDAC
Times
Times Sleps
walker2T-nne maddiun--p
wollerrTD-ffll-eeeee full--ll-pyylpp
Normalized Roum
Nocmalzzed Rotum
Normalized lizeed Rotum
100
75
50
25
EPIL' BDAC
EDAC
BDAC
Times Steps (1o6)
Times Steps (1e6)
Times Steps (lo6)
Figure 16: Comparative mean performance analysis of EPIC and EDAC across 18 MuJoCo tasks.
28

---

Published as a conference paper at ICLR 2026
hallcheetah
Nomallzz Retum
50
Nomullna Retum
100
-0
Nomaltzdd
50
10
EPIC
EPIC
EPIC BDAC
EDAC
2
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Times Staps (1e6)
Times Steps (106)
Times Steps (1e6)
holleheecta hemetaa-mmmmmmmaepeee
Michsetal full-roplay
Nommllzed Betm
100
Nomalized Retmm
Nomalizze Retum
50
20
EDAC
EDAC
EDAC
Times Steps (1))))
Times Steps (1e6)
Times Staps (1e5)
topper-randem
hopper-mclium
hopper-cxpent
Nommlizzd Rotum
1.00
100
75
Nomndizd
20
Normalznd
25
EPIC
BDAC
EDAC
BDAC
Times Steps (1e6)
Times Steps (1ef)
Times Steps (1o6)
hopper mediim--peii
hopper modium--pppayyy
hopper- full-nply
Normalized Roaur
Normelized Remin
100
100
Normilzed
75
50
EPIC
EPEC
BPIC BDAC
EDAC
25
BDAC
Times Steps
Times
walker2D random
walkerD medium
walker2D expect
Narmalaad Ratum
Normallaed Rttum
100
Normalized
50
EPTC
EPCC
EDAD
Times Stape: (leB)
Times Steps (106)
Times Stops (1o5)
wwlke:2D-nediinee enDeeedium-eepett
Nomalized Retum
-5
Normaitzed
Normaltzed
50
50
30
12
BPIC
EDAL'
BDAC
Times Steps (106)
Times Steps (166)
Times Stops (Lo5)
Figure 17: Comparative IQM performance analysis of EPIC and EDAC across 18 MuJoCo tasks.
29

---

Published as a conference paper at ICLR 2026
100
antimaze umaaze
100
untmaze verse
100
autimaze medium play
Noralltd Ratu
75
75
Nottaleed
50-
50
13
0-
0.9
0.2
0.4
0.5
0.5
1.0
0.0
0.2
0.44
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.5
1.0
Times Stepp (1cf)
Times Skepe (1e9)
Times Steps (1ef)
snimasze Eonge Iarge--
50
Nermdiza Raimm
Normaliza Retum
Nermolied Ratum
60
60
30-
20-
-0
1Q21
IQC
0-
0.0
0.2
0.4
0.6
1.0
0.00
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0..5
0.8
Times Slepo (1eh)
Times Stops (1en)
Times Steps (1o6)
Figure 18: Comparative mean performance of EPIC and IQL. across six AntMaze tasks.
pent-laman
pen-cluned
125
150
100
bomalleed
Noualled
Nomalazd
73
EP10
IQU
IQU
100
0.3
0.2
03.4
0.56
1.0
0.0
0.2
0.44
0.6
0.5
1.0
0.0
0.2
0.4
0.5
0.8
1.0
Times Skepp: (let))
Times Stepe c (le))
Times (teps(lee))
done human
door cloned
door expert
15
15
Nomaltzd iteum
to
210
Norallzd
10
100
IQL
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
Times Steps
(l)
Times Steps (1o6)
Times (teps((ee))
haammer-lemmen
hamtme--eonnd
lemmessssppr
20
150
Nocmallieed Retura
65
Noemallood
Noraalood
LO
En
10L
0.0
0.4
0.5
0.8
1.0
0.0
0.2
0..4
0.5
0.8
1.0
0.0
0.2
0.4
0.5
0.8
Times Staps (1.6)
Times Steps (1.6)
Times Steps (3c5)
relocste-hummn
relocste-cioned
120
relocete-expent
Nomuliaad Rohin
Nommlltzdd Retuu
100
Normalzed
80
IQ1.
IQ1.
60
101
0.0
0.2
0.4
0UR
1.0
0.0
0.2
0.4
3.6
0.3
1.0
0.0
0.2
0.4
0.5
0.8
Times Steps (le5)
Times Steps (leS)
Times Steps (leS)
Figure 19: Comparative mean performance of EPIC and IQL across 12 Adroit tasks.
30

---

Published as a conference paper at ICLR 2026
G LIMITATIONS AND FUTURE WORK
Despite promising results, several limitations remain. (i) Peer selection currently relies on Euclidean nearest neighbors the raw state space; high-dimensional, highly nonlinear, visual settings, Euclidean similarity may diverge from semantic similarity, yielding "pseudo" peers. (ii) Candidate actions are drawn from local neighborhoods of the dataset and thus are sensitive data sparsity or distributional skew, which can expose coverage gaps. (iii) The best- hyperparameters (e.g., PIC strength d) vary across tasks, making selection nontrivial and potentially brittle.
Future work includes: (i) retrieving peers in learned representation space via contrastive or met- ric learning (optionally action-come siiioncoonitionee), replacing raw Euclidean distance; and (ii) developing adaptive schemes for automatically tuning ó (e.g., uncertainty- constraint-driven adjustment) to improve robustness across diverse environments.
H USE OF LARGE LANGUAGE MODELS
Large Language Models were only used for minor language polishing. They were not involved in method design, theoretical analysis, or experiments, and therefore do not affect the originality of this work.
31


Reviews:
[{"review_id": "1oiSmWtk9s", "reviewer": "Reviewer_3S8J", "summary": "The paper proposes Peer-Influenced Constraint (PIC), a plug-in policy regularizer for offline RL that, for a query state, retrieves K nearest “peer” states from the dataset, collects their associated actions as candidates, and selects the best candidate by a critic (min over critics) to constrain the actor toward high-value in-distribution actions. The authors further introduce EPIC (Ensemble PIC), showing a claimed “Coupling Effect” between PIC strength and uncertainty estimation that purportedly enables smaller ensembles without losing performance. Experiments on D4RL (Gym-MuJoCo, AntMaze, Adroit) report competitive or SOTA results and improved efficiency vs. large-ensemble methods. Key ingredients include a KD-Tree for fast peer retrieval, a formal performance-gap bound under Lipschitz assumptions, and ablations over K, δ (PIC strength), and N (ensemble size).", "questions": "1. Distance metric & normalization. What exact preprocessing/normalization is used before KD-Tree (per-dimension scaling, whitening, PCA)? Have you tried learned embeddings (e.g., representation pretraining) for neighbor search, and how does that affect performance vs. raw states? \n\n2. Coupling effect mechanics. Please reconcile the narrative around δ’s effect on uncertainty. Under fixed N, does increasing δ increase or decrease Q-uncertainty on OOD actions in practice? Can you provide a controlled analysis (same seeds, same checkpoints) that tracks policy action-distribution shift and critic variance step-by-step?  \n\n3. Critic dependence & bias. Since a* is chosen via min over critics, how sensitive is PIC/EPIC to critic underestimation bias? Provide an ablation using single-critic and target-action noise or double Q with clipped targets to test robustness.  \n\n4. Candidate-set composition. Beyond nearest states, did you explore wider but weighted candidate pools (e.g., radius-based retrieval with distance-weighted selection), and what is the trade-off vs. compute? Any results on K beyond 50 and on adaptive K per state? 5. Generalization diagnostics. In AntMaze/Adroit, can you report how often the selected a* comes from (i) the current state vs. (ii) peers, and the distance of chosen peers? This would concretely show when PIC escapes local optima rather than re-selecting the behavior action.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Practical, plug-and-play idea. PIC cleanly reuses cross-state action candidates to relax over-conservative behavior-cloning constraints while staying in-distribution; integrates with TD3/SAC/IQL. \n\n2. Clear algorithmic specification. Definition of PIC distance and candidate selection with min-critic action choice is straightforward; EPIC generalizes this to N critics.  \n\n3. Empirical coverage & results. Extensive D4RL results show PIC-TD3 competitive with ensemble-free baselines and EPIC surpassing baselines on many tasks (Gym-MuJoCo, AntMaze, Adroit).  \n\n4. Efficiency angle. Claimed ability to use smaller ensembles while maintaining performance addresses a common pain-point in offline RL. Parameter studies (K, δ, N) are helpful for practice.", "weaknesses": "1. State-space similarity & representation. PIC hinges on Euclidean nearest neighbors in the raw state space (KD-Tree). In many high-dimensional or poorly scaled domains, Euclidean distance can be misleading; the paper lacks experiments comparing learned/state-normalized metrics vs. raw features, and ablations on feature scaling or representation robustness. \n\n2. Coupling-effect clarity. The section discussing how δ interacts with uncertainty occasionally reads inconsistently (e.g., whether increasing δ “raises” uncertainty vs. “reduces” overestimation leading to “lower” uncertainty). This needs a tighter, causal explanation and clearer metrics (Qmin/Qstd/Qclip) across datasets/time. \n\n3. Theoretical assumptions are strong/generic. The Lipschitz assumptions and the bound provide qualitative reassurance but do not uniquely characterize PIC’s effect beyond standard smoothness arguments. No finite-sample or function-approximation analysis is provided to justify behavior under approximate critics. \n\n4. Scalability of neighbor search. KD-Tree is efficient in moderate dimensions but can degrade with very large |D| and high-dimensional S; the paper acknowledges this limitation without proposing practical approximations (e.g., ANN indices, learned retrieval).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5j399v5Sjx", "reviewer": "Reviewer_yUTS", "summary": "This paper addresses the distributional shift problem in offline reinforcement learning, where policies trained on fixed datasets often fail to generalize. It proposes a novel policy regularizer called Peer-Influenced Constraint (PIC), which guides the learned policy using actions from similar \"peer\" states found within the dataset. The paper also identifies a \"Coupling Effect\" between this policy constraint and the uncertainty estimation of ensemble-based methods. Based on these ideas, it introduces EPIC (Ensemble Peer-Influenced Constraint), a method that integrates PIC into an ensemble framework to achieve strong performance with higher computational efficiency. The methods are evaluated on standard D4RL benchmarks, where they demonstrate state-of-the-art results.", "questions": "1.Could the authors please explain the rationale for the 200x difference in the hyperparameter η between the Adroit tasks (η=200) and all other tasks (η=1)? why do the dexterous Adroit tasks require such extreme critic diversity regularization?Why was this parameter excluded from the sensitivity analysis in Section 4.4 and F.6?   \nCould the authors provide more intuition on the mechanism behind the \"Coupling Effect\"? Specifically, why does constraining the policy to a high-value in-distribution action cause the Q-function ensemble to exhibit higher variance on OOD actions？", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The core concepts of Peer-Influenced Constraint (PIC) and the \"Coupling Effect\" are novel, intuitive, and provide a valuable new perspective on balancing policy and value regularization in offline RL.\nS2: The paper is supported by extensive and rigorous experiments on standard D4RL benchmarks (Gym-MuJoCo, AntMaze, Adroit), demonstrating strong, state-of-the-art performance across multiple task suites.   \nS3: The inclusion of targeted experiments on generalization, computational efficiency, and offline-to-online fine-tuning provides a comprehensive evaluation of the proposed methods.\nS4: The paper is clearly written, and the motivation is well-illustrated, particularly with the introductory gridworld example that effectively contrasts the proposed approach with baselines like TD3+BC.", "weaknesses": "W1: The core mechanism of PIC relies on finding nearest neighbors using Euclidean distance in the raw state space. This approach is known to be ineffective in high-dimensional spaces (e.g., image-based observations) due to the \"curse of dimensionality,\" where distances between points become less meaningful. This severely limits the method's scalability and general applicability. \nW2: The method appears to be highly sensitive to hyperparameters, which undermines its claim as a simple \"plug-in\" module. In particular, the hyperparameter η is set to 200 for all Adroit tasks but 1 for all others—a 200x difference that is not explained or analyzed in the parameter study. This suggests that achieving good performance may require extensive, task-specific tuning, a known issue that can obscure the true robustness of an algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vOOShzQppS", "reviewer": "Reviewer_d22U", "summary": "The paper proposes Peer-Influenced Constraint (PIC) and its ensemble extension EPIC for efficient offline reinforcement learning. The key idea is to exploit peer states within the offline dataset to form candidate action sets, allowing the policy to learn from similar states and select the highest-valued in-distribution action. When combined with ensembles, the method exhibits a Coupling Effect between peer constraint strength and uncertainty estimation, enabling robust OOD suppression with smaller ensembles.\n\nImportantly, EPIC can be viewed as an evolution of ensemble-based offline RL methods such as EDAC. While EDAC enhances SAC-N by diversifying ensemble critics to reduce overestimation bias, it still requires large ensembles (often 10–50 critics) and substantial compute.\nEPIC inherits EDAC’s ensemble foundation but integrates a PIC that reuses dataset structure — effectively reducing the reliance on large ensembles and improving generalization, **leading to performance enhancement in larger state-action domain**. This synergy between behavioral regularization and value uncertainty represents a notable advancement in the design of efficient, uncertainty-aware offline RL.", "questions": "**Questions relative to weakness mentioned earlier**\n\n- Could the authors elaborate on the assumptions underlying the peer-influenced constraint—specifically, the degree of state–action continuity required for the method to remain valid? My understanding is that PIC implicitly assumes that nearby states tend to share similar optimal actions, which is reasonable in continuous and smooth control domains. However, this assumption may not hold in small or discrete environments such as the one illustrated in Figure 1. Especially, this could have negative impression of selecting OOD action which is prohibited in offline RL setting. If this interpretation is correct, clarifying such assumptions in the paper would strengthen its scope and applicability. Otherwise, please correct me if I am mistaken in my understanding of Figure 1 and the intended domain of applicability.\n\n- How would PIC behave in discrete or non-smooth domains where neighboring states may not share similar optimal actions?\n\n - Would the method need to adapt (e.g., via learned similarity metrics or representation smoothing) to remain effective in such cases?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Strengths**\n\n* **Clear conceptual advance over EDAC**: achieves comparable or better uncertainty calibration with significantly fewer critics through the Coupling Effect, offering a well-grounded improvement in efficiency without sacrificing performance.\n\n* **Innovative peer-state reuse**: effectively leverages intra-dataset structure to improve generalization beyond strict behavior cloning. Moreover, the study convincingly demonstrates that such peer-state reuse directly contributes to reducing the required ensemble size while maintaining stable OOD suppression.\n\n* **Insightful analysis of the PIC–ensemble interaction**: I particularly appreciate the experiment in Figure 4(d), which clearly visualizes how the effect of PIC varies with ensemble size. The results show that while PIC’s contribution may saturate under very large ensembles, it plays a critical role in the small-ensemble regime—precisely where efficiency and computational savings matter most. This ablation provides strong empirical evidence that the proposed Coupling Effect is both interpretable and practically beneficial, reinforcing the paper’s claim of achieving “efficient” offline RL.", "weaknesses": "**Weakness**\n\nWhile the motivation experiment in Figure 1 illustrates the behavioral difference between TD3+BC and PIC, the chosen environment—a small and discrete gridworld—may not be fully representative of the setting where the proposed peer-influenced constraint operates most effectively.\nThe key assumption behind PIC is that nearby states tend to share similar optimal actions, which implicitly presumes a degree of continuity and smoothness in both the state and action spaces.\nSuch an assumption is reasonable in continuous and high-dimensional control domains (e.g., MuJoCo tasks) but may not hold in discrete or highly non-smooth environments, where small state changes can correspond to qualitatively different optimal actions.\nClarifying this assumption early in the paper would help readers understand the scope and applicability of PIC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JpIjqTtP2W", "reviewer": "Reviewer_dFV9", "summary": "This paper proposes a new regularizer for policy optimization in offline RL called PIC. Unlike conventional approaches that constrain the policy toward the specific action observed for a given state, PIC encourages the policy to align with actions drawn from a set of similar (peer) states, which can be interpreted as a relaxation of the conventional approaches. The experiment results show the proposed method achieves state-of-the-art results on D4RL benchmarks.", "questions": "- In Equation (7), it seems when j=1, \\hat{s}_j will always be s. Is this correct?\n- In Equation (9), why are alpha and delta both required? To my understanding, only one of the two is required to adjust the importance between the two terms.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is intuitive and tackles a valid limitation of the commonly used policy regularizer in offline RL.\n- The paper provided a theoretical justification of the proposed approach.\n- The paper achieves strong empirical results across multiple tasks in the D4RL benchmark.\n- The proposed method can be integrated into other (offline) RL methods like TD3 or IQL.", "weaknesses": "- The peer identification relies on norm distance in the raw state space, which may not align with semantic similarity in high-dimensional (for example, pixel) environments. The proposed method should be tested in such environments.\n- Figure 4 (c) seems to show that the proposed method is somewhat sensitive to the hyperparameter delta.\n- Table 5 shows the proposed method used different hyperparameters for each task in D4RL Mujoco. However, as far as I know, the baselines like CQL and IQL used the same hyperparameters for the Mujoco tasks. Therefore, this is not a fair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)