Robust Reward Alignment via Hypothesis Space Batch Cutting
Authors: Zhixian Xie
Haode Zhang
Yizhe Feng
Wanxin Jin
Venue: ICML 2025 poster (2025)
Keywords: Learning from Human Feedback
Inverse Reinforcement Learning
Preference Based Reinforcement Learning
Robust Learning
Primary area: reinforcement_learning->inverse

Abstract: Reward design in reinforcement learning and optimal control is challenging. Preference-based alignment addresses this by enabling agents to learn rewards from ranked trajectory pairs provided by humans. However, existing methods often struggle from poor robustness to unknown false human preferences. In this work, we propose a robust and efficient reward alignment method based on a novel and geometrically interpretable perspective: hypothesis space batched cutting. Our method iteratively refines the reward hypothesis space through “cuts” based on batches of human preferences. Within each batch, human preferences, queried based on disagreement, are grouped using a voting function to determine the appropriate cut, ensuring a bounded human query complexity. To handle unknown erroneous preferences, we introduce a conservative cutting method within each batch, preventing erroneous human preferences from making overly aggressive cuts to the hypothesis space. This guarantees provable robustness against false preferences, while eliminating the need to explicitly identify them. We evaluate our method in a model predictive control setting across diverse tasks. The results demonstrate that our framework achieves comparable or superior performance to state-of-the-art methods in error-free settings while significantly outperforming existing methods when handling a high percentage of erroneous human preferences.

Robust Reward Alignment via Hypothesis Space Batch Cutting
Zhixian Xie* Haode Zhang*?
Yizhe Feng? Wanxin Jin
Abstract
(Radosavovic et al., 2024; Tsounis et al., 2020; Hoeller et al., 2024). In these applications, reward/cost functions essen- tial, as they dictate agent behavior by rewarding/penalizing different aspects of motion in balanced manner. How- ever, designing different reward terms and tuning their rela- tive weights is nontrivial, often requiring extensive domain knowledge and laborious trial- andl-o.d- This is further am- plified in user-specific applications, where agent behavior must be tuned to align with individual user preferences.
Reward design in reinforcement learning and op- timal control challenging. Preference-based alignment addresses this by enabling agents to learn rewards from ranked trajectory pairs pro- vided by humans. However, existing methods often struggle from poor robustness to unknown false human preferences. In this work, we propose a robust and efficient reward alignment method based on a novel and geometrically interpretable perspective: hypothesis space batched cutting. Our method iteratively refines the reward hypoth- esis space through "cuts" based on batches of human preferences. Within each batch, human preferences, queried based on disagreement, are grouped using a voting function to determine the appropriate cut, ensuring a bounded human query complexity. To handle unknown erroneous prefer- ences, introduce a conservative cutting method within each batch, preventing erroneous human preferences from making overly aggressive cuts to the hypothesis space. This guarantees prov- able robustness against false preferences, while eliminating the need to explicitly identify them. We evaluate our method in model predictive control setting across diverse tasks. The results demonstrate that our framework achieves compa- rable or superior performance to state-of-the-art methods in error-free settings while significantly outperforming existing methods when handling a high percentage of erroneous human preferences. Paper website and code can be accessed via link.
Reward alignment automates reward design by enabling an agent to learn its reward function directly from intuitive human feedback (Casper et al., 2023). In preference bassdd feedback, human ranks a pair of the agent's motion trajec- tories, and then the agent infers a reward function that best aligns with human judgment. Numerous alignment methods have been proposed (Christiano et al., 2017; Ibarz al., 2018; Hejna III & Sadigh, 2023), and most of them rely on the Bradley-Tere Terry model (Bradley & Terry, 1952), which formalizes the rationality of human preferences. While effective, these models are vulnerable to false human pref- erences, where rankings may be irrational or inconsistent due to human errors, mistakes, malicious interference, or difficulty distinguishing between equally undesirable tra- jectories. When a significant portion of human feedback is erroneous, the learned reward function degrades substan- tially, leading to poor agent behavior (Lee et al., 2021a).
In this paper, we propose a robust reward alignment frame- work based on a novel and interpretable perspective: hy- pothesis space batch cutting. We maintain hypothesis space of reward models during learning, where each batch of human preferences introduces nonlinear cut, removing portions of the hypothesis space. Within each batch, human preferences, queried based on disagreement over the current hypothesis space, are grouped using a proposed voting func- tion to determine the appropriate cut. This process ensures certifiable upper bound on the total number of human preferences required. To tackle false human preferences, we introduce conservative cutting method within each batch. This prevents false human preferences from making overly aggressive cuts to the hypothesis space and ensures provable robustness, while eliminating the need to explicitly identify the false preferences. Extensive experiments demonstrate that our framework achieves comparable or superior per- formance sfate-of-the thr-----ttttttt methods in error-free settings
1. Introduction
Reinforcement learning and optimal control have shown great success in generating intricate behavior of an agen- v/rooot, such as dexterous manipulation (Qi al., 2023; Handa et al., 2023; Yin et al., 2023) and agile locomotion
'Equal contribution AAiizna State University, Tempe AZ 85283, United States. 'Shanghai Jiao Tong University, Shang- hai, China. Correspondence to: Zhixian Xie <zxxie@ edu>.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
while significantly outperforming existing approaches when handling high rates of erroneous human preferences.
where the quality of human feedback is achieved using ac- curacy thresholds and manual verification. The method may be difficult to use in practice due to the absence of ground truth and the high cost of manual inspection. (Heo et al., 2025) proposes a robust learning approach based on the mixup technique (Zhang et al., 2018), which aug- ments preference data by generating linear combinations of preferences. While this approach eliminates the need for additional training, false human preferences could propa- gate through the augmentation process, ultimately degrading learning performance.
2. Related Works
2.1. Preference-Batee Based Reward Learning
Learning rewards from preferences for reinforcement learn- ing agents, also named as preference- reinforcement learning (PirL) et al., 2017), was early studied in (Zucker et al., 2010; Akrour et al., 2012; 2014). Early work focused on learning weights for weighted-sumt feature rep- resentations. With neural network representations, scalable PbRL is developed in (Christiano et al., 2017), recently used for fine-tuning large language models (Bakker et al., 2022: Achiam et al., 2023). In those works, human preference is modelled using the Bradley-Terry formulation (Bradley & Terry, 1952). Numerous variants of PbRL have later been developed (Hejna III & Sadigh, 2023; Hejna et al., 2024; Myers et al., 2022; 2023). Recently, progress has been made to improve human data complexity. PEBBLE (Lee et al., 202 used unsupervised pretraining and expe- rience re-labelling to achieve better human query efficiency. SURF (Park et al., 2022) applied data augmentation and semi-supervised learning get a more diverse preference dataset. RUNE (Liang et al., 2022) encourages agent explo- ration with an intrinsic reward measuring the uncertainty of reward prediction. MRN (Liu et al., 2022) jointly optimizes the reward network and the pair of Q/policy networks in a bi-level framework. Despite the above progress, most methods are vulnerable to erroncous human preferences. As shown in (Lee et al., 2021a), PEBBLE suffers from a 20% downgrade when there exists a 10% error rate in preference data. In real-world applications of PbRL, real human users tend to make mistakes when providing feedback, due to mistakes, malicious interference, or difficulty distinguishing between equally undesirable motions.
The proposed method differs from existing methods in three key aspects: (1) it requires no prior distribution assumptions for correct or false human preferences, (2) it avoids addi- tional classification training to assess feedback quality and filter false preferences, (3) it directly updates the hypothesis space conservatively based on entire preference batches.
2.3. Active Learning in Hypothesis Space
Active learning based on Bayesian approaches has been extensively studied (Daniel et al., 2014; Biyik & Sadigh, 2018; Sadigh et al., 2017; Biyak et al., 2024; Houlsby et al., 2011; Biyik et al., 2024), and many of these methods can be interpreted through the lens of hypothesis space removal. In partieular, the works of (Sadigh et al., 2017; Biyik & Sadigh, 2018; Biyik al., 2024) are closely related to ours. They iteratively update reward hypothesis space using a Bayesian approach through (batches of) active preference queries. However, their methods are limited to reward func- tions that are linear combinations of preset features. More recently, (Jin et al., 2022; Xie et al., 2024) have explored learning reward or constraint functions via hypothesis space cutting, where the hypothesis space is represented as con- vex polytope, and human feedback introduces linear cuts to this space. Still, these approaches are restricted to linear parameterizations of rewards/constrainss
2.2. Robust Learning from Noisy Labels
In addition to their limitations to weight-featur parametric rewards, the aforementioned methods do not address robust- ness in the presence of erroneous human preference data. In this paper, we show that the absence of robust design makes these methods highly vulnerable to false preference data.
To tackle erroneous human preferences, recent PbRL meth- ods draw on the methods in robust deep learning (Yao et al., 2018; Lee et al., 2019; Lukasik et al., 2020; Zhang et al., 2018; Amid et al., 2019; Ma et al., 2020; Jiang et al., 2018; Zhou et al., 2020). For example, (Xue et al., 2024) uses an encoder-decoder structure within the reward network to mitigate the impact of preference outliers. RIME (Cheng et al., 2024) and CANDERE COACH (Li et al., 2024) han- dle noisy human feedback labels by fitering-RIne RIME uses KL-divergence to filter and flip corrupted labels, while CANDERE-COACH trains a neural classifier to predict false preferences. These methods rely on prior knowledge or assumptions of true human preference distribution for false label identification. Uni- Uni-RLHF (Yuan et al.) intro- duces an annotation platform with large-scale feedback dataset for reinforcement learning with human feedback,
3. Problem Formulation
We formulate a Markov Decision Process (MDP) with a parameterized reward as AS, Paya: po), in which S is the state space, A is the action space, ro S x A x Q - R is a reward function parameterized by 0 E θ, Payn : S x A + S is the dynamics and po is the initial state distribution. The parameterization of re can be a neural network.
We call the entire parametric space of reward functions, θ R", the reward hypothesis space. Given any specific reward 0 e θ the agent plans its action sequence {ao.r-1}
2

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
with starting state 8y maximizes the cumulated reward
is the reward gap between two paired trajectories. The term 2y is the sign determined by human label y. For true human preference (5" <'', " it is easy to verify that (2) will lead to following constraints on OH
Here,
T-1 1=0
Jo (ao:T-1) = gg--(x (80.
re( ar)
(1)
over time horizon T. Here, the expectation is with respect to initial state distribution, probabilistic dynamics, etc. The action and the rollout state sequence forms an agent trajec- tory& & = {No, ao...., Sr). J. With a slight abuse of notation, we denote below the reward of trajectory& & also as Jo (E)-
0μ E (0 E @lf((. .'. y 2 0}
(5)
This means that a true human preference will result in constraint (or a cut) in (5) on the hypothesis space θ and the true human reward 0 satisfies such constraint.
Suppose a human user's preference of agent behavior cor- responds an implicit reward function in the hypothesis space 0H E O The agent does not know θh. but can query for human feedback over a trajectory pair (t)) Human returns a preference label TTT for (5", base on the rationality assumption (Shivaswamy & Joachims, 2015;
With the above premises, we present the overview of the method of Hypothesis Space Batch Cutting (HSBC) below.
HSBC Algorithm
Christiano et al., 2017; Jin et al., 2022):
With initial hypothesis space θ₂ C O perform the fol- lowing three steps at iteration i = 0, 2,
J0 (5") S (E') otherwise
y Tee
0
Step 1:
[Hypothesis sampling] Sample an ensemble & of reward parameters Os from the latest hypothesis space O for trajectory generation.
We call (5", g² " a true human preference. Oftentimes, the human can give false preference due to mistakes, errors, intentional attacks, or misjudgment of two trajectories that look very similar, this leads to a false human preference, defined as (E", E² fabe), where
Step 2: [Disagreement-baseee human query] Use the ensemble & to generate N trajectory pairs based on disagreement for active human query and obtain batch of human preferences B, ((E),.-El.) (t,))t,)),+'
Jo Jo (5") S Jau (5t)
false
(3)
Step 3: [Hypothesis space cutting] Calculate the con- straint set (batch cuts) C,:
otherwise
Problem statement:
We consider human preferences are queried and received incrementally, (E.. Ek yr), k 1, ..., where yk = yk or yv false and k is the query index. We seek to answer two questions:
C. = [0) E 5
M1.)) 2 0, j= 1, 1,..., N}. (6)
and update the hypothesis space by
0 = θ ne,,
(7)
Q1: With all true human preferences, i.e., V/= vt= for = 2... how to develop query-efficient learning process, such that the agent can quickly learn 0H with a certifiable upper bound for human query count K?
fia 0
+
θ
Q2: When unknown false human preferences (3) exist, i.c., yk vf false 3k 1.,, how establish provably robust learning process, such that the agent can still learn 0,r regardless of false preferences?
fu11
f12 = 0
4. Method of Hypothesis Space Batch Cutting
Figure 1: Illustration of update from O to O Three constraints are induced from a preference batch of size 3, with the simplified notation (O) s(0,<), <2,, El,y W..)). ). Red arrows are the di- rections of constraints, i.c., the region of (0) 2 0}. New O++1 is the regions in θ satisfying all constraints.
In this section, we present an overview of our proposed Hy- pothesis Space Batch Cutting (HSBC) method and address the question of Q1. The proofs for all lemmas and theorems can be found in Appendix D and E.
As shown in Fig. 1, the intuitive idea of the HSBC algorithm is to maintain and update hypothesis space θ, starting from an initial set Op based on batches of human prefer- ences. Each batch of human preferences (Step 2) is actively queried. The human preference batch is turned into a batch cut (Step 3), removing a portion of the current hypothesis space, leading to the next hypothesis space.
4.1. Overview
We start with first showing how a human preference leads to a cut to the hypothesis space. Given a true or false human preference (E",E'. y). we can always define the function
5(0,5", 5', .v) = (1 (Jo(E)))) - Jo) (4)
3

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
From (7) in (Step 3), it follows O 2 , 02 2.... .... This means that the hypothesis space is not increasing. More importantly, we have the following assertion, saying the human reward is always contained in the current hypothesis all human preferences are true in a sense of (2).
Specifically, collect human preference B,, we only query
human using trajectory pairs (S... Ei.g i Vi.j) that satisfy
3 00,002 E G [0,, 1.00,,5,, fh. E:. Bir) 2
and f(02, Eh C vo.)) S 0, Vyn. (12)
Lemma 4.1. If all human preferences are true, i.e., ys.j = fiee Ji j. and OH E Oo then following HSBC Algorithm, one has 0H for all i I..
Intuitively, the disagreement basee query can ensure at least some portion of the hypothesis space is removed by the constraint ʃ(0, S.. E..j (i.j) 2 0, no matter what preference label yi.j human will provide. A geometric illustration is given in Fig. 2, with simplified notation fi,(0) == f(0, f(0.="
4.2. Voting Function for Human Preference Batch In the HSBC algorithm, directly handling batch cut C, and hypothesis space + can be computationally difficult, es- pecially for high-dimensional parameter space, e.g., neural networks. Thus, we next introduce the concept of "voting function" on the hypothesis space.
Vt.))- The above disagreement- based query strategies are also related to prior work (Chris- tiano al., 2017; Lee et al., 2021b).
Specifically, we define a voting function for a batch of hu- man preferences B;- = {(El.) (ig.Bi.)) as follows
fu 2 0
θ 00
i
Vi(0)= = Σ j=1 (U((,,5,, M.g))
(8)
0
++1
0₂0
01
i+1
fu 2 0
where H. is the Heaviside step function defined as H(x) =1 otherwise. Intuitively, any point o E θ in the hypothesis space will get "+1" vote if it satisfies f(0.2 St Vi.5) 2 0 and "0" vote otherwise. Therefore, (0) represents the total votes of any point in the hypothesis space after the ith batch of human preferences.
if x 2 0 and Hxx)
Figure 2: Illustration of disagroemennt-ascee preference query. Regardless of the human actual preference label (which only deter- mines which side is to cut), the disagreement condition will always guarantee at least a portion of the hypothesis space is removed.
With the disagreement-basee preference query in each batch, the following assertion provides an upper bound on the human query complexity of the HSBC algorithm given all true human preferences.
Thus, 0 E C, in (6) if and only if (0) = NN because As need to satisfy all N constraints f(0, E.,j: Et.j.B..) 2 for 1,2, ..., N. As the number of votes in (8) only takes integers, the batch cut C, can be equivalently written as:
Theorem 4.2. In the HSBC algorithm, suppose all true human preferences. Let P(g", x''" y be an unknown distribution of true human preferences. Define err(re) = (f E" , .y"" < 0}, which the probability that re makes different predictions than human. For any e, 5,
C = (0 (0) 2 N - 0.5).
Furthermore, the indicator of batch cut C, can be written as:
1e. (0) H(V N + 0.5).
(10)
P(err(re) 16)
(13)
Following (7) in the HSBC, since θ Oo n ni- 1-1 can define the indicator function of θ as:
1 Cc. we
holds with human query complexity:
1 K =O((dlog) + log (log 16)) log ?) (14)
1 (0) = Le.
II R Le. (0),
i21 (11)
with de,(0 be the indicator function of initial hypothesis space On- With the indicator representation of O in (11), we will show f can be sampled from θ (Step 1) using (11).
where N and I are the batch size and the iteration number, is the disagreement coefficient related to the learning prob- lem defined in Appendix E.2.3, and d is the VC-dimension (Vapnik, 1998) of the reward model defined in Appendix E. I.
4.3. Disagreement-Basee Query and its Complexity In the HSBC algorithm, not all batch cuts have similar effec- tiveness, i.e., some batch cut C, can be redundant without removing any volume of hypothesis space, leading to an unnecessarily human query. To achieve effective cutting, each trajectory pair (E%,)= Sl.g Mi.z) should attain certain dis- agreement on the current hypothesis space O,, before sent to the human for preference query.
The above theorem shows that the proposed HSBC algo- rithm can achieve Probably Approximately Correct (PAC) learning of the human reward. The learned reward function re can make preference prediction with an arbitrarily low error rate e under an arbitrary high probability - 8, under the human preference data complexity in (14). In practi- cal scenarios, the reward function ro often represented
4

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
by deep neural network. While Theorem 4.2 is stated in terms of VC-dimension, the measure is applicable to many neural network classes. For instance, in multilayer percep- trons (MLPs) with ReLU activations and bounded weights, the VC-dimension is finite and scales with the number of parameters and layers (Bartlett et al., 2019).
The following lemma states that with the above modification of C, the HSBC algorithm can still achieve the provable robustness against false human preference.
Lemma 5.2. With the conservativeness level and replac- ing (9) with C2 in (16), the HSBC Algoritium will have 0h E C, and 0H E O,, i=1,2,3...,, regardless of false human preference.
5. Robust Alignment
In this section, we extend the HSBC algorithm to handle false human preference data. We will establish a provably robust learning process, such that the agent can still Iearn Hr regardless of unknown false human preferences.
The lemma means by simply replacing (9) with (16) while keeping other settings unchanged, the HSBC algorithm robust against human false labels. (9) is special case of (16) since when & 0, [(1 2) [N] = N. Geometric interpretation of Lemma 5.2 is given below.
First, for false human preference (5",5', y it easy to verify that (3) will lead to
5.2. Geometric Interpretation
On & (0 E 00 (0, , ,/ (ub))) 2 0}
(15)
fLa
0
3
This means that a false human preference will mistakenly exclude the true 0n- In the HSBC algorithm, suppose batch Bn, of human preference includes unknown false hu- man preferences, i.c., 3j- = 1, 2..,
O
O
e 1
θ
fu1
fu
fua
f,a3 = 0
.... NN Dmj = f We
have the following lemma stating the failure of the method.
Figure 3: Geometry Interpretation of the robust HSBC with false preferences. Red arrows are the direction of the constraints, i... the 0 region satisfying (0) 2 D3. Left: y is false human preference, and thus simply taking the intersection will cut out O Ou- Right: in robust HSBC, the regions with V:(0)= 2 are kept, thus 0 is still contained in the new hypothesis space O.+1-
Lemma 5.1. Following the HSBC Algorithm, if there exists one false preference in m-th batch Bm, of human preferences, then HH &f Cm and for all i > m, 1 & O
Geometrically, Lemma 5.1 shows that any false human pref- erence in preference batch Bm will make the batch cut C. mistakenly "cut out" 0n, i.e., On E Cm. In the following, we show that the HSBC Algorithm will become provably robust by a slight modification of (6) or (9).
Fig. shows a 2D geometric interpretation of the robust hypothesis cutting from + O++1 in the presence of false human preferences in batch B,. Here, the batch size is N = 3, and only the third preference (42.a, Sl,, vf.t")) is false. We set 7 = 1/3. As shown in Fig. 3, since (E%.3 E1,3 false) is a false preference, will induce a con- straint fr. == f(0, S%.3. (10.5%,33.
5.1. Cutting with Conservativeness Level y
Our idea is to adopt worst-case, conservative cut to handle unknown false human preferences. Specifically, let conser- vativeness level y E [0, 1] denote the ratio of the maximum number of false human preferences in a batch of N human preferences. In other words, with 7, we suppose a prefer- ence batch has at most [yN] ([-1 false preference. 7 a worst-case or conservative estimate of the false preference ratio and is not necessarily equal to the real error rate, which is unknown. In practice, " can be treated as hyperparameter to reflect varying levels of human irrationality, or it can be adapted online based on run-time estimation of human behavior. In this work, we use as a constant hyperparameter for simplicity.
y..lee) 0 which 0,, does not satisfy. As result, simply taking the intersection of all constraints (0) (0) 2 0.j = 3, will rule out 0μ by mistake. However, in this case, one can still perform a correct hypothesis space cutting (containing 0h) using (17). Replacing N = 3 and 7 1/3 in (17), the indicator func- tion for the batched cut is cc, (0) ()) = HV) 1.5). This means by keeping the region in + with voting value Vi(0) to be 2 or 3, 0H E O In other words, by setting a mild voting threshold [[1 >)NN - 0.5, a broader hypothesis space containing 0μ can be preserved.
is the round-up operator)
6. Detailed Implementation
With the conservativenes: level 7, one can change (9) into
By viewing the case of all true human preferences special case with conservativeness level y = 0, we present the implementation of the HSBC algorithm in Algorithm 1. The details are presented below.
C (0) [(1 - 7)NJ 0.5),
(16)
where I-] is the round-down operator, and the indicator function of the batch cut C, thus can be expressed as
6.1. Hypothesis Space Sampling
The step of hypothesis space sampling is to sample an en- semble of M parameters, &, = (0)) M from the current
1 (0) = HVV - [(1 - N] + 0.5).
(17)
5

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
Algorithm I Implementation for HSBC Algorithm
6.3. Reward Ensemble Trajectory Generation
Batch size N. ensemble size M, conservativeness level Y. disagreement threshold 17, segment count per trajectory Z.
Input:
With the ensemble ε, sampled from θ; the reward used for planning and control is defined by taking the mean value of predictions from the reward ensemble, i.e.
for 0. do
Sample an assemble &, from current hypothesis space (if
0, randomly initialize Eg) with the method in Section
Tg, (8, a. = TeeEE,Te(s a)).
(21)
6.1. Filter and Densify & with the method in Section 6.2; //Hypothesis Sampling Based on ε., generate trajectory using sampling- -based MPC (Section 6.3). Select trajectory pairs based on disagreement score, and acquire a preference batch B, (Section 6.3);
In our implementation, dictive control (MPC) in the MJX simulator (Todorov et al., 2012) for trajectory generation. Half of the action plans gen- erated from the MPC are perturbed with a decaying noise for exploration in the initial stage of learning. It is possible to use reinforcement learning for trajectory generation.
sampling- sampling-based model pre-
I/Disarreement -assd human query Update the sampling objective function with (19) and (20) using the batch B,; Illypothesis space Cutting
6.4. Disagreement Scoring and Query
end for return E,
In our algorithm, generated trajectories are stored in buffer. When new trajectory is generated, we segment it into Z segments, and mix them with old segments for randomly picking segment pairs (5") . for disagreement test.
hypothesis space +, We let the initial hypothesis space O, R", that is, 1e, = 1, which guarantees possible H is contained in Og- In our implementation, we use neural networks rewards, thus Eg is randomly initialized with common distributions for neural network initialization.
For any segment pair (5"; {²),, we measure their disagree- ment on the current hypothesis space using the reward en- semble & We define the following disagreement score:
For > O, sampling & can be reformulated as an optimiza- tion with the indicator function (11), i.e.,
DISE, (5". .e)) "
(22)
where ng, is the number of Os in & such that Jo(E") > Jo ne N N n &, i.c., 0 E & makes one-sided judgment on (t))
If disagreement exist in
& C arg max 1e. (0) = arg max k=0 le. (0). (18)
0
DIS;, (5", =+-)== = O. When the disagreement is evenly split, the score 1. We use a disagreement threshold vj E (0, 1) to select valid disagreement pairs for human queries. A pair (E" { is rejected if DIS;, (E". E²) S 17; otherwise, the human is queried for a preference label y for (E". Trajectory generation and disagreementtbaseee query are performed iteratively until B, contains N pairs.
To use gradient-based optimizer, (18) needs to be differen- tiable. Recall in (17) the only non-differentiable operation is H. Thus, we use the smooth sigmoid function sig,()) with temperature to approximate H. In fact, other smooth functions with similar shape can be used. From (17), Lc. will be replaced by the smooth approximation:
EE,- .
1 c; siis
siga (f), -v1 (1))))))))) (19)
7. Experiments
We test our method on 6 different tasks, as shown in Fig. 4:
where a, B, y are the tunable hyperparameters, Therefore, sampling E, can be performed through
& C arg max II
1ca. (0).
(20)
To get diverse 0 samples in E,, we optimize (20) in parallel with diverse initial guesses. For > the optimization is warm-started using the previous E initial values.
6.2. Sample Filtering and Densification
As & are solutions to a smoothed optimization (20), there is no guarantee that the samples are all in θ, Thus, we use the original indicator function e (0) to filter the collected samples: remove Os that are not satisfying (e, Since E, after filtering may not have M samples, we propose a densification step to duplicate some Os (with also adding some noise) in & to maintain the same sample size M.
Figure 4: Task environments of experiments.
3 dm-control tasks: including Cartpole-Swingun Walker- Walk, Humanoid-Standup Standupp We directly use Cartpole, Walker and Humanoid to denote the tasks.
2 in-hand dexterous manipulation tasks: An Allegro hand in-hand reorients a cube and bunny object to any given
6

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
Ground Reward (Oracle)
Baseline
Ours
160
150
140
140
140
140
Cartpole
120
Raward
100
100
Beward
100
100
10
40
40
40
100
150
200
300
0
50
100
200
250
100
500
9
100
200
300
400
500
Query
Query
Qvery
500
500
500
400
400
Walker
Roward
100
100 200 103
400 500 650 730 B00
100
200
300
400 500 800 702 800
200
400
000 Query
500
1000
1200
200
400
000 Query
600
1000
1200
Query
Querry
225
225
225
225
200
230
250
Humanoid
Reward
150
75
25
200
600 Query
800
1000
200
400
600
800
1000
300
600
300 Query
1200
1500
300
600
900
1200
1500
Query
Query
-0.1
-0.1
-6.1
-0.3
Dexman- Dexman-Cube Cube
-0.2
-0.2
-0.2
-0.2
Reward
-0.3
-0.3
Deward
-0.3
-0.3
-0.
-0.4
-0.5
-0.7
-0.70
-6.7
Query
Quary
Dexman- Bunny
-0.2
Rovard
Roward
-0.4
-0.4
-0.4
-0.4
-0.3
-0.5
-0.5
-0.5
-0.8
-0.6
-6.6
-0.6
50
100
150 Query
200
250
300
100
150
200
250
200
50
100
150
250 Query
200
300 150 420
105
150
200 Query
250
300
250 400
Query
-20
-20
Go2-Standap
-40
-62
-20
-20
Roward
-00
-92
-10
-120
-120
-140
-100
140
140
-160
-180g
-10%
110g
-150
Query
Query
(a) False Rate
0
(b) False Rate 10%
(c) False Rate 20%
(d) False Rate 30%
Figure 5: Learning curves for different tasks (rows) under different (columns) false human preferences. The conservativeness level HSBC Algorithm is the same as actual human false rate. All results are reported over 5 runs. The results show our method significantly outperforms the bascline when the false rate is high, and has comparable performance when the false rate is 0 (no false preference).
500
500
500
500
402
300
Sare
300
200
200
200
200
10%
6.75
100
20%
102
1000
Number
(a) Ablation of
(b) Ablation of r/
(c) Ablation
(d) Ablation M
Figure 6: Ablation study in Walker Walk task for the choices of (a): conservativeness level (b) disagreement threshold (c): batch size
N. (d): ensemble size N. The learning curves are reported across 5 individual runs.

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
Table Results on Tasks over 5 runs
FALSE RATE 0 %
RATE 10%
FALSE RATE 20S.
FALSE RATE
TASKS
ORACLE
BASELINE
OURS
BASELINE
OURS
BASELINE
OURS
BASBLINE
OURS
CARIFOLE
148.6
143,3 18.3
137.8 -74
101.4
1.9.1
137,0 1K.
52.3 17688
130.7 124
42,8 1210
111.31183
WALKER
472.9
444.4
459.4
421.2
441.4
401.8
447.1
277.0
417.2
HUMANOID
210.1
107.0
1788
134.7 1
1758
148.0 1119
84.2 1152
117.9 1194
60.3 11300
1.7.0
DUXMAN-CURE
-0.139
-0.281
10.000
-0.207
10.043
-0.291
1oamz
-0.222 10005
-0.360 10.15:
-0.238 _00,00
-0.342 1i002
-0.227 10.ss
DEXMAN- BUNNY
-0.135
-0.352
29,090
+0.220
00.055
-0.356
20.204
-0.278
-0.404 +0.200
.0.236 =0,525
-0.505 #:0026
-0.263 +:0050
G02-STANDUP
-30.6
-62.1
-36.7
-101.2
1180
-85.5
-80.5
target. The two objects demand different reward designs due policy. We use Dexman- -Cbbe
show the human query complexity, we set the x-axis the
to the and Dexman- Dexman-Bunny to denote two tasks.
number of queries, and the y-axis is the performance of & evaluated by ground-truth reward. Quantitatively, Table I shows the evaluation reward results of all tasks.
Quadruped robot stand-up task: A Go2 quadruped robot learns a reward to stand up with two back feet and raise the two front feet. We use Go2-Standup to denote the task.
From both Fig. 5 and Table I, we observe a comparable per- formance of our proposed method and the baseline for zero false human preferences. Both methods successfully learn reward functions complete different tasks. As the rate of false preference increases, from 10% to 30%, we observe significant performance drop of the baseline method. In contrast, the drop of the proposed method over high rate of false human preference is not significant. These results demonstrate the robustness of the proposed method for re- ward learning against a high rate of false human preferences.
Detailed settings are in Appendix B.3. Similar MJPC (Howell et al., 2022) but to better support GPU-accelerated sampling-based MPC, we re-implemented all environments in MJX (Todorov et al., 2012) with MPPI (Williams et al., 2015) as the MPC policy. Note that our method implemen- tation is also applicable to other simulated environments, either by accepting slower policy inference in the absence of GPU parallelization, or by adapting alternative policy classes (e.g.. RL-based policies) to generate trajectories.
7.2. Ablation Study
Similar to (Lee et al., 2021b; Cheng et al., 2024), a "simulated human" is imple- mented as a computer program to provide preference for the reward learning. The simulated human uses the ground- truth reward re, (see the ground truth reward of each task in Appendix B) to rank trajectory pairs. To simulate different rates of false human preferences, in each batch, random selection of human preference labels is flipped.
Simulated (false) human preference:
Conservativeness level 7 Given fixed rate of false hu- man preference, We evaluate how different settings of con- servativeness levels 7 = 0, 10%, 20%, 30% affect the per- formance of the proposed method. Here, the actual human preferences have a fixed false rate of 20%. The ablation is tested for the Walker task, and the results over 5 runs are reported in Fig. 6(a). The results show that when y = O (i.e., the algorithm aggressively cuts the hypothesis space without conservatism), the learning performance drops sig- nificantly. When 7 = 30% exceeds the actual false rate (20%) but remains within reasonable range. it has little impact on learning performance. However, we postulate that, in general, a higher y increases query complexity, as less of the hypothesis space is cut per iteration.
Baseline: Throughout the experiment, we compare our method against a baseline implemented using the reward learning approach from PEBBLE (details in Appendix A).
HSBC Algorithm settings: In all tasks, rewards are MLP models. Unless specially mentioned, batch size N = 10, ensemble size M = 16, and disagreement threshold 17 = 0.75. In dm-controv tasks and Go2-Standup trajectory pairs of the first few batches are generated by a random policy for better exploration. Refer to Appendix C for other settings.
Disagreement threshold 1) We evaluate the performance of our algorithm under different disagreement thresholds 1 given all true human preferences. The test is in the Walker task. We set 17 to 0, 0.5. 0.75 and show corresponding learn ing curves in Fig. 6(b). The results show that disagreement- based query can accelerate the learning convergence.
7.1. Results
For each task, we test our method and the baseline each for independent runs with different random seeds. During the learning process, the reward ensemble & at the checkpoint iteration i is saved for evaluation. The evaluation involves using the saved reward ensemble & to generate new tra- jectory, for which the performance is calculated using the ground-truth reward. Each evaluation is performed in 5 runs with different random seeds. Evaluation performance at different learning checkpoints will yield the learning curve. We show the learning curves in Fig. 5. The mean and stan- dard deviation across different runs are reported. To better
Batch size N In Fig. 6(c), we show the reward learning performance for the Walker task with different choices of preference batch sizes, N 5, 10, 15, 20, under fixed rate 20% of false human preference. The results indicate a simi- lar performance, suggesting that the learning performance is not sensitive to the setting of preference batch sizes.
Ensemble size A In Fig. 6(d), we present the reward learning performance on the Walker task with varying en- semble sizes, M = 4, 8, 16, 32, under a fixed 20% rate of false human preferences and 1) = 0.75. The results show
8

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
Table Comparison with multiple other methods under 20% and 30% false preference
TASk
ORACLE
OueS
PEBBLE
RIME
SURF
MAE
5-CE
CARIPOLE-200
130.7 +2.0
+26.
98.0 +35.8
22599
CARTPOLE: 30%
148.6
111.3 166
42.8 +23.3
31,0=37.0
62.3 +42.0
59.9 -30.7
52.0 +30.5
WALKER-20K
472.9
447.0 144
401,9=57.6
408.4 #24.8
397.2 =70.
425.5 #30.2
410.8 =19.9
WALKER-30%
472.9
417.2
277.0
310.2 +4.00
292.0
288.3
345.6
that increasing the ensemble size can slightly improve the performance, but not significantly.
and thus "Stoc" teacher tends to provide very noisy labels, which hinders the convergence of the algorithm.
7.3. Comparison with Other Methods
Table 4: Evaluation under different B-Pref teacher models.
We compare our HSBC algorithm against two state-of-the- art preference-based reward learning methods, SURF (Park et al., 2022) and RIME (Cheng et al., 2024), the PEBBLE baseline, and two other robust learning methods, MAE (Ghosh et al., 2017) and t-CE (Feng et al., 2021). All meth- ods are evaluated on Cartpole and Walker tasks, with false preference rate of 20% and 30%, respectively. The com- parison results are given in Table 2. Our HSBC method outperforms all of other methods in robust reward learning under high error rates. Among all those comparing methods, RIME excels at handling false preference labels with its label denoising design. In the Walker task, using t-CE loss also achieves robust results. The detailed experiment set- tings and the corresponding learning curves for each method are provided in the Appendix F.
TEACHER
ORACLE
STOC
MISTAKE
MYOPIC
REWARD
148.6
93.6 1,900
122.6 1146
127.0 1:2.5
7.6. Evaluation on Real Human Data
We finally evaluate our HSBC algorithm using real human feedback on the CartPole and Walker tasks, with four hu- man volunteers. We report the performance of the learned rewards across participants. To efficiently bootstrap learn- ing, human feedback was collected after a small amount of simulated feedback -00 for CartPole and 100 for Walker. Each volunteer provided 50 preferences for CartPole and 100 for Walker. Due to the inherent noise of real human preferences, 7 was set to 40%. HSBC was compared against the PEBBLE baseline under identical conditions. The learn ing curves are presented in Fig. 7. The results demonstrate that HSBC achieves more stable convergence and superior performance when learning from real human feedback.
7.4. Statistical Analysis of Learned Rewards
To evaluate the consistency between the learned and ground- truth rewards, we compute the Pearson correlation coeffi- cient in three dm-control tasks under different false prefer- ence rates. For each task, we generate five 200-step trajecto- ries and report the mean and standard deviation of the corre- lation values. The results, shown in Table 3, indicate strong correlations, suggesting that the learned rewards align well with ground-truth for high rates false human preference.
150
500
140
Gute
PEIALE
400
Data
120
Human Start
Humun Statt
Reward
100
Rewaard
200
100
0
20
40
60
80
100
25
50
150 175 200 Total Query
100 125
Toral Query
Table 3: Correlation between learned and ground-truth rewards.
(a) Cartpole
(b) Walker
TAsK
FALSE RATE 0%
10%
20%
30%
Figure 7: Reward learning with real human preference. HSBC
CARTPOLE
WALKER
0,928 #0.025
135
0,858 =0.051
0.636
0.914 #0022
0.851 +00002
4:0000
outperforms baseline in reward performance and learning stability.
HOMANOID
0.657 1. aM
0.546 1n
0.500 in
8. Conclusion
7.5. Different False Preference Types
This paper proposes Hypothesis Space Batch Cutting (HSBC), method for robust reward alignment from hu- man preference that may include unknown false feedback. HSBC selects batches of human preference data based on disagreement and uses them to iteratively refine the hypoth- esis space of reward functions. Each preference batch leads to a voting function over the hypothesis space, which is then used to eliminate portions of the hypothesis space. To guard against false preference labels, conservative cutting strategy is proposed to avoid overly aggressive hypothesis cutting. HSBC is geometrically interpretable and certifiable on human query complexity. Across diverse control tasks, HSBC matches the and outperforms them under high false preference rates.
We further test our method under different types of irrational human preference labels (teachers) defined in B- B-Pref (Lee et al., 2021a), including "Stoc," "Mistake," and "Myopic" teachers (note we exclude "Equal" teacher, as ties are not considered in our formulation). The valuations are con- ducted on the Cartpole task with B-Pref hyperparameters set as B = 10.0 for Stoc. e 0.2 for Mistake, and y = 0.98 for Myopic. The conservativeness level of HSBC is fixed at 20%, and all other settings follow the original B-Pref setup. The results in Table 4 show our method handles "Mistake" and "Myopic" teachers well, but struggles with "Stoc" teach- ers. The reason could be that in the late stage of learning, the trajectories in pair are close enough in reward values,
methods with clean data

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
Impact Statement
learning of reward functions. In Conference on robot
learning, pp. 519-528. PMLR, 2018.
The HSBC method improves both interpretability and ro- bustness in reward alignment by offering a geometric per- spective on hypothesis space updates and certifiable bound on human query complexity. Its conservative cutting strat- egy enables reliable learning even under a high rate of false preference labels, without requiring explicit identification of those labels. As a result, HSBC is resilient to noisy or adversarial human feedback and remains simple to imple- ment, requiring no additional classification modules. These strengths make HSBC particularly well-suited for applica- tions in robotics, autonomous systems, and human-in -intiniiith------ th-th-- loop decision decision-making, where false inconsistent data are common and accurate alignment with human intent is criti- cal. By enhancing robustness while preserving interpretabil- ity, HSBC provides a principled framework for preference- based learning in uncertain real-world environments.
Biyik, E., Anari, N., and Sadigh, D. Batch active learn- ing of reward functions from human preferences. ACM Transactions on Human-Robe Interaction, 13(2):1-27, 2024.
Bιyık, E., Huynh, N., Kochenderfer, M. J., and Sadigh, D. Active preference- gaussian process regression for reward learning and optimization. The International Journal of Robotics Research, 43(5):665-6844 2024.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., Wang, T. T., Marks, S., Segerie, C.-R., Car- roll, M., Peng, A., Christoffersen, P. J., Damani, M., Slocum, S., Anwar, U., Siththaranjan, A., Nadeau, M., Michaud, E. J., Pfau, J., Krasheninnikov, D., Chen, X., Langosco, L., Hase, P., Biyik, E., Dragan, A., Krueger, D., Sadigh, D., and and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learn- ing Research, 2023. ISSN 2835-8856. URL https: openreview net / forum? vey Certification, Featured Certification.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I.,
Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., al. Gpt-4 technical report. arXiv preprint arXIv:2303.08874, 2023.
-eeeell,d-eeel,,,, D. Open problems
Akrour, R., Schoenauer, M., and Sebag, M. April: Active preference learning-based reinforcement learning. In Ma- chine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23, pp. 116- 131. Springer, 2012.
Sur-
Cheng, J., Xiong, G., Dai, X., Miao, Q., Lv, Y., and Wang, F.- Y. Rime: Robust preference- reinforcement learning with noisy preferences. In International Conference on Machine Learning, pp. 8229-8247, PMLR, 2024.
Akrour, R., Schoenauer, M., Sebag, M., and Souplet, J.-C. Programming by feedback. In International Conference Machine Learning, volume 32, pp. 1503-1511 JML. JMLR. org, 2014.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances neural information pro- cessing systems, 30, 2017.
Amid, E., Warmuth, M. K., Anil, R., and Koren, T. Robust bi-tempered logistic loss based on bregman divergences. Advances in Neural Information Processing Systems, 32, 2019.
Daniel, C., Viering, M., Metz, J., Kroemer, O., and Peters, J. Active reward learning. In Robotics: Science and systems, volume 98, 2014.
Bakker, M., Chadwick, M., Sheahan, H., Tessler, M., Campbell- Gailiigggil L., Balaguer, J., McAleese, N., Glaese, A., Aslanides, J., Botvinick, M., et al. Fine-tuning language models find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 35:38174 15:38176-333 18881998818, 2022.
Feng, L., Shu, S., Lin, Z., Lv, Li, L., and An, B. Can cross entropy loss be robust to label noise? In Pro- ceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence, pp. 2206-2215 2021.
Bartlett, P. L., Harvey, N., Liaw. C., and Mehrabian, A. Nearly-tight ve-dimension and pseudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research, 20(63):1-17, 2019.
Ghosh, A., Kumar, H., and Sastry, P. S. Robust loss func- tions under label noise for deep neural networks. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Biyik, E. and Sadigh, D. Batch active preference-basel
10

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
Handa, A., Allshire, A., Makoviychuk, V., Petrenko, A., Singh, R., Liu, J., Makoviichuk, D., Van Wyk, K., Zhurke- vich, A., Sundaralingam, B., et al. Dextreme: Transfer of agile in- an-hand manipulation from simulation to reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5977-5984, IEEE, 2023.
Langley, Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference Machine Learning (ICML 2000). pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Lee. K., Yun, S., Lee, K., Lee, H., Li, B., and Shin, J. Robust inference via generative classifiers for handling noisy labels. In International conference machine learning, pp. 3763-3772 PMLR, 2019.
Hanneke, S. A bound on the label complexity of agnostic active learning. In Proceedings of the 24th international conference on Machine learning, pp. 353-360, 2007.
Lee, K., Smith, L., Dragan, A., and Abbeel, P. B-pref: Benchmarking preference-based reinforcement learning. In 35th Conference Neural Information Processing Systems (NeurlPS). Neural Information Processing Sys- tems Foundation, 2021a.
Hejna, J., Rafailov, R., Sikchi, H., Finn, C., Niekum, S., Knox, W., and Sadigh, D. Contrastive preference learn- ing: Learning from human feedback without rl. In Inter- national Conference on Learning Representations, 2024.
Hejna III, D. J. and Sadigh, D. Few-shot preference learn- ing for human-in-the-logg rl. In Conference Robot Learning, pp. 2014-2025. PMLR, 2023.
Lee, K., Smith, M., and Abbeel, Pebble: Feedback- efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th Interna- tional Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6152 6163. PMLR, 18-24 24 Jul 2021b.
Heo, J., Lee, J., Kim, J., Kwak, M. G., Park, Y. J., and Kim, S. B. Mixing corrupted preferences for robust and feedhack-efficiet preference- reinforcement learn- ing. Knowledge-Band Based Systems, 309:1 12824, 2025.
Li, Y., Das, S., and Taylor, M. E. Candere-coach: Rein- forcement learning from noisy feedback. arXiv preprint arXiv:2409. 15521, 2024.
Hoeller, D., Rudin, N., Sako, D., and Hutter, M. Any- mal parkour: Learning agile navigation for quadrupedal robots. Science Robotics, 9(8))eea)77566, 2024.
Liang, X., Shu, K., Lee, K., and Abbeel, P. Reward un- certainty for exploration in preference-based reinforce- ment learning. In International Conference on Learning Representationss, 2022. URL https: /ppereviie.. net www.ooormmiiwwwwwooo 112rC.
Houlsby, N., Huszár, F., Ghahramani, Z., and Lengyel, M. Bayesian active learning for classification and preference learning. stat, 1050:24, 2011.
Howell, T., Gileadi, N., Tunyasuvunakool, S., Zakka, K., Erez, T., and Tassa, Y. Predictive Sampling: Real-time Behaviour Synthesis with MuJoCo. dec 2022. doi: 10.48550kaxX1.2211..0555 URL https://arre
Liu, R., Bai, F,, Du, Y., and Yang, Y. Meta-reward-nett Implicitly differentiable reward learning for preference- based reinforcement learning. Advances in Neural Infor- mation Processing Systems, 35:22270-22284 2022.
org/ arg/abs// 2212. 00541.
Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Lukasik, M., Bhojanapalli, S., Menon, A., and Kumar, S. Does label smoothing mitigate label noise? In Interna- tional Conference on Machine Learning, pp. 6448-6458. PMLR, 2020.
Jiang, L., Zhou, Z., Leung. T., Li, L.-J,, and Fei-Fei, L. Men- tornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning, pp. 2304-2313. PMLR, 2018.
Ma, X., Huang. H., Wang. Y., Romano, S., Erfani, S., and Bailey, J. Normalized loss functions for deep learning with noisy labels. In International conference on machine learning, pp. 6543-6553, PMLR, 2020.
Jin, W., Murphey, T. D., Lu, 7., and Mou, S. Learning from human directional corrections. IEEE Transactions on Robotics, 39(1):625 644, 2022.
Myers, V., Biyik, E., Anari, N., and Sadigh, D. Learning multimodal rewards from rankings. In Faust, A., Hsu, D., and Neumann, G. (eds.), Proceedings of the 5th Confer- ence on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 342-352. PMLR, 08-11 Nov 2022.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint orXiv:1412..60 /4/2. 2014.

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
Myers, V., Bryik, E., and Sadigh, D. Active reward learning from online preferences. In 2023 IEEE International Con- ference Robotics and Automation (ICRA), pp. 7511- 7518, 2023. doi: 10.
Xie, Z., Zhang, W., Ren, Y., Wang, Z.., Pappas, G. J., and Jin, W. Safe mpe alignment with human directional feedback. arXiv preprint vrXiv:2407.04216 2024.
Xue, W., An, B., Yan, S., and Xu, Reinforcement learning from diverse human preferences. In Proceedings of the Thirty-Third International Joint Conference Artificial Intelligence, UJCA 24, 2024. ISBN 978- 178-1-956792-44 1. doi: 10.249 10.24963/ijcci. 2024/586. URL ht /ttps:///o.... /doi. org / 10. 24 /t//://jjaaa http:///iiaaaaa222/52222 2024/586.
Park, J., Sco, Y., Shin, J., Lee, H., Abbeel, P., and Lee,
K. Surf: semi-super reward learning with data augmentation for feedback-efficiet preference-based re- inforcement learning, arXiv preprint arXiv:2203.10055, 2022.
Yao, J., Wang, J., Tsang. I. W. Zhang, Y., Sun, J., Zhang, C., and Zhang, R. Deep learning from noisy image labels with quality embedding. IEEE Transactions on Image Processing, 28(4): 1999-9922,2000
Qi. H., Kumar, A., Calandra, R., Ma, Y., and Malik, In-hand object rotation via rapid motor adaptation. In Conference on Robot Learning, pp. 1722-1732. PMLR, 2023.
Yin, Z.-H., Huang, B., Qin, Y., Chen, Q., and Wang, X. Ro- tating without seeing: Towards in-hand dexterity through touch. Robotics:
Radosavovic, I., Xiao, T., Zhang, B., Darrell, T., Malik, J., and Sreenath, K. Real-world humanoid locomotion with reinforcement learning. Science Robotics, 989)): eadi9579, 2024.
Science and Systems, 2023.
Yuan, Y., Jianye, H., Ma, Y., Dong. Z., Liang, H., Liu. J., Feng, Z., Zhao, K., and Zheng, Y. Uni-rlhf: Universal platform and benchmark suite for reinforcement learning with diverse human feedback. In The Twelfth Interna- tional Conference on Learning Representations.
Sadigh, D., Dragan, A. D.. Sastry, S. S., and Se- shia, S. A. Active preference-base learning of re- ward functions. In Robotics: Science and Systems, 2017. URL. https: sepp..ssattiiscooooo...ooo org//oozpuID::222555
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz. D. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. URL https: /ttp//ppeeeeeeeeeeeeeeeeeeeeeeee net forum?
Settles, B. Active Learning. Morgan & Claypool Publishers, 2012. ISBN 1608457257.
i id=r1Ddp1-Rb.
Shivaswamy, P. and Joachims, T. Coactive learning. Journal of Artificial Intelligence Research, 53:1-40, 2015.
Zhou, T., Wang, S., and Bilmes, J. Robust curriculum learning: from clean label detection to noisy label self- conrection. In International Conference on Learning Representations, 2020.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model- based control. In 2012 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems, pp. 5026-5033 IEEE, 2012. doi: 10. N0.1109/RRS.2012. 6386109.
Zucker, M., Bagnell, J. A., Atkeson, C. G., and Kuffner, J. An optimization approach to rough terrain locomotion. In 2010 IEEE International Conference on Robotics and Automation, pp. 3589-3595. IEFE, 2010.
Tsounis, V., Alge, M., Lee, J., Farshidian, F., and Hutter, M. Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning. IEEE Robotics and Automation Letters, 5(2):3699-3706, 2020.
Vapnik, V. Statistical learning theory. John Wiley & Sons google schola, 2:831-842, 1998.
Williams, G., Aldrich, A., and Theodorou, E. Model pre- dictive path integral control using covariance variable importance sampling, arXiv preprini arXiv:1509.0114 2015.
Wirth, C., Akrour, R., Neumann, G., and Fürnkranz, J. A survey of preference- based reinforcement learning meth- ods. Journal of Machine Learning Research, 18(136):
1 46, 2017.
12

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
A. Baseline Method for Comparison
We present the details of the baseline method used for comparison. It is inspired by the method PEBBLE (Lee et al., 2021b) on training the reward model, while uses the same setting to plan the trajectories and collect the preferences. The only differences between the baseline and our method are the ensemble size and the objective function. An ensemble of 3 reward models is maintained. After batches of preferences, the reward models tuned to optimize the Bradley- Terry loss function of all previous preferences:
C
EE J=
-P (Ek. > Ek,)) ) + u (E > Ek.
(23)
where
JD
PBT > E-)= Chase Ja (5") + exp Jo
(24)
The parameter Chase the temperature of the Bradley- Terrly model. In the dexterous manipulation tasks, Olbase 0.5. In all other tasks, Otbsse = 3. In the experiments of cartpole swingup, Otbase 10. The preferences are selected with disagreement threshold 17 0.8, which is similar our approach.
B. Detailed Description of the Tasks
B.1. DM-Control Tasks
We re-implemented three dm-control tasks in MJX to perform sampling-based MPC, similar to (Howell et al., 2022).
Cartpole We directly use the environment xml file dm-control, with the same state and action definition in the original library. The goal swing up the pole to an upright position and balance the system in the middle of the trail. Use yo, e to denote the angle and angular velocity of the pole, x; å to denote the horizontal position and velocity of the cart. The system is initialized with and velocities. (ær, sin yp, cos sp, èt .6 and is 1D control scalar signal for the horizontal force applied on the cart. We use a ground truth reward function:
and ip = along with normal noise with standard deviation 0.01 imposed on all joint positions
r(s, a) upright(s) * middle(s) smallettr((a) small_el(ss)
(25)
where upright(s) (cos y + 1)/2, middle(s) exp(-x²), small (4 exp((Aa²))/5, small_eel(s) + exp((0..223))2
Walker We directly use the environment xml file in dm-control. The goal is to make the agent walk forward with a steady speed. The system is initialized at standing position and normal noise with standard deviation 0.01 is imposed on all joint positions and velocities. Use z to denote the sum of 2 coordinate of the torso and bias 1.2, thus the value : = O when the walker stands up-straight. by to denote the torso orientation in xz at=::zz-aeee plaaee q to denote all joints between links and
to denote the torso linear velocity on x x axiis. (2, Pp 4, â., é, Sy. ④) and [0, 1]6 stands for torques applied on all joints. We use ground truth reward function:
3 standing(s) s) + upright(s)
r(s, a)
move(s)
(26)
where standing(s) clip(1 |z), 1), upright(s) = (cos yy + 1)/2, move(s) (1; 0,
Humanoid We directly use the environment XMI. file in dm-control. The goal is to stand up from a lying position. The system is initialized at a lying position and uniform noise between -0.01 and 0.01 is imposed on all joint positions and velocities. Use 2, quat to denote the z-coordinate and quaternion orientation of the humanoid torso. Use (Va, Uyy Wz], U2], w- [wtz wy, wz] to denote 3D linear and angular velocity of the torso. Use to denote all of the joints in the humanoid agent. s = Iz, guat, w, i] and a E [0, 1127 stands for torques applied on all joints. We use ground truth reward function:
r(s, a) = standing(a) (a) small.vels))
(27)
where standing(s) clip(=/1.2,0,) 1) and small.ve((s) = exp(-0.1 * Voz + 0z - 0.3 [w]l)
13

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
B.2. In-hand Dexterous Manipulation
We mainly focus on Allegro robot in-hand re-orientation of two different objects, cube and bunny. An allegro hand and an object are simulated in the environment. The allegro hand is tilted with quaternion [0.0, 0.82, 0.0, 0.57]. The objects have an initial position of [0.0, 0.0, 0.05) with quaternion [0, O, O, 1]. At the beginning of every trajectory planning during training. target orientation in the form of an axis-angle is randomly selected. The axis is uniformly randomly selected from 2, U. 2 axis and the angle is uniformly randomly selected from the set { + 2 +
2x T,; J.. This angle-axis rotation is then converted to the target quaternion Qoorg. The goal is to use the fingers of the robotic hand to rotate the object to the target orientation, as well as remain the object in the center of the hand. Use to denote the joint positions of the robot hand. p to denote the object position, Aquat to denote the quaternion different between current object pose and target pose (Aquat 1 (g (guurr9targ)) 9targ) and f stip: m. mftiip: I'tpp lheip to denote the positions of four fingertips (forefinger, middle finger, ring finger and thumb). The object and fingertip positions are multiplied by 10 to balance the scale. [g. p, Aquat, f ftip. m ftip. T. fesps thes E [0, 1] is the delta-pon celta-position command on all joints and is multiplied by 0.2 to generate actual control signals. The ground truth reward is designed to be:
r(8,0) a) - -cost_quat(s) 0osssss_pos(s) - 0.05cost_contacl/ (8)
(28)
Where cost.quut((s) = Aqual, cost_pos(s) = 1p - (0.0, 0.0, 0.01) * 10112 and cost_contact(ss) |pp - mupll' + 1pp-11 rfssppl + |lp theipll 11?
|lp
I seepll
+
In evaluation, 6 trajectories are planned by setting different target of = rotation on æ, y, a axis. The mean reward per trajectory per time step is calculated and recorded.
B.3. Go2-Standup
In this task, a go2 quadruped robot is simulated and the goal is to stand up on two back feet to keep the torso height on 0.6, and raise two front feet to the height of 1.2. The initial pose and the target pose are shown in Fig. 8. The system initialized a standing position on four legs, and uniform noise between -0.01 and 0.01 imposed on all joint positions and velocities. Use 2t, qual to denote the c-corrdinate and quaternion orientation of the robot torso. Use zff; Pry to denote 2 --cooddinates of two front feet and two rear feet. To balance the scale, all features related to the z-coordinates multiplied by 10. Use v = [izz Vy: U2), w [M2, Wy: wa] to denote 3D linear and angular velocity of the torso. Use q to denote all of the joint positions. 8 = [Z+, quol, 4, 2/ Erfs 0, w. q) and a E [0, 1]12 stands for the desired delta-position on all joints. We define the ground truth reward function as:
r(s, a) 2.5heightt 0.3feet.cost($
10 1006 "ang Lang.velcoos($$ 10 ve022velooss(s
(29)
where height costtts = (z 0.6)2, feet cost (s) =11=5s 1.211 2 + || 11 ang vel cost(ss) = || and vel costosss ||u||
(a) Initial Position
(b) Stand-up up Position
Figure 8: Illustration the Go2-Standup task, the goal is to reach a stand-up posture with two feet.
C. Model and Parameters in Experiment
Reward Neural Networks For all tasks, we use an MLP with hidden layers of 256 hidden units to parameterize reward functions. The activation function of the network is ReLU, In the dm-control tasks, the input is a concatenated vector of state & and action a. In dexterous manipulation and quadruped locomotion tasks, the input is purely the state & to perform
14

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
Table Learning Parameter Different Tasks
TASK
a
8
Z
Teval
Tseg
CARTPOLE
10.0
3.0
50/80
100
200
50
WALKER
5.0
3.0
2
80/120
150
500
50
HUMANOID
5.0
3.0
3
100/150
200
300
50
DEXMAN-
5.0
3.0
3
30/40
100
150
20
DEXMAN-BUNNY
5.0
3.0
30/40
100
150
20
Go2-STANDUP
5.0
3.0
80/120
150
100
25
Table 6: MPPI Parameter in Different Tasks
TASK
Num
Hor
A
Std
CARTPOLE
256/512
20/25
0.01/0.01
1.0/0.75
WALKER
512/1024
25/30
0.01/0.01
1.0/0.75
HUMANOID
1024/1024
25/25
0.01/0.01
1.2/0.75
DEXMAN- CUBB
1024/1024
515
0.01/0.01
1.0/1.0
DEXMAN-BURE BUNNY
1024/1024
5/5
0.01/0.01
1.0/1.0
GO2-STANDUP
1024/1024
25/25
0.005/0.01
1.0/0.75
faster learning. The output scalar value of the reward is scaled to the interval [-1, 1] using the Tanh function. In particular, the dexterous manipulation tasks, an intrinsic reward ..Stoos..contct(( is imposed to the network output, forming a predicted reward to encourage grasping the object and bootstrapping the learning.
Learning Settings The parameters a, B, Z, I, trajectory length T, segment length Tseg and evaluation trajectory length Teeat slightly varies from the tasks, Table 5. In the Table, the two number of batch number / corresponds to the number with irrationality {0, 10%} and {20%, 30%}. The Adam (Kingma, 2014) optimizer parameter is identical for all dm-control and Go2-Standup tasks, the learning rate is set to be 0.005 with weight decay coefficient 0.001. In dexterous manipulation tasks, the learning rate is 0.002 with weight decay coefficient 0.001.
MPPI Parameter The number of samples (Num), planning horizon (Hor), temperature (A) and the standard deviation (Std) of the normal sampling distribution determine the quality of trajectories planned by MPPI. These parameters varies from different tasks, see Table In the table, two numbers on each entry denotes the different settings in the training and evaluation stages.
D. Proof of the Lemmas
D.1. Proof of Lemma 4.1
Proof. By the definition of the in (2),
f(0, E.,j E v.ge) 2 0
(30)
combining the definition C, in (6), 0H E C,, Vi. Hypothesis space θ, = 2 7C - - Combining 0H E Oo and 0u E C,2 Vi, 0H E O, Vi.
D.2. Proof of Lemma 5.1
false) has false label, Thus, Proof. Without loss of generality, suppose jth preference in mtth batch (5 e 3 vm
fm. (0H) < 0 Which means 0H & C.. With definition of C. and θ, for all m.
(31)
c, C O. C C.
(32)
that leads to AF & , for all m.
15

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
D.3. Proof of Lemma 5.2
Proof. With the definition of y, there are most [+>]] incorrect labels and at least L(1 - N)NN [ correct labels in one batch
B, For all HH sr fi,(00) 2 0 holds for all tuples with correct labels (El. E. y ). As result, in the voting process of (1(0z) shown in (8), there will be at least [(1 - (1- "+1" votes and ( [1--)) With the modified definition of C; in (16), 0µ E C; and we have 0h E θ
E. Proof of the Theorem 4.2
The proof of the theorem follows the pipeline in Chapter 6 in (Settles, 2012). order for the paper to be self-contained, we include the complete proof here.
E.1. Reward-Induced Classifier
The preference alignment can be viewed as a classification task. Use the new notation x (5", g²) denote trajectory pair. This pair is fed the classifier predict 0-1 label y of which trajectory is better in human's sense. For any parameterized reward model re, classifier he can be induced by defining:
0, Jo 2 Jo(E²)
(x:) =
(33)
1, Jo( < Jo(E
Recall that function g is used to calculate the average reward on any trajectory. All of the classifiers induced by T9 parameterized by 0 E Oo forms a concept class H = { E TR"). We assume the VC dimension of H is a finite number d. Specifically, there is a human classifier correspond re,, which defines the label y corresponds to a::
0, Jen (5") Jon (5t) 1, Je (5") J0n (E')
y y= h= (z)
(34)
Using Pxy to denote the data-label joint distribution and use Px to define the marginal distribution of for simplicity.
There exists one aligned classifier h" E H such that:
Yx,y hx)
(35)
Define the error rate of any h E H as:
ehr(h) = Px(h(x) + h- hx))))
(36)
it is equivalent to express the preference prediction error rate err(ro) P(f(0,E",E ,"' ,hr) < <0) in Theorem as err(ro) =
err(h
E.2. Definition of some Helper Functions, Sets and Coeflicients
In this section we establish some definitions of helper functions, sets or coefficients, which are used in the main proof of Theorem 4.2.
E. VERSION SPACE
Similar to the definition of H. the classifier set generated by every Q can be notes as V, {ha O2}, which is typically
called version space in the context of active learning. By the definition of O, for all h V,, h can make perfect classfication of all data (x. y) formed by the trajectory pairs and labels in all previous batches Bo B Also, since θ C Ot. h* E V; for all the time. Because the hypothesis space for parameters is shrinking, i.e., θ O the version space is also shrinking such that V/ 2 Vi+1-
E.2.2. DIFFERENCE BETWEEN CLASSIFIESS ON E AND CORRESPONDING BALLS
The data distribution provides a straightforward measure between two classifiers h1, h2 E H. by directly examining the proportion of data that they behave differently. A difference (hh h2) can be defined as the probability that two classifiers
16

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
behave differently, under the data distribution E:
( h2) =
+ h2(z))
(37)
With this different as a distance measure, an p ball with radius p and center he can be defined as:
B(he, p) (h E H| h) p}
(38)
E.2.3. REGION DISAGREEMENT AND DISAGREEMENT COFFFICIENT
For given version space and use X' denote the instance space of x, the region of disagreement is a subset of the instance space:
DIS(V) = {or E X13,, h2 e V hi()) # h2(x))
(39)
With the definition of the region of disagreement, a disagreement coefficient was proposed by (Hanneke, 2007) to characterize the difficulty of active learning, which defined with the expression
Px (DIS( p)))
= sup p>0
(40)
The value Ç is determined by H and E. The greater Ç is, the more the volume of the disagreement region of the p-ball around /- scales with which signifies greater learning difficulty.
E.3. Passive PAC Sample Complexity Bound
Since the training set from the distribution Exv is perfectly seperateble as in, using the famous probably approximately correct (PAC) bound for passive learning, with arbitary and 6, P(err(h) I c) 2 1 - 8 can be achieved by perfectly fit training data with size ASS!
LPASS S (a= + log 1 ) o
(41)
E.4. Formal Proof Theorem 4.2
Proof. For arbitary e and 6, define batch size to be:
N = [ee(d log $1
1
(42)
with c as a constant and a specially chosen (the definition of which is shown later in the proof). With probability - 8' and for all E Vi+1:
df (x)) E DIS(V)) S N (d
N + log 1 8
(43)
d
This is because data batch B, selected by disagreement (12). By the definition of V.-1. all of the data B, comes from the set DIS(V,) and of the classifiers V4+1 perfectly fits this batch. By switching the PAC bound LPASS in Appendix E.3 with M and solving for get the RHS of the inequality.
With proper choice of c'. log N < log and the RHS of (43):
N + log gi) (d d d
1 d s
(44)
o
2
As result, for all h E V4+1.
err(h) Px(((()) # #x)))))
(45)
=Px(((z) #1(x) x (x DIS(),))+
(46)
Px(h(z) (x))= Px DIS(V,))
(47)
17

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
By the property Vi+1 C Vg, Vi and combining with h² E,,, (x(((a))) + (x))a DIS(V)) 0 and
err(h) (x DIS(V)) Px(DIS(V))
(48)
2
2
This means V1+1 S B(h,, (x(DI)(A)) with the definition of the disagreement coefficient ç,
2(
(D((IS(V++)) () & PX(SS((B(),, (x((IS(V)) ))
Px Px(DIs(V))
(49)
2(
Thus, let Un- Px (D after logz (Uo/c) batches of data and with 8' =8/1, for all h E V,. The relationship
err(h) S U,2-'=c
(50)
holds with a union bound probability 1--18' = 1 ò. With the definition of V,, for O,, err(re) = err(hg E Vi) s e. Thus, the total number of the queries to perform PAC alignment can be expressed as:
log 1/e log -
K = TN = O(((dlog log
(51)
8
This is the bound shown in Theorem 4.2 and completes the proof.
F. Additional Experiments
F.. Comparison with Multiple Reward Learning Methods
The comparison includes two advanced preference- based reward learning methods, SURF (Park et al., 2022) and RIME (Cheng et al., 2024), as well as the PEBBLE (l.ee et al., 2021b) baseline with two robust reward functions, MAE (Ghosh al., 2017) and (-CE (Feng et al., 2021). For fairness in all comparisons (including the PEBBLE baseline in Section A), replace the original RL policies in all methods with MPPI-based planners. The reward learning components remain consistent with their original implementations to ensure fair evaluation of planning performance.
For the comparison of RIME (Cheng et al., 2024) and SURF (Park et al., 2022), we used the same settings the original PEBBLE baseline for collecting trajectory segments. In RIME, KL-divergence between predicted preference probabilities and labels filters untrustworthy labels and flips them for improved learning. We set RIME parameters to a 0.25, (n.00005) Ie 1 1/60 for the Cartpole-Swingun task, and a 0.3, Sunx 2.2, Bmin 1.7, Tupper In (0.0055), & 1/100 for the Walker- Walk task.
Bmax 3.0, Piin 1.0, Tupper
For SURF, changed the length of collected segments to 60 and used temporal data augmentation to crop segments to fixed length of 50. We choose 0.95, μ = 1.0, and À 1.0 in both tasks. All algorithm parameters are selected to ensure the best performance of the baseline methods.
In MAE (Ghosh et al., 2017), the original loss function in PEBBLE baseline replaced with LMAE = Elý - Pol for robust reward learning. In t-CE (Feng et al., 2021), the loss replaced with L.CC E Ei= 1-6 Pa. Here, the one-hot version of the noisy label, and Po is the predicted probability of human preference trajectory pairs. We choose t 4 in the (-CE loss based on its best performance.
The learning curves are shown in Fig. 9.
18

---

Robust Reward Alignment via Hypothesis Space Batch Cutting
160
160
140
140
120
Ours
120
Ours
PEBBLE
PEBBLE
Reward
100
Reward
100
RIMÉ
80
SURF
80
SURF
MAE
MAE
60
60
t-CE
t-CE
40
Oracle
40
Oracle
20
20
100
200
300
400
500
100
200
300
400
500
Query
Query
(a) Cartpole, 20% false rate
(b) Cartpole, 30% false rate
500
500
400
400
Reward
300
Reward
300
Ours PEBBLE
Ours
PEBBLE
200
RIME
200
RIME
SURF
SURF
MAE
MAE
100
100
t-CE
t-CE
Oracle
Oracle
0
200
400
600
800
1000 1200
200
400
600 Query
800
1000 1200
Query
(c) Walker, 20% false rate
(d) Walker, 30% false rate
Figure 9: Learning curves of the methods used for comparison.
19


Reviews:
[{"review_id": "wKe7BoYq4k", "reviewer": "Reviewer_e9yZ", "summary": "The paper introduces Hypothesis Space Batch Cutting (HSBC), a framework for robust reward alignment in reinforcement learning (RL). HSBC addresses the challenge of learning reward functions from human preferences, particularly in the presence of false or noisy feedback. The core idea is to iteratively refine a hypothesis space of reward models by \"cutting\" regions inconsistent with human preferences. Batches of preferences are queried based on disagreement among current hypotheses, and a voting function aggregates these preferences to determine the cuts. To handle errors, a conservative cutting strategy ensures that up to $\\gamma$N false preferences per batch are tolerated. Theoretical guarantees include PAC learning bounds and robustness proofs. Empirical results demonstrate that HSBC outperforms representative methods like PEBBLE under high false preference rates (up to 30%) across diverse tasks (DM-Control, dexterous manipulation, quadruped locomotion).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer #e9yZ\n\nWe sincerely appreciate your thoughtful feedback and comments on our paper. Below, we address each of your concerns in detail. All our responses will be incorporated into the final paper.\n\n### 1. Comparisons with other robust PbRL methods?\nThank you for highlighting this point. Following the reviewer’ and other reviewers’ comments, in this rebuttal we have added the comparison with some other robust reward learning methods, including RIME[1], SURF[2], MAE[3] and t-CE[4]. The comparison is performed on the cartpole-swingup and walker-walk tasks, with 20% and 30% error rate.\n\nFor the comparison of RIME [1] and SURF [2], we used the same settings as the original PEBBLE baseline for collecting trajectory segments. In RIME, KL-divergence between predicted preference probabilities and labels filters untrustworthy labels and flips them for improved learning. We set RIME parameters to $\\alpha = 0.25$, $\\beta_{max} = 3.0$, $\\beta_{min} = 1.0$, $\\tau_{upper} = -\\ln(0.005)$, $k = 1/60$ for cartpole-swingup task and $\\alpha = 0.3$, $\\beta_{max} = 2.2$, $\\beta_{min} = 1.7$, $\\tau_{upper} = -\\ln(0.005)$, $k = 1/100$ for walker-walk task. For SURF, we changed the length of collected segments to 60 and use temporal data augmentation to crop segments of a fixed-length 50. We choose $\\tau = 0.95$, $\\mu = 1.0$ and $\\lambda=1.0$ for SURF in both tasks. All algorithm parameters are chosen for the best performance of the baseline methods.\n\nIn  MAE[3], the original loss function is replaced with $L_{MAE} = \\mathbb{E}|\\hat{y} - P_{\\theta}|$ for robust reward learning. In t-CE[4], the loss is replaced with $L_{t-CE} = \\mathbb{E}\\sum_{i=1}^t \\frac{(1-\\hat{y}^T P_{\\theta})}{i}$. Here, $\\hat{y}$ is the one-hot version of noisy label and $P_{\\theta}$ as the predicted probability of human preference on trajectory pairs. We choose $t=4$ in t-CE loss of its best performance.\n\nThe result (sum of reward) is shown in the table below:\n\n| Task|Oracle|Ours|PEBBLE|RIME|SURF|MAE|t-CE|\n|-|-|-|-|-|-|-|-|\n| Cartpole-Swingup-20% |  148.6  | $130.7 \\pm 2.0$  | $52.3 \\pm 26.8$  | $75.0 \\pm 45.3$  | $98.0 \\pm 35.8$  | $98.6 \\pm 25.9$  | $73.3 \\pm 16.3$  |\n| Cartpole-Swingup-30% |  148.6    | $111.3 \\pm 16.8$  | $42.8 \\pm 23.3$  | $81.0 \\pm 37.0$  | $62.3 \\pm 42.0$  | $59.9 \\pm 30.7$  | $52.0 \\pm 30.5$  |\n| Walker-Walk-20%    |  472.9    | $447.0 \\pm 14.4$  | $401.9 \\pm 37.6$  | $408.4 \\pm 24.8$  | $397.2 \\pm 30.7$  | $425.5 \\pm 30.2$  | $410.8 \\pm 19.9$  |\n| Walker-Walk-30%    |  472.9      | $417.2 \\pm 12.2$  | $277.0 \\pm 62.3$  | $310.2 \\pm 84.0$  | $292.0 \\pm 69.0$  | $288.3 \\pm 139.0$  | $345.6 \\pm 52.2$  |\n\nThe results show that the proposed HSBC method outperforms baselines in robust learning under high error rates. Among the baselines, RIME excels at handling false preference labels with its label denoising design. In the Walker task, using t-CE loss also achieves robust learning.\n\nWe will include the above baseline comparison in the revised version of the paper.\n\n\n### 2.  Validation in a practical setting?\nWe appreciate this observation. To address this, we performed HSBC on real-human feedback, please refer to the response of question 5 to reviewer #sofd.\n\n### 3.The theoretical bounds assume a finite VC-dimension, how does HSBC generalize to high-dimensional neural reward functions?\nWe thank the reviewer for the helpful comments. The VC-dimension is finite for certain neural network classes, such as multilayer perceptrons (MLPs) with ReLU activations, where it scales with the number of parameters and layers [5]. For such networks, the upper bound of the sample complexity applies. However, the exact VC-dimension of general neural networks, especially deep architectures with complex connectivity and unbounded weight norms, remains an open problem. In these cases, Theorem 4.2 provides a worst-case upper bound, which may be conservative. Tighter complexity bounds are an important direction for future work.\n\nWe appreciate your time and effort in reviewing our work. Your feedback has been invaluable in strengthening our manuscript.\n\n### Reference\n[1]Cheng, et al. \"Rime: Robust preference-based reinforcement learning with noisy preferences.\" arXiv preprint arXiv:2402.17257 (2024).\n\n[2]Park, et al. \"Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning.\" arXiv preprint arXiv:2203.10050 (2022).\n\n[3]Ghosh, , et al. \"Robust loss functions under label noise for deep neural networks.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 31. No. 1. 2017.\n\n[4]Feng, et al. \"Can cross entropy loss be robust to label noise?.\" Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence. 2021.\n\n[5]Bartlett, et al. \"Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks.\" Journal of Machine Learning Research 20.63 (2019): 1-17.", "claims_and_evidence": "The paper’s claims are generally supported by clear evidence.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are well-aligned with the problem:\n- Hypothesis Space Cutting: The geometric interpretation of hypothesis updates is intuitive and addresses the limitations of prior preference-based RL methods (e.g., PEBBLE’s vulnerability to noise).\n- Disagreement-Based Queries: Actively selecting trajectory pairs where current hypotheses disagree ensures efficient learning, as shown in the ablation study for $\\eta$ (Figure 7b).\n- Benchmark Tasks: The use of DM-Control, dexterous manipulation, and locomotion tasks (e.g., Go2-Standup) covers a broad range of control challenges, validating HSBC’s generality.", "theoretical_claims": "Theoretical claims are rigorously presented.", "experimental_designs_or_analyses": "The experimental design is generally sound, but minor issues exist:\n- Baseline Comparison: PEBBLE is used as the primary baseline, but comparisons with other robust Preference-based Reinforcement Learning (PbRL) methods (e.g., RIME [Cheng et al., 2024] or Xue et al. [2023]) would strengthen the evaluation.", "relation_to_broader_scientific_literature": "The paper situates HSBC within the PbRL literature, contrasting with prior work on hypothesis space reduction (e.g., Sadigh et al. [2017]) and robust learning (e.g., Heo et al. [2025]). Key contributions include:\n- Conservative Voting: A novel approach to handle false preferences without prior distribution assumptions, unlike methods like mixup [Heo et al., 2025].", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**\n- The combination of batch cutting, voting functions, and conservatism provides a unique approach to robust PbRL.\n- HSBC’s robustness is critical for real-world applications (e.g., human-robot interaction) where errors are inevitable.\n\n**Weaknesses**\n- All experiments are simulation-based. Validation in a practical setting would enhance credibility.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Scalability to Neural Networks: Given that the theoretical bounds assume a finite VC-dimension, how does HSBC generalize to high-dimensional neural reward functions used in experiments? Could the bounds be adapted for neural models, or is this a limitation of the current framework?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pQdAtNu9OP", "reviewer": "Reviewer_sofd", "summary": "This study addresses the challenge of reward design in reinforcement learning and proposes a robust and efficient preference-based reward alignment method, particularly for noisy human feedback. The method introduces a novel framework called \"hypothesis space batched cutting,\" which iteratively refines the reward hypothesis space using batches of human preferences. The method employs a conservative cutting mechanism to ensure robustness when facing incorrect preferences. In error-free settings, the framework performs comparably to PEBBLE, while significantly outperforming other methods under high error rates in preferences.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer #sofd\n\nWe sincerely appreciate your thoughtful feedback and constructive comments on our paper. Below, we address each of your concerns in detail. All our responses will be incorporated into the final paper.\n\n### 1. Does gamma really play an adaptive role?\nThank you for the comments. Currently, $\\gamma$ is fixed as a (conservative) estimate of the batch error rate, but our method could be extended to adaptively adjust $\\gamma$ if real-time error estimation is available, which we leave for future work.\n\n### 2.  It would be better to compare it with more powerful baselines \nThank you for the comments. In this rebuttal, following the reviewer’s and all other reviewers’ comments, we have added the comparison with some other robust reward learning methods. Please refer to the comparison results in our response of question 1 to reviewer #e9yz.\n\n### 3. Why are the experiments conducted in a custom environment from dmc? ...\nThank you for the comments. We chose to implement our approach in new benchmark environments because we need a GPU-accelerated simulation to accelerate MPPI policies with parallel sampling. All of our custom testing environments will be released to support future research and standardized benchmarking in this area.\n\nRegarding practicality in other simulated environments, the proposed method is directly applicable. Environments without GPU support or parallel simulation may limit the speed of MPPI-based policies. In addition, as we pointed in Section 6.3, it is possible to use other policies (like RL) to generate trajectories for comparison.\n\nAs for real-world transferability, recent works such as [5] have demonstrated that sampling-based control policies with parallelized simulation backends like MJX can be effectively transferred to real-world robotic tasks.\n\n### 4. Lack of discussion of other RL methods with noisy human feedback in related work and experiments.\n\nThank you for the comments. Both RIME [1] and CANDERE-COACH [2] improve learning from noisy labels by filtering human feedback—RIME uses KL-divergence to filter and flip corrupted labels, while CANDERE-COACH trains a neural classifier to predict preferences and filters based on discrepancies with real labels. Xue’s work [3] employs an encoder-decoder structure for reward models, estimating confidence levels from latent distributions and performing a weighted average for better predictions. Unlike these methods, ours does not explicitly assess feedback quality but instead updates the hypothesis space conservatively based on entire preference batches. Uni-RLHF [4] introduces an annotation platform and large-scale feedback dataset, using accuracy thresholds and manual verification, which are impractical for online reward learning due to the absence of ground truth and the high cost of manual inspection.\n\nIn the revised version, we will include the above discussion.\n\n### 5. Have you tried using real human feedback? \nThank you for the comments. We conducted a new experiment evaluating HSBC with real human feedback on CartPole and Walker. Four volunteers provided trajectory preferences, which were used for reward learning. HSBC ($\\gamma=0.4$) was compared to PEBBLE under the same settings as in the main paper, with a small amount of simulated feedback for pretraining. The results (sum of reward), presented below, show performance across different numbers of human preferences, with the first table for CartPole and the second for Walker.\n|# human preferences|0|10|20|30|40|50|Oracle|\n|-|-|-|-|-|-|-|-|\n|Ours (reward)|$45.1 \\pm 44.7$|$80.5 \\pm 50.8$|$132.8 \\pm 19.8$|$147.4 \\pm 11.1$|$132.5 \\pm 32.8$|$133.2 \\pm 31.2$|$148.6$|\n|PEBBLE (reward)|$43.3 \\pm 20.2$|$49.9 \\pm 8.3$|$49.4 \\pm 14.6$|$44.5 \\pm 17.3$|$63.2 \\pm 27.3$|$73.5 \\pm 40.2$|$148.6$|\n\n|# human preferences|0|20|40|60|80|100|Oracle|\n|-|-|-|-|-|-|-|-|\n|Ours (reward)|$181.7 \\pm 92.3$|$309.0 \\pm 33.4$|$301.9 \\pm 38.4$|$310.3 \\pm 41.7$|$331.4 \\pm 41.2$|$356.2.2 \\pm 66.0$|$472.9$|\n|PEBBLE (reward)|$131.3 \\pm 85.2$|$154.2 \\pm 78.9$|$175.5 \\pm 126.2$|$326.5 \\pm 15.2$|$200.1 \\pm 108.5$|$177.9 \\pm 123.6$|$472.9$|\n\nIt can be shown that our method can achieve more stable convergence and superior performance when handling human feedback.\n\n### Reference\n[1]Cheng, et al. \"Rime: Robust preference-based reinforcement learning with noisy preferences.\" arXiv preprint arXiv:2402.17257 (2024).\n\n[2]Li Y, et al. CANDERE-COACH: Reinforcement Learning from Noisy Feedback[J]. arXiv preprint arXiv:2409.15521, 2024.\n\n[3]Xue W, et al. Reinforcement learning from diverse human preferences[J]. arXiv preprint arXiv:2301.11774, 2023.\n\n[4]Yuan Y, et al. Uni-rlhf: Universal platform and benchmark suite for reinforcement learning with diverse human feedback[J]. arXiv preprint arXiv:2402.02423, 2024.\n\n[5]Li, et al. \"Drop: Dexterous reorientation via online planning.\" arXiv preprint arXiv:2409.14562 (2024).", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "see weakness", "essential_references_not_discussed": "see weakness", "other_strengths_and_weaknesses": "Advantages:  \n1. The paper is very clear, and the logic is easy to understand.  \n2. I can fully understand the motivation of this paper; performing effective RLHF learning on noisy data is very important issue.  \n3. The theoretical analysis is thorough, and the method makes sense. \n\nDisadvantages:\n\n1. Does gamma really play an adaptive role? \n2. Although the method is highly sophisticated, the experiments are the weak point of this paper. The paper only compares with the PEBBLE algorithm, which is a very basic baseline algorithm, while recently there have been more methods with better performance. Can HSBC work effectively across most algorithms? It would be better to compare it with more powerful baselines.\n3. Why are the experiments conducted in a custom environment from dmc? Most similar papers for online rlhf/pbrl use the B-Pref benchmark environment. Will you release these new environments as benchmarks?\n3.5. Similar to the above point, I observed that a possible reason might be that the environment requires large-scale parallelism to support MPPI. Would this affect the practicality in other environments or the transferability to real-world tasks?\n4. Another issue is the lack of discussion of other reinforcement learning methods with noisy human feedback in related work and experiments. Although this type of work is novel and important, there are still some papers that have conducted preliminary studies, such as: [1] Cheng J, Xiong G, Dai X, et al. RIME: Robust Preference-based Reinforcement Learning with Noisy Human Preferences[J].\n[2] Li Y, Das S, Taylor M E. CANDERE-COACH: Reinforcement Learning from Noisy Feedback[J]. arXiv preprint arXiv:2409.15521, 2024.\n[3] Xue W, An B, Yan S, et al. Reinforcement learning from diverse human preferences[J]. arXiv preprint arXiv:2301.11774, 2023.\n[4] Yuan Y, Hao J, Ma Y, et al. Uni-rlhf: Universal platform and benchmark suite for reinforcement learning with diverse human feedback[J]. arXiv preprint arXiv:2402.02423, 2024.\n5. Have you tried using real human feedback? Real human feedback should inherently contain a certain noise. You may refer to [4].", "other_comments_or_suggestions": "I think this is a good paper, but the experiments are somewhat weak, and it lacks discussion and comparison with relevant literature. If the author can provide reasonable rebuttals, I am willing to raise the score.", "questions_for_authors": "-----\n\nAfter rebuttal, I raised my score from 2 to 3.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TwiFOozVC1", "reviewer": "Reviewer_GLNZ", "summary": "The paper introduces a novel method called Hypothesis Space Batch Cutting (HSBC), which iteratively refines a space of potential reward functions by using batches of human preferences to make \"cuts\" based on a voting function. To handle potentially erroneous human feedback, HSBC employs a conservative cutting method within each batch, ensuring robustness against false preferences while still learning the desired reward function.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to reviewer #GLNZ\n\nWe sincerely thank you for your comments. Below, we address each of your comments in detail. All our responses will be incorporated into the final paper.\n### 1. There are multiple ways... What kind of errors can it handle more easily?\nThanks for your question. For different human error types in Bpref, our method cannot handle “Equal” preferences as we do not consider ties. However, our method can naturally handle “Skip” cases, as these can be simply excluded when constructing preference batches. We added evaluations on “Stoc,” “Mistake,” and “Myopic” teachers in the Cartpole task, following BPref hyperparameters: $\\beta = 10.0$ for \"Stoc,\" $\\epsilon = 0.2$ for \"Mistake,\" and $\\gamma = 0.98$ for \"Myopic\". The conservativeness level was set to 20%, with other settings following the paper. Results confirm our method’s robustness across these feedback types.\n|Teacher|Oracle|Ours|Stoc|Mistake|Myopic|\n|-|-|-|-|-|-|\n|Reward|$148.6$|$130.7 \\pm 2.0$|$93.6 \\pm 30.7$|$122.6 \\pm 14.6$|$127.0 \\pm 26.5$|\n\nThe results show our method handles “Mistake” and “Myopic” teachers well but struggles with “Stoc” teachers. This could be because in the late stage of learning, the trajectories are close enough in its rewards and “Stoc” teacher tends to provide very noisy labels, which hinders the convergence of the algorithm.\n### 2. How does voting affect the cuts?\nThanks for the question. We will use Figure 3 in the paper for illustration. The voting function $V_i (\\boldsymbol{\\theta})$ controls the aggressiveness of hypothesis space cuts. Retaining only the maximum-vote region $V_i (\\boldsymbol{\\theta})=N$ would remove all hypotheses disagreeing with any preference, assuming perfect feedback accuracy. However, human errors could wrongly eliminate the ground-truth reward (left panel of Figure 3). To address this, we set a mild vote threshold $V_i (\\boldsymbol{\\theta}) \\geq \\lfloor(1-\\gamma) N\\rfloor  -0.5$, preserving a broader hypothesis space containing $\\boldsymbol{\\theta}_H$ (Lemma 5.2), as shown in the right panel of Figure 3.\n### 3. How much does the learned reward deviate from ground truth?\nThanks for the question. To assess consistency, we compute the Pearson correlation between the learned and ground-truth rewards in Cartpole, Walker, and Humanoid. For each task, we generate five 200-step trajectories and report the mean and standard deviation. Results are below:\n|Task|false 0%|false 10%|false 20%|false 30% |\n|-|-|-|-|-|\n|Cartpole| $0.928 \\pm 0.025$  | $0.888 \\pm 0.031$  | $0.914 \\pm 0.022$  | $0.851 \\pm 0.062$ |\n| Walker| $0.584 \\pm 0.035$  | $0.636 \\pm 0.060$  | $0.598 \\pm 0.070$  | $0.430 \\pm 0.062$ |\n|Humanoid| $0.673 \\pm 0.070$  | $0.657 \\pm 0.066$  | $0.546 \\pm 0.134$  | $0.500 \\pm 0.079$  |\n\nThe above results show that the learned reward and groundtruth reward has good correlation. Also, this correlation is weaker when the error rate increases, which is consistent with the performance results.\n\n### 4. Comparison with SURF?\nThank you for the comments. In this rebuttal, following the reviewer’s and all other reviewers’ comments, we have added the comparison with some other robust reward learning methods, including SURF. Please refer to the response of question 1 to reviewer #e9yz.\n\n### 5. Is it necessary to use a sigmoid function in Eq 20...?\nNo, any smooth differentiable function with a similar shape to sigmoid can be used, for example, the tanh function.\n### 6. Ablation of the ensemble size M\nWe added an ablation experiment on ensemble size $M$ on the walker-walk task with 20% error rate. The final result is shown  below:\n|$M$|4|8|16|32|\n|-|-|-|-|-|\nReward|$433.9\\pm 12.17$|$446.7\\pm 14.3$|$447.0\\pm 14.4$|$450.2\\pm 9.22$\n\nWith the groundtruth result of $472.9$. It can be seen from the result that increasing the ensemble size $M$ can slightly improve the performance, but not signfiicantly.\n\n### 7. Better for clarity of the function f\nThanks for the comment. The function $f$ represents the signed reward gap between ${\\xi}^0$ and ${\\xi}^1$, with the sign determined by human label $y$. Because $y$ is the “index” of the preferred trajectory, we set the sign as $1-2y$ to ensure $+1$ when $y=0$ and $-1$ when $y = 1$.\n\nAs shown in Equ. (6), one human preference $({\\xi}^0_{i,j}, {\\xi}^1_{i,j}, y_{i,j})$, no matter true or false, imposes a constraint on hypothesis space using the inequality $f({\\theta}, {\\xi}^0_{i,j}, {\\xi}^1_{i,j}, y_{i,j}) \\geq 0$.\n\n### 8. Typo of Heaviside function\nThank the reviewer for point this out. This was a typo. The correct notation is $\\mathrm{\\mathbf{H}}(x) =1$ if $x \\ge 0$ and $\\mathrm{\\mathbf{H}}(x) = 0$ otherwise. We will fix this in the revised version.\n\n### Reference\n[1]Lee, Kimin, et al. “B-pref: Benchmarking preference-based reinforcement learning.” arXiv preprint arXiv:2111.03026 (2021).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "It would be more informative to test the robustness of the method against different types of human errors as per Bpref [1]\n\n1. B-pref: Benchmarking preference-based reinforcement learning.\" arXiv preprint arXiv:2111.03026 (2021).", "theoretical_claims": "I hace not checked the proofs", "experimental_designs_or_analyses": "Well presented with good ablations, but lacks comparisons with baselines.", "relation_to_broader_scientific_literature": "It could present an approach to improve the stability of preference based RL methods.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The method seems interesting and novel and seems to improve performance. However, the set of baselines used is small. It could benefit from comparing with more baselines.", "other_comments_or_suggestions": "1. The function f’s intuitive meaning and direct relationship to how a preference (true/false) constrains the hypothesis space could be emphasised better for clarity. Specifically, explaining why the $(1-2y)$ term effectively flips the sign of the difference in rewards based on the preference label would be helpful for a clearer understanding.\n\n2. Below Eq 8, the Heaviside function seems to be described incorrectly", "questions_for_authors": "1. The BPref paper cited in the paper shows there are multiple ways in which humans can be erroneous. How would HSBC deal with each of these individually? What kind of errors  can  it handle more easily?\n\n2. With regards to the geometric interpretation, how does voting affect the cuts?\n\n3. In practice, how much does the learned reward function (with HSBC) deviate from the ground truth?\n\n4. Wouldn’t SURF (Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning) also be a suitable baseline for comparison? \n\n5. Is it necessary to use a sigmoid function in Eq 20 or would any smooth differentiable function be sufficient?\n\n6. While the authors do a good job with the ablations, I wonder how the size of the ensemble M would affect the results.\nLine 329 (left) ‘to using’", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents Hypothesis Space Batch Cutting (HSBC), a method for robust reward alignment in reinforcement learning (RL), aiming to address the learning reward functions from human preferences in the presence of false or noisy feedback. The core idea is to iteratively refine a hypothesis space of reward models by "cutting" regions inconsistent with batches of human preferences, using a voting function and a conservative cutting method to handle potentially erroneous feedback. The method is theoretically grounded with PAC learning bounds and robustness proofs.

Reviewers generally appreciated the clarity of the paper, the motivation, and the theoretical underpinnings. The primary initial concern was the limited scope of experimental comparisons, which the authors addressed in their rebuttal by including additional state-of-the-art robust PbRL methods. Reviewers were also interested in the method's performance with real human feedback and under different types of errors, which the authors also explored in their responses. While some questions regarding the experimental setup and the use of parallelized environments were raised, the authors provided clarifications. 

One reviewer raised the score, while some reviewers were not responsive. The meta-review takes into the account the authors' responses to all the reviewers.
Decision: Accept (poster)