Relative Value Learning
Authors: Marc Höftmann
Jan Robine
Stefan Harmeling
Venue: ICLR 2026 Poster (2026)
Keywords: Relative Value Learning
On-Policy Actor-Critic
GAE
PPO
Primary area: reinforcement learning
TL;DR: Learn a pairwise, antisymmetric value-difference critic and reconstruct GAE from differences, giving a drop-in PPO critic that avoids gauge issues and performs competitively on Atari.

Abstract: In reinforcement learning (RL), critics traditionally learn absolute state values, estimating how good a particular situation is in isolation. Adding any constant to $V(s)$ leaves action preferences unchanged. Thus only value differences are relevant for decision making.
Motivated by this fact, we ask the question whether these differences can be learned directly. For this, we propose \emph{Relative Value Learning} (RV), a framework that considers antisymmetric value differences $\Delta(s_i, s_j) = V(s_i) - V(s_j)$. We define a new pairwise Bellman operator and prove it is a $\gamma$-contraction with a unique fixed point equal to the true value differences, derive well-posed $1$-step/$n$-step/$\lambda$-return targets and reconstruct generalized advantage estimation from pairwise differences to obtain an unbiased policy-gradient estimator (R-GAE). Besides rigorous theoretical contributions, we integrate RV with PPO and achieve competitive performance on the Atari benchmark (49 games, ALE) compared to standard PPO, indicating that relative value estimation is an effective alternative to absolute critics.

Published as a conference paper at ICLR 2026
RELATIVE VALUE LEARNING
Mare Höftmann, Jan Robine & Stefan Harmeling
Department of Computer Science, Technical University of Dortmund, Germany Lamarr Institute for Machine Learning and Artificial Intelligence {marc. hoeftmann, jan. robine, stefan, harmeling tu www.dddrttdddddo
ABSTRACT
In reinforcement learning, crities typically estimate absolute state values (s),, estimating how good a particular situation is in isolation. However, it turns out that only differences in value are relevant for control. Motivated by this, we propose Relative Value Learning (RV), framework that learns value differences directly via an antisymmetric function A (s;, sÿ) (s;) (s;)) We introduce a pairwise Bellman operator and prove is y-contraction with unique fixed point equal to the true value differences, derive well-posed -step, n-step and X-return targets and reconstruct generalized advantage estimation from pairwise differences obtain an unbiased we integrate RV with PPO and achieve competitive performance on the Atari benchmark (49 ALE games) compared to standard PPO, indicating that relative value estimation an effective alternative to absolute critics. Our code available at htttts::///ttttut com Hauf 3n/ / re ive va ue learning
gradient estimator (R-GAE). Beyond theoretical results,
MOTIVATION
In control, actions are chosen by comparisons, not by absolute magnitudes. What matters is how good one state (or action) is relative to another. Formally, for 7 V is uniquely determined by the Bellman equation, but shifting it by any constant leaves advantages and greedy choices unchanged. This gauge freedom makes the abso- lute scale behaviorally meaningless: only differ- ences matter. Advantages A (s,sa, a) or greedy action selection max, Q" a) imply that ab- solute scales are not behaviorally meaningful. Optimal control depends not on absolute mag- nitudes but exclusively
MAX(2, 4)
CARROT
relative differences. Despite this, standard value-base based RL trains a critic approximate V* (or Q") and treats dif- ferences as a derived quantity. This introduces unnecessary degrees of freedom (the unpinned offset), invites drift under reward shaping or base- line changes, and can be ill-posed in settings where only comparisons or implicit feedback are or human-in -ttte-in-the-
RELATIVE VALUE LEARNING
ABSOLUTE VALUE LEARNING
Figure 1: Relative Value vs. Absolute Value Learning.. RV (left) learns value differences be- tween states for decision making while AV (right) learns the value for each state in isolation and then decides for the best decision (e.g. by taking maximum in Q-learning).
given (e.g., preference-baseee loop RL.), precisely where the absolute scale is ambiguous while pairwise relations remain well- defined. A formulation that places relative infor- mation at the center would match the invariances of the decision making problem itself.
We therefore adopt the following viewpoint: make value differences the primary learning objective. Concretely, learn antisymmetric function Ae S S RR with Ag(st, 8,) Ls) s,) that approximates / (st) - V=(s)). Working directly with A eliminates the gauge degree freedom by construction and aligns the critic with the invariants already exploited by policy-gradient methods

---

value Published as a conference paper at ICLR 2026
through baselines. In addition, advantages can be reconstructed from pairwise differences without knowing the absolute value of any state, providing an unbiased volicy-gradien estimator (R-GAE). This perspective is not just aesthetic. It enables clean analytic foundation.
Our Contributions:
1. Pairwise Value Operator. and prove 7-contraction with a fixed point equal to true value differences.
We formalize Bellman operator antisymmetric functions
2. Value Targets. We derive I-step/ n-step X-return targets using only observable rewards and non terminal pairwise terms, ensuring well-posed bootstrapping targets.
3. Relative GAE (R-GAE). We show that GAE can be reconstructed from pairwise differ- ences that also results in an unbiased policy gradient estimator. In addition we derive the relationship between GAE and R-GAE and show that RV achieves competitive performance on Atari.
2 RELATED WORK
Classical value-based RL trains absolute state or action values using Bellman operators, e.g. TD(A),
DQN, Double DQN, Rainbow, and other actor critic variants (Sutton & Barto 2018 Mnih et al. 2013
Van Hasselt et al. 2016)] Wang et al. (20 Hessel et al. 2018)) Distributional critics (Bellemare et al.
2abee] et al. (2018) restructure the target but continue to operate in an absolute space. Although
adding constant to V or Q" leaves action preferences unchanged (Sutton & Barto, 2018). absolute critics still predict scalar on an arbitrary scale. By contrast, RV removes the offset degree of freedom
at the model level by learning antisymmetric value differences over state pairs, aligning the function class with the invariances of decision making. This invariance a special case of policy-invariant reward transformations (potential- based shaping) (Ng et al. 1999). Several methods estimate or emphasize ad advantages rather than values. Direct Advantage Estimation (DAE) Pan et al. (2022) A directly learns advantages A (s, a) to bypass value learning. Dueling networks decompose learning
Q" (s, a) = V (s) + (s,s, a) to improve sample efficiency and robustness Wang et al. 6b) Actor-critic variants such as A2C/A3C
al. (2016), JRRPOSSchllmn al. (2015a) and clipped PPO Schulman et al. (017)) then use baselines and advantage estimates within trust-region-style updates. In contrast, RV directly learns value differences (ss,,8) with an explicit pairwise Bellman operator to provide the actor relative advantages. RV's critic uses siamese difference head that enforces antisymmetry and zero self-difference by design. Earlier work in stochastic control has also advocated learning value differences rather than absolute values. [Brtsekas) ( (1997) introduces differential training of rollout policies, approximating cost-to-go differences with TD-style methods. In RL, pairwise objectives have appeared mainly in preference-based and human-in moman--n-thhe -ieeiiin-tthhooppppp RL (Christiano et al. 2017 Leike et al. 2018) or inverse RL, but not as a critic. Our work fills this gap.
3 RELATIVE VALUE LEARNING
We now present Relative Value Learning (RV), which learns an antisymmetric function over all state pairs with a neural network
Au S x S S
Aa(s;, 85) = -o
(1)
that tries approximate the value difference A (F 8,) V (s.) ) - (s)), under a fixed policy т. As a result, RV avoids exact value estimates during training and integrates naturally with on-policy actor-critic methods (e.g., PPO) by supplying relative advantages (R-GAE).
3.1 PRELIMINARIES
We consider a discounted Markov decision process (MDP) (S, A,r,, 7) with state space S, action space A, transition function P(s' s, a), bounded reward function r: S x A + [rmin Tmax]] and discount factor E [0, 1). Considering a stochastic policy πT aTaa s), the value function under IT is
V*(s) = Ex.P
> rst ax)
80 =
(2)
t=
2

---

Published as a conference paper at ICLR 2026
which satisfies the Bellman equation
v (s) Ex (11s) (s)) rx(() == Ba (rs,
(3)
Gauge Freedom and Relative Values. Shifted value functions do also satisfy a Bellman equation like Equation B (Lemma B.1 explainns how shape the reward). These shifts irrelevant for control, since the advantages or greedy action selection are invariant. Hence the absolute scale of V is not meaningful, and only differences of values are invariant. We therefore define the pairwise value difference
(8, 85) (*t)) (sy),, (5),
S.
Note that it is antisymmetric (8)), 85) (sj,,,, and satisfies A (s, 8) = 0 for all 8.
Pairwise Bellman Identity. Fix (J) E x S. Draw si ~ ~ s;) and s/ (- independently (only conditional on 8; 85 respectively). Subtracting the Bellman equations of for 8; and 8, gives the recursive identity
(5))))8)) (s;) (sj + 7 BB
A (5 (s)
(5)
Equation 5 depends only on observable one-step rewards via Tx and on pairwise differences at successors. It invariant to any additive shift of V* as we have shown in Lemma B.T
3.2 PAIRWISE BELLMAN OPERATOR
Fix policy We work the Banach space of bounded antisymmetric pairwise functions
F = { A R (ss, sy) = - ((s) 8s) | | < oo)
(6)
| sup
| 8,)1
(o:.5,)
In addition, let Ar" (s), 8,) ) = r" (s:) - r (sy) be the immediate reward difference.
Operator form. Let Let act on by (P))))()))))
(ss)) A(s), sf)]
Define the pairwise Bellman operator Tx F + b
(T+ 8y) == Ar"(s + (P.
(8)
This coincides with the definition from Equation 5 used in method when sf and s/ drawn independently from pa (-|s,) and pa (-1sg), respectively. Theorem 3.1 (Contraction and uniqueness). For any D1, E.F
ITT' 12 S7l1A A2llx
Consequently, by the Banach fixed-point theorem, Txx has a unique fixed point A F. Moreover, this fixed point equals the true value differences, i.e., A 85) (*, (V= (s)) for all s;) S.
Proof. The immediate difference TT cancels in A T,A2, hence for each (si,
T2 7 - s/>]
S 7 [E[[( (2)
$ y A Azlloe.
Since this bound holds for all pairs (s1,8;) thus taking the supremum over (si, 8j) concludes with
[|TA; - TA2ll I A Az
Related 'relative' value functions arise in wveaage-rwaard MDPs, where values defined only up to an additive constant and algorithms fix the gauge via relative value iteration (Abounadi et al. 2001 Puterman 2014 Bertsekas 2025).
3

---

Published as a conference paper at ICLR 2026
3.3 RELATIVE GENERALIZED ADVANTAGE ESTIMATION (R-GAE)
In this section, we define the R-GAE estimator which is in essence analogous to GAE (Schulman al., 20155b. Furthermore, derive the relationship between GAE and R-GAE and show that both estimators learn the same (optimal) policy. Then, R-GAE is combined with PPO (Schulman et al. 2017) to provide relative advantages. Hereby, one of our key theoretical contributions the fact that GAE can be reconstructed without knowing the exact value of a state.
Relative GAE. First, construct relative values Vo for each state of arbitrary environment rollout (sp,81:- $1, $T), where T denotes the rollout length. Note in our notation that 8 is not the environment's s start state, but it can be any state. This notation simplifies the following descriptions. We define the relative value sequence by telescoping the differences:
1-1
(o(s == O, Vo(+ = ( 1).
(9)
Note, that also possible to use a larger step size for calculating relative values, e.g. one can directly compute Vs,s, (s so). If Ae A", then the relationship becomes Vo(s;) (T((s) - (so). Analogous to GAE, define relative TD residuals
8, == + (V(44)1) Vo(s.),
(10)
and finally construct the relative GAE similarly
T- A = E
(72)
(11)
1=0
Essentially, RV acts the critic and replaces GAE with R-GAE in the PPO clipping objective. Furthermore, we give additional theoretical insight about R-GAE in the following. In short, these insights conclude that:
1. There exists a relationship between GAE and R-GAE that is given Å, A. Be. For more details, see Lemma 3.2
2. The optimal policy for both estimators is identical. See Corollary 3.3
Lemma 3.2 (Relationship between GAE and R-GAE), We call C := (so) the trajectory constant.
If Ao A3, "hen the following equality holds
À where A, the standard GAE computed from V and B, (1 -1-1)01
A + Bi,
(12)
Proof. By construction of the relative values, we have that V(s)) = V=((s) for all t. Hence inserting this form in Equation 10 gives
d. = rt ( (8t+1) C) (V: - C) = VV V (st) + + (1-7) +) C, (13)
where 6, is the TD residual. Therefore, for each timestep the relative advantage is
T-t A,
T-t
T-t
(12) 80s1 1=0
[na sess + +1
Σ 1=0
(22 = A, + Br,
10
Gauge Invariance vs. Trajectory-constant Baseline. To a better understanding of the trajectory constant B,, we discuss its role and properties in Appendix C Lemma B.1 states that adding any scalar E R. to V (together with the corresponding reward shaping) leaves action preferences, advantages and pairwise value differences invariant; hence the absolute gauge of V is behaviorally irrelevant. Lemma 3.2 however, shows when reconstruct advantages from pairwise differences by
4

---

Published as a conference paper at ICLR 2026
anchoring to zero on each rollout, there exists indeed a difference quantified by B₁. These statements compatible and are contradicting each other. Anchoring (so) 0 is a gauge fixing analogous to the differential (relative) value normalization used in average-reward MDPs (Abounadi et al. 2001 Puterman 2014 Bertsekas, 2025). The only difference is scope: Ours is per-trajectory and theirs is global, so unbiasedness (see Corollary 3.3) is preserved while trajectory-consten B3 may appear.
As remark, two-argument function
achieve pointwise equality with GAE, if we learned the non-antisynmetrri
==(=',))==>Vi0)
V(s),
(14)
so that the temporal-difference residual 6, = + A, ($t+1, st) is modeled precisely, The correspond- ing Bellman operator is indeed contractive self-map on the pairwise space, which can be derived analogous to Theorem 3.1 However, we tried to apply A, (s', s) in practice, but the results are worse compared to R-GAE. Therefore, we continue to work with R-GAE and show in Section how to reduce the variance of By.
Corollary 3.3 (Unbiasedness for policy gradient). Let V (() E, e log ( I st) A,] denote the standard policy gradient (PG) under Tœ. If the advantages in the score-finction estimator are replaced by the relative advantages from Lemma 3.2 i.e.,
V J(o
V log (s 8t) A.
then
VoJ(o) = Vo)(o).
(15)
A, + B, with B. ((1-(22)--11) 1-72
Proof. By Lemma 3.2 A,
as trajectory constant, hence
E V Tog s,) A
E. V a log Ta (at st) A.] V. log TT(a; st) B,]
(16)
Condition 8, by the trajectory prefix up to time (so that B, does not depend on ax) and then using the ccore-function identity gives
E V & log (a st) B, st] = B, (le V log To (az st)] O.
(17)
Taking expectations over t yields
E2
V a log (a | Ac]
[.[V (s 81) As),
(18)
which is precisely Equation
15
3.4 RELATIVE VALUE TARGETS
When approximating the pairwise Bellman operator (Eq. g from two sampled 1-step transitions T, (st, ai, ri, di 8141); T3 (sj, ajr d 81+1), terminal successor states make the naive bootstrap A (5 85+1) ill-posed because absolute values are not available in our formulation. For example, if 8,41 a terminal state (d.- 1), then need calculate
A(s++1, 85+1) = 0 V(5541) (V(541))
which is simply not accessible in that way. Therefore, we need to rearrange all bootstrapping targets terms of observable rewards and non terminal pairwise differences A). In this way the targets get well-posed and are compatible with the operator Tx on antisymmetric functions. Due to limited space, the complete formal derivation is given in Appendix A
1-step Target. Given the prediction Ag(s;, s5), the corresponding 1-step target takes the form
Vij (rr ry) + 7 dij,
(19)
where the bootstrap term dg depends on the successor terminal flags d,, d3 {0, 1}:
Ao(s $j41), Ao(s141, 8z) + Tzr
if d, = 0, d, = 0. if d, 0, d, = 1,
8.j
(20)
Ao(si, 88+1) - rir
if d, = 1, d, = 0,
Ag(se, 83) + r3 - Th
if = 1, d, = 1.
5

---

Published as a conference paper at ICLR 2026
When d,=d, =1, may replace the last line of Equation 20 by dey =0 (both successors absorbing
with zero value) to reduce variance at ends the two episodes. We use 6zy=0 by default, so the derived fourth case in Appendix A optional.
N-step Target. To extend beyond 1-step temporal difference targets, we define the pairwise n-step case by considering two trajectories
Ti = ((Si+k; rs ditk; 8: 1)-R+)))200
= [((s+R; TJt,, 8)+k1)))00
Then, the target becomes
n-1
* (ri+ - T5+k) + A Σ k=
(21)
with the assumption that neither trajectory terminates within the n-step window (i.e., di+ dj+ for nt). the trajectories have different length, then we take the minimum length. Note, that the final estimated difference Ae A0(stnn: 8s+n) needs consider the case distinction from Eq. 20
X-Return. Interpolating between gigh-bias/low-variane (n=1) and Monte-Carlo limits gives the pairwise A-return
yij (1 (1-1)
(A)
- (n)
A [0,1],
(22)
(n)
with the convention that Vij is truncated at the first encountered terminal using the case distinction
from Equation 20
4 RELATIVE VALUE INITIALIZATION
As demonstrated Lemma 3.2 the zero-anchor V (so) = 0 (see Equation 9 induces a trajectory- constant offset B, for R-GAE. If the unknown [C* is large (recall C' (s))s) then Be inflates the magnitude of A, It thereby can increase the variance of our volicy-gradien estimate (see Appendix C). The increased variance makes the credit assignment problem harder and we therefore seek a data-dependent initialization that drives EE [Br] R 0 over collected batch. Intuitively speaking, our solution is to rank trajectories relative to each other which can be understood as anchoring. For visual example, see Figure 2
4.1 TRAJECTORY RANKING
Given M training rollouts (m)m ) m=1 with states ((m) = (8) (m)
M
sqm) and done flags
(d(m) T-1
)-1 indicating whether s begins new episode. Define
(m)
(m(m) [0} U {te ......,T:: d(m =1,
(23)
as the index set where either sub-episode or new episode starts. In the following, we refer these (n) selected states as start states. Enumerate all N = Σm [((mm) start states as {s J. where each
[n) (m) for some l E ((m)) Let Ag S x S IR be the learned (noisy) difference model and Sstart = 8
define the N x N matrix of pairwise differences
Aij := Ao
Sstarty
(i)
start
ISijSN.
(24)
To obtain an offset estimation that is robust to prediction noise and is identifiable up to a batchwise constant (which sufficient for ranking), we estimate all start state offsets O via row-wise averaging and then subtract the batch minimum to get non-negative values with ranking
N
( Sstart
Σ Anjs
( = (
min
(
(25)
N
ISESN
1=1
Finally and to avoid overly complex notation, for any state s rollout, use the most recent start state in that same rollout, call it Sstart Set the relative value for as
Vo(s) (o(sum)) + ( ssar)) and use these values for R-GAE, In that way, each state gets its episode-specific offset.
(26)

---

Published as a conference paper at ICLR 2026
RV DIRPERENCES
ESTIMATE ORSSET
RELATIVE VALUES
Vo (ste)
Sp
Sg
80
71
90
O
00
Apo
((5))
DO
!
D0
(
sD
Aoo
(50)
RCTNNE OISSET
ADD OIFSET
RELATIVE VALUES WITH OFFSET
(o()
(a())) Va()) Ag(+t,
(o(
(s0)
usB
(o().) = Po(so) (o(de, so)
(=0)
Vo(o.) Vo(so) ( 40)
0
Refinement: min (O(so). (ss)) (ss))
(=0)) == O(so)
Figure 2: Trajectory Ranking. When training batches contain samples from more than episode, the initialization (su) 0 for each trajectory т; is not correct. The trajectories need be ranked relative to each other by adding an offset that is calculated with (ss, 85). ). Note that 8p, 80, 8n are start states of 71, 72, 73 indicated by color. For simplicity, assume in this figure that start states only present at t for each rollout, so we can think about each T as one episode.
TRAINING OBJECTIVE
The PPO objective uses relative advantages A, and the usual clipped surrogate:
ne (a, 8t)
Cpolicy (0) E, (m) clip(r.(0), e, +c)))) re( oold (e: 3t)
(27)
The critic loss and entropy bonus are
Ceiiii ( 1/2 ((.))>n (( 8,) Viy (n))
)
(en E, (T s,))],
(28)
where use the n-step value target (see Eq. 21 and finally optimize the combined loss
(( Spllicy (0) Cv Lenuic)))) Ce Lem
(29)
5.1 NETWORK ARCHITECTURE
For all Atari experiments we use the exact same architecture as PPO Schulmmulman et al. (201)). The pol- icy and relative value function share the CNN encoder (sne(s) Rd and then split their computation by using a single linear layer for their respective outputs.
Relative Critic. Relative values are obtained by applying the same encoder feue s) both states and projecting the difference of their embeddings. We do not use an additional target encoder or nop-gradients. Formally, the value difference for a state pair (si, 85) given by
Ao(s;, 9;) = 4 (fenc((s) fone(s;)) (9;)),
(30)
where d is the projection head. For fair comparisons, our experiments use single learned vector E IRd without bias term to ensure antisymmetry (o(s,, = Ao(sj. s;) and Ao s,) by design. It is also feasible to build & as an non-linear MLP that is antisymmetric in nature (e.g. by using tanh activations and no bias term linear layers). But so far, we have not observed additional improvements by using such non-linear heads.
EXPERIMENTS
We evaluate Relative Value Learning (RV) as a drop-in critic for on-policy policy-gradice methods the Arcade Learning Environment (ALB) for Atari. Observations follow the standard PPO
7

---

Published as a conference paper at ICLR 2026
preprocessing: random no-op resets, frame skip with max-over-twe, grayscale resizing to 84x84, stacking m=4 frames, and input scaling to [0, 1]. Networks use orthogonal initialization with a small policy logit scale (0.01) and unit scale for the value head, matching widely used PPO implementations. We run with EnvPool Weng et al. (2022) for high-throughput environment simulation and train for 40M frames (10M environment steps). For each game use 10 independent seeds and report performance as the average score over the last 100 training episodes. All hyperparameters stay
identical over 49 games and are reported in Appendix D
Table PPO+RV (ours) is competitive with PPO and DAE. Mean final scores (last 100 episodes) with standard deviation of PPO, DAE and our method (PPO+RV) after 40 M game frames.
Game
Schulman 2017
DAF
2023
PPO (ouns)
Alica
1860-3237668
1372.7#349.
1748.8:::00
Amidar
674 6::.6:2::00 108.5
121,
504.74:: 108.5
Ascault
4971 9926424
2205. 1 162.
3901 219.992
Asterix
4532 (et 157077
3760. 1::229
3862.2:4678.22
Asteroids
2097-55::88
1392.32:6.3
1489-331991.5
Atlantis
2311 81.5 01.5.0::55000000 52:5555
088811 221522
686. 14-51444511175
BonkHess
12280 2200.6222.7
257 193.2
1200900::7..5
BattlZone
17:366 10450
16.302.0 204.5
21 780.0:1 1516.1
ReamRiler
1500.02
172.7
2044.0:::00
Bowling
10.1:213.
1::9
46.3:177
Boxing
94.62209
25.9219.
94.8 94.82208
Breakrur
274.8=2012
284.9=28.2
263.04:27.7
Cemipede
4386.41::11
3915.
1226.
ChomperCommall
351 9,19
1387-321473.6
4435.24:900
CrapyClimber
110-20200::2477
112319.6261254
111 56452
DemonAilack
113778 28000
2477 34.443
8944.74:50 53:33333
DouilleDank
-14.921.3
1.0
23.33:5.3
Enduro
758. 3::312
0.0:10.0
1080 0.1 121.1
Ferhin
17.8
60.422.2
19.8:1522
Froeway
32.5 32.5220.3
19.7
31.9202
Frustbite
314
367.9:1278.2
522.6:1:99.
Gopher
2932 9 6870
1137:22156.8
3719. 1::24495
Gravitar
737.22 50.9
443.5250.1
1441. .0::561
lceloockr
- 4. 023.3
-..1
-3.9.10.4
Jameshund
507 81:21.1
568.9:1-33.7
Kangaroo
9928. 7 7 7089.9
1331 0::: 1060.8
3649.0-4-29400
Krull
7912 3:5.5
9034 01:51:0::777..
5870.911-4733
KingFaMaster
23:31 2251
20 2810
24 483.6116994
MuniszumaRevenge
42
MsPaemmnn
2096. 00096.5:1:555 157.1
2501 600.00.99
1721.2-1-30..
NumeThisGame
6264. 9 160.9
6016 6016.31::00 28333
6685.3:1:98
Piifal
-32 19,0
-3,10:233.1
27.33
Pong
20.71:02
20.710.3
16.8:1311
Phivatclaye
69 5--55.2
N6.0d: 14,9
17.66
Qbert
14 293 3:3.193.1
11 119.1:1::0000 3:6:5
15 261. 6 1.502.1
Riverraid
8393 6..365AA
1150..81.499.
9465.8 1062.5
ReadRuzner
25 076 108512
16 1466344207440
346.3+-6741..
Robutank
5.5.1.07
8.9:-2.7
19.5
Seaquest
1204 5:1-11111
1049.8:1-33.3
1659.7112004
Spacelnvaders
942 266.
914.9=2194
712.3+ 134.5
StarGunner
32 9:199
2:1
830.8 10990.8
Tennis
- 14 8::3.88
-17.6100
-31, 3.4.334
TimePPlot
43442 0-1-31111
7252.75-17336
10 1021227---42 7-4-492.0
Tutankham
254.4 4::22.:
196.0:126.2
224.2:1:3.11
UpNDown
95 446. 01:21884111
4364444446
113991 74217777
Venture
0.0-1:0.0
0.0::000
2.5-6:15.6
VideoPinball
37 389 0:1: 15876.6
23 958 34360
138 3.177727772
WizudUrWor
3
3361.36096.8
5484.1 14-2614
Zasxon
5008.7 1261.3
1612.2#17125
845.8 1567.9
Compute Resources. Each 40M-frame run completes in approximately 65 minutes on a single A100 GPU with 12 CPU cores. Across all 49 games and 10 seeds (490 runs total), this corresponds to 530 GPU-hours or 22. 10 A100-days.
Median
IQM
Mean
Optimality Gap
PPO+RV cono
PPO
DAE
0.6
0.8
1.0
0.60
0.75
0.90
4.8
5.2
5.6
6.0
0.36
0.40
0.44
0.48
Human Normaizzdd Score
Figure 3: Comparison between Methods. Aggregate metrics with 95% stratified bootstrap con- fidence intervals (Agarwal et al. 202I). Higher median, interquartile mean (IQM), and mean, but lower optimality gap indicate better performance.
8

---

Published as a conference paper at ICLR 2026
Led
19.00
14:0
2001
30 J014
3M
zon
VCM
LSP
ZCM sone
can Chospu Common
2ow ing
Buslig
Contacde
102
10000
1788
25
20:3
230
149
X0W 204
3M 204 30m 4dM Gopher
LOM zo1 3i 4MM Seaquett
33m 20W
UP ZCM 30
de Li 200 40 Doun wDurk
Dee ZOM 30M one
Crazyy Imber
NaPacman
Demo
2002
1543
1530
-22
2002
1000
1891
1003
2993
-30
20M 20N 20m noM
2M 204 Jam it
Los 20n 20r 1MM Prostbll
23m 20r 63M
Lar JCM 2an coftoday
Loe J0M 201e man aamabbrd
Lan 2CM JON Ga Cangaroo
Pi paniingbaryy
Troeway
Graviar
600
1009
299
1/1 2014 vin che
T4l GungTuMMdder
19e you via inu MenbruussSeuerge
70m 2ia vil eine Jamel NooGmme
tri rru ván 2anal
2ru 12v sin Peng
rin viM abn PreateCye
00:00
40:00
4000
Ober1
Riverakd
NaziPurree
tobecian:
Spacetradds
Star SanGumirr
Lanris
4C890
23-
Sane
10000
1000
2200
300330000 Timeflict
33/4 Viceo Pinzal
hutantham
UphDown
Venture
WitarsCWiorr
10000
200800
52899
203
Figure 4: Ablation for Value Initialization. This figure compares the performance difference between (see Equation 9) and offset initialization (see Equation 26 for relative values. Usually the proposed offset initialization is needed to improve credit assignment, but for some games the algorithm can handle zero initialization as well.
6.1 ARCADE LEARNING ENVIRONMENT (ALE)
Table presents the per-game scores of PPO (Schulman et al. 2017) and Direct Advantage Estimation (DAE) Paan et al., 2022) and extends the table with an additional column, PPO+RV. This column represents our algorithm where the absolute value critic is replaced by the RV critic. Across the benchmark, PPO+RV attains competitive performance: it exceeds PPO on out of 49 games (or 61%) and DAE on 37 out of 49 games (or 75%). In addition we report aggregated metrics in Figure for further analysis.
6.2 ABLATION STUDY
Furthermore, present the influence of relative value initialization in Figure In general, that relative ranking is needed, but sometimes actors are not bothered by the influence of B, which speculate is due the PPO clipping objective.
9

---

Published as a conference paper at ICLR 2026
7 LIMITATIONS
Gauge fixing induces a trajectory-constant baseline B, in R-GAE that inflates variance for long horizons when 7 but it is only partially reduced by relative value initialization. The enforced antisymmetry improves stability but restricts critic expressivity (our near-linear difference head may underfit complex value differences). Pairwise training is O(B²), where B is the batch size, in the naive form and relies on subsampling that trades compute for estimator variance. Trajectory ranking used for initialization assumes post-baseline values are on a comparable scale across trajectories. Ranking signal may become uninformative. Empirically, experiments are limited to PPO on discrete- action Atari. Generality continuous control, off- off-policy regimes, and preference-based settings remains to be demonstrated.
8 CONCLUSION
In this paper we established that reinforcement learning can operate on value differences rather than absolute values. We proposed Relative Value Learning (RV), gauge-invariant alternative to absolute critics that learns antisymmetric value differences A (s), = (s;) V (s;) as primitive representation. On the theoretical side, we defined pairwise Bellman operator that is 7-contraction bounded antisymmetric functions with unique fixed point equal to the true value differences, and we derived well-posed bootstrapping targets (I-step/n-step/AA that operate entirely on observable reward differences and non terminal pairwise terms. We further introduced trajectory ranking and reconstructed generalized advantage estimation from pairwise differences (R-GAE), showing that it ensures unbiased policy-gradient estimator and clarifying its relationship to standard GAF. Empirically, replacing the PPO critic with our relative critic gives competitive performance across Atari while simplifying the value learning objective the quantities that matter for control: differences rather than absolutes. We hope RV serves as a building block for algorithms that reason natively the relative domain where decisions are actually made.
ACKNOWLEDGMENTS
This research has been funded and supported by the Federal Ministry of Research, Technology and Space of Germany and the state of North Rhine- Westphalia as part of the Lamarr Institute for Machine Learning and Artificial Intelligence.
REFERENCES
Jinane Abounadi, Dimitrib Bertsekas, and Vivek S Borkar. Learning algorithms for markov decision processes with average cost. SIAM Journal on Control and Optimizution, 40(3):681-696 2001.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-2933 -93200, 2021.
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International conference on machine learning, pp. 449 458.. PMLR, 2017.
Dimitri P Bertsekas. Differential training of rollout policies. In Proceedings of the Annual Allerton Conference on Communication Control and Computing, volume 35, pp. 9-22. UNIVERSITY OF ILLINOIS, 1997.
Dimitri P Bertsekas. Neuro-dynamic programming. In Encyclopedia of optimization, pp. 1-6. Springer, 2025.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pp. 1096- 1105. PMLR, 2018
10

---

Published as a conference paper at ICLR 2026
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforeement learning. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: A research direction. arxiv 2018, arXiv preprint arXiv:1811.07871, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PmLR, 2016.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In leml, volume 99, pp. 278 287. Citeseer, 1999.
Hs Pan, Nico Gürtler, Alexander Neitz, and Bernhard Schölkopf. Direct advantage estimation. Advances in Neural Information Processing Systems, 18111889-1188 2022.
Martin I Puterman. Markov decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.024338 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0633: 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando De Freitas. Sample efficient actor-critic with experience replay. arXiv preprint 1eXiv:/611.00 2016a.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995-2003. PMLR, 2016b.
Jiyy Weng, Min Lin, Shengyi Huang. Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, Zhongwen Xu, and Shuicheng Yan. En vPool: A highly parallel reinforcement learning environment execution engine. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 22409-22421. Curran Associates, Inc., 2022. cc/paper paper, files Btl.ee/paaer//222///2 Datasets 8 Bcaa00eelbbbbodoofffa00777ee2222 Patat and Benchmarks
/ https//prooeedins..eriiii cc cc/
URL
pdf
11

---

Published as a conference paper at ICLR 2026
A TARGET DERIVATION FOR 1-STEP BOOTSTRAPPING
When approximating the pairwise Bellman operator from Equation by using two randomly sampled transitions
= (8i; Oi, Th, di, 8i+1),
(5j, Tj, dj, 83+1),
we can encounter terminal successor states that result in ill-defined targets for A (s1+11, 11441,88+1) We present four exhaustive cases that arise from the possible combinations of terminal and non-terminal successor states where d;, dj E {0, } are terminal state indicators for Si4-1 and 8j+1- In each case, derive equivalent expression for A(8 $j++1 by only using (1) observable rewards ris T and (2) non-terninal, well-defined value differences Ag(:,)). The rearrangement is then used for the bootstrapping target in Equation (20 to work for sampling.
Case 1: Both successor states are non-terminal (d; =, 0, 0). Then, the relative value
((84118541) = V(8(+1) - ((8++1)
(31)
well-posed and directly expressible with (0(8+1188+11
Assume the successor 8;-1 1 is terminal, and 8j41 is non-terminal (d; 1,d, 0). To address this, we derive a modified target:
Case 2:
A (s141, 85+1)
V(s+41) ((8++1) V(si41) V(5++1) + +V(s)) Vs:))
(811,84+1) - V(8i) (88+1)
(32)
(s; 83+1) - E, RR + R2+1 +...]
using
As1, ri
Case 3: Assume the successor 8,-1 1 is non-terminal, and is 8j+1 terminal (d, 0, 1). To address this, we derive a modified target:
V(ss-1) V(8++1)
V(8i-1) V((j+1) + +V(s) - V(s;) (88141,8;) + +V(8)) - V A(s141,8;) + Ex, [R. + 7R +...] A(si+1, 83) + +r₃
(33)
using Tj 2
Case 4: Assume that both successors are terminal (d; = 1, dj = 1). We derive a modified target:
A (8 8 V(8+11) V(8j+1)
V(8++1) V(8j+1) V(s.) - V(si) + V(s,) V(s;) 83)),;)) Ex, [Rg + 7 + +... + E., [R + 7 + +...]
(34)
using Fe.T, A(si, 33) r: +rū
For better training stability, simply say that
A A(s 1441 = -(8844)-V(8844))
(35)
since the value of both terminal states, by using MDP definition, is equal to zero.
12

---

Published as a conference paper at ICLR 2026
B GAUGE FREEDOM
Lemma B.1 (Constant-offset gauge invariance). Suppose A satisfies the Bellman equation
V Ta + V= For any define the shaped reward and shifted value
(s) (s) (1 (1-7)c,
(*) == (*)s
(36)
Then W satisfies the transformed Bellman equation
V r'x +
(37)
Moreover, if one equivalently defines the reward shaping r'(s, (), (s,a) + (1 7/ so that (s) = (Bawx(1l) (s, a)), then action preferences and pairwise differences are invariant:
(s, (s, and ( (7 sor all (st, 8,) E S x S.
(38)
Proof. By direct calculation.
/ + * = (rx + (1 7)c) + 7Pp +c)
(r+ ((=+PP)V") + - (1-7)c+
V" +c
VT
so V satisfies the transformed Bellman equation. For invariance: with ('(s r(s
(s) = rs, +
= (x=
(39)
Thus (s, (5' (s) = A A a). and = - (+)
A (s,..
Constant offsets are special case of potential-ban -based shaping (Ng et al. 1999
13

---

Published as a conference paper at ICLR 2026
C VARIANCE ANALYSIS OF THE RELATIVE POLICY GRADIENT
In Section we argue that the unknown trajectory constant |C| can increase the variance of the policy gradient estimator. Here, we provide the formal derivation supporting this claim and reference Figure 4 for empirical validation. First, recall from Lemma 3.2 that the relative advantage is given by A A, + Br where B, is trajectory constant determined by the initialization offset C. While Corollary 3.3 proves that the estimator remains unbiased (i.e., the first moment unchanged), the second moment differs.
Lemma C.1 (Variance Inflation). Let Istd = V log (a (8) A, be the standard gradient estimator and Grei = V log To((olss) be the relative gradient estimator. The variance of the relative estimator given by:
Var(gre) Var(gsta) E [I[V log (a 8 2 [A. log ( (8)))
(40)
Crucially, the strictly positive term E | & log ol? B,] scales quadratically with the trajectory offset C².
Proof. The variance of any estimator is defined as Var(g) [1
||]]] From Corollary 3.3
we know that Elgra]] = [[[saa]] = (Ø). Since the expected values are identical, the difference in variance is determined entirely by the second moment [3]
Let U V log (a) denote the score function. Expanding it for the second moment gives:
greer| 2| = ||122
(41)
= [A + B²)
(42)
|14/1112 + I|u Bi 2(4 (w Be)
(43)
Note that A, and B, are scalars, so we can factor them out:
11 1|2 = |uell" A +1| 1² + 2/ A.B,
(44)
1|gorall"
Recall from Lemma 3.2 that B, (1 (1-7) (1-))C Thus, B, is directly proportional to the initialization offset C.
1. Noise Term: The term B|[[u,]| [|2]]] is strictly non-negative. Since B, x C, this noise term scales with C2.
2. Correlation Term: The term 3EI|11 A,] represents the correlation between the true advantage and the offset. While this term can be negative, it scales linearly with C.
Consequently, for a sufficiently large uncorrected offset |C), the quadratic noise term (C will dominate the linear correlation term, resulting in increase in estimator variance. This necessitates the Trajectory Ranking strategy (from Section 4) to minimize |C] and drive B, Rs O. Empirically, our ablation in Figure 4 validates the importance of decreasing estimator variance.
In addition to validate B, empirically, 2 presents small example for a seven step trajectory. Given are rewards and true values to compute the true GAE for comparison to relative GAE:
[ro. [ro...., r3] [1, 1,1,, [V(ss)...... V(ss)) = 2,
In this example V(so) = 2, so one can compute all predicted differences B, using C' = 2. Over all time steps, the observed advantage differences match exactly the prediction B,, confirming the algebraic derivation from Lemma 3.2 In this example, we rounded to two decimal digits for clean visualization.
Figure [ generalizzs the intuition of Table by visualizing the prediction and decay of B, over a T = 128 timestep rollout, which has the same length as in our experiments. In the figure, also
14

---

A, Published as a conference paper at ICLR 2026
Table Empirical Evidence for the Trajectory-constant, This tables shows a small environment trajectory and validates the trajectory-constant baseline empirically. To compute true and relative GAE, we set y 0.9, À 0.8 and use C 2.
t
t
5
Values
1
V(se)
2
A(s;,
(s))
True & Relative GAE (y = 0.9,
0.8)
& 1.70
1.60
1.50
1.40
1.30
1,20
&
1.90
1.80
1.70
1.60
1.50
1,40
4.73
4.21
3.63
2.96
2.16
1,20
A,
5.35
4.79
4.15
3,41
2.51
1.40
GAE Differences (C = 2)
Å, Ar
0.61
0.58
0.52
0.45
0.34
0.20
Predicted via B,
0.61
0.58
0.52
0.45
0.34
0.20
Expected Difference to True GAE
0.75
C=
C=.1
C=1
0.50
C=2
C=5
Baseline B2
-0.25
0050
0075
20
40
80
100
120
Time step
Figure 5: Variance Visualization. This figure shows the expected difference B, to the true GAE A, using different baseline values C. Note, that the difference is higher for larger |C], because applies a stronger absolute discount.
consider the same hyperparameter setting (7 0.99, À 0.95) which influences the magnitude of B, At the start / = 0 of a rollout, one can see the absolute value of Bg is large but almost constant, and inflates the GAE value. Then after some time, B, falls off and finally (t 128) almost vanishes, so R-GAE x GAE for the last few timesteps. This can be demonstrated for arbitrary C.
15

---

Published as a conference paper at ICLR 2026
D HYPERPARAMETERS
We present the hyperparameter configuration for the Atari benchmark (ALE) which is used for all 49 games.
Table 3: Hyperparameters for PPO+RV used in experiments.
Hyperparameter
Value
Discount factor 7
0.99
GAE parameter À
0.95
N-step return target
Clip parameter
0.1
Number epochs per update
Minibatch size
128
Number parallel environments
Rollout length T
128
Learning rate
2.5 x 10
Optimizer
Adam
Adam epsilon
x 10
Entropy coefficient co
0.01
RV loss coefficient Go
1.25
RV value clipping
0.15
Gradient clipping (max-norm)
0.5
Additional Hyperparameters Details. For training the relative critic do not use purely random state pairing. With probability 33%, the second state is chosen from the same episode as the reference state 8,, encouraging temporally coherent comparisons. Otherwise, the partner is sampled randomly in the batch. See Appendix [ for an ablation study.
16

---

Published as a conference paper at ICLR 2026
E ABLATION ON PAIR SAMPLING
In this section, we present Table 4 as an ablation study for different pair sampling strategies ja that can be applied for Equation 28 In our case, "Biased" means that with probability p [0, 1] the second state 8j is randomly sampled from the same episode. In general, we can see that the sampling strategy has rather small influence except when p gets too large. Then there is small decrease in performance.
Table 4: Ablation of Pair Sampling. Mean final scores (last 100 episodes) with standard deviation of our method PPO+RV with different pair sampling strategies after 40 M game frames. The results average over five seeds.
Gouse
Raidom
Biosed (p 0.33)
Blased (p 0.66)
Alien
2123.44 1135..44-3340 444-66600
17466 5-1-491.7
1802. 00-000--6000.5
Assault
42144.3 12214.3.1:338
4018 & 8:1:144.1
3615 2:1:290.2
Astenix
4300, 2:1: 1024.8
4495 11100
3821 2: 132--61.
BattleZone
22 512., 0:1: 1008.6
22 576.66.. 0:1:921.1
21 (16.6..00 216.0.0:::222 1228.4
BeamnRider
21999. 2:t:573.9
19964 10::1330.1
1814.8#2247
Breakout
2:37.0:3
258 31:3355
241.7:47.6
9364.33::
9:17734.8
1181.11138.
Gopber
92224.7-+10911 1099.3
3085.7 1-13881-111311
3626 233333811
Gravitz
1414.244759.
1379 05--9.99
1232 0: 406.9
Icellocky
-3.7:1:0.6
3.7.
0.99:1.4
729-2
4:3393-7
3044
MsPaccan
18:40.0::147..
1783.41:22
9-18099:1-447.5
Namee Thastame
1:000.3::1114 1446.8
0880 2::9377
1890 296.
Qbert
714-17.3
799-19911
Robetank
22.551100
21 3::3
19.7
Senguest
1592.6-43377..
1612 4 10966
1236. 1 14:321.77
TimePilot
9701.6:1:455
9825 819.
3692.881.588
UpNDown
101 965
333778
VideoPiaball
144 597. 1177.1::550 89747.6
119 187 4-779888
82 002 4 4-02244-111000
WizardOWWe
6501.2::757.0
4890.002-99.9
5128.0=374.3
17

---

Published as a conference paper at ICLR 2026
F THE USE OF LARGE LANGUAGE MODELS (LLMS)
Finally, we report on the use of LLMs in this section. The image in Figure is generated with Gemini 2.5 Flash Image (Nano Banana). Furthermore, we used large language models (LLMs) writing assistants in the preparation of this manuscript. LLMs were employed to draft and refine text as well as to generate preliminary versions some proof derivations. All mathematical results, derivations, and theoretical claims were independently verified and validated by the authors.
18


Reviews:
[{"review_id": "yxz5x4iHmZ", "reviewer": "Reviewer_YGuv", "summary": "The paper proposes learning relative value differences compared to directly predicting state value. They integrate the proposed idea of relative value learning with PPO and evaluate its performance on the Atari benchmark.", "questions": "- Based on the above, could the authors differentiate their contributions from the mentioned prior work \\[1\\] and also report variance in performance on the benchmark?\n\n**References**  \n\\[1\\] Bertsekas, “Differential Training of Rollout Policies”, Proc. of the 35th Allerton Conference on Communication, Control, and Computing, Allerton Park, Ill., October 1997", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The idea of estimating relative values is well motivated and might additionally have benefits under function approximation settings (as noted in \\[1\\])  \n- The paper is well written and the details of the method and experimental setup are clearly outlined.", "weaknesses": "- The idea of using cost-to-go differences or relative value was explored earlier in \\[1\\], so perhaps the idea of estimating value difference is not necessarily novel. However, its application in DeepRL might be novel.  \n- The proposed approach does not necessarily improve over the PPO baseline in the evaluated benchmark, it would be good to also indicate the observed variance in scores to suitably contrast against the baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vkOd8umhf9", "reviewer": "Reviewer_5Wrj", "summary": "This paper introduces Relative Value Learning (RV), a reinforcement learning framework that focuses on learning pairwise value differences $\\Delta(s_i, s_j) = V(s_i) - V(s_j)$ instead of absolute value functions. The authors derive a “pairwise Bellman operator,” show that it is a γ-contraction, and propose a relative version of Generalized Advantage Estimation (R-GAE). They integrate RV with PPO and evaluate the approach on Atari games, reporting comparable or slightly improved performance in some cases.", "questions": "1. Since the pairwise Bellman operator is mathematically equivalent to subtracting two standard Bellman updates, what concrete advantage does this formulation bring in practice beyond removing the value offset?\n2. How sensitive is the performance of PPO+RV to the way state pairs are sampled?\n3. Have you tested RV in continuous-control or off-policy settings, where relative value estimation might interact differently with bootstrapping?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- The paper is clearly written and well organized\n- The empirical results are conducted on a reasonably broad benchmark", "weaknesses": "- Limited theoretical novelty. The main theoretical results are direct consequences of standard value-based RL theory. The proposed “pairwise Bellman operator” is simply the subtraction of two standard Bellman equations, and the contraction proof follows immediately from classical results. Similarly, R-GAE and the unbiasedness result are reformulations of known properties of GAE. While the relative formulation is conceptually neat, it doesn’t introduce new theoretical insights.\n- Marginal empirical contribution. The empirical results show that PPO+RV performs on par with PPO and DAE, but without consistent or significant improvement. This makes it difficult to justify publication at a top-tier venue like ICLR on empirical grounds alone.\n- Insufficient related work discussion. The paper cites relatively few references and overlooks related research on invariance, reward shaping, normalization, and average-reward formulations, which would help clarify where this work fits in the broader context.\n- Overall, the paper feels more like a conceptual reinterpretation of existing ideas than a substantial theoretical or algorithmic advance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZOsbiLyUWg", "reviewer": "Reviewer_8vTK", "summary": "The paper proposes Relative Value Learning (RV), which focuses on learning value differences between states, ∆(si, sj) = V(si) − V(sj), rather than absolute state values. It introduces a pairwise Bellman operator, proves its convergence, and derives targets for 1-step, n-step, and λ-return updates, as well as a policy-gradient estimator based on these differences (R-GAE). The method is integrated with PPO and tested on Atari games, showing performance comparable to standard PPO.", "questions": "- Given the case where both successors are absorbing with zero value, could that provide a way to anchor the values?\n- Would it be possible to visualize the value differences between states for a few states or trajectories?\n- Why does the paper focuses on 49 while the usual full set of ATARI games includes 57 games?\n- In the conclusion, you mention \"We hope RV serves as a building block for algorithms that reason natively in the relative domain where decisions are actually made.\". What would be the potential advantages given that performance-wise, it does not seem to improve learning? Any setting where this approach might be superior to usual Q-learning based for instance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies a meaningful gap in reinforcement learning: the focus on absolute state values when only differences matter for decision-making.\n- It provides well-defined mathematical formulations, including the pairwise Bellman operator, and proves properties like γ-contraction.\n- It is implemented with PPO, showing the approach is compatible with widely used RL algorithms. The experiments cover a broad benchmark (49 Atari games), providing evidence that the approach is competitive with standard methods.\n- The paper is overall well-written, with clear explanations.", "weaknesses": "- The paper does not beat state-of-the-art results in the experiments\n- The paper is well written with clear mathematical reasoning but I found some parts a little less straightforward in the explanations (e.g. relative value initialization, see also the question below)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7zvcAGLYU3", "reviewer": "Reviewer_K8y5", "summary": "This paper proposes relative value learning, which considers value differences between states instead of the usual absolute values of states. The authors show that value differences satisfy a kind of Bellman operator and inherit the contraction property, which can be utilized to learn value differences in a way similar to classical values (e.g., TD ($\\lambda$)). Similar to GAE, the value differences can also be used to construct advantage estimators using TD errors, and the authors prove that such \"advantages\" are unbiased for policy gradients. Empirical results for the ALE were given, and results show competitive performance compared to GAE and DAE.", "questions": "1. It looks like relative values can eliminate the shift invariance of the reward functions, and I wonder if there are remaining redundant degrees of freedom that can be fixed. Can the authors elaborate on this?\n2. While Section 4 suggests that $B_t$ can increase the variance of the policy gradient estimator and points the reader to Appendix C, this problem is not really addressed in Appendix C either, aside from an empirical demonstration. Can the authors provide a more in-depth analysis on this?\n3. Regarding the critic training, can the authors share more details on how the pair sampling affects performance (line 806-809)? Also, it is not clear to me how $\\mu$ (Eq 28) affects convergence theoretically. I imagine the $(i, j)$ have to follow a certain distribution that is related to the sampling policy, but this is not really discussed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is well-motivated (i.e., the absolute scale is irrelevant in terms of policy improvement), clearly written, and easy to follow.\nThe theoretical results showing that the value differences satisfy a Bellman equation and can be similarly estimated like classical value functions are quite intriguing, and might open up new ways to design RL algorithms.", "weaknesses": "While convergence results were given, it is not immediately clear whether learning the relative values actually provides any benefit over traditional absolute value estimators, and empirical results seem to suggest that the two are perhaps not too different in terms of policy optimization.\n\nRegarding the experiments:\n1. I would highly suggest reporting aggregated metrics (e.g., median, IQM, or similar) and learning curves alongside per-environment scores. Currently, it's very difficult to assess the effectiveness of the proposed approach simply from Table 1 and the description on line 431.\n2. It is now well-known that deep RL algorithms often come with high variance, and reporting statistical significance is necessary for comparing the performance of different algorithms [1].\n3. The comparison to DAE is not entirely fair, as the reported results [2] use a much larger number of actors (1024 compared to 8), corresponding to significantly fewer PPO iterations.\n\nThere are also some interesting details that I think are worth mentioning in the main text, such as how the state pairing is done in practice (line 806). \n\n[1] Agarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\"  NeurIPS 2021.\n\n[2] Pan, Hsiao-Ru, et al. \"Direct advantage estimation.\" NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)