Symmetric Reinforcement Learning Loss for Robust Learning on Diverse Tasks and Model Scales
Authors: Ju-Seung Byun
Andrew Perrault
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement learning
Cross entropy
Symmetric loss functions
Primary area: reinforcement_learning

Abstract: Reinforcement learning (RL) training is inherently unstable due to factors such as moving targets and high gradient variance. Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAIF) introduce additional challenges. For instance, diverse preferences complicate the alignment process, and prediction errors in a trained reward model can become more severe as the LLM generates unseen outputs. These RL challenges create confusion about whether the probability of an action for a given state should be increased or decreased, similar to the noise in labels for classification tasks. In this work, we focus on RL algorithms that share learning difficulties with cross-entropy loss, especially for low-probability predictions. To enhance stability, we adapt reverse cross-entropy (RCE) from supervised learning for noisy data, defining a symmetric RL loss. We demonstrate performance improvements across various tasks and scales. We conduct experiments in discrete action tasks (Atari games) and continuous action space tasks (MuJoCo benchmark and Box2D) using Symmetric A2C (SA2C) and Symmetric PPO (SPPO). Notably, SPPO shows strong performance across different hyperparameters. Furthermore, we validate the symmetric RL loss in the RLHF framework using PPO for natural language processing tasks such as IMDB positive sentiment and TL;DR summarization.

Symmetric Reinforcement Learning Loss for Robust Learning on Diverse Tasks and Model Scales
Ju-Seung Byun'
Andrew Perrault
Abstract
1. Introduction
Recent advancements in Large Language Models (LLMs) have shown impressive performance across various natural language processing tasks (Chung et al., 2022; Wei et al., 2023), robot control (Huang et al., 2022; Driess et al., 2023), and healthcare (Lee et al., 2023c; Huang et al., 2020). How- ever, as these LLMs are typically trained to predict the next word provided dataset, they require post-training pro- cessing to make them useful for particular tasks. Reinforce- ment Learning from Human Feedback (RLHF) trains LLMs to generate responses aligned with user preferences through human feedback. Additionally, Reinforcement Learning from AI Feedback (RLAIF), which leverages feedback from well-trained AI models, has also been employed (Lee et al., 2023a; Bai et al., 2022). Thus, adapting fundamental Rein- forcement Learning (RL) algorithms such as REINFORCE (Williams, 1992), A2C (Mnih al., 2016), and PPO (Schul- al., 2017) to suit the fine-tuning of LLMs for LLM tasks is an area of active interest ( (Ahmadian et al., 2024; Ouyang et al., 2022; Rafailov et al., 2023).
Reinforcement learning (RL) training is inher- ently unstable due to factors such as moving tar- gets and high gradient variance. Reinforcement Learning from Human Feedback (RLHF) and Re- inforcement Learning from Al Feedback (RLAIF) introduce additional challenges. For instance, di- verse preferences complicate the alignment pro- cess, and prediction errors in trained reward model can become more severe as the LLM gen- erates unseen outputs. These RL challenges cre- ate confusion about whether the probability of an action for a given state should be increased or decreased, similar the noise labels for classification tasks. In this work, we focus on RI algorithms that share learning difficulties with cross-entropy loss, especially for low-probability predictions. To enhance stability, we adapt reverse cross-entropy (RCE) from supervised learning for noisy data, defining a symmetric RL loss. We demonstrate performance improvements across various tasks and scales. We conduct experiments in discrete action tasks (Atari games) and con- tinuous action space tasks (MuJoCo benchmark and Box2D) using Symmetric A2C (SA2C) and Symmetric PPO (SPPO). Notably, SPPO shows strong performance across different hyperparam- eters. Furthermore, we validate the symmetric RL loss in the RLHF framework using PPO for natural language processing tasks such as IMDB positive sentiment and TL;DR summarization.
RL methods (Sutton et al., 2000; Sutton & Barto, 2018a) have lead substantial breakthroughs in tasks such as robot control and game playing. Still, they entail learning instabil- ity compared to supervised learning due factors such as moving targets, high-gradient variance, and training value functions. The RL literature has proposed various methods to make the RL process more robust, such as preventing overestimation with Double DQN (van Hasselt et al., 2015), reducing variance with Generalized Advantage Estimation (GAE) (Schulman al.. 2018), updates within the trust region (Schulman al., 2015; 2017), and encouraging di- verse behavior with Soft Actor-Critic (SAC) (Haarnoj et al., 2018). In addition to the methods devised specifically for RL problems, RL literature has also adopted supervised learn- ing techniques to make the learning process more robust. For example, ensembles have been used for more accurate value function prediction, while Layer Normalization and Batch Normalization have been employed to constrain pre- dictions for out-of-distribuun samples, thereby mitigating the overestimation and extrapolation.
'Department of Computer Science and Engineering, The Ohio State University. Correspondence to: Ju Ju-Scung Byun wwwyn..3@osu...ooo
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
RLHF (Ouyang et al., 2022; Lee et al., 2023b) and RLAIF (Lee et al., 2023a; Bai et al., 2022; Byun et al., 2024) po-

---

Symmetric RI Loss for Robust Learning
We were friends for 10 years, before got together. He than told me once about his...
Content:
Content: So Okay, I'm from New York study in Oregon for year. Recently friend of mine...
Content:
My Grandma and my aunt (her daughter-in-law) haven't spoken to each other in years...
Summary: <emply>
Summary: <empty?
Summary: Some summary
Reward: 3.14
X
Reward: 6.66
Reward: 6.40
Figure! 1. Example reward prediction errors a trained reward model for TL;DR summarization. The generated summary samples (left and middle) are both empty, yet they receive significantly different rewards. The middle sample is higher than some summarization (right) and even scores higher (6.66) than the average reward score of SPPO (6.13). The full text for these samples can be found Appendix 15.
tentially introduce additional training challenges. For exam- ple, these algorithms often receive feedback from multiple sources (humans or AI models) to align LLMs, and each feedback provider may have different preferences, mean- ing sample considered preferable by one provider could be deemed undesirable by another (Ethayarajh al., 2024; Chakraborty et al., 2024). In addition, RLHF and RLAIF of- ten leverage trained reward model to provide feedback samples generated by the LLM. This indirection raises the question: does the learned reward model provide the correct reward? The reward model has prediction errors itself (See Figure 1), but as the LLM is trained with RL, its outputs de- viate from the reward model's 's training dataset, introducing more errors (noise) in the reward model's predictions for -utto--disttibutionn samples.
PPO, naming them Symmetric A2C (SA2C) and Symmetric PPO (SPPO), and evaluate their performance across various tasks and model scales.
First, we assess the performance gains of SA2C and SPPO on Atari games (Mnih et al., 2016), which have discrete ac- tion spaces, as well on the MuJoCo benchmark (Todorov et al., 2012) and Box2D (Catto, 2011) environments, which have continuous action spaces. For these control tasks, we introduce noisy reward variant, hypothesizing that it will increase confusion in advantage prediction to better evalu- ate our method. Additionally, we test our method on RLHF tasks using LLMs, such IMDB positive sentiment analy- sis (Maas al., 201 1) and TL;DR summarization (Völske et al., 2017) 7). The IMDB task involves generating positive sentiment for given context and TL:DR is summarization task where an LLM is required to summarize content.
The challenges associated with RL, RLHF, and RLAIF, as mentioned above, can introduce confusion when calculat- ing advantage values in RL algorithms like A2C and PPO. Specifically, an action that should have a positive advantage value may have a negative sign the next update, depend- ing which samples (states, actions) generated and how the batch is composed during advantage normalization. The sign of the advantage determines whether the probabil- ity of a corresponding action for a given state increases or decreases in policy gradient algorithms. If the advantages are predicted incorrectly, this can lead to learning the opposite direction. We hypothesize that these difficulties are similar noisy classification tasks in supervised learning, where some labels are incorreet.
SA2C and SPPO demonstrate better performance improve- ments across diverse control tasks compared to A2C and PPO. Notably, both SA2C and SPPO perform well in set- tings with added noise the reward. Additionally, SPPO shows consistent performance improvements across various hyperparameters (Table 11), We analyze why SPPO ex- hibits more robust improvements than SA2C in Section 5.4. Furthermore, SPPO shows superior performance to PPO in RLHF tasks, such as IMDB positive sentiment and TL;DR summarization. We demonstrate that SPPO outperforms PPO on reward in both tasks, and SPPO's summarization significantly better, as measured by the win rate judged by GPT-4 Turbo
- 04-000--- -9)).
In this paper, we leverage a technique developed for clas- sification tasks with noisy labels, employing robust loss function to enhance the learning procedures of A2C and PPO. We define symmetric RI. loss, whose fundamen- tal mechanism aligns with the robust loss function used in supervised learning (Wang et al., 2019), improve the robustness of RL procedure for A2C and PPO (See Sec- tion 4.3). We apply this symmetric RL loss to A2C and
In summary, our key contributions are:
We propose the symmetric RL loss for A2C and PPO, along with the gradient analysis that aligns with the gradient behavior of robust loss functions used in noisy classification tasks in Section 4.3.
We conduct experiments across various environments
2

---

Symmetric RL Loss for Robust Learning
and model scales, demonstrating performance improve- ments to validate the symmetric RL loss for general control tasks and RLHF tasks in Section 5.
Learning from Al Feedback (RLAIF) (Lee al., 2023a; Bai et al., 2022) have contributed to the success of large language models (LLMs) by aligning them with user pref- erences. However, these methods require training a reward model and a value function. Each of these components has prediction errors, and finding appropriate hyperparameters for training requires significant effort. Direct Preference Optimization (DPO) (Rafailov et al., 2023) eliminates the cost associated with the reward model by rearranging PPO loss for ranking- bbaiig feedback (e.g., sample A is preferred over sample B). Ethayarajh et al. (2024) remove the re- quirement ranking-based feedback by modifying DPO loss further, allowing a model to be trained with bad or good labels. Additionally, Chakraborty et al. (2024) demonstrate that feedback from diverse people, each with different pref- erences, makes a single reward model difficult to reflect preferences correctly. Recent studies focus on sentence- level feedback (Lightman et al., 2023; Wang et al., 2024), but DPO and KTO cannot utilize sentence-level feedback. Therefore, we propose the reverse RL loss term, which can make PPO in existing RLHF methods more robust.
We analyze how PPO can introduce additional con- fusion in advantage estimates, which justifies using symmetric RL loss (See Section 5.4). This shows that SPPO demonstrates consistent improvement across a range of hyperparameters.
2. Related Work
We introduce robust loss functions studied in the context of noise in supervised learning classification tasks, Ghosh et al. (2017) prove that, in the presence of a noisy dataset, the mean absolute error (MAE) has a slower learning speed compared to cross-entropy loss (CE), but the model learns more robustly. Zhang & Sabuneu (2018) propose general- ized cross entropy loss g. which becomes CE when q 0, 1. By adjusting this parame- ter 0 I q S 1, robust learning is achieved in noisy datasets. The symmetric cross entropy (SCE) (Wang et al., 2019) that we mainly refer to suggests a symmetric cross-entropy loss. This loss not only considers the flow of information from the true distribution to the model's s predictions but also incorporates information flowing in the reverse direction. SCE works better than GCE in general, especially for data with high noise rates. Ma et al. (2020) introduce various loss functions and classify them into types: Active Loss and Passive Loss functions. They demonstrate that normalizing the loss can help improve robustness. They use a combina- tion of one active loss and one passive loss like SCE. We define a loss function that considers reverse information to match the RL version.
and becomes MAE when
3. Preliminaries
3.1. Reinforcement Learning
Reinforcement Learning (RL) formulates a Markov decision
process (MDP) (Puterman, 2014; Sutton & Barto, 2018b) defined by the tuple M = (S, P, R, 7, 11). At each timestep t, an action at E A is sampled from an agent's 's policy Tt 8t) for given state 8, E S. For the taken action at, the reward function returns a reward R(s;, at) where R S x A R, and the transition probability P(- st, ax) determines the next state $1+1- 7 is the discount factor, and represents the initial state distribution for so. The RL objective is to find the optimal 0 that maximizes the expected discounted sum of rewards:
In the RL literature, Wang et al. (2018) propose using con- fusion matrix to handle perturbed rewards. predicting surro- gate rewards for robust policy updates. While this method appears effective for Atari games, later research (Chen et al., 2024) shows that it does not outperform corresponding base- lines in continuous tasks. Additionally, introducing noise in RL has demonstrated performance benefits. For instance, Obando-Ceron et al. (2023) show that smaller batch sizes improve performance, and Schaul et al. (2022) present that policy churn aids exploration. These studies primarily con- duct experiments on Atari games, which require navigating many novel states. However, whether noise is beneficial or not in continuous action spaces remains debatable (Mai et al., 2022; Byun & Perrault, 2024). Our work proposes robust loss function designed to handle noise (confusion in advantage prediction) without judging whether the noise is beneficial or not.
Σ y R(sr, and (I)
0" = argmax
E So ( St+1 P(--P
9
3.2. A2C and PPO Algorithms
The Advantage Actor-Critic (A2C) algorithm (Mnih et al., 2016) is an actor-critic method that combines based and policy-based approaches. A2C uses the advantage func- tion A to reduce the variance in policy updates. The policy To is updated by following the gradient of the objective function to maximize the sum of rewards (Equation I):
VoJ(mT)
Σ V log (p(a st)Ass; a:)
0=
Reinforcement Learning from Human Feedback (RLHF) Ouyang al. (2022); Lee et al. (2023b) and Reinforcement
Proximal Policy Optimization (PPO) (Schulman et al., 2017) aims to update the policy within a trust region. This is
3

---

Symmetric RL Loss for Robust Learning
CrazyClimber
WizardOFWor
Ant-vé
Walker2d v
25
20
14
Changed Adveragee
20
tn
0
0.0
0.5
3.0
15
2.0
Emmsstapa
Total Eimestapo
Total Emmasoos
Bimssttpaa
Figure 2. Change of advantage rate (%): The graphs show how often the advantage signs flip in various environments as training progresses. Atari games, often over samples change signs, while in MuJoCo tasks, usually over 10% samples change signs after the advantage normalization. We use different random seeds for CrazyClimber and WizardOfWor, and 30 different random seeds for Ant-v4 and Walker2d-v4. The line is the mean the change ratio across the seeds, and the shaded area represents standard errors.
achieved through clipped loss function to ensure that the new policy does not deviate too much from the old policy. The PPO loss function can be written as:
combines CE and RCE losses (Equation 6),
Lace
a + B
(6)
where a and | are constants determining the contribution of each part. SCE demonstrates performance improvement across various noisy ratios and types. As mentioned in the introduction section, the RL training process can lead to noisy advantage predictions, so we propose a symmetric RL loss in the next approach section.
ppoo (0) = E, [min (r,(0) (0),( - +)AA)l
(3)
where r0) = (o(le))))
Teaia (0r]=,) is the probability ratio, and is small hyperparameter that controls the range of the clipping. The advantage function estimates how much better an action a is compared to the other actions for at given state 8. Both algorithms increase the probability of a for s if the corresponding advantage A(s, a) > and decrease if A(s, a) < O. In the next section, we introduce the connection between A2C and PPO with the cross-entropy loss for classification and define the symmetric RL loss.
4. Approach
This section introduces the reverse RL loss and proposes the symmetric RL loss for A2C (Mnih al., 2016) and PPO (Schulman al., 2017), RL version of Symmet- ric Cross Entropy (SCE) (Wang et al., 2019). A2C and PPO training procedures basically increase or decrease the probability of an action depending on the advantage sign, but the advantage prediction involves noise due to several factors. highly engineered reward function required to eliminate errors, and the trained reward model has pre- diction emror in RLHF (Ouyang et al., 2022) and RLAIF (Lee et al., 2023a; Bai et al., 2022). Receiving feedback from multiple sources further complicates the training of the reward model (Chakraborty et al., 2024). Additionally, the value function also has estimation errors, and the sign of the advantage in advantage normalization depends on how the batch is composed. PPO increases sample efficiency compared A2C, but the off-policy part can introduce con- fusion in advantage predictions (See Section 5.4). Similar to SCF, which is robust to noisy data, the symmetric RI. (4) loss enhances robustness in an RL environment that can introduce noise.
3.3. Symmetric Cross Entropy
Symmetric Cross Entropy (SCE) (Wang et al., 2019) is designed for noisy classification datasets. Cross Entropy (CE) loss (Equation 4) performs effectively when the data is clean; however, it encounters challenges in the presence of noise. Given a true distribution and a predicted distribution p. p is learned based on the information derived from q according to information theory. However, when is noisy, p can only approximate the true distribution to a limited extent. To address this issue, SCE incorporates information in the opposite direction through Reverse Cross Entropy (RCE) (Equation 5).
K Σ kml
Lee
[k(xx) lkgp(k(x)
Lyco Σ
pkk[x) ((klx)
(5)
4.1. Reverse Reinforcement Learning Loss
Given true (target) distribution q and a predicted distri- bution p. if q is noisy, training p can be challenging and p cannot accurately reflect the true distribution. Reverse Cross Entropy (RCE) considers the reverse information from p.
where le E (1...., ......, K} is a class and x is an input. RCE loss has been proven to be robust to a certain amount of noise, but the learning speed is too slow. Therefore, SCE

---

action probabilities set s as (e(s) = {To Th ..... To }. Note that we discretize the continuous action space for con- tinuous action tasks (Tang& Agrawal, 2020). One thing need to note is that when updating policy, vantages instead of label sets in RL. Advantages can have negative values (negative labels) unlike ordinary labels. We only consider the sign of the advantage' because this ad- vantage is the role of the label in supervised learning. For Symmetric RL, Loss for Robust Learning
We propose that the reverse RL losses for A2C and PPO also incorporate reverse information address noisy fac- tors in the RL training procedure. The RCE loss (Equation 5) defines log 0- Z where Z < O is some constant for (<{]x) = 0 (Wang et al., 2019). We also use this defini- tion for the negative advantage and this is also useful to prove the robustness of the reverse RL losses. For all tasks conducted in this paper, we use Z -. Note that the constant terms Z and / in Equation 7 and 9 are multiplied together, so we control the impact of the reverse RL loss solely by adjusting B. For example, (B 1.0, Z -1.0) and (3 = 10.0, Z = -0.1) yield the exact same results.
RL loss, in Section 4.2. We then analyze how these reverse RL losses contribute RL robustness in Section 4.3.
4.2. Symmetric Reinforcement Learning Loss
The Symmetric Reinforcement Learning (SRL) loss Lart con- sists of two parts like SCE (Equation 6): the original actor loss Lei (A2C PPO) and the corresponding reverse RL loss Leev (RA2C or RPPO). Lurl flexibly adjusts the symmet- ric learning framework with two additional hyperparameters (a > 0 and / > 0) as follows:
Lgil abn 3Leev
(9)
Suppose there exist Is actions and (0) indicates 7th action. (6) =s(a()|s) for a state s. Let's denote the possible
"g
We name A2C and PPO using the symmetric RL loss as Symmetric A2C (SA2C) and Symmetric PPO (SPPO), re- spectively. The meanings of a and [3 align with SCE, where a represents the degree of actively training a policy, and 3 serves as auxiliary support to stabilize the entire learning process. In the following section, we analyze the gradient of the two types of losses.
(²)).
ad-
4.3. Gradient Analysis
(t) and the corresponding advantage A(s, (9))= = A10), the sample-wise reverse A2C (RA2C) loss is:
sampled action probability "g
For an input x and the corresponding correct label k. the cross entropy (CH) loss gradient is [e[ V Vopo(k|xx) Smaller po values aggressively increase the magnitude of the gradient. CE loss rapidly increases uncertain predictions. If there is no noise, this method is correct, but it may lead to incorrect predictions on noisy datasets and excessive over- fitting (Zhang Sabuncu, 2018). A2C and PPO losses also have the same issue. For A2C, the gradient is simply mul- A(s.a)
ΣSENANA if A(O 20
()) A Z, -Tg
Leaze (T))
(7)
(1) A0) 2,
ΣjENANA Tg if A <
tiplied by an advantage A, i.e., (a(s)) (e In the case of PPO, the magnitude of the gradient tends to increase as the probability of an action decreases. Consider a sample that passes the clipping function: the difference between Told and is within the bound, As the denominator Fold gets smaller, the magnitude of the gradient increases.
For a positive advantage A, the difference between A2C's loss A log π and CE loss log p is that A2C can be con- sidered CE multiplied by the advantage. In terms of gradients, A is a constant, A2C reflects the information A times more strongly than the CE loss. Thus, we also re- flect the reverse direction A times more strongly. Similarly, (8) since PPO has Tala term in the loss, the sample-wise reverse
The symmetric RL loss gradient analy- sis aligns with the analysis of SCE, For simplicity, we set a and 33 to 1 and examine the gradient direction for two types of A2C loss (RL and reverse RL) with respect to the action logits z. We use the notation defined in Section 4.1 and introduce the case when A 0. For the full derivation including SPPO and A < 0, please refer to Appendix A. The sample-wise SA2C loss is:
Detailed Analysis:
PPO (RPPO) loss just introduces the additional constant (6) old for a sampled action probability To to consider the
(6)
same amount of reverse information:
EyEWNN 2 A
if A() 0
(1) L (1)) , oodd
A000
Σ ΣSENAA (0) <0
Lea2e Luze + Lraze
(10)
(8)
We define the symmetric RL loss, which consists of the orig- inal RL loss (A2C or PPO) and the corresponding reverse
The gradients for each part are:
(L (1)())
-
if (II)
We do not consider cases where the advantages he- cause they do not affect policy updates.
Ozy
1(07(u),
if Fy
5

---

Symmetric RI Loss for Robust Learning
Table Mean final scores and standard errors (over the last 10 episodes) of PPO and SPPO on Atari games, without and with binary symmetric channel (BSC) noise with crossover probability of 0.1 across seeds. Full results can be found in Table 10.
WITHOUT NOISE
<~ BSC((.1))
PPO
SPPO
PPO
SPPO
ALIEN
1128 + 105
1081 79
525 26
713 + 26
CENTIPEDE
2961 + 379
3694 + 224
4759 - 257
7525 + 769
CRAZYCLIMBER
86764 + 3568
103588 + 2871
71144 11060
99810 + 2487
GRAVITAR
371 47
442 + 67
269 +
332 + 61
QBERT
4352 + 128
4412 + 282
2827 + 1927
4020 2415
MSPACMAN
837 +
1204 86
704 41
1011 =
-T (T(() 1),
tasks (Catto, 2011) (Section 5.2) with continuous action spaces using Stable-Baselines? (Raffin et al., 2021). In these control tasks, we also create a variant of each that in- troduces reward noise, hypothesizing that will create more confusion in advantage prediction. SPPO performs better than SA2C for various reverse RI. loss hyperparameters f3. We also evaluate our method on IMDB and TL;DR datasets using TRIL (Chang al., 2023) to determine whether approach is practical for LLM tasks. We primarily present the experimental results for PPO in the main paper. In the latter part of this section, we analyze why our method works better with PPO than A2C (Section 5,4), conduct hyper- parameter sensitivity tests, and examine the training cost (Section 5.5).
O1 (A)))
i and A >
(12)
Özy
- A
i + and A >0
Thus, the SA2C loss gradient is:
(T 1) 10) 7x (1 (I
Vu < if y and A
Veraze <0
bnace 0zy
A0 -A (T(1)
Vn 20
Vi >
i + y and A > 0
(13)
5.1. Discrete Action Space Tasks
For both cases, the gradient directions of the RL (A2C) loss and the reverse RL (RA2C) loss are aligned. When i y and A > 0, the gradient of the RA2C loss is 1), reaching its maximum magnitude at ((1) 0.5 as parabolic function. This means that the accelerator helps the probability (1) increase most rapidly when the action to take is ambiguous. When i o and A > 0, the probability of actions other than (+) is re- duced, and this reduction is influenced by the confidence of both (+) and x(v), Specifically, the gradient of the RA2C loss ( (+())=(( (()), When both (6) and (()) are 0.5, representing the most ambiguous predictions, the acceler- ator aids the A2C loss in reducing r(v) most effectively. Thus, the RA2C loss helps deviate from ambiguous pre- dictions as an accelerator. SPPO's s loss gradients are also aligned like SA2C and follow the same mechanism (See Appendix B.2).
We first conduct experiments on Atari games (Mnih et al., 2016) that the action spaces are discrete to evaluate SPPO and SA2C. We primarily select 22 games based the reported score for A2C in Schulman et al. (2017), focusing games where the A2C scores are not close to 0, as this allows us to demonstrate meaningful score changes.
To introduce some reward noise, we simply flip the reward from 0 to I or from denote this noise setting as Binary Symmetric Channel (BSC). This setting is analogous ranking-basee feedback (Ouyang et al., 2022) from humans or AI, where evaluators may have different preferences, re- sulting in reversed scores. We observe that SA2C shows marginal improvements (Table 7), with a narrow range of effective hyperparameter values. In contrast, SPPO per- forms well in both noise-free and noisy environments (See Section 5.4 for discussion). Table presents partial results, while the complete results for SPPO, including training curves (Figure 4), can be found in Table 10. SPPO achieves 16 out of 22 wins in noise-free settings and 19 out of 22 wins in noisy settings.
with probability of 10%. We
potential problem in
5. Experiments
To validate the effectiveness of our algorithm, we conduct experiments on various tasks and models of diffferent scales. First, we experiment on Atari games (Mnih et al., 2013) featuring discrete action spaces (Section 5.1), as well as Mu- JoCo benchmark tasks (Todorov et al., 2012) and Box2D
6

---

Symmetric RL Loss for Robust Learning
Table Mean final scores and standard errors (over the last 10 episodes) of PPO and SPPO on Atari games, without and with binary symmetric channel (BSC) noise with crossover probability of 0.1 across 5 seeds. To leverage the reverse RL loss, we discretize the continuous action space. DPPO is added as another baseline (a 1.0, B 0.0). and DSPPO is our proposed method. Full results can be found in Table 12 and 13.
c <> NV 0.05²)
ANT
HOPPER
HALFCHEETAH
HUMANOIDSTANDUP
PPO
601 47
1936 + 147
2068 = 208
80945 2130
DPPO
1897 + 86
2153 + 106
2722 = 188
146038 + 1841
DSPPO
2095 102
2333 + 109
3118 + 195
145974 1: 2520
WALKER2D
SWIMMER
WALKEAL
LUNARLANDERCNTTIUOR
PPO
1270 107
44 + 3.0
158 15.2
181 + 13.8
DPPO
3419 = 100
57 + 3.6
274 + 7.1
281 + 5.7
DSPPO
3523 = 129
72 + 5.1
267 + 8.8
294 + 3.3
5.2. Continuous Action Space Tasks
ment classifier (Sanh et al., 2019) is used as a reward model to evaluate how positive a provided text is. The base policy is GPT-2 (Radford et al., 2019), which we fine-tune using PPO or SPPO. We evaluate this model based on the reward score and perplexity. SPPO shows improvement in both reward score and perplexity compared to PPO.
Next, we perform experiments on MuJoCo benchmark (Todorov et al., 2012) and Box2D (Catto, 2011) contin- uous action space environments. To utilize the reverse RL loss, we need other action probabilities for sampled action probability. However, conventional RL. uses a multivariate Gaussian distribution as policy, so it cannot provide the other action probabilities. Thus, we discretize the contin- uous action space (Tang & Agrawal, 2020), naming these methods DA2C and DPPO, and add them as additional base- line comparisons.
The second RLHF task is TL;DR summarization (Völske et al., 2017). The objective is to summarize Reddit posts. The reward model is fine-tuned GPT-J (Wang & Komat- suzaki, 2021) with LoRA adapters (Hu et al., 2021) by Chang et al. (2023). The training dataset for this reward model is the filtered dataset with additional human prefer- ence data used in Stiennon et al. (2020).
Note that discretizing the continuous action space generally works better than the original RL methods like A2C and PPO for these tasks if the continuous action space is dis- cretized with a sufficient number of bins. This discretized distribution can represent more complex distributions than diagonal Gaussian distribution (where the covariance is diagonal). We apply the reverse RL loss to both DA2C and DSPPO.
The base policy model an open-source GPT-J model (CarperAI /ppanaippnii...mm summarize tidr__ddddt______ _ft) with added LoRA adapters. Note that the open-source GPT-J mode often outputs empty summarizations for most evalua- tion data. Therefore, we report results after 10 epochs of RL updates as an alternative to SFT, as it begins to consistently summarize posts. We evaluate SPPO based on reward score, perplexity, and win rate. This win rate is judged by GPT-4 Turbo (OpenAI, 2024) by comparing the generated output and reference text. Even though the perplexity of SPPO is slightly higher than that of PPO, there is an improvement in the reward score and significantly increased win rate.
Since the reward functions in these environments are highly engineered, we perturb the reward function with Gaussian noise with a mean of O and a standard deviation of 0.05. Table 2 shows partial results for SPPO under noise settings. The full experiment results are in Table 12 and 13. Similar to the Atari game results, SA2C without noise shows tied performance in the noiseless setting, and improvements when the reward noise is introduced. SPPO consistently shows robust performance gains across wide range of / values for both settings.
We also conduct experiments using Qwen2-0.5h as the pol- icy model with a LoRA adapter, employing the same reward model and hyperparameters: a = 0.5 and B = (0.2, 20.0] for SPPO. Specifically, 3 0.2 is used for GPT-J, and 3 = 20.0 is used for SPPO in the continuous tasks. Overall, SPPO outperforms PPO (Figure 3). Notably, SPPO demon- strates a significant performance boost with / = 20.0, al- though its performance drops sharply after 300 epochs. In contrast, PPO and SPPO with 8 = 0.2 show a similar per- formance drop around epoch 900.
5.3. RLHF Tasks
The final tasks are RLHF tasks to assess our method's 's ap- plicability to large language models. The first task, IMDB positive sentiment, aims to generate positive sentiment con- tinuations for movie reviews (Maas et al., 2011). The senti-
7

---

Symmetric RL Loss for Robust Learning
Table RM Score indicates the reward model score, Perplexity measures the uncertainty of the model, and Win Rate is judged by GPT-4 Turbo by comparing the generated output and reference text. We use 4 different random seeds for each task.
IMDB SENTIMENT
TL;DR SUMMARIZATION
RM SCORE (1)
PERPLEXITY (L)
RM SCORE (t)
PERPLEXITY (4)
WIN RATE (t)
SFT
0.54 = 0.00
33,02 0.09
5.83 = 0.02
18.35 + 0.02
42,00 2.58
PPO
0.89 = 0.02
41.09 + 0.43
5.94 = 0.08
19.08 + 0.17
43.25 + 3.82
SPPO
0.92 + 0.01
40.60 + 0,44
6.13 + 0.02
19.27 + 0.21
52.50 + 2.40
off-policy, affecting advantage estimates. Second, PPO of- ten uses advantage normalization to restrict large advantage values from being involved with policy updates to stabilize the learning process. In addition, PPO often uses smaller mini-batch sizes (e.g.. 64), whereas A2C uses the entire dataset for policy updates. Many popular RL code baselines, such as Stable Baselines3 (Raffin et al., 2021), RL4LMs (Ramamurthy et al., 2023), TRL. (von Werra al., 2020), and TRLX (Havrilla et al., 2023) use PPO advantage normal- ization by default, whereas A2C does not. Our experiments the usefulness of advantage normalization also show that the performance increase in IMDB is greater than the performance decrease in TL:DR (Appendix 14).
Owen2 0.5B TL;DR Summarization
6.0
5.8
Reward
5.6
5.4
5.2
SPPO g-0.5, 2-20.0
SPPO a=0.5, p=0.2
PPO
5.0
0
50
100
150
200
250
300
Epoch
We cxamine the ratio of advantage sign changes before and after normalization for PPO in Atari games and MuJoCo tasks (Figure 2). This ratio varies across different environ- ments. The ad advantage sign changes usually exceed 5% for Atari games and 10% for MuJoCo and Box2D environments. These changes introduce the confusion, which makes the reverse RL loss more effective for PPO. This observation aligns with our motivation for using symmetric RL loss to handle noisy data, similar to how it is addressed in noisy classification tasks in supervised learning.
Figure 3. TL;DR summarization results with Qwen2-0.5B across 4 different random seeds. SPPO with 20.0 shows sharp performance surge, followed by a drop after 300 epochs. A similar drop accurs for SPPO with 0.2 and PPO around epoch 900.
In the introduction section, we mention that RLHF or RLAIF have additional errors due to a trained reward model. We check whether the trained reward model in TL;DR has reward prediction errors. Figure shows a dramatic exam- ple: the generated summary sample (left) and the middle sample were both empty, but their rewards show huge gap. The middle sample scores (6.66) better than those learned with an SPPO score 13). Wrong summaries, like empty, can score higher than summarized text (right). These cases are observed very often. This makes the RL training procedure more noisy and means that the sign of advantage changes depending on how the batch is composed. The more detailed texts for these samples are available in Appendix D.
Additionally, since A2C uses the entire dataset (rather than using advantage normalization with small batches) for the policy updates, it introduces less confusion in advantage prediction. As a result, SA2C demonstrates performance comparable to A2C in settings without reward noise (Table 7 and 8), and improvements in settings with reward noise (Table and 9), where advantage estimation is more likely to be confused.
5.5. Hyperparameters and Training Cost
5.4. Why SPPO Works Better Than SA2C
Although the symmetric RL loss introduces three additional hyperparameters (Equation 4.2): a, B, and Z, we simply fix 0.5 in all experiments to reduce the overall mag- nitude of the symmetric loss. Additionally, since / and Z are constants that are multiplied together, we can fix one and adjust the other. For example, (B = 1.0, Z = 1.0) and (3 = 10.0, Z -0.1) yield the same results. In our experiments, we fix Z 1 and adjust 3 to determine the influence of the reverse RL loss.
The motivation for using the reverse RL loss is to address the issue of ambiguity in advantage predictions (Section 4.3). We hypothesize that the PPO advantage prediction (sign) is less consistent than in A2C during policy updates, but this does not mean that PPO is worse than A2C. There are two main reasons why consistency is not maintained. First, PPO has improved sample efficiency compared to A2C, but after the first epoch, subsequent updates become
8

---

Symmetric RL Loss for Robust Learning
We test the sensitivity of for SPPO on Atari games with and without noise in the rewards. Table 11 presents the per- centage improvements compared to PPO. We exclude exces- sively large improvements (e.g., 2000%) to avoid skewing the average. These significant improvements typically result from PPO's training failure, while SPPO remains stable (Go- pher and WizardOfWor Figure 4). Fixing a = 0.5 and Z = -1, we vary 3 and observe consistent improvements, demonstrating SPPO's robustness across hyperparameters. Also, we use the default values of Stable Baselines3 (Raffin et al., 2021) ) for the other RL hyperparameters; more details can be found in Appendix C.I.
human feedback in llss, 2024.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite, K., Lovitt, L.,, Sellitto, M., Elhage, N., Schiefer, N., Mercado, N., DasSarma, N., Lasenby, R., Larson, R., Ringer, S., Johnston, S., Kravec, S., Showk, S. E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S. R., Hatfield- Dodds, Z., Mann, B., Amodei, D., Joseph, N., McCan- dlish, S., Brown, T., and Kaplan, J. Constitutional ai: Harmlessness from ai feedback, 2022.
The symmetric RL loss introduces the reverse RL loss term, which is essentially another form of cross-entropy that does not significantly increase training time. In practice, there is no increase in training time for the continuous tasks dis- cussed in Section 5.2 and the LLM tasks in Section 5.3, and a 10-20% increase for the Atari games in Section 5.1.
Byun, J.-S. and Perrault, A. Normality- guided distributional reinforcement learning for continuous control, 2024. URL attps://azzen wwwp/rgg////aeeee/e/////eeeeee/e /200. 13125.
6. Conclusion
Byun, J.-S., Chun, J., Kil, J., and Perrault, A. Ares: Al- ternating reinforcement learning and supervised fine- tuning for enhanced multi-mone chain-of-thougle rea- soning through diverse ai feedback, 2024, URL https: //arxiv. org/ abs 2abb//2/ 2407. 00087.
We present Symmetric RL loss, inspired by Symmetric Cross Entropy (SCE) (Wang al., 2019) from supervised learning, to enhance RL robustness. By incorporating re- verse information through SCE, we develop SA2C and SPPO, extending standard A2C and PPO algorithms. We test SA2C and SPPO on various discrete and continuous action space tasks and further evaluate SPPO on RLHF tasks like IMDB positive sentiment and TL;DR summariza- tion. Our results show that SPPO consistently outperforms PPO.We We attribute this PPO's off- off-policy components and advantage normalization with small batch sizes, which cause advantage sign changes (confusion). SCE helps stabilize training, addressing these challenges.
Catto, E. Box2d, a 2d physics engine for games, 2011, URL http://box2d.co org.
Center, O. S. Ohio supercomputer center, 1987. URL
Chakraborty, S., Qiu, J., Yuan, H., Koppel, A., Huang, F, Manocha, D., Bedi, A. S., and Wang, M. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences, 2024.
Acknowledgments
Chang, D., Brantley, K., Ramamurthy, R., Misra, D., and Sun, W. Tril: Transformers reinforcement and imitation learning library. https: /ttp:::///tthbb.... com/ Cornell -eaelllRRRrrriii 2023.
The authors would like to thank the Ohio Supercomputer Center (Center, 1987) for providing the computational re- sources used in this research.
Chen, X., Zhu, Z., and Perrault, A. The distributional re- ward critie architecture for perturbed-reward reinforce- ment learning, 2024. URL https: https://arxiv..rrt abs 2202..05777
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Chung, H. W. Hou, L., Longpre, S., Zoph, B., Tay, Y.. Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J.. Roberts, A., Zhou, D., Le, Q. V., and Wei. J. innructtio-iinetuueed Scaling instruction finetuned language models, 2022.
References
Ahmadian, A., Cremer, C., Gallé, M., Fadace, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. Back to basics: Revisiting reinforce style optimization for learning from

---

Symmetric RI Loss for Robust Learning
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I,, and Florence, Palm-e: An embodied multimodal language model, 2023.
Ma, X., Huang, H., Wang, Y., Romano, S., Erfani, S. M., and Bailey, Normalized loss functions for deep learning with noisy labels. CoRR, abs/2006. 13554, 2020, URL https:
13554.
Maas, A. L., Daly, E., Pham, P. Huang, D., Ng. A. Y., and Potts, C. Learning word vectors for sentiment analy- sis. In Lin, D., Matsumoto, Y., and Mihalcea, R. (eds.), Proceedings of the 49th Annual Meeting of the Associ- ation for Computational Linguisties: Human Language Techmologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL https: httpp:://aclaathoossyy. wwt
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization, 2024.
Ghosh, A., Kumar, H., and Sastry, S. Robust loss func- tions under label noise for deep neural networks, 2017.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor- critic: Off- Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018.
Mai, V., Mani, K., and Paull, L. Sample efficient deep reinforcement learning via uncertainty estimation, 2022. URL htwps:tts////cc //axxiv org/aba
01666.
Havrilla, A., Zhuravinskyi, M., Phung, D., Tiwari, A., Tow, J., Biderman, S., Anthony, Q., and Castricato, L. trlX: A framework for large scale reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 5578-8595, Singapore, December 2023. Association for Computational Linguistics. doi: URL https: ac antaoaooooooo logy- orgo wtgg/222/ 2023 emn p-ma--mm--- 5330.
Mnih, V. Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. A. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. URL abs/1312.5600
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lilli- crap, T. P., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. 2016. URL http: /tatt:://rxiv... org/abs/1600 01783.
10. 10.18653/v1/2023 wwtp:223emmmmmmmmmeme m30.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., and Chen, W. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685, 2021. URL https://arat org/ abs///b///// /bs./1/5......... 09685.
CORR,
Obando-Ceron, J., Bellemare M. G., and Castro, P. Small batch deep reiffrrceeet Jeram/ 2023. URL https:
Huang, K., Altosaar, J., and Ranganath, R. Clinicalbert: Modeling clinical notes and predicting hospital readmis- sion, 2020.
/ //arxiv..rr/abb//23333. 03882.
OpenAl. Gpt-4 technical report, 2024. URL https://
arxiv. hrtp:oo/////a////// 08774.
Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Lan- guage models as zero-shot planners: Extracting action- able knowledge for embodied agents, 2022.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C. L.,, Mishknn, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M. Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback, 2022.
Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., Bishop, C., Hall, E., Carbune, V., Rastogi, A., and Prakash, S. Rlaif: Scaling reinforcement learning from human feedback with ai feedback, 2023a.
Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y., Boutilier, C., Abbeel, P., Ghavamzadeh, M., and Gu, S. S. Aligning text-to-image models using human feedback, 2023b.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
Lee, P., Bubeck, S., and Petro, J. Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine. New England 2023c. doi: 10. 0010566NE//s722141884, URL https: www. nejm. org/ http/////ff///1/// 1056 /NEJMsx2214184
Journal of Medicine, 388(13):1233-1)
Rafailov, R., Sharma, A., Mitchell, E.,, Ermon, S., Manning, D., and Finn, C. Direct preference optimization: Your language model is secretly reward model, 2023.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step, 2023.
Raffin, A. baselines3 zoo. https: / giithub.... com/
DLR- D wtw.l-baaeeeeeee
2020.
10

---

Symmetric RL Loss for Robust Learning
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Ssable-baselines:: Reliable rein- forcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http: html.
Tang, Y. and Agrawal, S. Discretizing continuous action space for on-policy optimization, 2020.
Todorov. E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSI Inter- national Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 10. 109/TTO..201..63200
//jmlr. .amml.orrgeo org/ wwt.rrrrrggpppp//./22..
Ramamurthy, R., Ammanabrolu, P., Brantly, K., Hessel, J., Sifa, R., Bauckhage, C., Hajishirzi, H., and Choi, Y. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization, 2023.
van Hasselt, H., Guez, A., and Silver, D. Deep reinforce- ment learning with double q-learning, 2015.
Völske, M., Potthast, M., Syed, S., and Stein, B. TL;;R: Mining Reddit to learn automatic summarization. In Wang, L.,, Cheung, J. C. K., Carenini, G., and Liu, F. (eds.), Proceedings of the Workshop on New Frontiers in Summarization, pp. 59-63, Copenhagen, Denmark, September 2017. Association for Computational Lin- guistics. doi: 10. 1/.18653/1/W117445000 URL https: / //aclanntool
Sanh, V., Debut, L., Chaumond, J., and Wolf, T. Distilbert, distilled version of BERT: smaller, faster, cheaper and 108, 2019. URL http://
lighter. CoRR, arxiv. www.:.orggbb..... 911.000000000000
Schaul, T., Barreto, A., Quan, J., and Ostrovski, G. The phenomenon of policy churn, 2022. URL https:// arxiv.org abs 220//2206.00000 00730.
org 7-4508.
von Werra, L., Belkada, Y.. Tunstall, L., Beeching, E., Thrush, T., Lambert, N., and Huang, S. Trl: Trans- former reinforcement learning. https:// 2022 2020.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Muchine Leurning, vol- ume 37 of Proceedings of Machine Learning Research, pp. 1889-1897, Lille, France, 07-09 Jul 2015. PMLR. https: / httpp/ppocenng....// mlr htt.i/ns.mm......s//seeee pwwps/ess/// schulman15 5. html.
Wang. B. and Komatsuzaki, A. 6 Billion Parameter Autoregressive Language Model. https://github: wtt.//om/iii//eee. kingofloll/ mesh mesh-transformer jax, May 2021.
GPT-J-6B: A
Wang. J., Liu, Y., and Li, B. Reinforcement learning with perturbed rewards. CoRR, http: http://arxiv..oo org abs B1 01032.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. CoRR, abs/1707.06347 2017. URL /tt//abs/10 / 063477.
2018. URL
Wang, P., Li, L., Shao, Z., Xu, R. X., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math- Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations, 2024.
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation, 2018.
Wang. Y., Ma, X., Chen, Z., Luo, Y., Yi, J., and Bailey, J. Symmetric cross entropy for robust learning with noisy labels. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 322-330, 2019. doi: 10. 1109/ICCV.201.000411..
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P.
Learning to summarize from human feedback. CORR, abs/2009.01325, 2020. URL abs/2009.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F. Chi, E., Le, Q. and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models, 2023.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, USA, 201 8a. ISBN 978-0262039246.
Williams, R. J. Simple statistical gralient-following algo- rithms for connectionist reinforcement learning. Machine learning, 8(3-4):22--256, 1992.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018b.
Sutton, R. S., McAllester, D. A., Singh, S. P., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. Advances in neural in- formation processing systems, pp. 1057-1063, 2000.
Zhang, Z. and Sabuncu, M. R. Generalized cross entropy loss for training deep neural networks with noisy labels. CoRR, htts/1805.07866, 2018. URL http:://aart org/abs/ ht//////111....77

---

Symmetric RI Loss for Robust Learning
A. Gradient of RL loss and reverse RL loss
Suppose there exist actions, and (+)) indicates the ith action. Let ( = To (s()))s)) denote the policy for state s. The set (s)) (x/)
(1) "B
(k)) represents the possible action probabilities set for A indicates the corresponding "g
advantage of the sampled action ( for 8. Z < 0 is constant used in the reverse RL loss to handle the computational issue
where log 0 -00. For simplicity of notation, we drop 0, and from the policy Note that A and Z are not involved with the gradient as they are constants with respect to 0.
A.1. A2C Loss
The derivation of the A2C loss Laze with respect to logits is presented as follows: For i y/
1x(()
2zy
e 2 E E
nk
(14)
(E E )2
=1 (1)
For i #
0
e
Ozy
fzy y
Σ
w== B:u
e E"
(15)
(E Ew=l eFw )2
=1)(1)
The sample-wise A2C loss is:
Lazc( (1 ((0)) -A((
log (1)
(16)
For i y.
(1)) log न
Ozy
0zy
A i
log (1)
azy
(17)
A(s (t)
T(() Ozy
(0 (T( 1) by (14)
For * y.
Th.2
(5)
log π 6)
02g
2zy
A1) Fzy
log π (6)
(18)
(+) dm!! (
(()) Dzg
((v) by (15)
In summary, we have the following form for Laze (1)
Laze (T (1) A())
(T((
1),
if = y if + y
(19)
Ozy
111) (v),
12

---

Symmetric RI Loss for Robust Learning
A.2. Reverse A2C Loss The derivation of the reverse A2C loss Lrazc with respect logits z is presented as follows:
EsENNO -x() Z, Σ EjERN T0)
if A >
(20)
if A
For i = y and (()) > O,
OToaee
Σ
A
Bzy
dzy
aT(1)
Σ
Ozy
(E(
(21)
- Σ --(0)(() by (15)
A (1) (1 - 7 ())
Zx0 Zm 1)
For i # y and A(o) > 0,
0
A
dzy
0zy jE(
vn(())
-ー z
E
Bay
(x)
=
(22)
Ozg
Ozy
jE[A:
)
A
70 + m (1) + (0)
- a x()()))) x(1)) (1
by (14) and (15)
JE(
( 1T (1)
For = y and A < 0, the only difference from Equation 21 the negative sign, thus:
O
( (z) (T() -1) by (21)
(23)
dzy
For i # y and A < 0,, the only difference from Equation 22 is the negative sign, thus:
ATraze
(z) by (22)
(24)
Ozy
In summary, we have the following form for Laze (r(1),((0))
A Zo (v) (T(2) -1),
if = and A(0) > 0
OLyaze (L (4,04)))))
A(1 (0) ((),
if i * y and A() >
(25)
1zy
( Z1)) - 1),
if i and A(0) 0
( Zz (()
i + and A(0 < 0
13

---

Symmetric RI Loss for Robust Learning
A.3. PPO Loss
The derivation of the PPO loss Lpo with respect to the logits presented as follows. The PPO loss includes clipping function and minimum operation. When these conditions are not satisfied, there is gradient.
The sample-wise PPO loss is:
x A0
(pp ((0) oodd)
(26)
(8)
Vold
For = y.
σT pp
0
((
A10
8zy
0zy
π old
(1) θm (8)
(27)
Ozy
oold
A nr (n7( 1)
by (14)
(3)
Told
For i # y.
aL
0
n
40
0zg
2zy
(i)
A
Om (4)
(28)
(1) Ōzy
T Told
A n ( (r)
by (15)
Told
A.4. Reverse PPO Loss
The derivation of the reverse PPO loss Lrppo with respect to logits is presented as follows. As with PPO, the reverse PPO loss only considers samples that pass the clipping function and the minimum operation.
From Section A.2, we have the following form for Lrppo ( (6) 1(, n T(
1 2 (-(()
if and A
(0)
Rold
Lrppoo (7,())))))))) (2)
( Zx (())
if i * and A(0) > 0
Fold)
(1)
sld
(29)
0zy
(4) Zzr (9) (x
if y and A
Fgld (i) (v)g() ())
if + y and (
oold
B. Gradient Analysis of RL Loss and Reverse RL Loss
B.1. Symmetric A2C Gradient Analysis
The gradient analysis of the symmetric RL loss follows the SCE analysis. We adopt their analysis and extend it to cover the RL loss analysis. We set and with respect to the logits We show that the gradient directions for both types are the same and that the reverse RL loss helps deviate ambiguous predictions where the probability is around 0.5. We first show how the symmetric A2C (SA2C) loss behaves. Note that Z < is a constant used in the reverse RL loss to handle log = oo.
for simplicity and evaluate the gradient direction of both RL and reverse RL losses
Traae = Teaze + Teruze
(30)
14

---

Symmetric RI Loss for Robust Learning
For y and A 0,
① Lsa2e
aa dzy
abrazo
+ dzy
Ōzy
(31)
1 (x( -1) A Zx Zz (x V.
by (17) and (21)
V
For i # y and A > 0,
Ö Teaae
T.aee
ÖTraze
Bzy
Bzy
azy
(32)
A (v)
A -x (0)
by (18) and (22)
VLase 0
Lraze >
For i y and A(0 < 0,
0l L
ØL
Ozv
dzg
Ozy
(33)
A (T)()) - -1) 2x -1) by (17) and (21)
V Vluze 20
braze >
For i * y and A < 0,
0Luze
0Lra2e
0zy
Dzy
Ozy
(34)
10) (v)
-A Zx x()
by (18) and (22)
VL.sc <0
Viraae
For the above cases, the gradient directions of the RL (A2C) loss and the reverse RL (RA2C) loss the gradients. Essentially, the RA2C loss acts as an accelerator. In the case of = y and A > 0, the gradient of the RA2C loss is - (v) (v(v) - 1), with the largest gradient magnitude at (v) = 0.5 as parabolic function. In other words, the accelerator helps the probability (+) increase most quickly when is ambiguous which action to take. In the case of i + and (a) > 0, the probability of other actions except () is reduced, and this reduction influenced by the confidence of both (*) and (v)) Specifically, the gradient the RA2C loss is A Z (2) (1), When both (+()) and x(2) are 0.5,
SCE
indicating the most ambiguous predictions, the accelerator helps the A2C loss reduce / most aggressively.
When A00 < 0, the gradient direction is simply reversed. The behavior of the gradient itself remains the same as when A > 0. In the case of i = y. RA2C decreases the probability x(1) more when (w/) is around 0.5. For i * RA2C helps increase (v) more when both 7: and (2) are ambiguous (both around 0.5).
B.2. Symmetric PPO Gradient Analysis
Leppo = Lipo + Lyppo
(35)
For = y and 10) 0,
L azy
Opppo
Arrpoo
+
Ozy
Ozy
1 nr (T7(1) -1 1)
A 2242Zz000 (w) -1)
(36)
by (27) and (29)
(i)
(2) old
Told
V Lppo <0
V Lrppo < 0
15

---

Symmetric RI Loss for Robust Learning
ior + y and A 0,
Lappo
ppo Bzy
ÖLyppo dzy
+
Dzy
A x1() r
Zm)
(37)
by (28) and (29)
(3)
(4)
olld
Fold
Vlppo 0
V Lirpo >
For = porizy
and A(0) <
LLppo
@ppp dzg
aLpppo
Ozy
Ozy
10 (1)) (8) (6)
1)
A Z (v) (T(M) -
(38)
+
by (27) and (29)
π oold
Vold
V Lppo
V Vaypoo
For i * y and 166 < 0,
Leppo
aLppo dzg
aLrppo
+
d2g
dzy
(t)
A (v))
(39)
by (28) and (29)
(4)
(6)
Told
#ald
Ippe
V Irpppo
(4)
Basically, the mechanism of RPPO is the same as RA2C, except for oldd which does not change the gradient sign. Therefore, RPPO also helps PPO deviate from ambiguous predictions, acting an accelerator.
16

---

Symmetric RL Loss for Robust Learning
C. Experimental Setups and Results
C.1. Hyperparameters
We primarily follow the hyperparameter settings of RL Baselines3 Zoo (Raffin, 2020). Most hyperparameter values remain unchanged across environments. Only a and are adjusted for the reverse RL loss. For SA2C without noise, we use (ax 0.5, 3 5.0) for all environments. For SA2C with noise, we use (a 0.5, 8 1.0) for (Alien, MsPacman, Qbert, TimePilor, VideoPinball, Assault, Gravitar, StarGunner, UpNDown), and (a 0.5, 8 1.0) for others. For SPPO without noise, we use (ar 0.5, 3 1.0) for all environments. For SPPO with noise, we use (a< 0.5, 3 10.0) for all environments. We do not use any GPU for Atari games.
Atari Games:
Table 4. Hyperparameters for Atari games
WITHOUT NOISE
BSC(O.1)
SA2C
(a = 0.5.3 = 1.0)
(ALIEN, ASSAULT, GRAVITAR, MSPACMAN, QBERT, STARGUNNER, TIMEPHLOT, UPNDOWN, VIDEOPINBALLL) ALL OTHERS EXCEPT THOSE MENTIONED ABOVE
(a 0,5,3 5.0)
ALL ENVIRONMENTS
SPPO
(a 0.5, 3 1.0)
ALL ENVIRONMENTS
(a 0.5,3 3 10.0)
ALL ENVIRONMENTS
MuJoCo and Box2D: We use n_envs 4 and n.steps 8 for A2C and SA2C. We follow
default
hyperparameters (Raffin et al., 2021) for other settings. Only and are adjusted for the reverse RL loss. For table visibility, let (Ant = 1, BipedalWalker = 2, HalfChectah 3. Hopper 4, HumanoidStandup 5, InvertedDoublcPendduuum 6,
Lumarlanderrooniiuuuu = 7, Swimmer = 8, Walker2d = 9}. We do not use any GPU for these tasks.
Table 5. Hyperparameters for MuJoCO and Box2D environments
WITHOUT NOISE
~~ (O,00.05
SA2C
(o 0.5, 8 0.2)
(1, 9) (2.
(1)
(a 0.5, 8 0.5)
(a 0.5, 3 5.0)
(3, 6, 7)
(7)
(a = 0.5, = 10.0)
(2, 3, 4, 5, 6. 8, 9) ALL ENVIRONMENTS ALL ENVIRONMENTS ALL ENVIRONMENTS
Z = -1
ALL ENVIRONMENTS
(TIMESTEPS= 2e6) (NUMBER OF BINS= 11)
ALL ENVIRONMENTS
ALL ENVIRONMENTS
SPPO
(a 0.5, 8 20.0)
ALL ENVIRONMENTS
(1,7)
(a = 0.5, 8 25.0)
(2, 3, 9) (4, 8)
(a = 0.5.B 8 = 50.0) 7 = -1
ALL ENVIRONMENTS
ALL ENVIRONMENTS
(TIMESTEPS= 1e6)
(2
(2, 6)
(TIMESTEPS- 2c6)
(1. 4, (9)
(1, 4. 8) (9)
(TIMESTEPS= 5c6)
(TIMESTEPS= 1c7)
(5)
(S)
(NUMBER OF BINS= 11)
ALL ENVIRONMENTS
ALL ENVIRONMENTS
We basically use the provided implementation (Chang al., 2023) and follow their hypepparameters, with the addition of the advantage normalization step for PPO. The scripts used in our experiments are available in the code repository for further detail. We use a single Nvidia A100 (80GB) for our experiments.
IMDB and TL;DR:
17

---

Symmetric RI Loss for Robust Learning
Table 6. Hyperparameters for IMDB positive sentiment and TI.DR summarization
IMDB
TL;DR
PPO
MODEL:
GPT-2
GPT-J
UPDATES:
60
100
TRAJECTORIES PER UPDATE:
112
64
EPOCHS PER UPDATE
5
4
BATCH SIZE
28
32
LEARNING RATE
5E-6
5E-6
DISCOUNT FACTOR
0.99
GAE LAMBDA
0.95
0.95
CLIP RANGE
0.2
0.2
SPPO
(a 0.5, 8 0.4)
(a 0.5, / = 0.2)
C.2. Experimental Results: A2C and SA2C
Table 7. Mean final scores and standard erross (over the last 10 episodes) of A2C and SA2C on Atari games, without and with binary symmetric channel (BSC) noise with crossover probability 0.1 across seeds.
WITHOUT NOISE
BSC(0.1)
A2C
SA2C
A2C
SA2C
ALIEN
913 + 100
771 + 51
481 + 72
496 + 37
ASSAULT
1538 + 199
1061 + 41
287 226
399 + 133
ASTERIX
2308 + 86
2377 + 164
1403 + 305
1430 + 208
BEAMRIDER
1121
1335 +
1087 + 339
902 196
CENTIPEDE
3588 1 430
3574 1: 295
3108 1 243
3540 1 194
CRAZYCLIMBER
98774 1: 2516
99330 1. 4371
93042 1 8711
97058 1 6251
DEMONATTACK
4309 3:5
5017 1 625
30 1 21
19 1
FROSTBITE
255 2
257 + 3
241 +
286 +
GOPHER
960
1036 + 138
91
996 = 114
GRAVITAR
18
201 16
279 + 48
183 36
KRULL
1387
7672 + 819
7564 + 486
6337 + 754
MSPACMAN
1
1495 104
926 + 44
916 100
NAMETHISGAME
5945
5614
2280 + 257
2372 + 141
QBERT
1646 + 240
2103 + 261
620 96
641 77
RIVERRAID
4368 + 582
5461 1 456
1609 1 65
2511 1 190
ROADRUNNER
14971 + 1396
18624 +: 1812
5606 1788
3830 + 1517
SEAQUEST
836 7
988 +
650 +
653 22
SUANGUNNER
2222 - 114
1766 + 120
1194 + 645
622 -
TIMEPILOT
3992 + 198
3116 + 137
2232 + 259
3288 + 106
UPNDOWN
8313 1 1544
1638 1 761
4228 + 1187
7093 1 2772
VIDEOPINBALL
24948 3038
19618 1 1888
20319 1 2157
25035 1 3914
WIZARDOFWOR
824 136
674
125
496
752 156
WINS (SA2C)
12 22
15 / 22
18

---

Symmetric RI Loss for Robust Learning
Table Mean final and standard errors (over the last 10 episodes) of A2C and SA2C MuloCo benchmark tasks and Box21 environments without Gaussian noise actoss 30 seeds.
WITHOUT NOISE
ANT
HOPPER
HALFCHEETAH
HUMANOIDSTANDUP
A2C
757 +: 116
1410 + 112
1393 + 163
121850 + 4264
DA2C
2220 +: 96
1944 + 116
2325 209
152135 + 3937
DSA2C
2287 - 94
1797 + 139
2266 = 203
159142 + 129
WALKER2D
SWIMMER
BIPEDALWALKER
LUNARLANDER
A2C
1348 + 130
95.8 + 19.0
124 = 23
79.0 + 20.2
DA2C
2131 1 154
142.4 17.0
234 1: 22
176.7 20.9
DSA2C
1662 +: 128.5 16.2
274 16
221.2 10.7
INVERTEDDOUBLEPENDUIE
A2C
1670 + 500
DA2C
9139 + 94
DSA2C
9145 1
Table Mean final scores and standard errors (over the last 10 episodes) of A2C and SA2C MuJoCo benchmark tasks and Box2D environments with Gaussian noise (mean and standard deviation 0.05) across 30 seeds.
~ (O00 0.05²)
ANT
HOPPER
HALFČHEETAH
HUMANOIDSTANDUP
A2C
673 t 108
1083
1610 + 163
101064 + 4933
DA2C
1296 80
1323 + 87
1510 + 126
126241 + 3973
DSA2C
1520 83
1307 + 102
1696 + 163
128064 + 4391
WALKER2D
SWIMMER
BIPEDALWALKER
158 + LUNARLANDER
A2C
786 86
28.9 4.4
-3.7 15.9
DA2C
1599 138
36.8 + 4.6
210 = 21
106 + 20
DSA2C
1423 129
53.1 + 7.0
222 +: 20
179 + 12
INVERTEDDOURLEPENDDEE
A2C
3852 + 634
DA2C
7900 + 364
DSA2C
8323 + 217
19

---

Symmetric RI Loss for Robust Learning
C.3. Experimental Results: PPO and SPPO
Table 10. Mean final scores and standard errors (over the last 10 episodes) of PPO and SPPO Atari games, without and with binary symmetric channel (BSC) noise with crossover probability of 0.1 across sceds.
WITHOUT NOISE
(01
PPO
SPPO
PPO
SPPO
ALIEN
1128 105
1081 + 79
525 26
713 + 26
ASSAULT
3134 = 193
3385
2327 + 401
3698
ASTERIX
2599 = 101
2976 + 150
1272 + 106
1739 + 329
BEAMRIDER
2176 = 251
1635 + 404
1828 = 130
1580 = 96
CENTIPEDE
2961 379
3694 + 224
4759 257
7525 + 769
CRAZYCLIMBER
86764 = 3568
103588 2871
71144 + 11060
99810 2487
DEMONATTACK
7872 = 302
7901 + 455
14
132 + 13
FROSTBITE
268 +
286 + 6
509 + 108
23 + 16
GOPHER
787 = 48
875 + 78
478 = 38
7765 + 3366
GRAVITAR
371
442 + 67
2
332
KRULL
6628 = 417
7578
5602 481
9015
MSPACMAN
837 = 62
1204 1 86
704
1011 1 52
NAMETHISGAME
5665 = 280
5423 4 63
2681 + 143
5187 +: 247
QBERT
4352 = 128
4412 + 282
2827 + 1927
4020 = 2415
RIVERRAID
6128 = 272
6343
2460 127
3998
ROADRUNNER
28382 = 2254
22562 2875
1204 = 157
3830 = 1230
SEAQUEST
902 +
888 6
652 + 16
814 15
GUNNER
11848 722
14746 + 1876
1514 110
23250 + 6292
TIMEPHLOT
3850 151
3548 220
3506 318
3936 420
UPNDOWN
58289 21226
126830 27534
8815 + 1395
73490 33553
VIDEOPINBALL
22408 = 4292
29485 2851
31680 + 2318
37048 6989
WIZARDOFWOR
3186 256
3762 1 387
940 1 158
4442 1 1332
WINS (SPPO)
16 / 22
19 / 22
Table 11. Percentage improvement of SPPO over PPO. The percentage improvements are computed across 22 Atari games. We simply fix 0.5 reduce the total loss magnitude and vary / control the impact the RI. loss. We exclude very large improvements (e.g.. 2000%) from calculating the average. This large improvements result from PPO's significant learning failures.
a 0.5 IS FIXED
3 == 0.5
0 = 1,0
B 3= 5.0
10.0
25.0
SPPO UNDER 0% NOISE
7.83%
10.15%
24.98%
21.52%
18.92%
SPPO UNDER 10% NOISE
1.74%
21.89%
148.46%
166. 73%
136.50%
20

---

Symmetric RL Loss for Robust Learning
Aitee
Asssuit
Astenx
2520
BearRiisee
1250
3000
2500
2000
2000
800
2003
1903
3000
430
1003
1000
500
500
timestepss
timestops
Ied
timestops
timesteps
pede
CrazyC Imber
DemonAttack
3030
10000
Frstbrite
90000
8000
5030
dasoo
4000
20000
306
3030
timanioss
Caspher
Caavitarr
Kruli
MaPacman
33000
10000
3420
2220
400
8000
3000
300
0000
2500
100
4000
Garee
Obert
Diverraid
RoadFuunnar
6000
0003
0003
40500
5000
6003
25000
4020
4000
4000
2000
2003
2030
timestteps
Senqueest
StanGGnner
Timeiriot
UoNDown
200000
30000
20003
4000
352000
20000
15000
2000
300000
5
ocal Gimesteps
100a timesteps
timestops
timestops
VideoPibail
WizardDWWor
30000
2730
25300
20300
2003
37500
MS01 Cimesacpe
local Gimesaagp
Figure 4. Result of training plots for SPPO and PPO for Atari games. The blue line indicates the original PPO without any added noise, while the orange line represents SPPO without added noise. The green line indicates PPO with 10% noise, and the red line represents SPPO with 10% noise. We fix = 0.5 for all environments, with 8 = 1.0 for the experiments without noise and 3 = 10.0 for the noise environments.
21

---

Symmetric RI Loss for Robust Learning
Table 12. Mean final and standard errors (over the last 10 episodes) of PPO and SPPO on MuJoCo benchmark tasks and Box21 environments without Gaussian noise actoss 30 seeds.
WITHOUT NOISE
ANT
HOPPER
HALFCHEETAH
HUMANOIDSTANDUP
PPO
2068 + 166
2875 + 137
2282 +: 191
93763 + 3402
DPPO
2735 +: 109
2154 + 119
3478 + 279
176320 6538
DSPPO
2885 + 100
2299 + 115
4104 = 258
189301 + 5915
WALKER2D
SWIMMER
BIPEDALWALKER
LUNARLANDER
PPO
2793 + 199
112 + 5.0
247 + 8.3
134 + 10.9
DPPO
4443 1: 119
131 1 0.3
265 1: 15.5
241 +: 7.7
DSPPO
4587 +: 154
130 0.6
6.2
250 +: 6.9
INVERTEDDOUBLE
PPO
7454 + 394
DPPO
8928 + 136
DSPPO
9015 1
Table 13. Mean final scores and standard errors (over the episodes) PPO and SPPO on MuJoCo benchmark and Box2D environments with Gaussian noise (mean and standard deviation 0.05) across 30 seeds.
N(0,0.05%)
ANT
HOPPER
HALFCHEETAH
HUMANOIDSTANDUP
PPO
601 = 47
1936 + 147
2068 + 208
80945 + 2130
DPPO
1897 86
2153 106
2722 + 188
146038 1841
DSPPO
2095 102
2333
3118
145974 1 2520
WALKER2D
SWIMMER
WALKER
LUNARLADERCONTNNOO
PPO
1270 = 107
44 + 3.0
158 + 15.2
181 + 13.8
DPPO
3419 100
+ 3.6
274 7.1
281 + 5.7
DSPPO
3523 129
72 + 5.1
267 8.8
294 + 3.3
INVERTEDDOUBBLREENDIIE
PPO
8050 +: 244
DPPO
8963 +
DSPPO
9147 61
C.4. On and Off Advantage Normalization
Table 14, Comparison with and without advantage normalization over 4 different random seeds.
PPO
IMDB
TL:DR
WITHOUT A NORMALIZATION
0.77 + 0.01
6.06 + 0.02
WITH A NORMALIZATION
0.89 + 0.02
5.94 = 0.08
22

---

Symmetric RL Loss for Robust Learning
D. Examples of Reward Model Errors
Warning: This section contains harmful language.
Table Example showing trained reward model with errors that are not consistent empty outputs, the reward for an empty output is greater than that for non-empty summarization. [...] indicates omitted content for brevity.
SUBREDDIT: ARELSSIIONHIPS (SAMPLE ID: 37)
TITLE: I'M A DUMB [21] MALE AND SO I'M HAVING A LOT OF TROUBLE INTERPRETING THE SIGNALS
THAT THIS [21] GIRL MAY MAY NOT SENDING ME. A LITTLE HELP PLEASE?
POST: SO OKAY, I'M FROM NEW YORK BUT STUDY IN OREGON FOR MOST OF THE YEAR. RECENTLY
A FRIEND OF MINE WHO I WAS NOT REALLY CLOSE STARTED FACEBOOK MESSAGING ME, THAT WAS ABOUT 3 MONTHS AGO, SINCE THEN WE VE TALKED ALMOST EVERYDAY. [...] TRIED TO DO JUST THAT BUT SHE TOTALLY GAVE ME THE COLD SHOULDER; NOT BEING REALLY RESPONSIVE HA HANGING OUT, LEAVING FARLY THAT SHE WAS INTO ME JUST BECAUSE OUT OF THE BLUE SHE STARTED TALKING TO ME A LOT? IS SHE TRYING TO PLAY HARD TO GET? AM LOOKING WAY TOO INTO THIS AND MAYBE SHE WAS JUST OCCUPIED THAT WEEKEND? I REALLY HAVE NO IDEA HOW TO EVALUATE THIS. Do ANY OF YOU GUYS HAVE ANY SUGGESTIONS/IDEAS"
ETC... AM I WRONG
ORIGINAL ASSUMPTION
GENERATED SUMMARY: <EMPTY>
REWARD MODEL OUTPUT: 6,66
SUBREDDIT: ARELSSIIONHHPPAAA ADVICE (SAMPLE ID: 60)
TITLE: MY BF [23] DOESN'T SPEAK OF HIS CHILDHHOD, BUT [[F22] KNOW HE'S TRAUMATIZED.
ONCE POST: WE WERE FRIENDS FOR 10 YEARS. BEFORE WE TOGETHER. HE THAN
ABOUT HIS TERRIBLE CHILDHOOD, (HE TOLD ONLY HIS FRIENDS HIS STORY) NOw WE WE'RE COUPLE FOR QUITE A FEW MONTHS AND WELL, SOMETIMES THERE'S STUFF I KNOW THAT REMINDS HIM OF HIS CHILDHOOD, BUT IT'S LIKE HE'S FORGOTTEN THAT HE HAD TOLD ME. [...] AND STUFF LIKE WATCHING TV ABOUT RAISING CHILDREN. WE TALK ABOUT HOW WE'RE GOING TO RAISE OURS IN FUTURE AND THAT WON'T WILL STRIKING, THE THINGS HE THINKS ARE IMPORTANT ARE ALWAYS THE THINGS HIS PARENTS SHOULD HAVE DONE, TO SAVE HIM FROM THE TRAUMATIZING STUFF.) JI KNOW HE LIKES TO PUT HIS PROBLEMS FAR AWAY. BUT ON THE OTHER HAND, I'M HIS GIRLFRIEND NOW AND WE RE PRETTY SERIOUS, ISN'T KNOW SECRET/WON'S 'T TELL,, AND MOST OF ALL, ALWAYS THERE FOR HIM? WHAT DO YOU THINK? GENERATED SUMMARY: <EMPTY>
HORRIBLE
THE PARENTS ON TV. (BUT
GOOD
ABOUT MAYBE ONCE,
REWARD MODEL OUTPUT: 3.14
SUBREDDIT: R/ASKREDDIT (SAMPLE ID: 27)
TITLE: DEAR REDDIT, WHAT SILLY/IRRELEVANT/REDIIIC
FAMILY MISCOMMUNICATIONS HAVE
LEAD TO FEUDS LASTING YEARS?
POST:MY GRANDMA AND MY AUNT (HER DAUGHER-IN-AWE HAVEN'T SPOKEN TO EACH OTHER IN YEARS OVER A PHONE THAT DIDN'T GET HUNG UP. MY AUNT AND UNCLE SCREEN THEIR CALLS AND RETURN THEM ONE TIME, MY GRANDMA CALLED AND LEFT A MESSAGE THEN THOUGHT SHE HUNG UP THE PHONE, FEW MINUTES LATER- MY GRANDMA WAS TALKING WITH SOMEONE IN HER HOME AND USED THE WORD BITCH THIS WAS ALL RECORDED ON MY AUNT AND UNCLE'S ANSWERING MACHINE AND MY AUNT ASSUMED IT WAS ABOUT HER AND HASN'T SPOKEN TO NOR SEEN MY GR GRANDMA UPWARDS OF 5 YEARS. [...] WHY WASTE TIME THE TIME YOU HAVE SOMONE? WHY CONTINCE GRANDMA HAS A DAUGHTER WHO LIVES WITH HER AND LIKES TO BE IN OTHER PEOPLES BUSINESS- THINK SHE IS ALSO PART OF THE PROBLEM HERE AS SHE WON'T DROP IT EITHER. GRANDMA IS INNOCENT BUT HAS A DAUGHTER AND DAUGHTER-IN-LAN WHO WON'T TT GROW UP AND DROP IT GENERATED SUMMARY: GRANDMA AND AUNT HAVEN'T SPOKEN IN YEARS OVER A PHONE THAT DIDN'T GET HUNG UP. GRANDMA WANTS TO RECONCILE AND CLEAR THE AIR, BUT AUNT WON'T GO NEAR HER, WON'T LET HER HUSBAND AND KIDS GO THERE, AND AVOIDS.
FREQUENTLY
GRUDGE? To COMPLICATE
FURTHER,
REWARD MODEL OUTPUT: 5,40
23


Reviews:
[{"review_id": "jPzVWCMbtP", "reviewer": "Reviewer_epYB", "summary": "The paper proposes a new family of loss, symmetric A2C and symmetric PPO loss for RL tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and valuable feedback on our paper. We hope the response below addresses your concerns.\n\n**Weakness 1**\n> The approach enumerates all possible actions in the loss, which can be computationally feasible for small/medium discrete action spaces (e.g., Atari), but becomes very large and inefficient for language modeling.\n\n=> In large language models, the next token (action) is sampled via softmax, which already provides probabilities for all other (non-chosen) tokens—so there's no need for additional computation to obtain them. Incorporating the probabilities of other tokens during backpropagation introduces some additional computation, but it is roughly equivalent to adding a single MLP layer. Given the overall size and structure of LLMs, this does not significantly impact training speed. In fact, SPPO (the symmetric RL for PPO) showed faster training than PPO for some seeds—i.e., within the overhead of adding one MLP, GPU memory allocation conditions had a greater effect. We briefly address training speed in the last paragraph of Section 5.5.\n\n**Weakness 2**\n> More recent studies on LLM alignment typically use newer models (e.g., Llama 2 / 3, Qwen 2.5 or beyond) and richer feedback datasets (e.g., Ultrafeedback, Chatbot Arena open data, Nectar etc.).\n\n=> These methods were motivated from the perspective of noisy rewards in standard deep RL tasks. We show that they perform well across a range of RL tasks and limited experiments in RLHF. It is difficult to cover every setting---we think that the promising nature of this approach is established by our experiments. We promise that if the paper is accepted we will add Qwen2.5 on the TLDR task to the results table.", "claims_and_evidence": "Claims: The paper asserts that its policy gradient formulation with the newly proposed loss achieves superior or at least competitive performance in Atari benchmarks, and that using GPT-J with RLHF on TLDR/IMDB datasets demonstrates improved alignment.\n\nEvidence: For Atari, the evidence primarily comes from charts showing improved scores vs. baseline agents, which seem reasonably convincing if one accepts the standardness of Atari benchmarks. For RLHF, the evidence includes limited experimental data on GPT-J with relatively small and somewhat outdated datasets, making it less clear if the approach scales to more advanced language models and richer feedback datasets.", "methods_and_evaluation_criteria": "Methods: The main methodological innovation is the introduction of new loss functions (Equations 8 and 9) that factor in reward signals and the policy gradient. However, the approach enumerates all possible actions in the loss, which can be computationally feasible for small/medium discrete action spaces (e.g., Atari), but becomes very large and inefficient for language modeling.\n\nEvaluation Criteria: Benchmarking on Atari is fairly standard; the paper also attempts to assess alignment improvements on TLDR/IMDB datasets. However, these datasets and the GPT-J baseline may not fully reflect current state-of-the-art for language model RLHF, limiting the generalizability of the reported results.\n\nMore recent studies on LLM alignment typically use newer models (e.g., Llama 2 / 3, Qwen 2.5 or beyond) and richer feedback datasets (e.g., Ultrafeedback, Chatbot Arena open data, Nectar etc.). Incorporating these newer data sources and more modern baselines would better situate the method in the current literature and could potentially unlock more significant empirical gains.", "theoretical_claims": "The derivation looks good to me", "experimental_designs_or_analyses": "The experiments on Atari appear consistent, using standard protocols (training steps, seeds, reported scores). For the RLHF experiments, the design is somewhat limited: the use of GPT-J with TLDR/IMDB is not clearly motivated given the more up-to-date large language models and more comprehensive feedback datasets available.", "relation_to_broader_scientific_literature": "The paper builds on standard RL methods and extends them with an additional symmetric term.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Please see above", "other_comments_or_suggestions": "Please see above", "questions_for_authors": "Please see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VvyB9XIg2a", "reviewer": "Reviewer_xvu3", "summary": "Reinforcement learning (RL) training is inherently unstable due to factors such as moving targets and high gradient variance. Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAIF) introduce additional challenges. For instance, diverse preferences complicate the alignment process, and prediction errors in a trained reward model can become more severe as the LLM generates unseen outputs. These RL challenges create confusion about whether the probability of\nan action for a given state should be increased or decreased, similar to the noise in labels for classification tasks. In this work, the authors focus on RL algorithms that share learning difficulties with cross-entropy loss, especially for low-probability predictions. To enhance stability, they adapt reverse cross-entropy (RCE) from supervised learning for noisy data, defining a symmetric RL loss. They demonstrate performance improvements across various tasks and scales.\n\n# Update after rebuttal\nAll my concerns have been addressed thanks to the authors' efforts in rebuttal. In all, I decided to raise my score to weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and valuable feedback on our paper. We try to handle your questions and provide additional details below.\n\n**Please let us know if our responses resolve your questions and concern. If so, we would greatly appreciate your consideration in updating your score. We’re also happy to continue the discussion if you have further questions. Our answers are as follows:**\n\n**Question 1**\n> I don't understand why adding the reverse advantage loss could help relieve the problem of estimation error resulting from noisy human feedback and multiple sources of scaled models.\n\n=> In the Gradient Analysis section, we describe how the reverse RL loss helps the learning procedure in the presence of noise, acting as an accelerator. We believe you would agree that when noise is present, the policy lacks certainty about which action to take. Since the reverse RL loss gradient has its maximum magnitude at 50\\% (a parabolic function) and its direction aligns with the original RL loss, it helps the policy move away from this ambiguous state. Please refer to further details in Section 4.2 and Sections A.1 to A.4 in the Appendix. We provide a comprehensive derivation and analysis for all cases.\n\n**Question2**\n> Why bother to use symmetric CE loss other than directly train on large amounts of data sets of diverse tasks? \n\n=> Increasing the data amount is one potential solution, but Reinforcement Learning (RL), especially with large models, has burdensome such as generating actions and  getting the corresponding rewards. For the rewards, we would need a highly engineered reward function, human, or AI evaluators (which can also have noise). Therefore, achieving better performance using the same amount of data is obviously advantageous. The symmetric RL loss (or symmetric CE loss) facilitates this and is easy to implement. Since action (or token) sampling is done through the softmax, the other probabilities can be obtained naturally for symmetric losses.\n\n**Question 3**\n> What is the role of $Z$ in equation 7?\n\n=> $Z$ was introduced in previous work [1] on noisy classification datasets to address the issue of $\\log 0 = -\\infty$, which cannot be used when updating a neural network. To resolve this numerical problem, $-\\infty$ is replaced with a negative value $Z$, which has been shown to still satisfy the conditions of a robust loss function. In our case, we also use $Z$ to handle negative advantages (see Section 4.1 for more details)\n\n**Question 4**\n> Why SPPO still performs well in noisy-free setting?\n\n=> Compared to A2C, PPO has off-policy update parts and typically uses smaller batch sizes (e.g., 64), whereas A2C processes all data points at once (i.e., no batch). Additionally, PPO applies the advantage normalization by default, which can introduce sign changes in the advantages. While PPO is more sample-efficient than A2C, these characteristics introduce sources of noise—even in noise-free settings. Our symmetric RL loss helps mitigate these noisy factors, which can lead to confusion in advantage estimation, thereby improving performance even in clean environments (see Section 5.4 for details).\n\n**Question 5**\n> What is the benefits of advantage normalization with small batch sizes?\n\n=> While some studies [2, 3] suggest that smaller batch sizes can be more beneficial than larger ones, our main point is not directly about batch size. In practice, we commonly use batch sizes such as 32, 64, or 512 for PPO, whereas A2C processes the entire dataset without batching.However, after applying advantage normalization, the signs of the advantages can flip depending on how the batch is sampled. For example, a given sample $(s,a)$ might end up with a positive or negative advantage solely based on the composition of the batch. This introduces confusion, which can be interpreted as a form of noise. In such cases, the symmetric loss can help mitigate the impact of that noise. \n\nThat said, this doesn’t mean advantage normalization is harmful. Although it can cause sign changes in advantages, it also helps limit the influence of extremely large advantage values, thereby stabilizing the policy update process (Please refer to Section 5.4).\n\nIn summary, the symmetric RL loss retains the benefits of advantage normalization while further alleviating the instability caused by sign changes.\n\n**Weakness 1**\n> Baselines are too few for comparison in Table 2.\n\n=> We further provide results for PPO in a noise-free setting in Table 12, including an additional environment. Additionally, we report A2C results in Tables 8 and 9 to support further comparison.\n\n[1] Symmetric Cross Entropy for Robust Learning with Noisy Labels, ICCV 2019\\\n[2] The Phenomenon of Policy Churn, Neurips 2022\\\n[3] Small batch deep reinforcement learning, NeurIPS 2024", "claims_and_evidence": "The SPPO applies symmetric loss by adding traditional RL loss with a reverse RA2C or RPPO loss. And the result in Table 1 somewhat validates the claims that SPPO avoids value estimation errors due to the sign of the advantage in advantage normalization dependent on how the batch is composed.", "methods_and_evaluation_criteria": "I don't understand why adding the reverse advantage loss could help relieve the problem of estimation error resulting from noisy human feedback and multiple sources of scaled models.", "theoretical_claims": "The authors use gradient analysis to formulate the policy update and show the convergency guarantee in appendix.", "experimental_designs_or_analyses": "Experimental design is valid and analysis is comprehensive.", "relation_to_broader_scientific_literature": "It has a large impacts to the domain of RLHF with noisy reward lables and non-stationary feedback signal.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\n- Formulation is clear and the idea combines symmetric cross entropy and A2C/PPO framework.\n- It provides theoretical proof of gradient analysis of RL loss and reverse RL loss.\n\nWeakness:\n- Motivation is unclear about why bother to use symmetric CE loss other than directly train on large amounts of data sets of diverse tasks.\n- Baselines are too few for comparison in Table 2.", "other_comments_or_suggestions": "Add more recent RL baseline comparison in Table 2 and 3.", "questions_for_authors": "- What is the role of Z in equation 7. \n- Why SPPO still performs well in noisy-free setting.\n- What is the benefits of advantage normalization with small batch sizes.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rhFijKgLWg", "reviewer": "Reviewer_7vHN", "summary": "The manuscript introduces a symmetric reinforcement learning (RL) loss function designed to improve the robustness of RL algorithms like A2C and PPO. The proposed symmetric RL loss is inspired by reverse cross-entropy (RCE) used in noisy classification tasks, and the authors apply it to both discrete (Atari) and continuous (MuJoCo, Box2D) RL tasks. The experiments demonstrate that Symmetric A2C (SA2C) and Symmetric PPO (SPPO) outperform traditional A2C and PPO, particularly in environments with added reward noise.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and valuable feedback on our paper. We hope the response below addresses your concerns.\n\n**Weakness 1**\n> Insufficient Comparison with Other Robustness Methods: While the paper focuses on the symmetric RL loss, it lacks detailed comparisons with other established methods for robust RL, such as Double Q-learning or SAC, which could provide a clearer context for its benefits.\n\n=> Rather than expanding to additional types of reinforcement learning algorithms, we focused on those whose working mechanisms align with the symmetric loss formulation in noisy classification settings. Instead, we aimed to demonstrate the effectiveness of the symmetric RL loss across a diverse range of problem domains—including discrete action spaces, continuous action spaces, and NLP tasks—highlighting its applicability even to large language models.\n\n**Weakness 2**\n> Limited Theoretical Justification: The paper would benefit from a more detailed theoretical explanation of why the symmetric RL loss is particularly effective in addressing the noise in reward prediction, especially when compared to existing techniques like Generalized Advantage Estimation (GAE).\n\n=> In all our experiments, we used Generalized Advantage Estimation (GAE), and we will explicitly clarify this in the experimental section. Since GAE depends on the reward, noise in the reward can lead to confusion in advantage estimation. Our symmetric RL loss acts as an effective accelerator to mitigate this issue. We provide a detailed analysis of how this acceleration benefits the learning process in Section 4.3 and Appendix A.1 to B.2.\n\n**Weakness 3**\n\n> Hyperparameter Sensitivity: The sensitivity of the performance to hyperparameters such as $\\alpha$ and $\\beta$ is not sufficiently explored.\n\n=> We provide additional hyperparameter sensitivity analysis in Table 11 of the Appendix for SPPO, both with and without reward noise, showing consistent improvements across a range of values. Note that $Z$ and $\\beta$ can be treated as a single hyperparameter (see the first paragraph of Section 4.1 for details).\n\n**Weakness 4**\n\nScalability and Computational Cost: More extensive analysis is needed to understand how these hyperparameters affect training and performance stability.\n\n=> The symmetric RL loss is scalable, even for large language models. When selecting the next action (token) via softmax, the model also produces probabilities for non-selected tokens. Computing gradients with respect to these non-selected probabilities introduces some additional overhead, but this cost is roughly equivalent to adding a single MLP layer (we will include this explanation in the paper). Given the overall size and structure of LLMs, this additional cost does not significantly affect training speed. In fact, SPPO showed faster training than PPO for some seeds—i.e., within the overhead of adding one MLP, GPU memory allocation conditions had a greater effect. We briefly address training speed in the last paragraph of Section 5.5.", "claims_and_evidence": "Strengths\n1. Strong Experimental Setup: The manuscript provides comprehensive experiments across diverse environments (Atari, MuJoCo, Box2D) and real-world applications (IMDB, TL;DR summarization), demonstrating the generalizability of the approach.\nWeaknesses and Areas for Improvement\n1. Insufficient Comparison with Other Robustness Methods: While the paper focuses on the symmetric RL loss, it lacks detailed comparisons with other established methods for robust RL, such as Double Q-learning or SAC, which could provide a clearer context for its benefits.\n2. Limited Theoretical Justification: The paper would benefit from a more detailed theoretical explanation of why the symmetric RL loss is particularly effective in addressing the noise in reward prediction, especially when compared to existing techniques like Generalized Advantage Estimation (GAE).\n3. Hyperparameter Sensitivity: The sensitivity of the performance to hyperparameters such as α and β is not sufficiently explored. More extensive analysis is needed to understand how these hyperparameters affect training and performance stability.\nScalability and Computational Cost: The manuscript does not provide enough information regarding the computational complexity and scalability of the proposed method, especially in large-scale environments or real-time applications.", "methods_and_evaluation_criteria": "Clear Results: The experiments clearly show the effectiveness of SPPO over PPO, especially in noisy reward environments.", "theoretical_claims": "3. Methodology\n3.1 Symmetric RL Loss\n\t• Section 4, Paragraph 2: The explanation of the reverse RL loss and its application to A2C and PPO is somewhat vague. The connection between the noisy advantage estimates in RL and the reverse cross-entropy loss is not fully established. \n\t○ Suggestion: Provide a more detailed explanation of how the reverse RL loss functions in practice, particularly focusing on how it corrects the advantage sign confusion caused by noisy rewards.\n3.2 Symmetric A2C and PPO\n\t• Section 4, Equation 9: The use of constants α and β is introduced, but the rationale for their selection is not clear. \n        ○ Suggestion: Discuss how the values of α and β are chosen. Are they determined through cross-validation, or are they fixed based on prior empirical knowledge? A sensitivity analysis of these hyperparameters would be valuable.", "experimental_designs_or_analyses": "4. Experiments\n4.1 Atari Games\n\t• Section 5.1: The results on Atari games show SPPO performing well in noisy settings, but the authors do not discuss the implications of these results for the real-world application of RL algorithms. \n\t○ Suggestion: Provide insights into how the improvements observed in Atari games can be generalized to real-world tasks, particularly those that involve more complex reward models or environments.\n4.2 MuJoCo and Box2D Tasks\n\t• Section 5.2: While the performance of SPPO is evaluated on continuous action tasks, the impact of reward noise is not thoroughly discussed. \n        ○ Suggestion: Include a more detailed analysis of how the added noise specifically affects the training process and how the symmetric RL loss mitigates these effects in continuous action spaces.", "relation_to_broader_scientific_literature": "5. Conclusion\n\t• Section 6, Paragraph 2: The conclusion briefly mentions future work but does not propose specific directions for improving or extending the symmetric RL loss method. \n       ○ Suggestion: Discuss potential future improvements, such as exploring the scalability of the method in large-scale environments, or applying the approach to other RL algorithms or tasks (e.g., multi-agent systems).", "essential_references_not_discussed": "Scalability and Computational Cost\n\t• Provide a discussion on the computational cost of the symmetric RL loss, especially in large-scale or real-time settings.\n\t• Analyze the scalability of the approach across different model architectures.", "other_strengths_and_weaknesses": "Please refer to Claims and Evidence.", "other_comments_or_suggestions": "1. Abstract\n\t• Suggestion: The abstract outlines the problem and proposed solution but lacks quantitative details on the performance improvements achieved by SPPO over PPO, especially under noisy conditions. \n\t○ Recommendation: Include key findings, such as the percentage improvement in reward scores or stability in noisy environments.\n2. Introduction\n\t• Section 1, Paragraph 4: The authors mention that \"RL methods introduce challenges such as moving targets and high gradient \n          variance,\" but they do not sufficiently explain how these issues are specifically mitigated by the symmetric RL loss. \n       ○ Suggestion: Provide a clearer link between the inherent challenges of RL and how the proposed symmetric RL loss addresses these challenges. Specifically, discuss how reverse cross-entropy aligns with the noise in reward models.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7qCYO4VrKv", "reviewer": "Reviewer_ANjE", "summary": "This paper proposes a “Symmetric Reinforcement Learning Loss” (SRL) to improve the robustness of policy-gradient algorithms—namely A2C and PPO—when facing noisy or inconsistent advantage estimates. \nThe core idea is to adapt the concept of “symmetric cross-entropy,” to the RL setting, originally developed for classification under label noise. Concretely, the authors add a “reverse RL loss” term that, together with the standard policy-gradient loss, forms a “symmetric RL loss.” They name the resulting methods SA2C (Symmetric A2C) and SPPO (Symmetric PPO).\nThe paper demonstrates that in both standard control tasks (Atari, MuJoCo, Box2D) and RLHF-style language tasks (IMDB positive-sentiment generation and TL;DR summarization), the symmetric loss improves performance over standard A2C/PPO—especially when reward signals (or advantage estimates) are noisy or suffer from misalignment.\nThe authors attribute the gain primarily to PPO’s increased susceptibility to advantage “sign flips” (confusion) and show that introducing the reverse RL term helps correct or dampen these detrimental effects.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and constructive feedback on our paper. To address your questions and concerns, we have provided detailed responses below.\n\n**Question 1**\n> PPO practitioners have found much larger batch sizes of 100k samples to work much better than smaller batch sizes\n\n=> Thank you for pointing out this work that uses very large batch sizes. However, the effectiveness of batch size remains somewhat controversial—likely depending on the environment—as other studies [1, 2] argue that smaller batch sizes can be more beneficial. The success of the work you mentioned may be due to its specific setting, where over 10,000 agents can generate a massive number of data points. In general, we believe that, given typical computational and cost constraints, updating an LLM with a batch size of 100k is often not feasible in standard setups.\n\n**Question 2**\n>  Is the improvement from simply ignoring reward-model mistakes or from genuinely mitigating advantage confusion?\n\n=> Rather than focusing on reducing the error rate of a trained reward model (which is certainly one way to improve performance), our method directly targets the mitigation of advantage confusion. Even when there is no reward model error, advantage confusion can still arise due to techniques like advantage normalization (Figure 2). We observe that SPPO still shows improvements in the noise-free setting (Section 5.4).\n\n**Question 3**\n> Comparing the effect of bigger vs. smaller batch sizes on advantage confusion\n\n=> Thank you for suggesting this interesting experiment. If the paper is accepted, we will include results evaluating our method across a range of batch sizes. However, we believe the effect is quite task-dependent. Based on our current understanding, Atari games tend to perform better with smaller batch sizes [1, 2], while MuJoCo tasks may benefit from larger batch sizes [3].\n\n**Weakness 1**\n> The method adds extra hyperparameters $(\\alpha, \\beta, Z)$, which can be a drawback for ease of adoption\n\n=> While the symmetric RL loss introduces three hyperparameters, $\\beta$ and $Z$ can be treated as a single term. To demonstrate ease of adoption, we conducted a hyperparameter sensitivity analysis for $\\beta$ while keeping $\\alpha$ and $Z$ fixed—particularly in the PPO setting (see Table 11 in the Appendix). We observed consistent performance improvements across a wide range of $\\beta$ values, suggesting that the method is easy to apply in practice.\n\n**Weakness 2**\n> While the paper shows that advantage sign flips frequently occur, the discussion is largely heuristic rather than formal.\n\n=> We provide empirical evidence of sign flip ratios in Table 2, evaluated across multiple random seeds for a diverse set of environments (5 for Atari games and 30 for the MuJoCo benchmark). We believe that advantage sign changes are obviously  expected after advantage normalization, but we will include results from additional environments to further support this observation.\n\n**Weakness 3**\n> For the RLHF tasks, all evaluation rests on an imperfect reward model and GPT-4-based comparison (no direct human eval). Though typical in modern LLM research, more thorough human-based or multi-metric evaluations could be enlightening.\n\n=> We used GPT-4 as the evaluator and measured win rates based on two comparison examples, following the AlpacaEval [4]. Our experiments involve relatively smaller models and simpler tasks, where GPT-4’s evaluations have been shown to align closely with human judgments.\n\n**Thank you for suggesting REDQ, DroQ and CrossQ. We will make sure to cite them in the our paper.**\n\n[1] The Phenomenon of Policy Churn, Neurips 2022\\\n[2] Small batch deep reinforcement learning, NeurIPS 2024\\\n[3] Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation, ICLR 2022\\\n[4] Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators, COLM 2024", "claims_and_evidence": "- Main Claim: Adding a symmetric term to the actor loss, which mirrors the standard policy-gradient update in reverse, yields more stable and robust learning, leading to higher final performance across diverse tasks and scales. Instabilities are hypothesized to be mainly caused by advantage \"sign flips\" which are emphasized through advantage normalization based on mini batch statistics.\n\n- Evidence: The paper reports results on 22 Atari games (with and without artificially flipped rewards), several MuJoCo/Box2D continuous-control tasks (with injected noise), and two RLHF tasks (IMDB sentiment and TL;DR summarization). In nearly all tested scenarios with noise or sign confusion, SPPO outperforms PPO, often by a substantial margin. Plots show that advantage sign flips are common, which corroborates the main motivation. In RLHF experiments, SPPO achieves higher model-based reward scores and higher human-like preference judgments (via GPT-4) than PPO.\n\nA different potential fix to the problem of the high fraction of advantage sign flips would be to use significantly larger mini batch sizes. To the best of my knowledge, PPO practitioners have found much larger batch sizes of $\\gt 100k$ samples to work much better than smaller batch sizes (Rudin et al. 2022, Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning).\n\nWhile the results are overall convincing, the paper primarily focuses on final performance metrics (rewards, policy returns, or model-based reward scores) and includes standard deviation ranges across multiple seeds. The paper would benefit from presenting the results visually (training curves or bar charts) rather than in tabular form, which is very hard to interpret well. Further, it could benefit from using more thorough metrics like IQM and confidence intervals aggregated over multiple environments/experiments (Agarwal et al. 2022, Deep Reinforcement Learning at the Edge of the Statistical Precipice).", "methods_and_evaluation_criteria": "-", "theoretical_claims": "I did not check the derivations.", "experimental_designs_or_analyses": "- Soundness/Validity: The designs follow standard practice in RL research: multiple seeds, standard baselines (A2C/PPO vs. SA2C/SPPO), well-known benchmark environments. The chosen “noise injection” strategies are straightforward and sensible to highlight the method’s robustness. The RLHF experiments use standard reward-model training and GPT-4 preference judgments, which is a recognized approach in alignment research.\n- Potential Issues: RLHF tasks rely on pretrained reward models, which can itself be noisy. While that’s precisely the paper’s motivation, it slightly complicates attributions of success (i.e., is the improvement from simply ignoring reward-model mistakes or from genuinely mitigating advantage confusion?). The authors do partially demonstrate examples (e.g., empty summaries yielding higher reward) to show how confusion arises. Overall, the experiments appear valid, though it would be beneficial to see more fine-grained ablations (for example, comparing the effect of bigger vs. smaller batch sizes on advantage confusion).", "relation_to_broader_scientific_literature": "The work draws work with robust supervised learning under label noise (symmetric cross-entropy). It also builds on well-known RL fundamentals (A2C, PPO) and RLHF techniques. The paper cites relevant prior methods for dealing with overestimation, distribution shifts, and training instability (Double DQN, GAE, trust-region methods). This connection to “robust supervised loss” methods is the main novelty. The authors also mention alternative strategies for RLHF like ranking-based or preference-based RL. Overall, the references and positioning in the literature appear reasonable.\n\nIm the introduction the authors further mention the use of supervised learning techniques in RL (ensembles, layernorm, batch norm), without any citations. These works should be cited: \n- REDQ: Chen et al. 2021 \n- DroQ: Hiraoka et al. 2021\n- CrossQ: Bhatt et al. 2024", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "Strengths:\n- Simple, elegant idea that is easy to integrate with existing policy-gradient code.\n- Demonstrates robust gains across various tasks (discrete/continuous control, language tasks) and with artificially perturbed rewards and real RLHF-based noise.\n- Thorough experiments with multiple seeds and hyperparameter sweeps.\n\nWeaknesses:\n- The method adds extra hyperparameters (α, β, Z), which can be a drawback for ease of adoption—though the authors fix α and Z in their experiments.\n- While the paper shows that advantage sign flips frequently occur, the discussion is largely heuristic rather than formal.\n- For the RLHF tasks, all evaluation rests on an imperfect reward model and GPT-4-based comparison (no direct human eval). Though typical in modern LLM research, more thorough human-based or multi-metric evaluations could be enlightening.", "other_comments_or_suggestions": "-", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel symmetric RL loss for mitigating noisy reward and preference data problems for RL or RLHF, motivated by the previous reverse cross-entropy loss for classification. The gradient analysis supports an effective acceleration away from abstract states by the reverse RL loss in noisy cases, and experimental results on diverse RL tasks and some small NLP tasks show that the proposed Symmetric PPO (SPPO) improves performances especially for noisy data. The proposed loss seems to be technically sound. The authors sufficiently address most concerns raised by the reviewers, except the lack of empirical validation on RLHF tasks with recent well-known benchmarks and LLMs. Based on the consensus among the reviewers, I would recommend this paper to be accepted. However, the authors need to perform experiments on recent RLHF tasks with cutting-edge LLMs.
Decision: Accept (poster)