Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data

Authors: Jeonghye Kim, Yongjae Shin, Whiyoung Jung, Sunghoon Hong, Deunsol Yoon, Youngchul Sung, Kanghoon Lee, Woohyung Lim
Primary area: reinforcement learning
Keywords: Offline-to-Online Reinforcement Learning, Offline Reinforcement Learning, Penalizing Infeasible Actions, Layer Normalization, Reward Scaling​

Abstract:
Reinforcement learning with offline data often suffers from Q-value extrapolation errors due to limited data, which poses significant challenges and limits overall performance. Existing methods such as layer normalization and reward relabeling have shown promise in addressing these errors and achieving empirical improvements. In this paper, we extend these approaches by introducing reward scaling with layer normalization (RS-LN) to further mitigate extrapolation errors and enhance performance. Furthermore, based on the insight that Q-values should be lower for infeasible action spaces—where neural networks might otherwise extrapolate into undesirable regions—we propose a penalization mechanism for infeasible actions (PA). By combining RS-LN and PA, we develop a new algorithm called PARS. We evaluate PARS on a range of tasks, demonstrating superior performance compared to state-of-the-art algorithms in both offline training and online fine-tuning across the D4RL benchmark, with notable success in the challenging AntMaze Ultra task.

Decision: reject
Reviewer scores: 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729500596652

thread_type:
official_review

<thread_object>
creator: reviewer_h6w1

<content>
summary:
The authors present a concise yet effective algorithm called PARS, which introduces Reward Scaling combined with Layer Normalization (RS-LN) to mitigate extrapolation errors and enhance the offline and offline-to-online performance. The **experimental results** are particularly **impressive**.

However, I have some questions: 1) How does PARS differ from other related algorithms? Such as [1] 2) Could the authors clarify how the algorithm's design specifically contributes to its performance advantage? (see question section)

If you can given clarification, I think this paper will be much more comprehensive.

strengths:
The advantages of PARS are primarily reflected in two key aspects:

- PARS is a straightforward approach that integrates Layer Normalization and reward scaling with TD3+BC, making it easy to implement.

- PARS demonstrates strong performance across both offline and offline-to-online tasks from a variety of domains, including Androids, Mujoco, and Antmaze, making it highly scalable and showcasing competitive experimental results.

weaknesses:
The experimental performance of PARS is impressive. However, the authors do not thoroughly discuss the relationship between the performance improvements and the algorithm's design. Additionally, it is necessary to provide clearer distinctions between PARS and similar algorithms, such as [1], as mentioned in the questions section. Specifically, please see the **question section**.

questions:
**Q.1** Please clarify the advantages of your research's LN compared to that in [1] ? And could you demonstrate your methods' advantage over other offline-to-online algorithms? Such as [1], Cal-QL, RLPD, PEX.

**Q.2** Subsequently, please clarify why your algorithm performs better in online fine-tuning. 

Cal-QL [2] suggests that value misalignment in both offline and online settings may be a reason for poor performance during online fine-tuning. If you chose to utilize layer-normalization, it implies that your algorithm may learn more conservative Q-values, potentially leading to a more conservative performance during online fine-tuning. However, the algorithm in [1] improves upon soft-actor-critic (SAC) during online fine-tuning, where maximizing entropy encourages the agent to explore more diverse samples, thus alleviating the conservatism inherited from the offline stages. 

Meanwhile, I can give you some related studies for reference. Recent research has shown that appropriately incorporating BC terms during the online process may enhance online performance [3]. Could you please give much more analytical explanation such as [3], at the very least about the potential reasons, thank you.

**Q.3** Please clarify the innovations of your algorithm compared to [1]. Specifically, is the improvement in your algorithm mainly due to the combined use of reward re-scaling and layer normalization (LN), or is it due to the combined use of TD3-BC, and the reward scaling algorithm? Thank you.

*If TD3+BC is not the majority reason for PARS' high efficiency, please answer **Q.4***, otherwise, you shouldn't have to answer **Q.4**

**Q.4** Could you please conduct additional experiments to verify the extensibility of PARS. For instance, you could replace TD3+BC with other methods, especially algorithms beyond policy constraints, such as CQL/IQL etc.

# Reference

[1]Philip J. Ball, Laura Smith, Ilya Kostrikov, Sergey Levine, Efficient Online Reinforcement Learning with Offline Data.

[2] Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, Sergey Levine, Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning

[3] Seohong Park, Kevin Frans, Sergey Levine, Aviral Kumar. Is Value Learning Really the Main Bottleneck in Offline RL?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730766401625

thread_type:
official_review

<thread_object>
creator: reviewer_uzz9

<content>
summary:
This paper uses reward scaling and penalizing techniques for offline RL. The proposed method is not novel but effective. Experiments show the advantages of the proposed method.

strengths:
The writing of the paper is very clear and easy to follow.

The idea of the paper makes sense and is intuitive. The idea of imposing penalizing infeasible actions is simple but effective. It does not need additional efforts to train a model to classify the OOD and ID.

The experiments of the paper are sufficient and valid.

weaknesses:
The proposed method is very simple (which can also be regarded as an advantage). It is unclear to me why the layer normalization and reward scaling can mitigate OOD overestimation. The paper could elaborate more insight on this.



I appreciate the Didactic exmaple in Sec 4.1, which is good and intuitive. However, it's an example. More analysis and discussion would be good. Please see my questions below.





Missing Ablation studies:

- The paper can also compare with method that use a behavior model to classify the ID and OOD. Could the PA also show benefits over that?



Minor comments:

- Line 019: lower than what?
- Line $s_t,a_t$ should be defined before using them,.
- Line 089: the returns should be expected returns.p

questions:
Regarding the Didactic example in Sec 4.1:

- The examples are all positive. What happens if the values are negative? By the way, what is the sign of the reward in the practical environments used in the paper?
- It seems from both Figure 3 and the experimental results, with LN, larger $c_{\text{reward}}$ always leads to better results. Could you elaborate on the reason? And is it correct that we can choose a larger $c$?

Would the method and the main result also work and hold for a discrete action space?

The proposed method shows large improvements on AntMaze? Could the author elaborate more on these results?

I don't really understand why $L_I = U_I$ in Line 501. Why are they equal?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730776794630

thread_type:
official_review

<thread_object>
creator: reviewer_6jbm

<content>
summary:
This paper incorporates reward scaling with layer normalization and infeasible action penalization into existing offline reinforcement learning algorithms, specifically TBC+BC with critic ensemble, to enhance their performance. The proposed method is evaluated on various offline and offline-to-online benchmarks and compared with several baseline methods. The results show that the proposed method outperforms the baseline methods on the chosen tasks.

strengths:
- The presentation is clear and easy to follow.
- The proposed method is incremental but appears effective.
- The experimental evaluation is thorough and covers a wide range of tasks.

weaknesses:
- The motivation for penalizing infeasible actions is not clear. As shown in Figure 2(c) and (d), it seems penalizing OOD actions within the feasible region will lead to the same optimal policy (e.g., argmax Q) as RS-LN and PA. In this case, the proposed method is equivalent to RS-LN and PA, though the absolute Q values are different. 

- Section 3 criticizes that using a critic ensemble significantly increases the training complexity. However, the proposed method also uses a critic ensemble (4 for AntMaze, 10 for MuJoCo and Adroit), which undermines the advantage of the proposed method and makes its motivation less convincing. A comprehensive comparison of the proposed method without a critic ensemble against other baseline methods would be helpful to clarify the contribution of the proposed method.

- Some of the results are not consistent with the claims in the paper. Please see the questions section for details.

questions:
- Figure 3 shows results from a single seed on the toy example. Is this conclusion consistent when using different seeds? Will the shape worsen or converge to a pattern when the reward scaling factor is larger?
- Line 231 states, 'This regulated reduction in Srank prevents overfitting by minimizing the learning of irrelevant noise'. How can this conclusion be drawn from the results?
- Figure 7 shows that a larger reward scaling factor leads to better performance. Is this a general conclusion for other tasks? Is there any limitations about the reward scaling factor?
- Line 270 states, 'the performance does not heavily depend on the values of U when these values are set as 100 to 1000 times the boundary of the feasible region'. However, the results in Figure 7 show that performance decreases when U is large on 3 out of 4 tasks. Can you clarify this inconsistency?
- See Weaknesses.

I am willing to increase my score if all my concerns are addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731119622170

thread_type:
official_review

<thread_object>
creator: reviewer_ypm4

<content>
summary:
The paper introduces the PARS algorithm, which aims to address Q-value extrapolation errors prevalent in offline reinforcement learning (RL). The authors propose a twofold approach combining (1) Reward Scaling with Layer Normalization (RS-LN) and (2) Penalizing Infeasible Actions (PA). The former stabilizes the Q-function learning by scaling rewards while employing layer normalization to manage output variances effectively. The latter penalizes Q-values for infeasible action regions to prevent overestimation outside the feasible data coverage. The paper shows that PARS, built upon the minimalist TD3+BC algorithm, outperforms existing state-of-the-art (SOTA) methods in offline, and offline-to-online setups, particularly in the challenging AntMaze environment.

strengths:
- The paper demonstrates the effectiveness of the proposed PARS method through comprehensive empirical results across various domains, showing that it outperforms or matches strong baselines. 
- The method’s simplicity and minimal code changes required for implementation make it attractive for practitioners looking to enhance existing RL models without significant overhead.
- The paper is well-written, with clear visualizations that effectively convey the concepts and intuitions behind the approach. This aids in understanding the methodology and its application.

weaknesses:
- The way infeasible regions and reward scaling are defined—through arbitrary thresholds without adaptive or data-driven characterization—limits the method's novelty.
- While the paper mentions following prior work for hyperparameter tuning, the strategy of tuning hyperparameters with online interactions contadicts offline RL, which undermines the problem’s nature. Using Off-Policy Evaluation methods for hyperparameter selection, or fixing a set of hyperparameters for each environment would be better aligned with Offline RL.
- The paper does not discuss offline RL methods that do not query out-of-distribution (OOD) actions, such as SQL [1] , XQL [2],  and IQL. and how PARS ideas can be leveraged by those.

[1] Offline RL with no OOD actions: In-sample learning via implicit value regularization. ICLR 2023.

[2]  Extreme q-learning: Maxent rl without entropy. 2023

questions:
- Please check weaknesses.
- How do you define the feasible boundary for the action space in your method? Specifically, what do  l_i  and  u_i  represent? Are they derived as the minimum and maximum a_i within the dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732154082495

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common response

comment:
We sincerely thank all the reviewers for their detailed comments on our work. The reviewers' feedback has provided us with a valuable opportunity to address unclear points and strengthen the contributions of our work. In this common response, we address questions that reviewers have commonly raised. In particular, we have clarified our motivation and contributions and highlighted the revisions made in the updated paper.

&nbsp;

### **[C1] Motivation and contributions of PARS**

Our goal with PARS is to *"design an RL algorithm that performs well both offline and during the online fine-tuning phase, while utilizing reduced complexity in methods and implementation.”*

For strong offline performance, some previous works have aimed to lower Q-values for OOD actions by effectively distinguishing between ID and OOD regions and employing various methods to reduce Q-values within the OOD region. **In contrast, PARS does not directly enforce lower Q-values for OOD actions within the feasible region, due to the challenges of precisely reducing values in the OOD region and the potential side effects during the online fine-tuning phase. Instead, it aims to fully leverage the network's expressivity and adaptability, which naturally results in a reduction in Q-values.**

- Other methods often establish a clear boundary between ID and OOD regions, whereas **PARS maintains smoothness at the ID/OOD boundary**, enabling flexible exploration and better adaptation to distribution shifts during online fine-tuning phase.
- PARS is very simple yet highly effective across domains, dataset qualities, and training phases.

In the manuscript's Section 3, we compared prior works that aimed to perform critic regularization with PARS. Additionally, since our goal is not just to design an algorithm that performs well during the offline phase but also excels in the online fine-tuning phase, we provide a detailed explanation in the revised paper's Appendix B on how PARS offers advantages over other prior algorithms from the perspective of online fine-tuning.

&nbsp;

### **[C2] Paper modification**

The sections that were revised or added in the revised paper are as follows. The revised parts are highlighted in blue. Additionally, typos and unclear words pointed out by the reviewers have also been corrected.

- [modified] Section 3. Critic Regularization For OOD Actions
    - We clarify the differences between methods that rely solely on the critic ensemble and those that perform OOD penalization within the feasible region, ensuring that the advantages of RS-LN and PA are more distinctly highlighted.
- [Added] Appendix B.  In-Depth Comparison with Prior Works
    - Compared to prior works, we analyze the strengths of PARS in the context of online fine-tuning, supported by figures and additional experiments.
- [Added] Appendix D.  In-Depth Analysis of RS-LN Effectiveness
    - We provide an analysis of the potential reasons behind the effectiveness of RS-LN.
- [Added] Appendix F.2. Extension to In-Sample Learning-Based Methods
    - We present results demonstrating the extension of the PARS concept to in-sample learning-based methods.
- [Added] Appendix J. Didactic Example with More Seeds and Increased $c_\text{reward}$

----- Additional Updates -----

- [Added] Appendix B.2. More Related Works
- [Added] Appendix D.2. Comparison of TD3+BC and IQL Critic Updates and the Effectiveness of RS-LN
- [Added] Appendix K. Why is the combination of TD3+BC with RS-LN and PA effective?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732599136979

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A summary of further paper revisions during the rebuttal period

comment:
Dear Reviewers,

Thanks to the reviewers' ongoing feedback and dedication to our work, various improvements have been made to the paper during the rebuttal period. We would like to summarize these updates. Through extensive discussions, **we believe the paper has been updated to better convey the motivation behind PARS, the reasons for its strong effectiveness, and its versatility.** The potential to develop a simple yet powerful RL algorithm validated through PARS is expected to make RL more practical and inspire a variety of future works.

As the rebuttal period has been extended, we are willing to address and incorporate any remaining discussion points.

- **Regarding Motivation → Section 3 & Appendix B**
    
    We discuss the advantages of RS-LN and PA over other prior methods during both the offline phase and online fine-tuning. Appendix B specifically delves into the effectiveness of these methods in the context of online fine-tuning.
    
- **Regarding the Versatility of the PARS Idea → Appendix F**
    
    We apply the concept of PARS to in-sample learning-based methods and confirm that RS-LN is also effective for such methods.
    
- **Regarding the Effectiveness of RS-LN → Appendix D**
    
    We explore the potential reasons behind the effectiveness of RS-LN and discuss how RS-LN can be universally effective for TD3+BC and in-sample learning-based algorithms such as IQL.
    
- **Regarding the Combination of TD3+BC with RS-LN & PA → Appendix K**
    
    We discuss why RS-LN and PA show greater effectiveness specifically with TD3+BC compared to in-sample learning-based methods like IQL.
    

We look forward to further discussions and welcome any additional feedback to enhance the paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper argues that reward scaling (multiplying the reward by a scalar) with layer normalization in the value network, in addition to penalizing the value of infeasible actions can be an effective technique for offline reinforcement learning. The paper discusses experimental results on a number of standard benchmark problems.

The discourse with the reviewers revolved around the details of this simplistic approach, e.g., why is it necessary to penalize infeasible actions if they are never going to be sampled, and whether reward scaling can really be this effective. In this discussion, a few points seem to have been missed.

First, multi-layer perceptrons are biased, at early stages of training, to low-frequency functions. This is why penalizing the value for infeasible actions makes the predicted value function smoother (it fills in the gaps at intermediate actions regardless of whether they are good or bad). As such, smoother values need not necessarily be good for exploration or offline RL. If one trains for more iterations, this bias is rectified, and therefore one will not see smooth value functions. Altogether, it is not clear why one would impose this penalty. Second, the whitening occurring in layer normalization helps in fitting MLPs more quickly, it is not clear whether from the evidence in this paper whether it has anything to do with offline RL. Reward scaling is an interesting idea, but without further evidence of _why_ this particular approach helps in offline RL (perhaps it does so because scaling up the loss helps the user control the relative magnitude of extrapolation bias), it is difficult to argue for this paper. I would encourage the authors to investigate why these ideas work in the specific context of offline RL. The didactic examples and toy examples in the Appendix are very interesting, but not directly related to the problems being investigated in this paper.