Action Mapping for Reinforcement Learning in Continuous Environments with Constraints

Authors: Mirco Theile, Lukas Dirnberger, Raphael Trumpp, Marco Caccamo, Alberto Sangiovanni-Vincentelli
Primary area: reinforcement learning
Keywords: Constrained MDPs, continuous action space, deep reinforcement learning

Abstract:
Deep reinforcement learning (DRL) has had success across various domains, but applying it to environments with constraints remains challenging due to poor sample efficiency and slow convergence. Recent literature explored incorporating model knowledge to mitigate these problems, particularly through the use of models that assess the feasibility of proposed actions. However, integrating feasibility models efficiently into DRL pipelines in environments with continuous action spaces is non-trivial. We propose a novel DRL training strategy utilizing action mapping that leverages feasibility models to streamline the learning process. By decoupling the learning of feasible actions from policy optimization, action mapping allows DRL agents to focus on selecting the optimal action from a reduced feasible action set. We demonstrate through experiments that action mapping significantly improves training performance in constrained environments with continuous action spaces, especially with imperfect feasibility models.

Decision: reject
Reviewer scores: 5, 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1730198963986

thread_type:
official_review

<thread_object>
creator: reviewer_ykmm

<content>
summary:
This paper proposes a strategy called action mapping for RL in continuous environments with state-wise constraints. The idea is to learn a latent action space and an action mapping model, with which the policy samples a latent action and the latent action is further mapped to a feasible action. The proposed method is evaluated in a robotic arm end-effector pose positioning task and a path planning environment, showing better performance than several existing methods in terms of higher returns and lower constraint violations.

strengths:
- The target problem to solve in this work is of great significance to many real-world applications.
- The related works are introduced and discussed satisfactorily.

weaknesses:
- Although the authors mentioned the reference [Theile et al., 2024], I found the content in Section 4 overlaps largely with the content in Section 3 and Section 4 of [Theile et al., 2024]. For example, Equation 16 in this paper is almost the same as the JS loss in Table 2 of [Theile et al., 2024]. Therefore, the novelty and contribution of this paper are questionable.
- The whole training process is not clear. Adding a pseudocode of the proposed algorithm will help. Especially, the training of the feasibility model is not clear enough. In addition, many technical details are missing. Please see my questions below.
- The idea of action mapping is closely related to the research on action representation learning [1-4]. The illustration in Figure 1 is very similar to the concept presented by Figure 1 in [1]. These related works should be included in the related work section for a detailed discussion.

### Minors

- The symbol $J$ in Equation 1,2 and Equation 3,4 are inconsistent.
- The legends in Figure 4 are too small.

---
### Reference

[1] Yash Chandak, Georgios Theocharous, James E. Kostas, Scott M. Jordan, Philip S. Thomas. Learning Action Representations for Reinforcement Learning. ICML 2019: 941-950

[2] Boyan Li, Hongyao Tang, Yan Zheng, Jianye Hao, Pengyi Li, Zhen Wang, Zhaopeng Meng, Li Wang. HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation. ICLR 2022

[3] Pengjie Gu, Mengchen Zhao, Chen Chen, Dong Li, Jianye Hao, Bo An. Learning Pseudometric-based Action Representations for Offline Reinforcement Learning. ICML 2022: 7902-7918

[4] William F. Whitney, Rajat Agarwal, Kyunghyun Cho, Abhinav Gupta. Dynamics-Aware Embeddings. ICLR 2020

questions:
1. For the training in Section 4.1, does it require a ground-truth $g(s,a)$?
2. What is the bound of the output of $\pi_f$ and $\pi_o$?
3. What are the implementation details of the proposed method, e.g., network structure, hyperparameters?
4. For Figure 4, why is SAC+Replacement not included in Figure 4c? And where is the constraint violation plot for SAC?
5. Throughout the paper, it seems that the definitions in Equation 5-7 are not necessary, as in Section 4 and the experiments only the function $g$ is assumed. Can the authors provide more explanation on this point?
6. How many seeds/trials are used in Figure 4?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730455053556

thread_type:
official_review

<thread_object>
creator: reviewer_vbay

<content>
summary:
This paper proposes an action mapping method that distinguishes between feasibility and objective policies during training. By pretraining the feasibility policy first and then training the objective policy, the approach enables more efficient learning within a reduced action set. Experimental results demonstrate that the proposed method results in fewer constraint violations and achieves higher returns compared to previous action replacement, resampling, and projection methods.

strengths:
1. The method for training the feasibility policy is novel, allowing for fewer constraint violations and higher returns compared to other methods.
2. Experiments were conducted in environments requiring constraints, such as a robotic arm task and a spline-based path planning.
3. The approach is straightforward and can be combined with any RL algorithm.

weaknesses:
1. The assumption that the feasible policy can be pretrained seems overly strict. Pretraining requires prior knowledge of the cost function $C^\tau(s;\pi)$ and the feasibility model $G(s,a)$, which may be difficult to assume in general.
2. The experimental environments appear limited. It would be beneficial to include comparisons in environments like Safety Gym or other constrained RL environments.
3. There is a lack of baseline algorithms. Currently, the comparisons are limited to variants of action mapping, such as action resampling and projection. Direct comparisons with a wider range of methods, including Lagrangian approaches, would strengthen the evaluation. Even if these methods are primarily designed for standard CMDPs rather than SCMDPs, adjusting the constraint thresholds to be more strict would allow for a fair comparison under the SCMDP constraints used in the original experiments.

questions:
1. **[About Weakness 1]** Isn’t the required setup for the proposed method too strict? Other constrained RL methods estimate the cost function without prior knowledge, obtaining cost information similarly to rewards. In such cases, would the proposed action mapping approach still be applicable?

2. **[About Weakness 2]** Could you provide experimental results in a wider variety of environments?

3. **[About Weakness 3]** Could you also show performance comparisons with other constrained RL methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695902173

thread_type:
official_review

<thread_object>
creator: reviewer_ymya

<content>
summary:
This paper tackles the problem of how to efficiently train agents in environments with constraints (like a robotic arm avoiding obstacles or an aircraft maintaining non-holonomic constraints). Traditional DRL approaches struggle with poor sample efficiency and slow convergence in such constrained environments. The authors propose "action mapping," which decouples the learning process into two parts: first training a feasibility policy that learns to generate all feasible actions for any state, then training an objective policy to select optimal actions from this feasible set, effectively transforming a state-wise constrained Markov Decision Process (SCMDP) into an unconstrained MDP. They validate their approach through two experiments - a robotic arm end-effector pose task with perfect feasibility models and a path planning problem with approximate feasibility models - demonstrating superior performance compared to common approaches like action replacement, resampling, and projection.

strengths:
- The most significant contribution is how action mapping allows agents to express multi-modal action distributions through a simple Gaussian in latent space, improving exploration. Ability to plan with approximate feasibility models is a notable advantage since perfect models are rarely available in practical applications.

weaknesses:
- The paper omits constraint violation plots for the path planning task, making it impossible to verify claims about performance with approximate feasibility models.
    
- The feasibility model is a critical component of the proposed architecture, yet the paper lacks essential analysis and ablations of this component. Key questions remain unanswered: How is state space sampling performed during training? What metrics determine sufficient training of the feasibility model? How does the quality/approximation level of the feasibility model impact overall performance? Without these analyses, it's difficult to understand the method's robustness and its applicability to scenarios where perfect or near-perfect feasibility models aren't available.

Minor Comments:

- Typo on line 406, “actiosn”

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718361763

thread_type:
official_review

<thread_object>
creator: reviewer_8uny

<content>
summary:
The authors introduce a novel approach known as action mapping (AM) within the context of deep reinforcement learning (DRL), showcasing its effectiveness, particularly when utilizing approximate feasibility models. Their results suggest that the integration of approximate model knowledge can improve training performance and enable agents to represent multi-modal action distributions, thus enhancing exploration strategies. By applying AM to both PPO and SAC, which represent on-policy and off-policy RL algorithms respectively, the authors provide comparative experimental results. These findings demonstrate that the AM method can transform state-wise constrained Markov Decision Processes (SCMDP) into Markov Decision Processes (MDP), thereby enhancing the sample efficiency of the original algorithms.

strengths:
1. The motivation for this work is clear: it addresses the SCMDP problem by utilizing a model to learn a feasible action space, effectively converting SCMDP into an MDP and improving the algorithm’s sample efficiency.
2. The paper provides a step-by-step explanation of related concepts, making it very accessible to readers who may not be familiar with the field.
3. The description of the action and state spaces for the tasks in the experiments is clear, and the importance of the AM algorithm is effectively illustrated through visualizations at the end of the experimental section.

weaknesses:
1. The absence of accompanying code makes it difficult to replicate the experimental results.
2. The experimental section appears somewhat limited, as it only tests the method in two environments. This reduces the persuasive power and credibility of the results.
3. The action mapping approach is not end-to-end, requiring pre-training with trajectory data before its application. This introduces additional costs, which are not adequately discussed in the paper.

questions:
1. The paper's theme is closely related to Safe RL, as mentioned. However, many Safe RL algorithms are not included as baselines for comparison. What is the rationale behind this omission?
2. While PPO+Replacement has slower learning efficiency, it strictly ensures the satisfaction of constraints. This property is valuable in some online environments, but AM lacks this capability. Are there any proposed methods to address this limitation?
3. In Figure 4(c), the performance of AM-SAC suddenly increases at 7500 steps. How can this phenomenon be explained?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732789182416

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
List of Changes

comment:
We would like to thank the reviewers for their constructive feedback. We tried to address all of the questions and concerns and believe it made the paper stronger. We hope the reviewers have the time to check the updated manuscript to see whether their concerns were sufficiently addressed.

List of major changes (chronologically):
- Emphasize the difference to Theile et al. (2024)
- Discussed the action representation literature
- A thorough description of the objective policy and the overall training procedure is added in Section 4.2 and Algorithms 1+2
- A comparison with Lagrangian approaches is added
- Combined AM-PPO and action replacement in the robotic arm environment 
- Discussed the assumption of having a feasibility model in the Conclusion
- Provided a detailed description of the feasibility policy training in DRL environments in Appendix A
- Added the network architecture and hyperparameters in Appendix B
- Added a description of the Lagrangian algorithms in Appendix C.2
- Added a study on the approximation accuracy of the feasibility model in the path planning environment and its impact on learning performance in Appendix G
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a novel method for training a feasibility policy in safe reinforcement learning, demonstrating promising results in terms of constraint violation reduction and improved returns.  However, the submission suffers from several weaknesses that prevent its acceptance in its current form. While the core idea is interesting and the approach is conceptually straightforward, the assumption of a pretrainable feasibility policy appears overly restrictive and lacks generalizability. Furthermore, the experimental evaluation is limited in scope, both in terms of environments and baseline comparisons. Expanding the experiments to include more diverse and challenging constrained RL environments, along with a broader range of baseline algorithms (including adapted Lagrangian methods), would significantly strengthen the paper.  Given these limitations, particularly the strong assumption and limited empirical validation, we reject this submission but encourage the authors to address these concerns and resubmit a revised version to a future venue.