Title: Globally Optimal Policy Gradient Algorithms for Reinforcement Learning with PID Control Policies
Venue: NeurIPS 2025
Accepted: yes
Keywords: policy gradient

proportional-integral-derivative control

reinforcement learning

non-convex optimization
Abstract: We develop policy gradient algorithms with global optimality and convergence guarantees for reinforcement learning (RL) with proportional-integral-derivative (PID) parameterized control policies. RL enables learning control policies through direct interaction with a system, without explicit model knowledge that is typically assumed in classical control. The PID policy architecture offers built-in structural advantages, such as superior tracking performance, elimination of steady-state errors, and robustness to model error that have made it a widely adopted paradigm in practice. Despite these advantages, the PID parameterization has received limited attention in the RL literature, and PID control designs continue to rely on heuristic tuning rules without theoretical guarantees. We address this gap by rigorously integrating PID control with RL, offering theoretical guarantees while maintaining the practical advantages that have made PID control ubiquitous in practice. Specifically, we first formulate PID control design as an optimization problem with a control policy that is parameterized by proportional, integral, and derivative components. We derive exact expressions for policy gradients in these parameters, and leverage them to develop both model-based and model-free policy gradient algorithms for PID policies. We then establish gradient dominance properties of the PID policy optimization problem, and provide theoretical guarantees on convergence and global optimality in this setting. Finally, we benchmark the performance of our algorithms on the controlgym suite of environments.


Peer reviews:
comments: SUMMARY:
This paper proposes policy gradient algorithms for reinforcement learning with PID-parameterized control policies. The method leverages the practical effectiveness of PID controllers while providing theoretical guarantees on convergence and global optimality in both model-based and model-free settings.

STRENGTHS AND WEAKNESSES:
**Strengths:**

- The mathematical notation is consistent and clearly presented, which aids understanding of the theoretical analysis.

**Weaknesses:**

- The main limitation lies in the novelty of the contribution. Prior work has already explored reinforcement learning for fine-tuning structured controllers such as MPC and PID [1, 2, 3]. The paper would benefit from a clearer distinction between its contributions and those of existing methods.
- The experimental results are difficult to interpret. In Figure 1, all curves appear very close together, making it hard to evaluate relative performance. Additionally, many results are reported without uncertainty estimates (e.g., error bars), which weakens empirical claims.

[1] Gros, Sébastien and Mario Zanon. “Data-Driven Economic NMPC Using Reinforcement Learning.” *IEEE Transactions on Automatic Control* 65 (2019): 636-648.
 [2] Hardt, Moritz et al. “Gradient Descent Learns Linear Dynamical Systems.” *arXiv preprint* arXiv:1609.05191 (2016).
 [3] Westenbroek, Tyler et al. “Feedback Linearization for Unknown Systems via Reinforcement Learning.” *arXiv preprint* arXiv:1910.13272 (2019).

QUESTIONS:
1. Why are the gradients derived from Equation 3 instead of directly from Equation 1? Clarifying this design choice would help in understanding the derivation and optimization approach.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 1
ORIGINALITY: 2
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper bridges the gap between reinforcement learning and PID control by introducing policy gradient (PG) methods tailored for PID architectures. While PG methods have strong theoretical guarantees in classical control problems, their application to PID—despite its widespread use—has been largely heuristic. The authors propose an optimization-based framework for PID control, derive exact gradient expressions, and develop two key algorithms: a model-based method for full PID control, and a model-free method specifically for PI control. The model-based approach leverages system knowledge to achieve global convergence with linear rates, while the model-free algorithm provably learns optimal PI parameters using stochastic policies, offering global guarantees under weaker assumptions.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. This paper is written in a very elegant and fluent manner, making it easy to understand. It introduces the problem through a clear presentation of the background and proposes a solution supported by solid theoretical analysis.\n\n2. The paper establishes global optimality, convergence rates, and sample complexity guarantees for policy gradient methods applied to PID and PI control. Specifically, it proves gradient dominance for the model-based PID setting and derives weak gradient dominance and sample complexity bounds for the model-free PI algorithm. \n\n3. The paper compares its methods with both PG-based LQR algorithms and PPO-based approaches, effectively resolving concerns regarding performance and practical applicability.\n\n**Weaknesses:**\n\n1. The visualization in the experimental section could be improved. For example, in Figure 1, it is recommended to truncate the curves to a smaller number of samples (e.g., the first 500) to better illustrate the dynamic changes. Currently, the mean trajectories are almost completely overlapped with the 1-SD bands, making it difficult to distinguish between different methods and limiting the readability and comparability of the results.\n\n2. Since both PPO and LQR are included as baselines in the experimental section, it would be better to move the comparison with LQR into the main text to facilitate a clearer comparison of the results.\n\n3. If possible, the stability experiments related to Figure 6 would be better presented in the main text to more intuitively demonstrate the results. Currently, the experimental section takes up less than one page and feels somewhat flat.\n\nMinor: \n- Second equation in equation (3) should be: $g_{t+1} = \\overline{A}g_t + \\overline{B}u_t$\n\nQUESTIONS:\n1. Are Assumptions 4.3, 4.5, and 4.6 likely to be satisfied in practical implementations?\n\n2. When extending to constrained optimization settings, what needs to be achieved, and what is the main obstacle?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a control approach that optimizes PID parameters using gradient descent (PG) methods. The problem is first formulated as an LQR problem; however, rather than adopting PG methods commonly used for LQR, the study directly computes the derivatives of each variable. This leads to a thorough mathematical derivation and establishes theoretical guarantees for convergence.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe article addresses a clearly defined problem with a well-structured and logical approach. It discusses the issue that although the gain matrix K is fully determined, there may not always exist a feasible decomposition with respect to K. Instead of directly applying the PG method commonly used in LQR [17], the authors compute the gradients directly for the PID control problem. The paper progresses at a clear and efficient pace, addressing both model-based and model-free scenarios. Furthermore, by omitting certain differential terms, the method achieves a balance between theoretical soundness and practical applicability.\n\nWeaknesses:\n\nAlthough the paper integrates reinforcement learning with PID control, the applicability of the method is inherently limited due to the simple structure of PID controllers, which restricts their effectiveness to a narrow range of control problems.\nThe derivation is based on linear systems, and the paper does not address or discuss the extension to nonlinear systems.\n\nQUESTIONS:\nIn the understanding of model-free methods, sigma (σ) and the Gaussian mechanism indeed play crucial roles. As discussed in Section 3.2, the importance of the stochastic policy π(·|g) becomes evident, particularly in ensuring that π̃ₖ(u|g) > 0 holds for all states g and actions u.\n\n1. \"This suggests that in certain situations, introducing randomness is necessary in order to avoid issues that may arise from deterministic controllers μ̃ₖ(g).\" While this statement highlights the necessity of randomness in avoiding problems associated with deterministic controllers. Does the presence of randomness also play a fundamental role in enabling the existence and effectiveness of policy gradients?\n2. Why is the deterministic controller μ̃ₖ(g) unable to satisfy the condition π̃ₖ(u|g) > 0 when σ approaches zero? What are the theoretical implications of this limitation for the well-posedness of the policy gradient formulation?\n3. How does this limitation impact the solution of the cost minimization problem within the framework of model-free reinforcement learning? Furthermore, in practical implementations, how should the value of σ be tuned to strike an appropriate balance between exploration and control performance?\n\nLIMITATIONS:\nDespite the integration of reinforcement learning with PID control, the overall approach is limited by the inherent structural simplicity of PID controllers. As a result, the range of control problems that can be effectively addressed is relatively narrow.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two algorithms (model-based and model-free, respectively) to integrate classic PID control within the RL framework, enabling the automatic learning the PID parameters. They do so using RL with theoretical guarantees of convergence and global optimality.\n\nThe proposal aims to bridging classic control theory with RL while retaining the practical robustness of PID controllers.\n\nExperiments are performed in the Chemical Reactor and a 48-dimensional state-space LA University Hospital environments, showing that their proposal converges faster than a PPO, as it requires less exploration.\n\nSTRENGTHS AND WEAKNESSES:\nSTRENGTHS:\n- It address the relevant problem of how to improve the learning of RL in scenarios where PID controllers already work very well.\n- The manuscript provides algorithms for model-free and model-based.\n- Proofs of convergence and global optimality guarantees are provided for the two algorithms.\n- The experiments are well-described and the analysis backed by the results.\n\nWEAKNESSES:\n- While interesting, the proposal seems quite incremental, (though offering theoretical guarantees is excellent).\n- Applicability outside problems that can be controlled with PID is unclear (i.e. not clear if all problems could be somehow modeled as PID-controlled, and let the algorithms find the parameters, even though it would be hard for a human).\n- The assumptions about the structural conditions make the proposal very concrete (though the amount of PID controllers out there is huge).\n\nQUESTIONS:\n- The authors acknowledge that further work is needed to extending the mathematical framework to encompass multi-objective trade-offs. Can they offer a brief explanation as to how they think it can be done?\n- Will this approach be always preferable to the usual heuristic tuning of PID? If not always, when not?\n- What is the rationale for choosing those two particular testing environments? They seem to be relative easy to solve.\n- Is there any type of overhead (reserouces/time/extra data/....) required for PG4PID and PG4PI? If so, could you quantify them for your examples?\n- Is this proposal applicable to more complex real-world systems as it is? Or some prerrequisites or preprocessing are needed? Will they work or are there any limitations to be aware of?\n\nLIMITATIONS:\nLimitations are listed in the conclusions:\n- no multiobjetive yet.\n- no hard constraints on states and inputs.\nhowever, authors do not anticipate how they think they could be addressed.\n\nNo potential negative societal impact is foreseen.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}