Globally Optimal Policy Gradient Algorithms for Reinforcement Learning with PID Control Policies
Authors: Vipul Kumar Sharma
Wesley A. Suttle
S Sivaranjani
Venue: NeurIPS 2025 poster ()
Keywords: policy gradient
proportional-integral-derivative control
reinforcement learning
non-convex optimization
Primary area: theory
TL;DR: We develop policy gradient algorithms with global optimality and convergence guarantees for reinforcement learning with PID control policy parameterization.

Abstract: We develop policy gradient algorithms with global optimality and convergence guarantees for reinforcement learning (RL) with proportional-integral-derivative (PID) parameterized control policies. RL enables learning control policies through direct interaction with a system, without explicit model knowledge that is typically assumed in classical control. The PID policy architecture offers built-in structural advantages, such as superior tracking performance, elimination of steady-state errors, and robustness to model error that have made it a widely adopted paradigm in practice. Despite these advantages, the PID parameterization has received limited attention in the RL literature, and PID control designs continue to rely on heuristic tuning rules without theoretical guarantees. We address this gap by rigorously integrating PID control with RL, offering theoretical guarantees while maintaining the practical advantages that have made PID control ubiquitous in practice. Specifically, we first formulate PID control design as an optimization problem with a control policy that is parameterized by proportional, integral, and derivative components. We derive exact expressions for policy gradients in these parameters, and leverage them to develop both model-based and model-free policy gradient algorithms for PID policies. We then establish gradient dominance properties of the PID policy optimization problem, and provide theoretical guarantees on convergence and global optimality in this setting. Finally, we benchmark the performance of our algorithms on the controlgym suite of environments.

Globally Optimal Policy Gradient Algorithms for Reinforcement Learning with PID Control Policies
Vipul K. Sharma
Wesley A. Suttle U.S. Army Research Laboratory
Edwardson School of Industrial Engineering
Purdue University
Adelphi, Maryland 20783
West Lafayette, IN 47906
wesley, www.te...eerreereeeee.e..... mi
sharm 1thaamm978purduee edu
Sivaranjani
Edwardson School of Industrial Engineering Purdue University
West Lafayette, IN 47906
sseetha@@urduu edu
Abstract
We develop policy gradient algorithms with global optimality and convergence guarantees for reinforcement learning (RI.) with copporionalinneegaaaddiiiivttiii (PID) parameterized control policies. RL enables learning control policies through direct interaction with system, without explicit model knowledge that is typically assumed in classical control. The PID policy architecture offers built-in structural advantages, such as superior tracking performance, elimination of steady-state er- rors, and robustness to model error that have made it widely adopted paradigm in practice. Despite these advantages, the PID parameterization has received limited attention in the RL literature, and PID control designs continue rely heuristic tuning rules without theoretical guarantees. We address this gap by rigorously integrating PID control with RI., offering theoretical guarantees while maintain- ing the practical advantages that have made PID) control ubiquitous in practice. Specifically, we first formulate PID control design as an optimization problem with a control policy that is parameterized by proportional, integral, and derivative components. We derive exact expressions for policy gradients in these parameters, and leverage them to develop both model-based and model-free policy gradient algorithms for PID policies. We then establish gradient dominance properties of the PID policy optimization problem, and provide theoretical guarantees on conver- gence and global optimality in this setting. Finally, we benchmark the performance of our algorithms on the control gym suite of environments.
Introduction
Reinforcement learning (RL) for control, particularly policy gradient (PG) methods, have received significant attention in recent years (see (25) for comprehensive survey). This has been spurred by theoretical advances in characterizing the convergence, sample complexity, and global optimality
of PG methods in classical control settings like the linear quadratic regulator (LQR) [18] 28 31].
where the underlying problems have challenging, often non-convex optimization landscapes. Further developments have also extended such theoretical guarantees to additional control problems including
stabilization (36). He and Hox control 448] 21 noisy LQR
(3 20 output-feed feedbbck (17) and
partially observed problems like linear quadratic Gaussian (LQG) control 44
16 30). nonlinear
control (42), and robust control problems including adversarial disturbances
[4
49
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The -intoonaalniteer--ddreeee attention within the RL for control literature, despite its prominent place within control theory and engineering practice [ PID control offers built-in structural advantages due to its control law being parameterized by three interpretable components: proportional action to shape stability and transient performance, integral action to eliminate tracking errors, and derivative action based on anticipated future error to enhance robustness. Due to these properties, PID control is a widely adopted paradigm in several domains including industrial control, aerospace, and robotics applications [10), with over 90% controllers used in industry employing PID architectures [22] Further, the output- feedback architecture and low-dimensional parameterization of PID control make it promising avenue for policy- gradient based exploration. However, most PID control design approaches remain
(PID) control architecture has received surprisingly limited
policy-gradient heuristic, relying on engineering intuition and lacking provable guarantees performance, optimality, or even stability 40
In this paper, we bridge this gap by rigorously integrating PG approaches from RI. with the PID) control parameterization, enabling us to obtain convergence and global optimality guarantees enjoyed by PG methods for control while maintaining the practical advantages that have made PID control ubiquitous in practice. Specifically, we first provide an optimization- control problem, and derive exact policy gradient expressions in the PID parameters. Based these gradient expressions, we develop two new PG algorithms, a model- -bsed approach for PID control and model-free approach for PI control. We prove that the objective of these problems enjoys the gradient dominance property, establish convergence rates to global optimality for our model- based approach, and characterize the sample complexity needed to achieve approximate global optimality for our model-free algorithm. Finally, we benchmark the performance of our algorithms on the cont ro gym suite of control environments (5T). We summarize our core contributions as follows:
formulation of the PID
Policy optimization framework and algorithms. Our optimization- based formulation of the PID problem and our proof that its objective enjoys gradient dominance together establish critical link between the PID control problem and the literature on PG methods for control. Our derivation of exact gradient expressions for this problem also paves the way for the development of principled PG algorithms for tuning PID control policies. Specifically, we develop two new PG algorithms: a model-based method leveraging system identification to learn PID control policies, and a model- free method for learning PI control policies. In the model-free setting, we bypass the need for model knowledge by applying the classic policy gradient theorem, and derive a novel algorithm that employs stochastic policies during training to provably learn the optimal parameters of an underlying deterministic PI control policy. This allows us to provably achieve approximate global optimality in a model-free manner on PI problems, resolving an important open problem.
Global optimality, convergence, sample complexity. We establish glohal optimality, convergence, and sample complexity guarantees for our algorithms. For our model -aaee approach, we leverage gradient dominance to establish a linear convergence rate to global optimality. In the model-free setting, we prove that our optimization objective enjoys weak gradient dominance and establish convergence rate and sample complexity guarantees to global optimality. In the model-free setting, the analysis is particularly challenging due to the use of stochastic policies to learn the parameters of an underlying deterministic policy. To overcome this, we establish weak gradient dominance under stochastic policies, then use this to derive the algorithm's 's sample complexity for achieving approximate global optimality. While there are some recent results (33) on thcoretical guarantees for learning deterministic policies via stochastic gradients, these analyses assume global Lipschitz properties, which are not satisfied by PID) control policies. In contrast, we obtain theoretical guarantees of convergence and sample complexity for class of problems that only satisfy less restrictive local Lipschitz properties. In addition, our model-free approach yields a first-order PG algorithm that is applicable not only to PI control but also to general LQR problems (which constitute a special case of our framework where the control policy is parameterized only by the proportional component of the PID policy), whereas only zero-order methods (where gradients are estimated using only objective function evaluations) were previously available [8].
1.1 Related Work
While to our knowledge there is no literature integrating policy optimization theory with PID control, there is a rich literature on PG methods for problems like LQR, as well as attempts from control theory to introduce optimization objectives into PID control, which we review here.
2

---

Policy gradient methods. Convergence of PG methods to global optimality, often referred to as global convergence the machine learning community, has received significant attention in recent
years, starting with the fundamental results of [8] 29 26]. Subsequent works addressed optimal
convergence rates [3], sample complexity 77. regret 6 [5]. and even global convergence for general
utilities (50) 41] and deterministic policies (33]. A key feature of these analyses lies in establishing gradient dominance of the optimization objective, which allows global optimality to be established despite the typical non-convexity of the problem. Within RL for control, PG methods for a variety of control problems [25] have been shown to enjoy gradient dominance, particularly LQR and its
variants where model- and zero-order model-free PG methods have been developed [18 28). PG for least squares problems has been analyzed [13], sample complexity results for PG methods have been provided [14), and system identification based approaches developed [46]. Despite the wide range of works on PG for control, to our knowledge, globally convergent PG methods for PII)
control remain unexplored in the literature. We close this gap in this paper.
Optimization in PID Control. While most PID designs rely practice on heuristic tuning rules and simplified models, there have been attempts to formalize PID control design
optimization prob- lem [ 5 24). Constrained optimization approaches for PID control with robustness specifications 34
19 and linear and bilinear matrix inequality-basee tuning for single and multi-variable settings have also been proposed [1]] 45 91. However, all of these methods rely on frequency domain system descriptions, making them difficult integrate into RL theory, where state space descriptions
fundamental. Critically, these approaches lack theoretical guarantees on optimality and convergence. Recently, (37] proposed a new optimization framework connecting model-based PID control to the standard continuous-time LQR problem. We adopt a similar approach to formulate an optimization problem for PID control design, albeit in the discrete- setting. Importantly, work goes far beyond the scope of (37 by providing exact policy gradient expressions and rigorous theoretical guarantees in both the model-based and model-free settings.
2 Problem Formulation
We focus on the following optimal control problem with linear system dynamics and proportional- integral-derivatiive (PID) control policies, given by
minimize
E
Σ
Me
Y ye + ui Ru:
subject to To+1 Ax, + Buc. y== C.t. Io D,
where z, E R" is the system state at time t 2 0, y/ E I2 represents the output, and u, E RT denotes the output of the control policy. We henceforth adopt the scalar input and output setting (m, p 1) for ease of exposition, but we note that all of our theory can be extended to the multivariable setting. The control policy is parameterized by the PID parameters K Kp, K1, KD R,
[Kp, K,, KDl", where
Dit1
U KPy KD
K₁
and >> 0 is the sampling time of the discrete-time dynamics. The system is initialized at zo D according to some probability distribution D over the state space, Y E RP*P and R E Rm are positive definite matrices, and A 5ax= and B e R=xm the system and the input matrices. Intuitively, for a classical tracking problem, where the objective is to drive the system state to track an external reference, the proportional parameter Ky, responds to the current tracking error, the integral parameter K2 addresses long-term errors by accumulating the error signal over time, and the derivative parameter K D predicts future errors by evaluating the rate of change of the error. The related
PID control problem presented in equations (1 in the PID parameters K is equivalent problem, as formalized in the following proposition. We defer all proofs to the appendix.
Proposition 2.1. The optimization problem is equivalent the following problem:
00
minimize
J(K) = B
Σ
ge Q9. + uug Ruu
(3)
1=
subject to gc+1 = Are + Bue go ~ D.
3

---

where
9t :=
:-
Σ 1/1
Caa.
20 0
(4)
[t]
i=0
A
A
0
B
0
0
B
Q
YC
Y is as in and go D is used to signify
C
I
O
3o ~ D, 20 = O with a slight abuse of notation. The control policy in 3 is given by
Ut gt.
(5)
where the control parameter K is obtained from the PID parameters K = [Kp, K1, Ko) as
KDCB
K x(K) 1
KpC + KD C(A-I)
K1
(6)
7
Proposition 2.1 extends the continuous time results in [37 to our discrete-time PID setting, establish- ing an equivalence between the PID control problem and LQR problem an extended state space. By (6). the parameter is completely determined given the PID parameters K. For this reason, in what follows we will abuse notation and use JK) and (K) interchangeably, depending on context. Specifically, when we write J(K) and [(K), we consider J( K)) and [ K)), respectively. It is important to note that PG methods for LQR (e.g.. (18) do not apply directly to the PID control problem, since, though K in (5) is completely determined given the PID parameters K by ( there does not necessarily exist a feasible decomposition of K in terms of K. For this reason, it is not possible to use existing PG methods for LQR to solve (3, then translate the resulting policy into feasible PID policy. Fortunately, possible to perform PG descent directly the PID parameters, To achieve this, we derive gradient expressions in the PID parameters in the following section.
Algorithms
In this section we develop PG algorithms for optimal tuning of PID and PI controllers. In order to achieve this, we first derive policy gradient expressions for the objective (3) with respect to the PID parameters in Theorem 3.1 then build this result derive algorithms. The first algorithm is model-based approach that leverages system identification [27] to learn a controller for the full PID problem. The second algorithm is model-free approach for learning PI controller. We assume throughout that the matrix C appearing in (T) is known, which is common in practice, since the output y is typically designed depending on the application hand.
PID Gradient Expressions. We begin by deriving expressions for the gradients of J(K) with respect to Kp, KG, KD. We will use these expressions the remainder of this section to develop algorithms for performing gradient descent in the PID parameters. In order to obtain closed-form
gradient expressions for J(K) (K) in terms of the PID parameters K, we first develop additional notation
concerning the closed-loop dynamics of (3. First, from g++1 = Ag: - Bu, and u = Kgr, we get
9++1 = Axgi
AK À BK.
(7)
Expanding the definitions of A, B yields
A BFbKPC BFoKn c(A-D)
-BFbKi
Ax
(8)
C
where
FD [1 KpCB
(9)
For this closed-loop system, the corresponding cost function is given by ((R) [g0 Pk
where Px is the solution to the discrete algebraic Riccati equation
AKPKAR
Pk + +Q+ KT RK = O,
(10)
and is obtained from K via 6 We now provide gradient expressions for the objective of the PID parameters.
3 in

---

Algorithm PG4PID
Algorithm 2 PG4PI
1: Input: Tolerance E, stepsizes ap, 01, aDD,
1: Input: Tolerance E, stepsizes ap, az, trajec-
trajectory length N
tory length N, noise parameter o > 0
2: Initialize: p,, KT. D. set t <
2: Initialize: Ko [ KY set t f- 0
3: Generate 20; uo...., J,,, UN with (5)
3: repeat
4: Form XN. UN from the above trajectory
4: Sample go
D
5: Estimate A, B from XN, UN using
5: Generate
6: repeat
go: u0, CO:
........9--- 1N-1,,
7: Compute K2 from K, (Kp, Kj. Kb
with T
using (6)
6:
C kk= Cc
NN
8: Estimate Pk, using
88
7: Compute
V
log R (uolgo),
9: Compute Ek. using
P, I using
22
23
10: Compute
8: Kt
KP
V (K,), , J(K), (KoJ(K))
apC, V Kp log R (wolgo)
11:
from Theorem 3.1
Kptl + Ki, V Kp ((K))
9:
KL+1 x Kj as C.CVV
log & (ualg0)
10:
1+1+1
12:
Kit1_ + Kj an J
11:
until
V J(K)) <E
13:
Kttl -- kb oD kb (Ki)
14:
t
15:
until
[ <
Theorem 3.1. The gradients of the objective (KK of B
with respect to Kp, K1, KD are given by
VKpJ(K) = 2FDEREKKT
(11)
VK,J(K) = FF
(12)
J(K) = 2FD (a1 (A (A- -)= - 0203);
(13)
where Ts [n Onxi], To 10ix 11. a (r + KnCB)
02
(r
K CT Kr + (A- -ATT +T? (rTTKp), KD), a3 CB, and
Ex = (R+ B PKA, (14)
ΣK E
gigt
(15)
Recall that Ex and Σg functions of K and therefore of K by gradient expressions provided by Theorem 3.1 we turn next to developing algorithms using them.
6 Equipped with the explicit
3.1 Model-based Policy Gradient
We now derive our model-based PG algorithm for solving I Our approach is model-based in the sense that it relies on explicitly estimating the system matrices and and the solution Pи (IO) from collected trajectory data. Our algorithm builds on the approach proposed in (46), with the key difference that our procedure for gradient estimation procedure requires estimation of fewer system matrices, thereby enjoying superior computational efficiency.
to the Riccati equation
To estimate A and B, follow least-squares system identification approach similar that proposed in (27)). Given trajectory data 2o, uo. I1, 41,.... It-1, M441 collected from the system, for some t O. and taking k < t, let X₁+ [zo F1 ... Ix-1], U2 [uo U1 [p=1] denote matrices whose columns are formed, respectively, by the states ,0...., It-1 and inputs uo..... U4-1- Using this trajectory data, we can estimate A and B by simply computing the least-squares estimator of the error ||X (AX, B in terms of and B, as described Section A.2.2 the Appendix. To estimate Px, for K, at time 2 0, we use the iterative Riccati equation solution approach described in Section A.2.2 in the appendix. Combining the above procedures for estimating A, B, and Px with the gradient expressions of Theorem 3.1 provide the model-based PG algorithm in Algorithm
5

---

3.2 Model-free Policy Gradient
Though the model-based Algorithm T achieves a globally optimal PID controller as shown in Section 4 direct estimation of A, B. and Pk can be computationally burdensome in practice. To address this, we now propose model-free PG approach to learn globally optimal PI controllers without the need to estimate the system dynamics. The key to our approach is to use stochastic policy centered around the deterministic PID policy (5). whose form allows us to apply the PG theorem (43) to perform model-free learning of optimal PI control policies. While the use of Gaussian policies is widespread in the RI. literature, the key innovation in our work is the use of the PID parameterization
(5) for the mean. This allows to combine the PG expressions of Theorem 3.1 with the PG theorem
43 to directly tune the parameters of our control policy in a model-free manner.
For state g in the augmented state space of problem (3), consider the deterministic policy Mg (g(g)) (g) Kg g of (5). We define a stochastic policy based on this deterministic controller by
=
(w- (u (u- (o))
& (ulg)
(16)
0v22
where o > 0 is user-specified constant. Notice that (6) is simply the probability density function corresponding to Ni (g). f),
Gaussian random variable with mean (*(9)) and covariance o² where / is the identity matrix. Sampling a control input u ~ x g (+lg) is thus equivalent to using the control f (g) plus zero-mean Gaussian noise. Since the policy in (16) is stochastic, it can be used in the policy gradient theorem, which states in the finite-horizon case that, for initial go and horizon N,
N-1 I
V((K) = E Σ
elgr, ) V log T (wol90))
(17)
: of Qg: uf Ru, This expression enables truly model-free approaches to solving (3). since it provides way minimize costs given access only to information about costs and the policy Tg- Note that, in order for (17) to be well-defined, it is critical that Tg (ulg) > O, for all g. u. The stochastic policy x(-lg) satisfies this property, while the deterministic controller p (9). which can be viewed as the limiting case of (16) as a + 0, does not. This highlights the necessity of introducing the stochastic policy (16 in order apply (17).
where c(ge; u=):=
Notice that (17) holds whether the gradient is taken with respect K or K, again by equation (6) In our setting, if the score function Vk, log (uulgo) (o) is well-defined and efficiently com- putable, for each = P, T. D, then (7) can be used to approximate V J(K) by sampling go D, Ui ~ (190), then collecting an N-length trajectory go: uo 91, M1,....9N' 11. UN-1, where N is sufficiently large. We therefore need expressions for computing V log क (ulg), for
i P,I, D. First notice that, for i P, I, D. equation (16) and the chain rule give
VK, log R (ulg) 2 (u 1 (g)
(18)
Thus, to apply 17 all we need are the following expressions for VK,MR i Theorem 3.2. For K defined in (6. we have
V VKPMR (g) -T(r KDCB)
(19)
VK (g) = T KDDD)
(20)
V VKDHR(g) = -DDB
1)T'g - KgCBB.
(21)
It is clear from Theorem 3.2 that VKPH& (g) and V KKHHR R (g) depend on B, while V K (g) depends on both A and B. Due to the presence of KD in the denominator, however, we note that fixing KD 0 eliminates this dependence on A and B in (19 (0)), allowing us to estimate gradients
for PI policies in a truly model-free way. Combining (18) with Theorem 3.2 we immediately have:
Corollary 3.3. For PI control, with Kn O, we have
Vxp log (ulg) T CT
(ug(g) u)
Vx, log 9 TTT (MR( u)
(22)
(23)
o²
In the convergence results of the following section, a in 22 23 is chosen to be small enough that suitable gradient dominance and convergence results can still be recovered. In practice, the choice of

---

comes down to ensuring numerical stability, and in our experiments we observed that Algorithm 2 remained stable for wide variety of a values. Pseudocode for our model-free approach based on Corollary 3.3 is provided in Algorithm 2 Though our model-free algorithm provides only PI instead of PID control, we emphasize that is still broadly applicable, since active tuning of the derivative component of PID controllers is uncommon in practice (4]
4 Convergence Analysis
In this section, we provide convergence and global optimality guarantees for Algorithms Tand 2 We first provide a detailed overview of our results. Denote the expected costs under the deterministic policy (5) and the stochastic, Gaussian policy (16), respectively, by
(K) B
Σ 1=0
ye Yu + us Ruc
(24)
(K) 1. R
Σ 1=0
ye yu + un Ru].
(25)
and let K* and K& denote the global minimizers of 24 and (25). respectively. Note that, due to the structure of problem (3). such optimal parameters are guaranteed to exist and to be unique (18). In what follows, we first establish that the objective (24) enjoys the gradient dominance property in the PID parameters K, which implies that any first-order stationary point of (4) is globally optimal. This key result, which leverages Proposition 2.1 and our PID policy gradient results of
Theorem 3.1 lays the foundation for the global optimality guarantees for both algorithms. Second, establish the convergence rate of Algorithm 1 to the global minimizer K of (24). Finally, provide convergence rate and sample complexity results demonstrating that Algorithm 2 achieves an approximately globally optimal solution of (24). We highlight that existing analyses of PG for LQR [18] and global optimality of stochastic PG methods for learning deterministic policies [33 require gradient dominance and globally Lipschitz system dynamics, whereas we improve on these results in our analysis by requiring only weak gradient dominance [32] and locally Lipschitz system dynamics.
PID Gradient Dominance. We now establish gradient dominance of the objective (3) in the PID pa- rameters K, which implies that any algorithm that achieves first-order stationary point of the problem fact achieves global optimality. For any matrix M, let p(M) and min (M) denote its spectral radius
and minimal singular value, respectively, and let MPID (VVKK)) (VKKI)T (VKKD)T Finally, define the set of stabilizing PID parameters as
K = [(K,,K, KD) R3 p(A BK) < K satisfies 6
(26)
We have the following result.
Theorem 4.1. Let K = min Rek J J(K). Then
(((K) ((RK) S a V [() for all K K,
(27)
where ā = =E Jmmin (Ex) ) ||MrDll K obtained using with K R and, Σx- is obtained from 15 with K = K*
As mentioned above, Theorem 4.1 implies that any first-order stationary point of J(K) is in fact globally optimal, i.e., that if K' satisfies V J(K') 0, then K" is optimal, since J(K') = (KK).. For this reason, the development of PG algorithms for 3 is well-motivated: PG algorithms performing gradient descent in K on this problem can be reasonably expected to achieve global optimality. We next leverage Theorem 4.1 to show that Algorithms [ 2 do precisely this.
4.1 Model-based policy gradient for PID
We now establish the convergence rate of Algorithm I to the global minimizer K of (4). As we will see, Algorithm I Jaciivess global optimality due to two factors: (i) the algorithm directly learns the parameters of the deterministic controller (5) and (ii) the problem 3 enjoys gradient dominance in the PID parameters, as established in Theorem 4.T above. The following result, which partially extends the result [13] Theorem 3] to the PID setting, establishes the convergence rate of Algorithm to the globally optimal objective function value.
7

---

Theorem 4.2. Let > O be as in Section A.3.1 Fix r/ E (), let Algorithm D run with stepsizes op, OLD, = = and let (Kt) denote the sequence of parameters generated. Then there exist (0,1) such that
(K.) JR() S x² ( (J.
(28)
This theorem establishes linear convergence of Algorithm to a globally optimal solution in the case where A, B and Pk, estimated perfectly at each step. This result be generalized to incorporate estimation error in the system identification procedure by following analysis techniques such as those in [46]. Since we focus the development of globally convergent PG methods for PID control in this paper, we leave this for future work.
4.2 Model-free policy gradient for PI
We now turn to providing convergence and global optimality guarantees for Algorithm Specifically, establish convergence rate and sample complexity results for achieving within 8 > of an e-global solution of (24), where 3 is controlled by choice of variance o² and rollout length N in Algorithm 2 It is important to observe that, despite the fact that Algorithm Saathhmmaattue actuully optimizes (5
it is nonetheless able to achieve near-optimality for (24) for appropriate choice a and N. This is because the objective (25) enjoys weak gradient dominance,
prove in Theorem 4.4, Similar to [33). the reason that approximate instead exact c-optimality is achieved is due to the use of the stochastic version (T6) of the deterministic controller
now establish that the objective (25) of Aigorithmmeejoy the weak gradient dominance property studied in (33), which is critical to establishing the global optimality guarantee below. We prove this result under the following mild assumption, which a technical condition ensuring that the difference between the costs (24 and (25) remains bounded.
Assumption 4.3. The variance of the stochastic policy 16 at time 2 is of = o2/ t)
Assumption 4.3 is akin to technical assumptions on decreasing step size that are common across most gradient-based algorithms, including stochastic gradient descent (SGD). This assumption can likely be relaxed in practice, since we observe across all our experiments in Section that PG4PI achieves convergence even with constant o,, that is aN o, Vt 0. In addition, our experimental ablation studies in A.4.2 also demonstrate convergence for a wide range of fixed values of a.
Theorem 4.4. Let Assumption 4.3 hold and let Kū be a minimizer of 25
We have
(k) J (Ka) s aV (K) +
(29)
for all Ke K, where K is in 26 and 3 2
3
With Theorem 4.4 in hand, we turn our main result. Denote the estimate obtained from 17 by
V J (K)
cgg, u. V log & (wolgo).
(30)
We will need the following conditions on 30
Assumption 4.5. There exists finite V such that, for all K, go. uo and N E N, we have
Var V J [K)l < V/N.
Assumption 4.6. For { (K,} generated by Algorithm we have & | & 1, for all T.
Assumption 4.5 a mild condition also assumed in 333. and can likely be shown to hold under reasonable conditions a and mild continuity and differentiability properties of J In practice, we can enforce this assumption by choosing a sufficiently long rollout length, which is common strategy to decrease variance of PG estimates in RL. Assumption i.6 for technical convenience Corollary A.9 s proof and can be dispensed with by using appropriate gradient normalization in Algorithm We observed in our experiments in Section that the proposed algorithm PG4P1 achieves convergence even without such gradient normalization, indicating that Assumption 4.6 can be relaxed in practice. We are now ready to provide our sample complexity result for Algorithm
8

---

HD Turling PPO
-00
New Eamin Caatiah
-30
140 Rold
-40
20
66
220
Numberr Semples
Figure I: Reward curves for Algorithm| 1 (PG4PID), Algorithm 2 (PG4PI), and PPO-based PID on the 8-dimensional Chemical Reactor and 48-dimensional LA University Hospital environments. Plots show mean and one standard deviation over multiple independent training runs.
PCGR
LOR Chemical
F64M
-25
cenoc
120
AG0
1.00
20 1.0
20
6G.
Figure 2: Stabilization and reference tracking curves for Algorithm 2 (PG4PI) and model-free LQR, on the 8-dimensional Chemical Reactor environment. Plots show that both PG4PI and LQR stabilize the states to origin; however, LQR fails to track the unit-step reference input in contrast to the perfect tracking obtained with PG4PI.
Theorem 4.7. Let Assumptions 4.3 4.5 and 4.6 hold. Let Algorithm run for T > 0 iterations with rollout length N > 0 and stepsizes = 1) satisfying n S min 1 5² Nå
Im co IT v }. Let
(Kitt=o..... denote the sequence of iterates generated and B 2 [|] ? Then, we have
S V 462N
If Vnaz N
B B[J 11 ( or 1 1-1
+
(31)
where Cj m, (Kt)) Jo (KK) B/3} for 0 k max(1(Ko), ((Kr)), where 1(K) is the local Lipschitz constant defined in Corollary A.8
T, and, IT
Moreover, to guarantee E|J J (K*) I c 3, for 0. setting the stepsize to be N N 1 412 V yields sample complexity most
NT
16a V Lax
log
Cr
(32)
Theorem 4.7
provides useful tool for striking balance between suboptimality and sample com- plexity: choosing a sufficiently small and N sufficiently large improves the convergence rate and suboptimality as quantified (3T). while (32) quantifies the corresponding increase sample com- plexity. We emphasize that [33] assumes the objective and its gradient are globally Lipschitz, while the objective of (3) known to only satisfy local Lipschitz properties [8 28). To overcome this, we
quantify the maximum possible deviation that can occur given the local Lipschitz properties over the
entire sequence of parameter iterates, then incorporate this into the bounds 31 32 See Appendix
A.3 for details.

---

5 Experiments
Experiment setup. We evaluate the performance of Algorithms and and compare with baseline methods on two environments from the control gom library [5T]: the 8-dimensional state-space Chemical Reactor and 48-dimensional state-space L.A University Hospital environments. Both environments are tracking problems, where the objective is to stabilize the system and ensure that the output tracks the reference input. On each environment we ran independent replications (see appendix for details) of Algorithms I and 2, the PG method for LQR from [18). and the Stab lebeleen Base ines [38] implementation of PPO 39 with policy parameterization mapping states to PID parameters. We collected rewards (i.e.. negative costs), tracking error, time- tome-domain tracking information, and state evolutions encountered during training. For our figures, we plotted reward, tracking performance, tracking error, and state evolution as function of the number of samples used, i.e., the number of interactions the algorithm performed in the environment during training. We provide experiments illustrating reward performance for Algorithms T 2
and PPO in Fig. [ tracking performance
and error in Appendix A..4TT ablation studies comparing various values of a for Algorithm 2 in Appendix A.4.2 and comparison with model-free LQQR in Appendix [A.43]] We also compare the tracking performance of our proposed model-free PG4PI algorithm with model-free LQR, on the Chemical Reactor environment, in Figure 2 The code for this implementation is publicly available at
com/ sha raaarmml22 12 256 6/RL- Se///RLopptt oott-pptimml... pid
General discussion. As illustrated in Fig. | for Algorithms I 2 the rewards accumulated during training rapidly converge to zero on both environments, indicating that both algorithm rapidly learn to stabilize the system and track the reference output. PPO, on the other hand, requires a large number of samples before it begins to learn to stabilize the system, and its tracking performance remains far worse than that of our proposed methods. The superior performance of Algorithms and Iis due to the fact that they directly search over the space of PID parameters and are guaranteed to converge to global optimality, while PPO searches over the much larger space of all policy parameterizations mapping states to PID control parameters and may only achieve a local optimum or saddle point of this much larger, highly nonconvex problem where gradient dominance likely does not hold. Finally, our ablation experiments studying the impact of a on performance of Algorithm 2 indicate that the choice of a does not have a noticeable impact convergence in practice.
Benchmarks vs. LQR. We implement PG-based LQR algorithms from [88 on both the environments. From Figures 2 and 6 in the Appendix, we see that the PID control policy achieves zero steady-stan tracking error in contrast to non-zero tracking error with the LQR policy, due to the inclusion of the integral parameter in the PID control policy. Importantly, LQR/L.Q variants are known to be fragile to model error [15). In contrast, PID policies can yield higher robustness margins in practice due to the output-feedback structure and low dimensionality. We demonstrate this in Fig. ii the Appendix, where a very small perturbation of the matrix A ( [|AA|| = 0.5) breaks stability even in model-based LQR, while our model-free PI policy maintains stability and tracking.
6 Conclusion
In this paper, we proposed and studied new optimization framework bridging the gap between PG methods for control and the PID control architecture. Building on this framework, we developed model-based and model-free PG methods for learning PID and PI control policies. respectively, then established convergence, global optimality, and sample complexity guarantees. Important future directions include extending our mathematical framework to encompass multi-objectiiv trade-offs arising from classical control specifications through mixed PID Ha/Ho> controller parameterizations, as well as incorporating hard constraints on system states or outputs, which will involve extending our theoretical framework to constrained optimization settings via safety- constrained exploration (42 or safety filters [2] to obtain safe PID policies.
References
[1] Alekh Agarwal et al. "Optimality and approximation with policy gradient methods in markov decision processes". In: Conference on Learning Theory. 2020, pp. 64-66.
[2] Karl J Åström. "PID controllers: theory, design, and tuning", In: The international society of measurement and control (1995).
10

---

[3] Karl Johan Äström and Tore Hägglund. Advanced PID control. ISA-The Instrumentation, Systems and Automation Society, 2006.
[4] Karl Johan Áström and Tore Hägglund. "The future of PID control". In: Control engineering practice 9.11 (2001), pp. 163--1175.
[5] Karl Johan Åström, Hélène Panagopoulos, and Tore Häggllun. "Design of PI controllers based on non-convex optimization". In: Automatica 34.5 (1998), pp. 585-601.
[6] Qinbo Bai, Washim Uddin Mondal, and Vaneet Aggarwal. "Regret analysis of policy gradient algorithm for infinite horizon average reward Markov decision processes". In: AAAI Conference on Artificial Intelligence. 2024.
[7] Amrit Singh Bedi et al. "On the hidden biases of policy mirror ascent in continuous action spaces". In: International Conference on Machine Learning. 2022, pp. 1716-1731.
[8] Jalaj Bhandari and Daniel Russo. "Global optimality guarantees for policy gradient methods". In: Operations Research (2024).
[9] Fernando D Bianchi, Ricardo J Mantz, and Carlos Christiansen. "Multivariable PID control with set-point weighting via BMI optimisation", In: Automatica 44.2 (2008), pp. 472-478.
[10] Rakesh P Borase et al. "A review of PID control, tuning methods and applications". In: International Journal of Dynamics and Control (2021), pp. 818-827.
[11] Stephen Boyd, Martin Hast, and Karl Johan Åström. "MIMO PID tuning via iterated LMI restriction". In: International Journal of Robust and Nonlinear Control 26.8 (2016), pp. 1718- 1731.
[12] Lukas Brunke et "Safe learning in robotics: From learning- basedd control safe reinforce- ment learning". . In: Annual Review of Control, Robotics, and Autonomous Systems 5.1 (2022),
pp. 411-444.
[13] Jingjing Bu and Mehran Mesbahi. "Global convergence of policy gradient algorithms for indefinite least squares stationary optimal control". In: IEEE Control Systems Letters 4.3 (2020), pp. 638-643.
[14] Sarah Dean et al. "On the sample complexity of the linear quadratic regulator". In: Foundations of Computational Mathematics 20.4 (2020), pp. 633-679.
[15] John Doyle. "Guaranteed margins for LQG regulators". In: IEEE Transactions on automatic Control 23.4 (2003), pp. 756-757.
[16] Jingliang Duan et al. "On the optimization landscape of dynamic output feedback linear quadratic control". In: IEEE Transactions on Automatic Control 69.2 (2023), pp. 920-935.
[17] Ilyas Fatkhullin and Boris Polyak. "Optimizing static linear feedback: Gradient method". In: SIAM Journal on Control and Optimization 59.5 (2021), pp. 3887-3911.
[18] Maryam Fazel et al. "Global convergence of policy gradient methods for the linear quadratic regulator". In: International conference machine learning. PMLR. 2018, pp. 1467-1476. [19] Olof Garpinger. "Design of robust PID controllers with constrained control signal activity"'. In: (2009).
[20] Benjamin Gravell, Peyman Mohajerin Esfahani, and Tyler Summers. "Learning optimal con- trollers for linear systems with multiplicative noise via policy gradient". In: IEEE Transactions Automatic Control 66. 61 (2020), pp. 5283-5298.
[21] Xingang Guo and Bin Hu. "Global Convergence of Direct Policy Search for State-Feed Feedback Hos Robust Control: A Revisit of Nonsmooth Synthesis with Goldstein Subdifferential" In: Advances in Neural Information Processing Systems 35 (2022), pp. 32801-32815.
[22] Tore Hägglund and José Luis Guzmán. "Give us PID controllers and we can control the world".
In: IFAC- PaC-PPapesOOnLine 58.7 (2024). pp. 103-108.
[23] Ben Hambly, Renyuan Xu, and Huining Yang. "Policy gradient methods for the noisy linear quadratic regulator over a finite horizon".. " In: SIAM Journal on Control and Optimization 59.5
(2021), pp. 3359-3391.
[24] Martin Hast al. "PID design by convex-concave optimization". In: 2013 European Control Conference (ECC). IEEE. 2013, pp. 4460-4465.
[25] Bin Hu et al. "Toward theoretical foundation of policy optimization for learning control policies", In: Annual Review of Control, Robotics, and Autonomous Systems 6.1 (2023),
pp. 123-158.

---

[26] Yanli Liu et al. "An improved analysis of (variance-reduced) policy gradient and natural policy gradient methods". In: Advances in Neural Information Processing Systems 33 (2020), pp. 7624-7636.
[27] Lennart Ljung. "System identification". In: Signal analysis and prediction. Springer, 1998, pp. 163-173.
[28] Dhruv Malik et al. "Derivative-free methods for policy optiiization Guarantees for linear quadratic systems". In: Journal of Machine Learning Research 21.21 (2020), pp. 1-51.
[29] Jincheng Mei et al. "On the global convergence rates softmax policy gradient methods", In: International Conference on Machine Leurning. 2020, pp. 6820-6829.
[30] Hesameddin Mohammadi, Mahdi Soltanolkotabi, and Mihailo R Jovanović. "On the lack of gradient domination for linear quadratic Gaussian problems with incomplete state information". In: 2021 60th IEEE Conference on Decision and Control (CDC). IEBE, 2021, pp. 1120-1 124. Hesameddin Mohammadi et al. "Convergence and sample complexity of gradient methods for the model-free linear-quadratee quadratic regulator problem".. In: IEEE Transactions on Automatic Control 67.5 (2021), pp. 2435-2450.
[31]
[32] Washim Mondal and Vaneet Aggarwal. "Improved sample complexity analysis of natural policy gradient algorithm with general parameterization for infinite horizon discounted re- ward markov decision processes". In: International Conference on Artificial Intelligence and Statistics. PMLR. 2024, pp. 3097-3105.
[33] Alessandro Montenegro al. "Learning Optimal Deterministic Policies with Stochastic Policy Gradients". In: Forty-first International Conference on Machine Learning. 2024.
[34] Hélène Panagopoulos, KJ Åström, and T Hagglund. "Design of PID controllers based on constrained optimisation". In: IEE Proceedings Control Theory and Applications 149.1 (2002),
pp. 32 40.
[35]
Bhrij Patel et al. "Towards Global Optimality for Practical Average Reward Reinforcement Learning without Mixing Time Oracles", In: Forty-forst International Conference on Machine Learning. 2024, pp. 39889-39907.
[36] Juan Perdomo, Jack Umenberger, and Max Simchowitz. "Stabilizing dynamical systems via policy gradient methods". In: Advances in neural information processing systems 34 (2021), pp. 29274-22222 29226....
[37] Boris T Polyak and Mikhail V Khlebnikov. "New criteria for tuning PID controllers". In: Automation and Remote Control 83.11 (2022), pp. 1724-1741.
[38] Antonin Raffin et al. "Stable-baselines3: Reliable reinforcement learning implementations".
In: Journal of machine learning research 22.268 (2021), pp. 1-8.
[39] John Schulman et al. "Proximal policy optimization algorithms". In: arXiv preprint arXiv: 07777700647460 (2017).
[40] Oluwasegun Ayokunle Somefun, Kayode Akingbade, and Folasade Dahunsi. "The dilemma of PID tuning". In: Annual Reviews in Control 52 (2021), pp. 65-74.
[41] Wesley A Suttle. Alec Koppel, and Ji Liu. "Occupancy information ratio: Infinite-horizon, information -iirecteddd, parameterized policy search". In: SIAM Journal on Control and Opti- mization 62.6 (2024). pp. 3145-3171.
[42] Wesley A Suttle et al. "Sampling- based safe reinforcement learning for nonlinear dynamical systems". " In: International Conference on Artificial Intelligence and Statistics. PMLR. 2024, pp. 4420-4428.
[43] Richard S Sutton et al. "Policy gradient methods for reinforcement learning with function approximation". In: Advances in neural information processing systems 12 (1999). [44] Yujie Tang, Yang Zheng, and Na Li. "Analysis of the optimization landscape of Linear Quadratic Gaussian (I.QG) control". In: Mathematical Programming 202.1 (2023), pp. 399- 444.
[45] Qing- Qing-Guo Wang al. PID control for multivariable processes. Vol. 373. Springer Science & Business Media, 2008
[46] Farnaz Adib Yaghmaie, Fredrik Gustafsson, and Lennart Ljung. "Linear quadratic control using model-free reinforcement learning". In: IEEE Transactions on Automatic Control 68.2
(2022), pp. 737-752.
12

---

[47] Kaiqing Zhang, Bin Hu, and Tamer Basar. "On the stability and convergence of robust adversarial reinforcement learning: A case study on linear quadratic systems". In: Advances in Neural Information Processing Systems 33 (2020), pp. 22056-22068
[48] Kaiqing Zhang, Bin Hu, and Tamer Basar. "Policy Optimization for H2 Linear Control with Mo. Robustness Guarantee: Implicit Regularization and Global Convergence". In: SIAM Journal on Control and Optimization 59.6 (2021), pp. 4081-4109
[49] Kaiqing Zhang al. "Derivative-free policy optimization for linear risk-sensitive and ro- bust control design: Implicit regularization and sample complexity". In: Advances in neural information processing systems 34 (2021). pp. 1949-2964
[50] Kaiqing Zhang et al. "Global convergence of policy gradient methods to (almost) locally optimal policies". In: SIAM Journal on Control and Optimization 58.6 (2020), pp. 3586-361
[51] Xiangyuan Zhang al. "Controlgym: Large-scale control environments for benchmarking rein- forcement learning algorithms". In: 6th Annual Learning for Dynamics & Control Conference.
PMLR. 2024, pp. 181-196.
13

---

A
Appendix
Here, we present proofs of all the results in the paper, and additional details for each section.
A.1 Proofs and Additional Details for Section 2
Proof of Theorem
2.7 From = Cr², we can write
vz Y = C2fC = = c = ge Qgi.
(33)
Now, the PID control policy from (2) can be expressed as
U1 KpCr KDC Tk+1 K2
(34)
Substituting 21+1 Ar Bus. can be further written
= - KpCCr - KDC Ara + Bug - St. Kizt
(35)
Rearranging the above equation, we get
1 1 (KPC+++
KDCB
KD C(A xe + K(2))
U4 = - [1
(36)
Substituting the control policy from
336 into the dynamics Ie+1 Axx + Bur, we can write
D++1 = Ax B[1 KDCB (KrC + KD C(A
11--1 C(A-1)
+ K121)
(37)
From 21 = Ei=0 y/s we can write 24+1 = Yt + 24 = Cx + 24. Now, we have
1141
Ax, BB1 + KOOBI-
(KpC + KD C(A-11) + K1z))
Gt+11
2++1
Cx2 + 2t
Ax Cut + Zt
-B + KyCB
((KPC+ + KD C(A-11) )azt + Kizz)
+
Axi Ca, + 2t
B
KuCB KoCBB-I((Ks) -I(KKCC + K C(A 1)
1 1
Jo + K
(38)
Recalling from 6 that
KDCB
K
1
KpC+ + KD C(A-N K1]
(39)
5
we can write
gt+1 Age + But;
(40)
A 0
where A=
B
and Uy = -Kg.. Finally, from 33 we have that
C T
0
J(K) = E
9i TY th + uu Ruy = E
Σ
o6 Q9i + uf Ruy)
(41)
t=
whence the PID control problem can be rewritten as
80
minimize J(K) = Σ
gi
T Qg.
ug Rut
(42)
subject 9++1 Ar, + Bug, 9o D.
with u, -Kg, where K is obtained from PID parameters K using 6
14

---

A.2 Proofs and Additional Details for Section 3
In this section, we provide detailed proofs of each of the results of Section We begin by defining the cost-to-go at a fixed initial state 9u by
(x( = gō Pk90-
(43)
From 10). the cost-to-go can also be written as
(90) = 90 (A + Q*K**+HHH RK)90.
(44)
Notice that J(K) -D [Vx (go)]. This cost-to-go expression will be useful when calculating
gradients with respect to the the individual PID control parameters Kp, K1, KD in the following gradient expressions.
A.2.1 Proof in subsection
En route to proving Theorem 3.1 we establish the following helper lemmas that provide derivatives
with respect to the parameters in K of certain expressions involving Ax and KJ RK that arise in the main results follow. We first begin with the derivative of the expression v" Axgo, where is arbitrarily chosen vector with appropriate dimensions and go is the initial state.
Lemma A.1 (Helper Lemma for Theorem B.I]. Let v be a vector of suitable dimensions such that o A Aggo is scalar, where go is the initial condition. Then, the matrix derivatives V VK, v² Ag for all E (P, I, D) are given as
V KKpU Akgo
FD L..B"ogh'11 BTTIC
(45)
VK T Akyo = FT ogb TT.
(46)
VKDU" Akgo -a1 [BT "gg (a2203 - (A- (A-1)((*)
(47)
where, T₁ = Ih Oaxi] T2 = 10 1. on = (r + KDCB) 02
(T +
( CT Kp - (T ((---) "T
KD + +TI KD), as CB, and FD [1 KpCB
-1
Proof. From 8). we know that 91
Akgo. and therefore
æ - - BFbKPC BFbKD C(A-I) pro BFDKI
(48)
1 CCo + 20.
We note that the variables FD, KP, Ki, KD are scalars, and, that the matrix Pk is symmetric. Since we know that ž does not contain any of the Kp, Kr, Kn parameters, we can write
VK Akgo = V T T1 for all i E (P,I,D,
(49)
where Vin denotes the the first n-elements of u. Then, we can write
Ky T 2 VKn BF BFyKpCou),
where
VKp BFDK = "T BF BFDCro.
due to the fact that Kp is scalar. Observe that, uT,, uT B. Thus,
T Akgo -v- BFDCrO- VK
(50)
Since the above expression is a scalar, we transpose the right hand side to obtain
V VKP T Akgo BB uuu CT
(51)
From the fact that xo = Tu we can write
VK UT Akgo BD vgh IITC.
(52)
Similarly, from 44 we can write
V
T Ki 21
(53)
15

---

Then, since = T99 we can obtain the gradient
VK, Ak9o -FD T
(54)
We can further expand the above as
1+ KaCH (KpODoKD C(A-1) - K)z0))
V "n = -V KD ( B-
C(A- do K120)] KD
1
--lvim
1+ KnCB
T
1+ KOCB (V VKD(KPCZOk C(A-1)
-uT B
K1z0)),,
T
-v B(KpCroKK C(A 1)
CB
1
20 K120))
7 (1 + KaCRy
T
-6 1
C(A
mo)).
KnCB
T
Simplifying the above expression, we obtain
VKoU" Ak9o = ai [B" ugb (0203 - T (A (A- CA"
(55)
We next provide derivatives of the expression LoK'K K2 RK go with respect K, where go is the initial state.
Lemma A.2 (Helper Lemma for Theorem 3.1 The matrix derivatives V V..g KT RK for each i e D3 are
V 9 T K" RK RK gooo C
C
(56)
V VK,9h 06 K2 RKgo go RK goge IT
(57)
Ko K2 RKgo = 221 RK 9ogd TH (A -) - agaas]]]
where, T,
In Onxi] T3
[0 11. an
(r + KDCB) 02 (T
+ TT (rTB) Kp + (A- K>
KD), 03 = CB, and FD [1+ -
Proof.
We begin by noting that FD. Kp, K1, KD. R are scalars. Now, for all i (),I,D), we can
write
V K. T T RK go- 2V w" Kgo.
(58)
where w is given by w = R Kgo = RKg RKg..
go. Then, from Kgo = FD(KPC + KD C(A-T)
FpK 20, get
VKY nh Kgo = V Kr "h FDKpCro = w" FDCry = g6 K" RFDCCo
(59)
Since the above gradient is scalar, we take transpose on the right hand side and get
VK,W" Kgu- = FDRK good C..
(60)
We note that zo = Tego. and, therefore we can write
V VKP w Kgo = FD RK Rogo T C.
(61)
Similarly, we can write
VK, w Kgo = FDRK go9 T..
(62)
We now calculate VKD n Kgo. We first expand the gradient as follows:
VKD w T Kgo = w ((KpC + KD C(A- - 1)
200 - FnKrzo) VKo PD
C(A ((A-T)) 11) - -120)
+FDV T [KpC+KK
C(A 1) 1)000=0 -FD (T + CB)2
TCB
-KPC + KDD
30o
+ + FD( FFp( C(A -I)
)zo.
16

---

Expanding these terms, we finally get the following expression
Kx T go = 01 R goo [TH (A - -22"3))
where the expressions for d1, 02 03 defined in statement of Theorem 3.1
We are now ready to prove the Theorem 3.1
Proof of Theorem 3.7 For each i (P,I,)) we have
V K, (90) =V K.9h PK90
(63)
=V K. A Px Ak + Vx.9 Q90 + KK K Tot KT
RK go.
In the above expression, we first focus on V Vk. A P which we can rewrite as
K.90 PK A 2V v" Akgo VK, Pk91.
(64)
where v - A and 9I = Ax90-
However, we note that VX (g)) = of PK9i and thus
V K. Rh PK VV Ak9u + Vk, VK()).
(65)
Then, from Lemma A. we obtain
Kr, v" Akgo FD Pk Ax C
KK, 0 Akgo = FDB PK A
- VK 0" Akgo = an [B" PK Ax - (A- (A-
Moreover, know from Lemma A.2 that
V K K RK = 2F go9o T CT
V Ki oh K" RK 2FDRK gogo I
V Ko T KT RKgo 201 RK gogu T T (A - 1 -a aga3)]]
Combining these expressions,
write
VK, (90) = 2F PK A T CT + VK, +F I CT
CT.
(66)
Note that Kı VK (91) can again be written as
V (g1) = 2F Px A TT + VKK vx(g)) PK Proceeding in this manner, we have
(67)
KK (9)
Σ B TI. TI CF TT
(68)
In light of this, we can write
-(-F) PKAK PAAK FT
(69)
Noting that
EK = (R+ - -B -B PK + RK,
(70)
we obtain
[K, OF.
(71)
Similarly, we can obtain
V ((K) = E (A- -I -
(A-
17

---

A.2.2 Estimation of Matrices Pk) in Algorithm I
In the next two subsections, we provide details regarding the estimation of matrices (A, Pk). We begin by describing a standard system identification approach that we use to obtain matrices (A, B). which are required to estimate the gradients utilized in the update rules of Algorithm |
System Identification for (A, B). We follow the standard least squares system identification ap- proach in [27] to estimate the matrices A and B. Given trajectory data zo. uo, 2,, Ui...... Tt++, U4+1 collected from the system, for some t and taking k S t, we first consider the following dynamical equation
XR+1 = AX2 + BUKr
(72)
where
X5 [oo I1 xk-1];
(73)
U2= = [up 4k-1
(74)
We then define the least squares estimation error as
E(A, = || - (AXx + BU&)1?
(75)
Defining
Xx
0 = [A BJ, M
(76)
U
and noting that Xk+1 O.M. we obtain the following estimate
= 1 MM
(77)
for the system matrices (A, B). With the estimation technique for (A, B) described, we estimate the matrix PK, by the method described in the next section to ultimately estimate the gradients J(K), for each E D).
Iterative Method for Estimating Pk. Consider the following Riccati equation for our reformulated system in Proposition
2.1 where K is obtained from K using O
Px = (Ā - BKY² (x - BK) + Q + KTRK.
(78)
Right multiplying the above equation by of and left multiplying it by 9t. we can write
9. PK9t = o. (A BK) Px(A BK)g: + 9. Q9: + of KT RKg.. Using ge+1 Age- + Bu, Un K gt, we rearrange the above expression and write
(79)
of PKg: - s PKg++1 = of Q9: + ot K gr-
(80)
We can now use 80 to estimate the matrix Px by collecting sufficient samples of g.. Consider
gt := of Qg. + of KTT RKg.
(81)
and the vector Pk E (m+1)) is obtained by vertically concatenating the column vectors of Pk- Before defining the data set for the estimation Pk. we review some preliminary results from matrix theory.
For vector w E R and matrix U R=**, we know that w TUw = Σn Ej= Us where U is the (ih element of U, and w, is the i-th element of the vector Consequently, we can write w """ = Eni of matrix U and the Kronecker product w @ w, respectively.
= U.W., where U. and w are the i-th elements of the vectorized version
Using the above facts, we can write
gt PKg. I PKg++1 = Px. (g. & 9t gt+1 @ St+1);
(82)
from which we obtain
(g. 8 gt ge+1 @ 9(+1)" T Pk gr.
(83)
18

---

We now define matrices G3 and Dn for k 2
m(x+1)
+ + 1, as
G [g0 @ 9r 91
91
9k-1 9k-1),
(84)
O2 [qu 91
4k-1l
(85)
Observe that these matrices can be constructed from the set Xg. defined 73)
Then, we can write
CLv yk
(86)
where Po" Gy = (Gx GR+1)", and yk Og. Finally, we minimize the squared error
E(v) | C
(87)
to obtain
v (G) Gx) G yk.
(88)
In this manner, we estimate the vectorized version of Pk.
A.2.3 Proofs in Subsection 3.2
In this subsection, provide proofs of results in Subsection 3.2 calculating policy gradients
in the model-free setting that are used the update rules of Algorithm 2
We first begin with the
proof of Theorem 3.2, where expressions for the score functions with respect to each of the PID parameters are provided.
Proof of Theorem 3.2 Given (x(g) Kg -11 + KnCB 1 [[KpC'+ + KoC(A-1)) it immediately follows that
K₁2).
VKrKK(g) T(T KDCBB
(89)
Vk,uk(g) -T(T + KDCB)-TT.
(90)
To compute KK (g), first write
K KDCB )K((g) (KpC+ KDC(A-1) (1+
)r KJ2.
5
Then, applying the operator KR on both the sides, we obtain
Vko KDCB
[k(g)) C(A l)x
or, cquivalently,
pk CB
+ (1+ KDCB
]KoHu(()) C(A- 1)x
Rearranging the above expression, we get
VkoKx(g) (r + KnC 1)x Px(g)CB).
where we establish expressions for the score function with respect to PI parameters that do not require the knowledge matrices (A, B).
We now move on to proving Corollary 3.3
Proof of Corollary
3.3
From
18 we know that
V Kn log (ulg) = u u- (x
V kn
(91)
0²
u - u-Mk(9 (K(9))
V K, log (ognx(ulop
V (g
(92)
o²
From Theorem
3.2 with Kp = 0, we get
pk(g)
V Kı log T (ulg)
g TTI C.T
(93)
o²
[k(g)
V K, log T (ulg)
g TT.
(94)
o²
19

---

A.3 Proofs and Additional Details for Section 4
We begin with proving that the objective of problem enjoys the gradient dominance property in K.
Proof of Theorem 4.1 We first define the set of stabilizing control parameters as
I. = {K 11 p(A BK) < <1
(95)
Then, know from Lemma 3 in [8] that the objective in Proposition 2.1 with the control policy in enjoys the gradient dominance property in K, that is,
[J(K) - J(K')] s [[)]) for forall all K E.K,
(96)
where K, K are obtained from K, K using ( and, a= |E ll (
Since,
replace J(K) with J(K) Further, we note that JK)(K)
[V (K) (K) JK)J( We can rewrite <J(K) using the chain rule, as
dk
dJ(K) = (VKKp)VXx (VKK) + (F
dK
or, equivalently,
( = K + m1 V K, (K) + mDmDVVK (K),,
where the matrices m m given by mp V KK my V and my V VKKD. Then, we can write the following:
VKJ(K)= = V (
(97)
where
mp
MPID
mı
(98)
mp
Applying the norm operator on both sides, obtain VKJ(KK) ( MPID
Then,
using the face that |/ [[)]] = ||MPDD || [
we have
[J(K - [KK)] S || | R. ( Finally, from the equivalence ((K (K)., we have
J(K) ((K))J(K()) ((x*)) s a M|MPlll J(K]
A.3.1 Proofs in subsection 4.1
In this section, we present the proof of Theorem 4.2 First, we define relevant variables from Theorem 3 in [13) for our formulation and then adapt their sample complexity result for our setting. We begin with the following conditions on the step size 17. Let v7 > 0 be such that
p(A BK₁) 1,
(99)
Tr K (K) (K) (1 -o))))) > O,
where K, = KP (K) and Amaz (R + RT
BTBBBPP PR
Proof of Theorem 4.2 We note that JKJP((K = 2E Ek. where Σx
E E
9 and Ex
given (14). Comparing to the gradient of the function f(K) from 13 Proposition 1], have
:= Eg. Y, := ΣK,, Ax, = A BK, and é,, = ( KJD Ex, (1- na>ix)) in our
setting. Then, from [13 Theorem 3] for all 0,
(K,)) (K*) S k¹ (Ko) (K*))
20

---

We note that K is function of K by ( and, K := p Kj Kpl' corresponds to K", the global minimizer of J J ( Thus, with slight abuse in notation, we can write
J2())) S & -
We now move on to convergence results for model-free case.
A.3.2 Proofs in Subsection 4.2
First, recall that for the model-free Pl algorithm where KD 0, modify the PT control parameter
to be
Kp
K =
(100)
K1
Consequently, from (6) with KD = 0, we get
K PKpC Kr],
(101)
equivalently,
K = SK,
(102)
where
= IC 1].
(103)
Further, define the matrix Mps
KK VKK1
Mr =
(104)
Second, under Assumption 4.3 the stochastic policy in T6 becomes
(w-H (o1)) (n- (u- (l))
Tx (ulg))
21.5T
(105)
2
that is, us ~ 0² / for all 22 O. We can also write the following:
dt = ug (g:) + Nis
(106)
where ~ ((0,,o2(tt 1)3), for all / 2 0. We note that the {m: are i.i.d. random variables and independent of go-
We now present helper results that we will use the proof of Theorem 4.7 We begin by obtaining a uniform bound the magnitude J (K) JD((R)
Theorem A.3. The costs JR and J satisfy
|R 7022
<
6 for all K E k.
(107)
Proof. Since the control input u, depends on 9o through Ia and {m the expectation in will be with respect to {n and 9o- Furthermore, by (6) we can write J& J K and (P( (K). We expand the expression for JC(K) to get
25
((K B4 Eg
/ Yuu + (ux(9)) + n1) R(ux(g)) +z))
(108)
Further expanding this expression gives
(C(K) = 1. EE
" Yuu + "x(9;) Ruk(9)) + 2n RuK + nz Rny]-
(109)
21

---

We observe that the expectation in (K) is with respect to Pg = Kg. with K given by ( which implies that the expectation is taken with respect go D, and thus
(K) g
33 Yu + (9(9)))
R (g)(
(110)
Thus, we can further write the following:
= =
+ mn 0 goo
2n Rpg (9;)(9))> + +1 Rnjl.
(111)
1=0
First, since {n:}s are independent of go. we have
[
E go
D
Rmjl
= (n
Rmyl
(112)
2=0
j=0
Then, from definition of variance and the fact the En, [ns]] = 0, for all j 2 0, we get
o²
lim [ T T4000
7 Rnp
lim 100
(113)
(1 + 1)2
Since 1 Et=o (e+1)²
6. we therefore have
T
Ro
lim
E
(n Tm
n
T
Rny
(114)
T-000
==0
Second, we examine the expression lim (m) Eg Eo= 2n Rp (9;)] and focus on the j-th term, that is, 1 [2n Rug (a:)).
We note that } are i..d. and that the control input Pe (g;) only depends on the first j 1 random variables. We therefore have
(m: J. Eg. 12 Ru (0,)] 2IE,,, In E H.D P.B.B Ego RR (0;))-
(115)
However, since E,, [n}]] O. we conclude that En 1. Eg 2n Rug ( = 0 and thus
lim (m
g
2nf Ru k (g;))
(116)
T-000
j=0
Therefore we can rewrite
111 using
113 and
116
R
(K = J(K)
(117)
Taking norms on both sides in 117 and applying the triangle inequality finally establishes that
||||o>>
(R(( - -P)))
6 for all K E R.
(118)
now show that the added noise in the stochastic policy has no effect on the gradient.
Theorem A.4. The gradients V JD and V JG are equivalent for all K R, that is
V R J (K) = V &
(119)
22

---

Proof.
From
117
we know that
R J -
(120)
6
for all R E K. Applying the operator Vx R on both sides, we immediately obtain the desired equality
as
J D V V J = 0, for all K E.
(121)
R
11
Leveraging the last two results, we now bound the optimality gap arising due to use of the stochastic policy (16). Theorem A.S. Suppose RG E largming (JG(()) and K E arg ming JD (K). We have
|R|o0>>
(K( (K(())) S
(122)
3
Proof.
Consider the following:
Jg - J2 = ()9(K)) - (K() + -Ka)) (K())
(123)
1 (K()) - (K() + 1 (Ka) - JP
From the definition of (KG)) we have that (K() (K*). Moreover, since J (KG) Ju (K+) + Ro²x² Combining the above two results and then applying Theorem A.3 we get that
6 we have J (Kt) 2 Jd (Kt). We thus have (((K 2 (K()) 2 JuC (Kt).
| |||a2x
-2(())
+ -2((K))_((1)))))
(124)
3
We are now ready prove Theorem 4.4
establishing weak gradient dominance of the cost (R)
Proof of Theorem 4.4 We first rewrite
(K(K) (( (Ka) = - -1000)---00000---000------------
(125)
-K(()) + )(()))
Rearranging terms gives
(K) (Ka) = - (P()) -P(())
(126)
J2()) ([(
Using the triangle inequality, we obtain
[)((K) ( -C(K)))(((a))) <
+ 1)( [p())
(127)
+ + +Z) + J - (K()a
From Theorems A.3 and A.5
we have
R2 2||
(C( (Ka)
(2(()) - JD (R())
(128)
6
From Theorem 4.1 can write
(() (K()) <
|||lo=22
n²
V
|
2 R 2
(129)
6
23

---

or, equivalently,
(K) J (K3) S
2 21|
+
V R (K) /
(130)
3
Then, following Theorem
A.4 finally get
21
(k) - J (KG)
a R JZ))
(131)
We ready to present result the local Lipschitzness of the LQR problem from (28). adapted for our PID control formulation. Recall that the equivalent problem our setting is given by Proposition 2.1 with control parameter K, where K is obtained from using (6).
Lemma A.6. There exist constants (m((K) (,(K), ((K) for a given control parameter K such that for every K' satisfying || K|| S (.) we have
(K V JKJ S [(K)]KK - K|
(132)
Proof. Since we know that (KK) = J(K), the proof follows from Theorem A.4 combined with Lemma 5 from (28
We establish similar local Lipschitzness result for our cost gradient V R J in terms of the parameter K.
Proposition A.7. We have that mmin (Mri) 0.
Proof. From 102 we can write
1 Kp = OCT T CT
(133)
K, = KT.
(134)
Taking derivatives of Kp and K2 with respect to K, we get
VK CCT CT₁
(135)
VK Ts.
(136)
Therefore, write Mp1
CCr CT: T3
CTr
Mp
(137)
and, consequently,
1 CeT CT. TT
(oo CT,TT C c (z(CT.)
CoT CT,T"
Mpr
(CT,)TT) T.]
(138)
T.T!
From the definition of T. and T2, have
T.I I.TT 1, I,T.T! = 0,
which yields
MprM = ालटीयय OCT 0
(139)
1
Finally, we establish that
Tmin (Mp1) = V (MpI(M)) V minn CCT 13 > 0,
(140)
where Aman represents the minimum eigenvalue.
24

---

Corollary A.8. The gradient V ( is locally Lipschitz in parameter K, that is, for all K' such
that K' - K I m(K), we have
V V ( ((()) R |K--R) K||
(141)
Proof. Consider the left hand side of [14] and write
h
V k (K (K') - V k
(142)
Then, from Theorem A.4 we know that V JaJa(K) (KJR((); Therefore, we get
h V Jo (K') JK)
(143)
Applying the chain rule on (K) we get
VKJ V K (K) + V K VK,
(144)
which can be rewritten as
V Kp (K) J (K)
V (K) vV VKK?]
(145)
V Wanj (K)
Note that J2(K sequently, we have
then and therefore JRJ M V J Con-
VK, Jb (K).
h V (F
(146)
Let Tmin Tmin (Mps) represent the minimum singular value of Mp.. Then, we can write
h 2 miin (MP1
V JRJ2((K)) - (K)
(147)
Since Tmin (Mp1) = Jmin ( (MP,) and that Tmin (Mp1) > from Proposition (A.7) we can write
| J - V (R)2(K) S Tmin (MP) I(K) ||K" K||
(148)
Now, defining VK- K' K and V K K, then we can write
AK |AKrC AKJ],
(149)
or, equivalently,
AK SSA.
(150)
Applying the Cauchy-Schwarz inequuill we have
(
<
I(K) K K
(151)
mmin (Mpr)
From Theorem A.4 we can finally write
V (C(") v Jo s s((R)) K 1K'- WP-Rt - =/|
(152)
where
I(K) = minn (Mp1)
I(K) ||S||
(153)
which completes the proof.
Before moving to proving Theorem 4.7 we present weak gradient dominance result in parameter K, from Theorem 4.4
25

---

Corollary A.9 (Weak Gradient Dominance). Suppose that Assumption 4.6 holds for the gradient V VRJO(K) Then the following version of weak gradient dominance holds
-
v
[6 for all K E K,,
(154)
3
where ā = -|2ix Imin (2 ||Mp ?
Proof. The above claim immediately follows from the fact that | [V 2 (R) due to Assumption 4.6 and the fact that ā > O.
We now provide proof of the sample complexity result of Theorem 4.7
Proof of Theorem 4.7 Using the fact that S | and Corollary
A.9
write
( = a V R (K)
+ 3/3.
(155)
Define (K() (Ka) + 3/3. Then, from 155 we have the following:
1 à
max - -(()) < [R)
(156)
We note that, for time-step 1. 2 0.
(K (a = (K JG (K.) +
-
(157)
Using the local Lipschitz property from Corollary A.8 and the second-order Taylor expansion gives
2 (Kt), R+t1 K2) + ((Ki) || Kell2.
(k(1))) < (V K. J
(158)
and we therefore have that
-(()))) < (Ki) - (K())) + (VK Kc+1 Ki)
-
1( I(i,)
(159)
+
|i K.ll..
2
Recalling the update rule Ket1 = K. - 17t V J (Ke), by the unbiasedness of the gradient estimator we have
-C < (K.) J (Ka)
n. V JC
I(Ki) 1/1 |
(160)
Let (KK,, me: 0 < k < f) denote the alagerrra generated by all randomness in the system up until time i, and E.[-] = F.- Taking expectations over the foregoing yields
Eg (((K( - (K()
< E2 (Ki) - J (K m R (Ki), V VR (K()))
(161)
((Ki) +
2
26

---

Given E, V J J (K,) = V R ( from the fact the our estimator is unbiased, and recalling the definition of the variance of random variable, we can write the following
Ee - -
- (K( - J. - VV -n (Kt).
R (.
+ 1(K))
2 11e Var VV J. (K)) I(Ki)
n [E, R J
(K)) |
(K)) - Jo (Kt) 1/| | J (K:)
2
+ 2 Th Vi ar V R J (K,) ((Ki), 1i
1(K))
| Rr (K,)) k
(KK)) - JoH (KG) m 1(Ke)
| k
2
m1
2
1(Ki) 1/ V ar J (Ki)).
+
2
(162)
Using Assumption 4.5
we finally obtain
Et J (Re+1) J (Ka)
S (g((K) Je (K²) - 77 (1 ((1)) 2
m1) || | |
(163)
I(K,) V ni N''
For a sequence of Lipschitz constants {((K))) suppose that the largest Lipschitz constant is given by lmaa Further, suppose that we pick a step size т/t S T then it follows that (1 - A 1/t 2 4 We can then re-write the above inequality as follows
Ee -F S
-
T1 2
R
(i))
+
2 1(Ke)
2
11
V
N
(164)
At this point, we can now follow the proof of Theorem 6.1 in 33
with L2
lm to obtain the bound
T
n ma
V
IT
Vnăa
B [KT) - (KK)) s 3/33+ + cr 1 11-
+
(165)
4 N
N
where we note that, for finite T, we can replace the mm by (Tu. From Theorem A.5
we have
J - (K()) v 2|
|0
23/3.
(166)
6
Note that, by 117 we have Jz(KK) JR( Ro²=" Furthermore, since JP (KK)) js (D(KK), we obtain
2
RO
(K())) 2 J2(
(167)
6
In other words, we have that (K*) is the global minimum. Noisily perturbing the function (K) to yield J (K) leads to a minimum that satisfies (k() < J& (K²), or, equivalently,
1 - (KK+) 2 O. From the fact that x S ||ا||| this implies JK(()) J[(KK)) < 23/3,
(168)
i.e., that
- (K()) 2 (K() - 23/3.
(169)
We therefore finally obtain
T
B[J - JJ SB 1- \1-V
17 V mz 462N
IT
Vnoa
+
ma
N
The remainder of Theorem 4.7 now follows in a straightforward manner from the rest of the proof of Theorem 6.1].
33
27

---

A.4 Experiment Details
We note that both the environments rea and lah have sealar input and output spaces. our experiments, we set R = Y 1.0, where matrices R and Y are as in (I. Further, the cost accumulated over each length-N N rollout, at time t, is given by
++N-1
Σ
(0) Qu: + +uu Rui).
(170)
For the experiments generating reward curves in Figures we ran 100 trials of our proposed algorithms and trials of the PPO algorithm. The simulation hyperparameters for our experiments are provided in Tables ! 3 and Finally, we ran our experiments on Windows laptop, configured with 6-core 17-8750H, 2.20GHz CPU, an NVIDIA GeForce RTX 2070 GPU, and 32GB RAM.
Table 2: Simulation Parameters for PG4PI
Table I: Simulation Parameters for PG4PID
Parameters
Values
Parameters
Values
No. of episodes T
10
No. of episodes T'
10
Rollout length N
100
Rollout length N
100
Step sizes {ar,, aI,
0.0001
Step sizes {ap,ay, 02, ap)
0.0001
Noise variance
0.1
Tolerance
0.001
Tolerance
0.001
Sampling time r
0.1 sec
Sampling time r
0.1 sec
Table 4: Simulation Parameters for PPO
Policy learning rate
0.0003
Table 3: Simulation Parameters for
Palue learning rate
0.0003
Model-free LQR
Entropy coefficient
0.0
Parameters
Values
Clip range
0.2
No. of episodes T
10
Weight decay
0.0
Rollout length N
100
Layer size
Step size
0.0001
Batch size
Tolerance
0,001
Buffer size
60000
Sampling time r
0.1 sec
Number of epochs
10
Rollout length
8
Discount factor
0.99
A.4.1 Validation rea and lah
We present the state, input tracking, and tracking error trajectories for our experiments the Chemical Reactor (rea) and the LA University Hospital (lah) control environments. Recall that we consider the unit step tracking task which entails two subtasks: i) system output tracking unit step input, and ii) stabilizing the closed loop system. The step tracking performance is shown in Figures 3a| and 3b with the model-based PID Algorithm PG4PID, and Figures [c and [3d] with the model-free PI Algorithm PG4PI. We also plot corresponding trajectories for the tracking error (deviation from reference step input) in Figures and Figures[3ejand]1 with PG4PID, and Figures 3g and 3h with PG4PI. Finally, we illustrate the efficacy of our model-based and model-free algorithms for the task of system stabilization. The corresponding state trajectories are illustrated in Figures 4a and 4b with PG4PID, and Figures and 4d with PG4PI.
A.4.2 Ablation Studies
We carry out ablation studies on the impact of the variance parameter a on the convergence of our Model-free PI Algorithm, PG4PI, on both the Chemical Reactor and LA Hospital environments. The results are presented Figure 5 We find that varying the a does not have a significant impact on algorithm performance for small (between 0.0001-2); however, some performance degradation occurs for larger variances (a 3). In general, a is chosen to small enough that suitable gradient dominance and convergence results can still be recovered. In practice, the choice of comes down
28

---

to ensuring numerical stability, and we find from our ablation study that Algorithm 2is is stable for wide range of values.
A.4.3 Benchmarking vs. PG based LQR
We present the results of our benchmarking experiment where we implement both a model-free (zero-order optimization method) PG method and model-based (using exact gradient expressions) In model-based LQR, we use estimates of matrices (A, B) to compute the policy gradient, while we use their Algorithm 1 in [18] to estimate the gradient in the model-free method. The experimental parameters for model-free LQR are presented Table 3 We note that our proposed model-free Algorithm follows first-order optimization method, that our model-free PID control policy outperforms both model-basee based and model-free LQR control policies in both tracking behavior and robustness to model errors, discussed in Section
method for tuning LQR control policies from [18) in Figures (
A.5 Broader Impacts
This work is primarily theoretical and pertains the development of policy gradient based algorithms to design PID control policies, and does not have any direct societal impact.
29

---

PG4PID Chemical Reector
PG4PID on LA Unversity hospital
1.
0.0
heur
-200
1200
000
Number Samples
(a) Tracking performance for unit step refer- ence lah environment with PG4PID.
(b) Tracking performance for unit step refer- ence rea environment with PG4PID.
FG4PI on LA University hosptta
PG4Pi on Chemical Reactor
01-
0.2
Tooking
-35
Tocking
-.3
-1.
-1.4
-23
Seep Inpuc
Step trpus
-25
3200
3200
Number Samples
Number Samples
(c) Tracking performance for unit step refer- ence lah environment with PG4PI.
(d) Tracking performance for unit step refer- ence in rea environment with PG4PI.
PG4PID Chemical
PC4PD on an LA Unwersity hospital
3.0
2.5
2.0
0.3
6.0
200
420
623
662
3200
Number
(c) Tracking error with unit step reference lah with PG4PID.
(f) Tracking error with unit step reference in rea with PG4PID.
PGdPI an LA Unveersity hospital
PGtP a Chemical Resctor
ny
Tracking Enor
10
23
Trockng Ereer
65
0.
1200
Samples
Sampless
(g) Tracking error with unit step reference 1ah with PG4PI.
(h) Tracking with unit step reference in rea with PG4PI.
Figure 3: Output trajectories and tracking errors in response to unit step reference input signal, and illustrating perfect tracking with PG4PID and PG4PI.
30

---

PG4PID on a Chemical Reactor
PG4PID LA Unversity hospital
0.0
-0.5
-1.0
States
Stales
-3.0
-3.5
200
400
600
800
1000
Number of Samples
Number of Samples
(a) State trajectories in lah with PG4PID.
(b) State trajectories in rea with PG4PID.
PG4PI LA Unversity hospital
PG4PI Chemical Reactor
0.5
-3.5
-3.0
Statls
States
-2.0
-30
200
400
600
600
1000
200
400
600
000
3000
Number of Samples
Number of Semples
(c) State trajectories in lah with PG4PI.
(d) State trajectories in rea with PG4PI.
Figure State trajectories for the lah and rea environments illustrating stabilization with PG4PID and PG4PI.
31

---

PG4PI on Chemical Reactor
PG4PI on a Chemical Reactor
-20
-20
Reward
-40
000
Verience 0.0011
Verience 0.1
Verience 0.0055
Verience 0.5
Variance 0.001
Veriance 1.0
-60
Variance 0.005
Veriance 155
Variance 0.01
Veriance 2.0
Variance 0.05
Veriance- 25
Variance
Variance
Number Samples
Samples
PG4PI LA University hospital
PG4PI LA University hospital
-1000
-2002
-3000
Reward
3000
4000
Variance
-1000
Variance
Veriance
-1000
-5000
-6000
200
400
600
800
1000
200
400
600
B0D
1000
Number of Samples
Number of Samples
Figure 5: Reward curves for the PG4PI Algorithm on the 8-dimensional Chemical Reactor and the on the 48-dimensional LA University Hospital environments. Plots contain two of experiments, run for smaller set and a larger set of variances a. While smaller variances (a < 2) have negligible effect on the convergence rates, slower convergence is observed for some larger variances (a 3).
32

---

Model Free LQR on Chemical Reactor
Model Free LQR vs. PG4PI on Chemical Reactor
04
-05
2.5
Tacking
-0.5
-24
-25
-1.
-36
Unit Step Reference
LOR Controller
-33
-2.
HD Concrslier
Samples
Samples
Model Free LQR University hospital
Model Free LQR vs. PG4PI LA University hospital
1.0
-05
-14
Tracding
-0.5
-24
-25
-15
-2.
-33
200
400
600
800
1000
200
400
600
B0D
1000
Number of Samples
Number of Samples
Figure 6: State stabilization trajectories and tracking performance for unit step reference with model- free PG with LQR policy and model-free PG4PI on the Chemical Reactor and LA University Hospital environments. While the Model-free LQR controller stabilizes the states for both the environments, there a non-zero steady state tracking error for the unit step input, while the model-free PI controller achieves both stabilization and perfect tracking with zero steady-state error.
33

---

Model Based LQR on Chemical Reactor
Model Based LQR on a Chemical Reactor
-100
-100
Tracking
States
200
-300
-400
Uni Step Reference
System Dutput
200
400
600
800
1000
200
400
800
800
1000
Number of Samples
Number of Semplles
Model Free Pl on a Chemical Reactor
Model Free Pl on Chemical Reactor
-0.5
Tracking
States
-1.5
-0.5
-2.0
-1.0
-2.5
-1.5
-3.0
Stop
System Outout
600
800
1000
200
400
800
000
1000
Number of Samples
Number of Samples
Figure 7: Comparison of model- PG with LQR policy and model-free PG4PI with small model error for the Chemical Reactor environment. LQR is highly fragile and exhibits instability with model error of || 0.05 with model-based implementation, while our model-free PI controller achieves stabilization and perfect tracking with zero steady-state error.
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims our abstract and introduction accurately reflect the paper's scope and contributions. All the theoretical and experimental results are aligned with the claims made in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings. is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
much the results
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We clearly discuss all the theoretical and experimental assumptions behind our
work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
35

---

Justification: We provide the assumptions behind the theoretical results in Sections 2, 3 and 4. The detailed proofs of all results are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the experimental details are described in Section 5 and in the Appendix, and all the code and data are submitted along with the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution
specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
36

---

Answer: [Yes]
Justification: All the code and data are submitted along with the paper, and will be open- sourced upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips. cc/
pub 1 c gu des Cooeeu'mm ss onno1 cy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips.o cc/ httppssccppbllcc/ceee wwwpic/quuuieaoooppopppppppppppp
for more details. The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
weddeeuumissiii
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Yes, the paper details the experimental setting and benchmarks in Section and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them. The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes, we report the mean and one standard deviation of all reward curves in our experiments over 100 independent training runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
37

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The computational resources utilized for the experiments are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://hwwss/...ep........//a/aa
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics, and ensured that our paper conforms to these regulations.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper is primarily theoretical, and does not have any immediate societal impact. We discuss this the Appendix A.5.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses
(e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
38

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not pose any such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: All the datasets, environments, and the benchmarks evaluate our algorithms are cited in the main text and our code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper sepperswthhooe com. / dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
39

---

Justification: All the code and data are submitted along with the paper, and will be released publicly with detailed documentation upon publication.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their
submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicabll, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper is mainly theoretical and does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper is mainly theoretical and does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We have not utilized LLLMs in any of the research described in this paper.
40

---

Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
/c0erpss.cc//oofffeeeee//2333 LLM) for what should or should not be described.
Please refer to our LLM policy
41


Reviews:
[{"review_id": "f43Vq2yFH1", "reviewer": "Reviewer_6Xsd", "summary": "This paper proposes two algorithms (model-based and model-free, respectively) to integrate classic PID control within the RL framework, enabling the automatic learning the PID parameters. They do so using RL with theoretical guarantees of convergence and global optimality.\n\nThe proposal aims to bridging classic control theory with RL while retaining the practical robustness of PID controllers.\n\nExperiments are performed in the Chemical Reactor and a 48-dimensional state-space LA University Hospital environments, showing that their proposal converges faster than a PPO, as it requires less exploration.", "questions": "- The authors acknowledge that further work is needed to extending the mathematical framework to encompass multi-objective trade-offs. Can they offer a brief explanation as to how they think it can be done?\n- Will this approach be always preferable to the usual heuristic tuning of PID? If not always, when not?\n- What is the rationale for choosing those two particular testing environments? They seem to be relative easy to solve.\n- Is there any type of overhead (reserouces/time/extra data/....) required for PG4PID and PG4PI? If so, could you quantify them for your examples?\n- Is this proposal applicable to more complex real-world systems as it is? Or some prerrequisites or preprocessing are needed? Will they work or are there any limitations to be aware of?", "limitations": "Limitations are listed in the conclusions:\n- no multiobjetive yet.\n- no hard constraints on states and inputs.\nhowever, authors do not anticipate how they think they could be addressed.\n\nNo potential negative societal impact is foreseen.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "STRENGTHS:\n- It address the relevant problem of how to improve the learning of RL in scenarios where PID controllers already work very well.\n- The manuscript provides algorithms for model-free and model-based.\n- Proofs of convergence and global optimality guarantees are provided for the two algorithms.\n- The experiments are well-described and the analysis backed by the results.\n\nWEAKNESSES:\n- While interesting, the proposal seems quite incremental, (though offering theoretical guarantees is excellent).\n- Applicability outside problems that can be controlled with PID is unclear (i.e. not clear if all problems could be somehow modeled as PID-controlled, and let the algorithms find the parameters, even though it would be hard for a human).\n- The assumptions about the structural conditions make the proposal very concrete (though the amount of PID controllers out there is huge).", "was_revised": true, "final_justification": "I acknowledge the effort from the authors on preparing the rebuttal and offering clarification to the raised points. Responses were appropriate and relevant, and they also reflected a deep understanding of your work and its place in the SOTA. I maintain my rating as I think this is an interesting paper, backed with evidence and well-defended in the rebuttal. However, additional work/experiments could have been done to make it rounder and for it to merit a stronger acceptance.", "rebuttal": "Thank you for your time and efforts in reviewing our work, and for the positive evaluation of the theoretical guarantees in our work. We address your questions below.\n\n**a) On the novelty of our contribution:**\n\n>While interesting, the proposal seems quite incremental, (though offering theoretical guarantees is excellent). Applicability outside problems that can be controlled with PID is unclear (i.e. not clear if all problems could be somehow modeled as PID-controlled, and let the algorithms find the parameters, even though it would be hard for a human).\n>The assumptions about the structural conditions make the proposal very concrete (though the amount of PID controllers out there is huge).\n\n**Response:** Thank you for allowing us to further clarify the novelty and applicability of our proposed approaches. To the best of our knowledge, there is no existing literature on policy gradient algorithms for PID control with ***provable theoretical guarantees***. While we do not claim that all problems could be controlled by PID controllers, we would like to emphasize this is a **large and very useful class of control policies** that is nearly ubiquitous in a variety of real-world industrial applications (chemical process control, robotics, aerospace), as the reviewer also rightly points out. In fact, **>90% of controllers in industry employ PID architectures**, despite lack of theoretical guarantees [1].  The PID policy also offers built-in structural advantages including enhanced robustness as compared to LQR/variants and a low-dimensional parameterization that allows us to develop gradient-based algorithms with fast convergence properties. Yet, PID policies have received limited attention in the RL community. We also demonstrate these advantages in our benchmark experiments comparing the performance of our proposed designs with LQR (Figures 5 & 6) and PPO policies (Figure 1).\n\nWe would also like to take this opportunity to highlight that obtaining theoretical guarantees on global optimality for this important class of problems is highly challenging. We derive novel policy gradient expressions for the PID control problem in both the model-based and model-free settings (Theorems 3.2 & 4.1, Corollary 4.2). We also establish gradient dominance of this problem (Theorem 3.3), which allows us to provide proofs of optimality, convergence, and sample complexity (Theorems 5.1, 5.3, & 5.6). In the model-free setting, we propose a novel policy gradient approach (Algorithm 2) where the key innovation stems from the fact that we are provably learning the optimal parameters $\\tilde{K}^*$ of a deterministic PID controller while updating the stochastic policy of eq. (15), wherein: (i) the PID gradients derived in Sec. 4.2 are new and necessary, and (ii) the guarantees of Sec. 5.2 require substantial innovation over existing analyses.\n\nTaken together, these are nontrivial, novel contributions that extend policy gradient algorithms to the widely useful class of PID parameterized policies.\n\n[1] Hägglund, T. and Guzmán, J.L., 2024. Give us PID controllers and we can control the world. IFAC-PapersOnLine, 58(7), pp.103-108.\n\n**b) On extension to multi-objective settings:**\n\n>The authors acknowledge that further work is needed to extending the mathematical framework to encompass multi-objective trade-offs. Can they offer a brief explanation as to how they think it can be done?\n\n**Response:** Thank you for raising this question. This extension is a subject of future work, as obtaining theoretical guarantees in multi-objective optimization in general, and multi-objective RL in particular is a challenging problem. Specifically, building on works such as [2-4], we can address this problem by scalarization (linearly combining multi-objective trade-offs to obtain a single-objective problem). The key challenge would be to derive policy gradient expressions and establish gradient dominance for this new objective with PID parameterized policies, to obtain policy gradient algorithms with theoretical guarantees to achieve pareto-optimality. \n\n[2] K. Van Moffaert, M. M. Drugan and A. Nowé, \"Scalarized multi-objective reinforcement learning: Novel design techniques,\" 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), Singapore, 2013, pp. 191-199\n\n[3] Van Moffaert, K. and Nowé, A., 2014. Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Learning Research, 15(1), pp.3483-3512.\n\n[4] Petchrompo, S., Coit, D.W., Brintrup, A., Wannakrairot, A. and Parlikad, A.K., 2022. A review of Pareto pruning methods for multi-objective optimization. Computers & Industrial Engineering, 167, p.108022.\n\n**c) Comparison to heuristic tuning:**\n\n>Will this approach be always preferable to the usual heuristic tuning of PID? If not always, when not?\n\n**Response:** Thank you for raising this question. Our policy gradient PID approach is preferable to heuristic tuning, for two reasons: \n\n(i) We derive ***exact analytical expressions for the policy gradients*** in terms of the policy parameters from Theorem 3.1 and Corollary 3.3, which allows for **fast convergence** of our algorithms even compared to other gradient-based tuning methods like PPO (see experiments in Section 5). In contrast, **heuristic tuning involves trial-and-error** and searching over a large non-convex space of policy parameters that will yield slower convergence.\n(ii) Our algorithms are accompanied by **theoretical proofs** of convergence to **globally optimal solutions**. In contrast,  **heuristic tuning is highly likely to yield local minima** due to the non-convex landscape of the underlying optimization problem. \n\nGiven the additional page allowed for the final version, we will add an additional benchmark experiment demonstrating this limitation of heuristic training with respect to our algorithms. \n\n**d) On choice of environments:**\n\n>What is the rationale for choosing those two particular testing environments? They seem to be relative easy to solve.\n\n**Response:** We chose standard ControlGym environments to demonstrate our proposed approach. We would like to highlight that the environments have high-dimensional state spaces (8- and 48- dimensional state spaces respectively for the chemical reactor and LA hospital environments) that pose a challenge to classical design approaches. In fact, in our experiments, PPO required a large number of samples (nearly 30000, see Fig. 1) for training with no proof of optimality and LQR was shown to be non-robust under dynamical errors in these environments (Fig. 6), while our algorithms demonstrate fast convergence in only a few samples (Fig. 1). Finally, we note that environments such as a chemical reactor and building control (LA Hospital) represent important applications for PID control in the industry, further motivating the choice of environments.\n\n**e) On resource overhead:**\n\n>Is there any type of overhead (reserouces/time/extra data/....) required for PG4PID and PG4PI? If so, could you quantify them for your examples?\n\n**Response:** Since our proposed method has a structured policy parametrization and explicit gradient expressions, the simulations only needed modest computational power; specifically, we ran our experiments on a a 6-core i7-8750H, 2.20GHz CPU, an NVIDIA GeForce RTX 2070 GPU, and 32GB RAM (line 759). We do not require any additional resource overhead or data beyond those detailed in Appendix A.4.\n\n**f) On applicability to real-world problems:**\n\n>Is this proposal applicable to more complex real-world systems as it is? Or some prerrequisites or preprocessing are needed? Will they work or are there any limitations to be aware of?\n\n**Response:** We believe that our proposed method can be used to solve a wide class of real-world control problems. \n(i) First, as we discussed in our earlier response, PID control is applicable to a wide variety of complex real-world applications, including aviation, robotics, power plants, chemical plants etc.\n(ii) We do not need preprocessing of any of the components in our proposed method; rather, we directly compute exact policy gradient expressions and utilize them in our policy updates.\n(iii) As mentioned in the Conclusion of our paper, we propose to extend this approach to multi-objective and constrained settings in the future, which will further expand the applicability to real-world settings.\n\n**g) On constrained design:**\n\n>no hard constraints on states and inputs. however, authors do not anticipate how they think they could be addressed.\n\n**Response:** While our algorithms can be extended to include constraints through soft penalties or regularization in practical settings, recovering theoretical guarantees on convergence while guaranteeing hard constraint satisfaction at all times is a challenging open problem. Specifically, there are two key challenges: (i) Our problem involves deterministic dynamics, which is common in most control problems. Unlike stochastic environments where the analysis of policy gradient methods is simplified by assumption of suitable ergodicity and irreducibility properties of the underlying Markov decision process, proving well-definedness of the cost under constrained policies for deterministic systems will require establishing instead of simply assuming these ergodicity and irreducibility properties, which is challenging in general. (ii) The addition of constraints further complicates this analysis, as it can potentially disconnect the set of reachable states and render the objective function ill-defined, requiring careful analysis. While such problems are beyond the scope of this work, we will include a brief discussion on these challenges in the final version.\n\nWe hope that we have been able to address your concerns through the above responses. We look forward to any additional questions you may have, and hope that you will consider raising your evaluation score of our work accordingly."}, {"review_id": "0Yur104rC2", "reviewer": "Reviewer_fNRn", "summary": "This paper presents a control approach that optimizes PID parameters using gradient descent (PG) methods. The problem is first formulated as an LQR problem; however, rather than adopting PG methods commonly used for LQR, the study directly computes the derivatives of each variable. This leads to a thorough mathematical derivation and establishes theoretical guarantees for convergence.", "questions": "In the understanding of model-free methods, sigma (σ) and the Gaussian mechanism indeed play crucial roles. As discussed in Section 3.2, the importance of the stochastic policy π(·|g) becomes evident, particularly in ensuring that π̃ₖ(u|g) > 0 holds for all states g and actions u.\n\n1. \"This suggests that in certain situations, introducing randomness is necessary in order to avoid issues that may arise from deterministic controllers μ̃ₖ(g).\" While this statement highlights the necessity of randomness in avoiding problems associated with deterministic controllers. Does the presence of randomness also play a fundamental role in enabling the existence and effectiveness of policy gradients?\n2. Why is the deterministic controller μ̃ₖ(g) unable to satisfy the condition π̃ₖ(u|g) > 0 when σ approaches zero? What are the theoretical implications of this limitation for the well-posedness of the policy gradient formulation?\n3. How does this limitation impact the solution of the cost minimization problem within the framework of model-free reinforcement learning? Furthermore, in practical implementations, how should the value of σ be tuned to strike an appropriate balance between exploration and control performance?", "limitations": "Despite the integration of reinforcement learning with PID control, the overall approach is limited by the inherent structural simplicity of PID controllers. As a result, the range of control problems that can be effectively addressed is relatively narrow.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe article addresses a clearly defined problem with a well-structured and logical approach. It discusses the issue that although the gain matrix K is fully determined, there may not always exist a feasible decomposition with respect to K. Instead of directly applying the PG method commonly used in LQR [17], the authors compute the gradients directly for the PID control problem. The paper progresses at a clear and efficient pace, addressing both model-based and model-free scenarios. Furthermore, by omitting certain differential terms, the method achieves a balance between theoretical soundness and practical applicability.\n\nWeaknesses:\n\nAlthough the paper integrates reinforcement learning with PID control, the applicability of the method is inherently limited due to the simple structure of PID controllers, which restricts their effectiveness to a narrow range of control problems.\nThe derivation is based on linear systems, and the paper does not address or discuss the extension to nonlinear systems.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and efforts in reviewing our work and for the positive evaluation of our contributions.  We address your individual concerns below. \n\n**a) On the applicability of PID controllers:**\n\n>Although the paper integrates reinforcement learning with PID control, the applicability of the method is inherently limited due to the simple structure of PID controllers, which restricts their effectiveness to a narrow range of control problems. \n\n**Response:** Thank you for allowing us to further clarify the novelty and applicability of our proposed approaches.\n\nTo the best of our knowledge, there is no existing literature on policy gradient algorithms for PID control with ***provable theoretical guarantees***. While PID controllers have a simple parametric structure, we would like to emphasize this is a ***large and very useful class of control policies*** that are nearly ubiquitous in a variety of real-world industrial applications (chemical process control, robotics, aerospace). In fact, **>90% of controllers in industry employ PID architectures**, despite lack of theoretical guarantees [1].  The PID policy also offers built-in structural advantages including enhanced robustness as compared to LQR/variants and a low-dimensional parameterization that allows us to develop gradient-based algorithms with fast convergence properties. Yet, PID policies have received limited attention in the RL community. We also demonstrate these advantages in our benchmark experiments comparing the performance of our proposed designs with LQR (Figures 5 & 6) and PPO policies (Figure 1). \n\nTherefore, we believe that obtaining provably optimal and convergent policy gradient algorithms for the PID parameterization is **important and useful in a wide variety of applications**.\n\n[1] Hägglund, T. and Guzmán, J.L., 2024. Give us PID controllers and we can control the world. IFAC-PapersOnLine, 58(7), pp.103-108.\n\n**b) On extension to nonlinear systems:**\n\n>The derivation is based on linear systems, and the paper does not address or discuss the extension to nonlinear systems.\n\n**Response:** Thank you for bringing up this direction. We acknowledge and will add this consideration to our discussion in the final version. In this paper, we focused on linear systems with quadratic costs to obtain theoretical guarantees on global optimality and convergence for PID parameterized policies that is lacking in the existing literature. However, extension to nonlinear systems is an interesting yet challenging problem. Specifically, we believe that our model-free algorithm is particularly promising for this extension, since the gradients do not depend on the model parameters. Then, the challenge will be to establish gradient dominance, convergence and sample complexity guarantees with the underlying nonlinear dynamics, which will be the subject of future work. \n\n**c) On the stochastic policy in the model-free PG4PI algorithm:** \n\nWe address the following questions jointly. \n\n>\"This suggests that in certain situations, introducing randomness is necessary in order to avoid issues that may arise from deterministic controllers μ̃ₖ(g).\" While this statement highlights the necessity of randomness in avoiding problems associated with deterministic controllers. Does the presence of randomness also play a fundamental role in enabling the existence and effectiveness of policy gradients?\n\n>Why is the deterministic controller μ̃ₖ(g) unable to satisfy the condition π̃ₖ(u|g) > 0 when σ approaches zero? \n\n**Response:** Indeed, building a stochastic policy centered around the deterministic controller **plays a fundamental role** in allowing us to compute explicit policy gradient expressions (line 215) and employ the policy gradient theorem to develop a model-free PG algorithm. In fact, this is **a key novelty of our approach**. \n\nTo see why the **deterministic controller cannot satisfy the condition** $\\pi(u|g) > 0$, we observe that the main technical challenge is that gradient in (16) needs to be well-defined in order to apply the policy gradient theorem to obtain a model-free algorithm. For a deterministic policy $\\mu_{\\tilde{K}}(g)$, the probability density $\\pi(u|g) = \\delta(u - \\mu_{\\tilde{K}}(g))$ is zero for all $u \\neq \\mu_{\\tilde{K}}(g)$, making $\\log \\pi(u|g) = -\\infty$ and its gradient $\\nabla \\log \\pi(u|g)$ undefined almost everywhere. By introducing the stochastic policy (15) centered around the deterministic policy, we can ensure  $\\pi(u|g) > 0$ for all $u, g$, making (16) well-defined, thus allowing us to apply the policy gradient theorem. \n\n>What are the theoretical implications of this limitation for the well-posedness of the policy gradient formulation?\n\n**Response:** As discussed above, we emphasize that the \"stochastification\" of the policy ensures $\\pi(u|g) > 0$ for all $u, g$ is not a limitation, but rather **essential to ensuring well-posedness** of the formulation. In fact, the stochastic policy is necessary to guarantee that the policy gradients with respect to the PID parameters are well-defined, enabling us to obtain score functions (Theorem 3.2) and consequently the policy gradient estimates in a purely model free manner (Corollary 3.3) and establish a model-free PG algorithm for PID control. \n\n>How does this limitation impact the solution of the cost minimization problem within the framework of model-free reinforcement learning?\n\n**Response:** There is **no limitation with respect to cost minimization** and the solution is not affected by this choice of stochastic policy. In fact, we **theoretically guarantee** that the model-free algorithm PG4PI provably converges to the **globally optimal solution** (Theorem 4.7). This is once again enabled by the fact that we obtain exact gradient expressions with respect to our policy parameters (Corollary 3.3) and establish the gradient dominance properties  of the problem (Theorem 4.4) with the stochastic policy centered around the deterministic controller.\n\n**d) On the choice of $\\sigma$:**\n>In practical implementations, how should the value of σ be tuned to strike an appropriate balance between exploration and control performance?\n\n**Response:** Thank you for this question on the tradeoffs involved in choosing the variance $\\sigma$. From our ablation studies (Fig. 4), we find that varying the $\\sigma$ does not have a significant impact on algorithm performance for small values ($\\sigma$ < 1.0); however, some performance degradation occurs for larger variances ($\\sigma=3$). Thus, in practical implementations, the performance and convergence profile of the algorithm can be maintained for a wide range of $\\sigma$, allowing for choice based on other factors such as exploration and numerical stability. \n\nWe hope that we have been able to address all your concerns through the above responses. We look forward to any additional questions you may have, and hope that you will consider raising your evaluation score of our work accordingly."}, {"review_id": "2BJdLISwuH", "reviewer": "Reviewer_1Cp5", "summary": "This paper bridges the gap between reinforcement learning and PID control by introducing policy gradient (PG) methods tailored for PID architectures. While PG methods have strong theoretical guarantees in classical control problems, their application to PID—despite its widespread use—has been largely heuristic. The authors propose an optimization-based framework for PID control, derive exact gradient expressions, and develop two key algorithms: a model-based method for full PID control, and a model-free method specifically for PI control. The model-based approach leverages system knowledge to achieve global convergence with linear rates, while the model-free algorithm provably learns optimal PI parameters using stochastic policies, offering global guarantees under weaker assumptions.", "questions": "1. Are Assumptions 4.3, 4.5, and 4.6 likely to be satisfied in practical implementations?\n\n2. When extending to constrained optimization settings, what needs to be achieved, and what is the main obstacle?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. This paper is written in a very elegant and fluent manner, making it easy to understand. It introduces the problem through a clear presentation of the background and proposes a solution supported by solid theoretical analysis.\n\n2. The paper establishes global optimality, convergence rates, and sample complexity guarantees for policy gradient methods applied to PID and PI control. Specifically, it proves gradient dominance for the model-based PID setting and derives weak gradient dominance and sample complexity bounds for the model-free PI algorithm. \n\n3. The paper compares its methods with both PG-based LQR algorithms and PPO-based approaches, effectively resolving concerns regarding performance and practical applicability.\n\n**Weaknesses:**\n\n1. The visualization in the experimental section could be improved. For example, in Figure 1, it is recommended to truncate the curves to a smaller number of samples (e.g., the first 500) to better illustrate the dynamic changes. Currently, the mean trajectories are almost completely overlapped with the 1-SD bands, making it difficult to distinguish between different methods and limiting the readability and comparability of the results.\n\n2. Since both PPO and LQR are included as baselines in the experimental section, it would be better to move the comparison with LQR into the main text to facilitate a clearer comparison of the results.\n\n3. If possible, the stability experiments related to Figure 6 would be better presented in the main text to more intuitively demonstrate the results. Currently, the experimental section takes up less than one page and feels somewhat flat.\n\nMinor: \n- Second equation in equation (3) should be: $g_{t+1} = \\overline{A}g_t + \\overline{B}u_t$", "was_revised": true, "final_justification": "The authors provided a thorough and thoughtful rebuttal. Their clarifications regarding technical assumptions, visualization improvements, and the restructuring of experimental results were clear, well-motivated, and addressed my key concerns. The response demonstrated a strong command of both the theoretical underpinnings and practical considerations of their proposed framework.\n\n- Resolved Issues:\n1. Assumptions 4.3, 4.5, and 4.6 were justified convincingly as standard or practically relaxable in implementation, with supporting ablation evidence and reference to widely used techniques such as gradient normalization.\n\n2. The authors gave a thoughtful and technically grounded discussion on the challenges of extending the framework to constrained settings, demonstrating a clear awareness of open research directions.\n\n3. They committed to improving visual clarity (e.g., Figure 1), correcting minor errors (e.g., Equation (3)), and moving important experiments (such as LQR and stability analysis) into the main paper, which will substantively enhance the readability and completeness of the empirical section.\n\n- Remaining Suggestions:\n1. While space limitations are understandable, the planned integration of empirical results into the main text is an important improvement, and I encourage the authors to ensure that future readers benefit from these clarifications.\n\n2. The theoretical results are a clear strength of the paper, and the planned discussion on constrained settings—even if brief—will make the work more forward-looking and comprehensive.\n\n- Overall Evaluation:\n\nThe rebuttal not only addressed the main concerns but also improved my overall confidence in the paper’s contribution and presentation. Based on the clarity, responsiveness, and technical depth of the rebuttal, I have raised my score accordingly. I look forward to seeing the revised version.", "rebuttal": "Thank you for your time, efforts, and feedback, and for the encouraging evaluation of our work! We address your comments below.\n\n**a) On Assumptions 4.3, 4.5, and 4.6:**\n\n> Are Assumptions 4.3, 4.5, and 4.6 likely to be satisfied in practical implementations?\n\nThank you for bringing up this question on the practicality of our assumptions and allowing us an opportunity to clarify. We address each assumption individually below, highlighting how all of them can be relaxed in practice.\n\n*(i) Assumption 4.3:*\nThis is akin to technical assumptions on decreasing step size assumption that are common across most gradient-based algorithms including stochastic gradient descent (SGD). In practice, we can  select any decreasing sequence $\\{\\sigma_t\\}$ as an alternative. In fact, in our experiments, we observed that all our experiments converged even with a *constant* $\\sigma_t$, that is $\\sigma_t=\\sigma, \\forall t \\geq 0$. We also demonstrate through ablation studies that our algorithms retain performance and convergent behavior for a wide range of $\\sigma$ (Fig. 4) Therefore, we believe that this assumption can be relaxed in practical implementations.\n\n*(ii) Assumption 4.5:*\nTheoretically, as mentioned in lines 290-292, this assumption is expected to hold in general, with reasonable conditions on sigma (small enough, as shown in our ablation studies in Fig. 4) and a differentiable cost function with reasonable Lipschitz bounds. In practice, we can obtain gradient estimates with bounded variance by choosing a sufficient rollout length and obtaining a sufficient number of trajectory samples, which is a common strategy to decrease variance in all RL implementations.\n\n*(iii) Assumption 4.6:*\nThis assumption is purely for technical convenience, as mentioned in lines 292-293. In practice, we can relax this assumption by gradient normalization whenever the gradient norm exceeds unity. This is a common strategy in most learning algorithms, including neural network training and policy gradient methods, to avoid exploding gradients [1]. However, in our experiments, we observed that our algorithms showed convergence even without gradient normalization, suggesting that this assumption can also be relaxed in practical implementations. \n\n[1] Chen, Z., Badrinarayanan, V., Lee, C.Y. and Rabinovich, A., 2018, July. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning (pp. 794-803). PMLR.\n\n**b) On extension to constrained optimization:**\n\n> When extending to constrained optimization settings, what needs to be achieved, and what is the main obstacle?\n\nWhile our algorithms can be extended to include constraints through soft penalties or regularization in practical settings, recovering theoretical guarantees on convergence while guaranteeing hard constraint satisfaction at all times is a challenging open problem. Specifically, there are two key challenges: (i) Our problem involves deterministic dynamics, which is common in most control problems. Unlike stochastic environments where the analysis of policy gradient methods is simplified by assumption of suitable ergodicity and irreducibility properties of the underlying Markov decision process, proving well-definedness of the cost under constrained policies for deterministic systems will require establishing instead of simply assuming these ergodicity and irreducibility properties, which is challenging in general. (ii) The addition of constraints further complicates this analysis, as it can potentially disconnect the set of reachable states and render the objective function ill-defined, requiring careful analysis. \n\nWhile such problems are beyond the scope of this work, we will include a brief discussion on these challenges in the final version.\n\n**c) On visualization of reward curves:**\n\n>The visualization in the experimental section could be improved. For example, in Figure 1, it is recommended to truncate the curves to a smaller number of samples (e.g., the first 500) to better illustrate the dynamic changes. Currently, the mean trajectories are almost completely overlapped with the 1-SD bands, making it difficult to distinguish between different methods and limiting the readability and comparability of the results.\n\n**Response:** Thank you for the constructive feedback. We agree, and will truncate the first two plots appropriately so that it becomes easier to visualize the variations in both the subplots in the final version.\n\n**d) On experimental results in the main text:**\n\n>Since both PPO and LQR are included as baselines in the experimental section, it would be better to move the comparison with LQR into the main text to facilitate a clearer comparison of the results.\n\n>If possible, the stability experiments related to Figure 6 would be better presented in the main text to more intuitively demonstrate the results. Currently, the experimental section takes up less than one page and feels somewhat flat.\n\n**Response:** Thank you for this suggestion. We agree - while we presented the LQR and stability experiments in the Appendix due to space limitations, given that the final version allows for an additional page, we will move these results to the main text to facilitate ease of comparison, make the experiment section more self-contained, and strengthen the demonstration of the proposed algorithms.\n\n**e) Correction to Equation 3:**\n\n>Second equation in equation (3) should be: \n$g_{t+1} = \\bar{A}g_t + \\bar{B}u_t$\n\n**Response:** Thank you for pointing out this error. We will fix this in the final version.\n\nWe hope that we have been able to address all your concerns. We look forward to any additional questions you may have, and hope that you will consider raising your evaluation score of our work accordingly."}, {"review_id": "71bW7gMrtC", "reviewer": "Reviewer_bcra", "summary": "This paper proposes policy gradient algorithms for reinforcement learning with PID-parameterized control policies. The method leverages the practical effectiveness of PID controllers while providing theoretical guarantees on convergence and global optimality in both model-based and model-free settings.", "questions": "1. Why are the gradients derived from Equation 3 instead of directly from Equation 1? Clarifying this design choice would help in understanding the derivation and optimization approach.", "limitations": "Yes", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- The mathematical notation is consistent and clearly presented, which aids understanding of the theoretical analysis.\n\n**Weaknesses:**\n\n- The main limitation lies in the novelty of the contribution. Prior work has already explored reinforcement learning for fine-tuning structured controllers such as MPC and PID [1, 2, 3]. The paper would benefit from a clearer distinction between its contributions and those of existing methods.\n- The experimental results are difficult to interpret. In Figure 1, all curves appear very close together, making it hard to evaluate relative performance. Additionally, many results are reported without uncertainty estimates (e.g., error bars), which weakens empirical claims.\n\n[1] Gros, Sébastien and Mario Zanon. “Data-Driven Economic NMPC Using Reinforcement Learning.” *IEEE Transactions on Automatic Control* 65 (2019): 636-648.\n [2] Hardt, Moritz et al. “Gradient Descent Learns Linear Dynamical Systems.” *arXiv preprint* arXiv:1609.05191 (2016).\n [3] Westenbroek, Tyler et al. “Feedback Linearization for Unknown Systems via Reinforcement Learning.” *arXiv preprint* arXiv:1910.13272 (2019).", "was_revised": true, "final_justification": "Thanks for the rebuttal. I am still concerned about the novelty of the method, where many works used RL to tune the parameters of classic controllers. And experimental results are also limited. I will keep my score.", "rebuttal": "Thank you for your time and efforts in reviewing this work.\n\n**a) On novelty and significance of our work:**\n> The main limitation lies in the novelty of the contribution. Prior work has already explored reinforcement learning for fine-tuning structured controllers such as MPC and PID [1, 2, 3]. The paper would benefit from a clearer distinction between its contributions and those of existing methods.\n> [1] Gros, Sébastien and Mario Zanon. “Data-Driven Economic NMPC Using Reinforcement Learning.” IEEE Transactions on Automatic Control 65 (2019): 636-648. \n> [2] Hardt, Moritz et al. “Gradient Descent Learns Linear Dynamical Systems.” arXiv preprint arXiv:1609.05191 (2016). \n> [3] Westenbroek, Tyler et al. “Feedback Linearization for Unknown Systems via Reinforcement Learning.” arXiv preprint arXiv:1910.13272 (2019).\n\n**Response:** We respectfully disagree: to the best of our knowledge, there is no existing literature on policy gradient algorithms for ***PID control with theoretical guarantees***. We emphasize that none of the works that the reviewer cites deal with the PID parameterization. More precisely, [2] focuses on system identification and does not consider a control design problem, [3] develops methods to linearize a nonlinear system through feedback, and [1] focuses does not explicitly tune structured policies (only providing LQR policy as case study), and requires assumptions such as strict dissipativity, which do not hold for the general class of problems we consider.\n\nIn fact, most prior works in RL-based tuning of structured controllers have focused on the widely studied LQR architecture [4]. Additionally, while MPC provides a powerful model-free control design framework, it does not yield structured policies in general. Meanwhile, PID controllers are **ubiquitous in most practical applications** including robotics, aerospace, and chemical process control, with >90% of controllers deployed in industry being PID controllers [5]. Yet, to the best of our knowledge, there is no work that rigorously addresses the PID policy parameterization in RL with theoretical guarantees.\n\nImportantly, our **contribution goes well beyond applying existing RL methods to tune a structured controller**. Rather, we develop a **complete policy gradient theory with provable guarantees on convergence and global optimality** for this widely useful class of policies. \n\nSpecifically, we derive novel policy gradient expressions for the PID control problem in both the model-based and model-free settings (Theorem 3.1 & Corollary 3.3). In contrast to existing approaches, which use zero-order methods to estimate the gradients, these exact gradient expressions allow us to design first-order PG algorithms that are also broadly applicable to general LQR problems, which are a special case of our framework. We also establish gradient dominance of this problem (Theorems 4.1 & 4.4), which allows us to provide proofs of optimality, convergence, and sample complexity (Theorems 4.2, 4.7). In the model-free setting, we propose a novel policy gradient approach (Algorithm 2) where the key innovation stems from the fact that we are provably learning the optimal parameters of a deterministic PID controller while updating the stochastic policy of eq. (15); here (i) the PID gradients derived in Section 3 are new and necessary, and (ii) the guarantees of Section 4 required substantial innovation over existing analyses. \n\nWe also demonstrate through experiments that **our algorithms outperform direct application of RL to tune the structured controller** (see benchmark on tuning PID controllers with PPO Section 5).\n\nTaken together, these are ***nontrivial, novel contributions*** that provide provably convergent and optimal PG algorithms tailored to the important and widely useful class of PID control policies.\n\n**b) On experimental results:**\n>The experimental results are difficult to interpret. In Figure 1, all curves appear very close together, making it hard to evaluate relative performance. Additionally, many results are reported without uncertainty estimates (e.g., error bars), which weakens empirical claims.\n\n**Response:** Thank you for this comment. All reward curves in Figure 1 are shown with error bands, but we agree that they may be hard to discern at the current scale. We will increase the size of Fig. 1 so that all reward curves and error bands are clearly discernible. While it is challenging to incorporate error bands in the state trajectory plots due to high-dimensional state spaces of our environments (8- and 48-dimensional respectively), we will include error bands in our tracking plots in Figure 2 in the final version. \n\n**c) On policy gradient derivation:**\n> Why are the gradients derived from Equation 3 instead of directly from Equation 1? Clarifying this design choice would help in understanding the derivation and optimization approach.\n\n**Response:** We appreciate this question, though we would like to clarify that this is precisely the key technical challenge addressed by our paper. As discussed in lines 131-140, it is not feasible to directly derive gradient expressions from Equation 1 due to the fact that though $K$ in (6) is completely determined given the PID parameters $\\tilde K$, there does not necessarily exist a feasible decomposition of $K$ in terms of $\\tilde K$. For this reason, it is not possible to use existing methods to derive policy gradient expressions directly from the problem in Equation (1), and then translate the resulting controller into a feasible PID controller. Therefore, it is necessary to perform PG descent directly in the PID parameters. To achieve this, we derive gradient expressions in the PID parameters for the reformulated problem in Equation (3), as detailed in Section 2. \n\nWe hope that we have been able to address your concerns. We look forward to any additional questions you may have, and hope that you will consider raising your evaluation score of our work accordingly.\n\n[1] Gros, Sébastien and Mario Zanon. “Data-Driven Economic NMPC Using Reinforcement Learning.” IEEE Transactions on Automatic Control 65 (2019): 636-648 \n[2] Hardt, Moritz et al. “Gradient Descent Learns Linear Dynamical Systems.” arXiv preprint arXiv:1609.05191 (2016) \n[3] Westenbroek, Tyler et al. “Feedback Linearization for Unknown Systems via Reinforcement Learning.” arXiv preprint arXiv:1910.13272 (2019)\n[4] Hu, B., Zhang, K., Li, N., Mesbahi, M., Fazel, M. and Başar, T., 2023. Toward a theoretical foundation of policy optimization for learning control policies. Annual Review of Control, Robotics, and Autonomous Systems, 6(1), pp.123-158\n[5] Hägglund, T. and Guzmán, J.L., 2024. Give us PID controllers and we can control the world. IFAC-PapersOnLine, 58(7), pp.103-108"}]

Author rebuttal:

Meta-review: The authors study policy-gradient methods for PID-based policies. The scope is thus narrow, but of great practical relevance.
As convincingly reiterated by the authors during the discussion, this work provides the first theoretical guarantees for this specific setting. The value of this contribution outweights the remaining concerns of the reviewers on the limited scope and experiments.
The reviewers' suggestions should be included in the final version.
Decision: Accept (poster)