# Linear Quadratic Optimal Control Design: A Novel Approach Based on Krotov Conditions

Authors: Avinash Kumar; Tushar Jain
Journal: Mathematical Problems in Engineering
 (2019)

---

## Abstract

This paper revisits the problem of synthesizing the optimal control law for linear systems with a quadratic cost. For this problem, traditionally, the state feedback gain matrix of the optimal controller is computed by solving the Riccati equation, which is primarily obtained using calculus of variations- (CoV-) based and Hamilton–Jacobi–Bellman (HJB) equation-based approaches. To obtain the Riccati equation, these approaches require some assumptions in the solution procedure; that is, the former approach requires the notion of costates and then their relationship with states is exploited to obtain the closed-form expression of the optimal control law, while the latter requiresa priori knowledge regarding the optimal cost function. In this paper, we propose a novel method for computing linear quadratic optimal control laws by using the global optimal control framework introduced by V. F. Krotov. As shall be illustrated in this article, this framework does not require the notion of costates and any a priori information regarding the optimal cost function. Nevertheless, using this framework, the optimal control problem gets translated to a nonconvex optimization problem. The novelty of the proposed method lies in transforming this nonconvex optimization problem into a convex problem. The convexity imposition results in a linear matrix inequality (LMI), whose analysis is reported in this work. Furthermore, this LMI reduces to the Riccati equation upon imposing optimality requirements. The insights along with the future directions of the work are presented and gathered at appropriate locations in this article. Finally, numerical results are provided to demonstrate the proposed methodology.


---

## Body

## 1. Introduction

Optimal control is a heavily explored and still developing area of control engineering where the objective is to design a control law so as to optimize (maximize or minimize) a given performance index (cost functional) while driving the states of a dynamical system to zero (regulation problem) or to make output track a reference trajectory (tracking problem) [1]. The generic optimal control problem (GOCP) is given as follows: notation: throughout this article, lowercase alphabets represent scalar quantities, lowercase bold alphabets represent vector quantities, and uppercase alphabets represent matrices, and also, the quantities with “star” in the superscript correspond to the optimal trajectory.The GOCP computes an optimal control law which minimizes the performance index/cost functional(1)Jxt,ut,t=lfxtf+∫t0tflxt,ut,tdt,subject to the system dynamics x˙t=fxt,ut,twithxt0∈ℝn,t∈t0,tf, to give the desired optimal trajectory x∗t. Here, lxt,ut,t is the running cost, lfxtf is the terminal cost, xt∈X⊂ℝn is the state vector, and ut∈U⊂ℝm is the control input vector to be designed. Also, lf:ℝn⟶ℝ and l:ℝn×ℝm×t0,tf⟶ℝ are continuous.Since, the aforementioned problem corresponds to optimization of the cost functional subject to dynamics of the system considered and possibly constraints on input(s) and/or state(s), the calculus of variations (CoV) is generally employed to address optimal control design problems [1, 2]. The assumption of existence of optimal control is usually the first step while using CoV techniques. Subsequently, the conditions which must be satisfied by such an optimal control law are derived. Hence, only necessary conditions are found, and sufficiency of these conditions is not guaranteed. Furthermore, the obtained control law is usually only locally optimum. Nevertheless, there are results available in the literature which provide restrictions under which the necessary conditions indeed become sufficient and the global optimal control law is obtained [3–5]. Note that, in solving optimal control design problems, the CoV-based methods use the notion of so-called costates (which are not actually present in the system). Moreover, in the solution procedure, the existence of a linear relationship between the states and the costates is exploited to compute the closed-form expression of the optimal control law (this is particularly true for linear quadratic problems) (see [6] for more details).Alongside CoV, another tool, namely, dynamic programming (DP) (introduced by Bellman), has also been explored to solve optimal control problems. The application of DP to optimal control design problems for continuous linear systems leads to the celebrated Hamilton–Jacobi–Bellman (HJB) equation which also gives a necessary condition for optimality [1]. Nevertheless, this equation also provides sufficiency under the following mild conditions [2, 7]: a there exists a continuously differentiable optimal cost function and b the gradient of optimal cost function with respect to the state vector equals the costate which corresponds to the optimal trajectory. For example, consider the optimal control design problem for the system [8] x˙t=xtut with the performance measure as Jxt,ut,t=∫1∞x2t+u2tdt. For this problem, the optimal cost function is J∗x∗t,u∗t,t=x∗t, and hence, the HJB equation is not defined at x=0 because of nondifferentiability of J∗x∗t,u∗t,t. From the aforementioned observations, a solution method for optimal control design which does not require these conditions is desirable. As shall be demonstrated in this article, the Krotov solution methodology is indeed such a methodology. In fact, this methodology provides sufficient conditions for existence of the global optimal control law without using the notion of costates and any a priori information regarding the optimal cost function [2].Starting from the sixties, the results on sufficient conditions for the global optimum of the optimal control problem were published by Krotov [9, 10]. These conditions have been derived from the so-called extension principle [11]. The first step, while employing these conditions, is a total decomposition of the OCP with respect to time via an appropriate selection of the so-called Krotov function [2, 12, 13]. Once such a decomposition is obtained, the problem is reduced to a family of independent elementary optimization problems parameterized in time t. It has been shown in [2] that the two problems—the original OCP and the optimization problem resulting from decomposition—are completely equivalent. The method, however, is abstract in the sense that the selection of the Krotov function is not straightforward and is very problem specific [11]. A number of works have used the Krotov methodology for solving OCPs encountered in control of structural vibration problems in buildings [12], such as MEMS-based energy harvesting problem [13], magnetic resonance systems [14], quantum control [15], and computation of extremal space trajectories [16, 17]. However, the equivalent optimization problems in all these articles are nonconvex, and hence, the iterative methods, one of them being the Krotov method, are employed to obtain their solutions. To address this issue, we propose a novel method to directly (noniteratively) synthesize optimal controllers for linear systems using Krotov sufficient conditions. The innovation in our approach lies in transforming the nonconvex optimization problem into a convex optimization problem by a proper selection of Krotov functions. Upon convexity imposition, for the linear quadratic regulation (LQR) problem, a matrix inequality is obtained which can be easily converted to a linear matrix inequality (LMI) using Schur complement. Following the argument “However, it is much less appreciated how the other relations (LMIs) enter in to the theory” in [18] (regarding the LMIs whose derivation and analysis are less appreciated as compared to those of the famous Riccati equation), this work demonstrates how this LMI enters into the theory. Specifically, it results in the convexity imposition on the equivalent optimization problem (which is obtained by decomposition of the original optimal control problem using the Krotov function). Moreover, analysis of the obtained LMI is also reported in this article. In [19], the methodology was demonstrated for finite horizon linear quadratic optimal control problems, and the Krotov function was taken to be a positive definite quadratic function. This article presents a rather detailed discussion of the results along with extension of the methodology to infinite horizon problems. It differs from the study [19] in that the Krotov function here is taken as a quadratic function; neither symmetry nor positive definiteness is imposed upon this function. Moreover, the analysis of the LMI which results from convexity imposition is also presented. Considering the aforementioned points and given the fact that the Krotov framework remains highly unexplored in the literature (to the best of authors’ knowledge), this article may serve as a background for further exploration of this framework to more involved control problems, viz., nonlinear optimal control problems, distributed optimal control problems, etc. In summary, the contribution of this work is as follows: i it exhaustively describes the methodology for solving the standard linear quadratic optimal control problems (both finite and infinite horizon) using Krotov sufficient conditions, ii it solves the equivalent optimization problems via convexity imposition: a technique which is not used in the previous work which uses Krotov conditions and then the analysis of resulting LMI is presented, and iii it provides the insights which result upon synthesizing the optimal control laws and may also lay the foundation upon which the Krotov sufficient conditions may be employed for solving more complex optimal control problems, viz., nonlinear optimal control problems.The rest of this article is organized as follows: In Section2, the preliminaries of linear quadratic optimal control problems are discussed and the solution methodologies based on the calculus of variations- (CoV-) based method and Hamilton–Jacobi–Bellman (HJB) equation-based method are outlined. The assumptions as encountered in these approaches are also discussed. In Section 3, the background literature of Krotov sufficient conditions and their application to the problems considered are detailed. This section also discusses a number of insights which result while solving the considered optimal control problems. These insights are gathered as remarks at appropriate locations in this section. In Section 4, the analysis of LMI obtained upon convexity imposition is presented. The Krotov iterative method is also discussed in brief in this section. In Section 5, the proposed method is demonstrated through numerical examples. Finally, the concluding remarks and future scope of the work are presented in Section 6.


## 2. Preliminaries and Problem Formulation

In this section, solution procedures of linear quadratic regulation (LQR) and linear quadratic tracking (LQT) problems using CoV- and HJB equation-based approaches are briefly discussed in order to highlight the assumptions used in these approaches.


### 2.1. OCP 1 (Finite Horizon LQR Problem)

Compute an optimal control lawu∗t which minimizes the quadratic performance index/cost functional(2)Jxt,ut,t=0.5xTtfFtfxtf+0.5∫t0tfxTtQtxt+uTtRtutdt,subject to the system dynamics x˙t=Atxt+Btut and drives the states of the system to zero (regulation). Here, xt0=x0 is given, xtf is free, and tf is fixed. Also, Qt≽0andRt≻0,∀t∈t0,tf.The solution using the CoV technique comprises the following four major steps:(i)
Formulation of Hamiltonian function: the Hamiltonian for the considered problem is given as(3)ℋx,u,λ=12xTtQtxt+12uTtRtut+λTtAtxt+Btut,whereλt is the costate vector.(ii)
Obtaining the optimal control law using the first-order necessary condition: the optimal control lawu∗t is obtained as(4)∂ℋ∂u=0⟹u∗t=−R−1tBTtλ∗t.(iii)
Use of state and costate dynamics and a transformation to connect states and costates for allt∈t0,tf: the boundary conditions (i.e., tf being fixed and xtf being free) lead to the following boundary condition on λt: λ∗tf=Ftfx∗tf. Then, the transformation(5)λ∗t=Ptx∗t,to connect costate and state vectors is used to compute the optimal control law as(6)u∗t=−R−1tBTtPtx∗t.(iv)
Obtaining the matrix differential Riccati equation: finally, taking the derivative of equation (5) and substituting the state and costate relationship, the following matrix differential Riccati equation (MDRE) is obtained which Pt must satisfy for all t∈t0,tf:(7)P˙t+PtAt+ATtPt+Qt−PtBtR−1tBTtPt=0.Furthermore, the solution using the HJB equation requires(8)∂J∗∂t+ℋx∗t,∂J∗∂x∗,u∗t,t=0,∀t∈t0,tf,where J∗x∗t,t is the optimal cost function and u∗t is the optimal control law. To solve (8), the boundary condition J∗x∗tf,tf=1/2x∗TtfFtfx∗tf is used with J∗x∗t,t assumed to be(9)J∗xt,t=12x∗TtPtx∗t,where Pt is a real, symmetric, positive-definite matrix to be determined. Substituting (9) into (8), we get(10)12x∗TtP˙tx∗t+12x∗TtPtAtx∗t+12x∗TtQtx∗t+12x∗TtATtPtx∗t−12x∗TtPtBtR−1tBTtPtx∗t=0.This equation is valid for anyx∗t if(11)P˙t+Qt+PtAt+ATtPt−PtBtR−1tBTtPt=0.Finally,Ptf=Ftf, and thus, the solution is the same as that obtained using CoV. Summarizing the above, the global optimal control law is given by(12)u∗t=−R−1tBTtPtxt,where Pt is the solution of(13)P˙t+PtAt+ATtPt+Qt−PtBtR−1tBTtPt=0,with the boundary condition Ptf=Ftf.


### 2.2. OCP 2 (Finite Horizon LQT Problem)

Compute an optimal control lawu∗t which minimizes the following quadratic performance index/cost functional:(14)J=0.5eTtfFtfetf+0.5∫t0tfeTtQtet+uTtRtutdt,where et≜zt−yt subject to the system dynamics(15)x˙t=Atxt+Btut,yt=Ctxt,such that the output yt tracks the desired reference trajectory zt. Here, et≜yt−zt is the error vector, xt0=x0 is given, xtf is free, and tf is fixed. Also, Qt≽0andRt≻0,∀t∈t0,tf.Similarly to the solution of LQR, the CoV- and HJB equation-based approaches yield the optimal control law as(16)u∗t=−R−1tBTtPtx∗t+R−1tBTtgt,where Pt and gt satisfy(17)P˙t+PtAt+ATtPt+CTtQtCt−PtBtR−1tBTtPt=0,∀t∈t0,tf,g˙t+At−BtR−1tBTtPtTgt+CTtQtzt=0,∀t∈t0,tf,with boundary conditions Ptf=CTtfFtfCtf and gtf=CTtfFtfCtf, respectively. Note that, in the HJB approach, the optimal cost function has to be guessed.Although the CoV- and HJB equation-based approaches as described above are widely employed for solving OCPs, there are some assumptions associated with these approaches in their solution procedure. Specifically, the CoV-based approach uses the notion of costates and their relationship with states for all time (5) to compute the optimal control law. Similarly, the HJB equation-based approach requires the existence of the continuously differentiable optimal cost function, and its gradient with respect to the state is the costate corresponding to the optimal trajectory [7]. Thus, the information about the optimal cost function must be known a priori. The angle of our attack is to synthesize an optimal control law using Krotov sufficient conditions, where the above issues are not encountered in the solution procedure. However, it is well known that the control law using these conditions is synthesized through an iterative procedure. The main nontrivial issue to be tackled is how to obtain noniterative solutions of optimal control problems using Krotov conditions. The next section answers this question for linear quadratic optimal control problems.


## 2.1. OCP 1 (Finite Horizon LQR Problem)

Compute an optimal control lawu∗t which minimizes the quadratic performance index/cost functional(2)Jxt,ut,t=0.5xTtfFtfxtf+0.5∫t0tfxTtQtxt+uTtRtutdt,subject to the system dynamics x˙t=Atxt+Btut and drives the states of the system to zero (regulation). Here, xt0=x0 is given, xtf is free, and tf is fixed. Also, Qt≽0andRt≻0,∀t∈t0,tf.The solution using the CoV technique comprises the following four major steps:(i)
Formulation of Hamiltonian function: the Hamiltonian for the considered problem is given as(3)ℋx,u,λ=12xTtQtxt+12uTtRtut+λTtAtxt+Btut,whereλt is the costate vector.(ii)
Obtaining the optimal control law using the first-order necessary condition: the optimal control lawu∗t is obtained as(4)∂ℋ∂u=0⟹u∗t=−R−1tBTtλ∗t.(iii)
Use of state and costate dynamics and a transformation to connect states and costates for allt∈t0,tf: the boundary conditions (i.e., tf being fixed and xtf being free) lead to the following boundary condition on λt: λ∗tf=Ftfx∗tf. Then, the transformation(5)λ∗t=Ptx∗t,to connect costate and state vectors is used to compute the optimal control law as(6)u∗t=−R−1tBTtPtx∗t.(iv)
Obtaining the matrix differential Riccati equation: finally, taking the derivative of equation (5) and substituting the state and costate relationship, the following matrix differential Riccati equation (MDRE) is obtained which Pt must satisfy for all t∈t0,tf:(7)P˙t+PtAt+ATtPt+Qt−PtBtR−1tBTtPt=0.Furthermore, the solution using the HJB equation requires(8)∂J∗∂t+ℋx∗t,∂J∗∂x∗,u∗t,t=0,∀t∈t0,tf,where J∗x∗t,t is the optimal cost function and u∗t is the optimal control law. To solve (8), the boundary condition J∗x∗tf,tf=1/2x∗TtfFtfx∗tf is used with J∗x∗t,t assumed to be(9)J∗xt,t=12x∗TtPtx∗t,where Pt is a real, symmetric, positive-definite matrix to be determined. Substituting (9) into (8), we get(10)12x∗TtP˙tx∗t+12x∗TtPtAtx∗t+12x∗TtQtx∗t+12x∗TtATtPtx∗t−12x∗TtPtBtR−1tBTtPtx∗t=0.This equation is valid for anyx∗t if(11)P˙t+Qt+PtAt+ATtPt−PtBtR−1tBTtPt=0.Finally,Ptf=Ftf, and thus, the solution is the same as that obtained using CoV. Summarizing the above, the global optimal control law is given by(12)u∗t=−R−1tBTtPtxt,where Pt is the solution of(13)P˙t+PtAt+ATtPt+Qt−PtBtR−1tBTtPt=0,with the boundary condition Ptf=Ftf.


## 2.2. OCP 2 (Finite Horizon LQT Problem)

Compute an optimal control lawu∗t which minimizes the following quadratic performance index/cost functional:(14)J=0.5eTtfFtfetf+0.5∫t0tfeTtQtet+uTtRtutdt,where et≜zt−yt subject to the system dynamics(15)x˙t=Atxt+Btut,yt=Ctxt,such that the output yt tracks the desired reference trajectory zt. Here, et≜yt−zt is the error vector, xt0=x0 is given, xtf is free, and tf is fixed. Also, Qt≽0andRt≻0,∀t∈t0,tf.Similarly to the solution of LQR, the CoV- and HJB equation-based approaches yield the optimal control law as(16)u∗t=−R−1tBTtPtx∗t+R−1tBTtgt,where Pt and gt satisfy(17)P˙t+PtAt+ATtPt+CTtQtCt−PtBtR−1tBTtPt=0,∀t∈t0,tf,g˙t+At−BtR−1tBTtPtTgt+CTtQtzt=0,∀t∈t0,tf,with boundary conditions Ptf=CTtfFtfCtf and gtf=CTtfFtfCtf, respectively. Note that, in the HJB approach, the optimal cost function has to be guessed.Although the CoV- and HJB equation-based approaches as described above are widely employed for solving OCPs, there are some assumptions associated with these approaches in their solution procedure. Specifically, the CoV-based approach uses the notion of costates and their relationship with states for all time (5) to compute the optimal control law. Similarly, the HJB equation-based approach requires the existence of the continuously differentiable optimal cost function, and its gradient with respect to the state is the costate corresponding to the optimal trajectory [7]. Thus, the information about the optimal cost function must be known a priori. The angle of our attack is to synthesize an optimal control law using Krotov sufficient conditions, where the above issues are not encountered in the solution procedure. However, it is well known that the control law using these conditions is synthesized through an iterative procedure. The main nontrivial issue to be tackled is how to obtain noniterative solutions of optimal control problems using Krotov conditions. The next section answers this question for linear quadratic optimal control problems.


## 3. Computation of Optimal Control Laws

In this section, solutions of the LQR and LQT problems using Krotov sufficient conditions are detailed.


### 3.1. Krotov Sufficient Conditions in Optimal Control

The underlying idea behind Krotov sufficient conditions for global optimality of control processes is the total decomposition of the original OCP with respect to time using the so-called extension principle [2].


#### 3.1.1. Extension Principle

The essence of the extension principle is to replace the original optimization problem with complex relations and/or constraints with a simpler one such that they are excluded in the new problem definition but the solution of the new problem still satisfies the discarded relations [11].Consider a scalar-valued functionalIv defined over a set M (i.e., v∈M) and the optimization problem as follows.Problem 1.
Findv¯ such that d=infv∈MIv where d≜Iv¯. Instead of solving Problem 1, another equivalent optimization problem is solved. Let L denote the equivalent representation of the original cost functional. Then, a new problem is formulated over N, a superset of M, as follows.Equivalent Problem 1. Findv¯ such that e=infv∈NLv where e≜Lv¯.
The equivalent problem is also called the extension of the original problem. The method of choosing of theequivalent functional L is not unique, and the selection is generally made according to specifications of the problem under consideration. This freedom in the selection of the equivalent functional can be exploited to tackle the generic nonconvex optimization problems. Also, it is necessary to ensure that Iv=Lv,∀v∈M, so that the optimizer v¯ is actually the optimizer of original Problem 1 [2]. The idea behind the extension principle is illustrated in Figure 1. Clearly, the application of the extension principle requires appropriate selection of the equivalent functional L and the set N.Figure 1
(a) Extension Principle. (b) Idea behind selection of solving function.
(a)
(b)


#### 3.1.2. Application of Extension Principle to Optimal Control Problems

The equivalent problem of the GOCP which is obtained by the application of the extension principle is given in the following theorem which provides a concrete definition of the equivalent functionalL and the set N.Theorem 1.
For the GOCP, letqxt,t be a continuously differentiable function. Then, there is an equivalent representation of (1) given as(18)Jeqxt,ut,ts=sfxtf+qxt0,t0+∫t0tfsxt,ut,tdt,where(19)sxt,ut,t≜∂q∂t+∂q∂xfxt,ut,t+lxt,ut,t,sfxtf≜lfxtf−qxtf,tf.Proof.
See Section 2.3 in [2] for the proof.
The equivalent functionalJeqxt,ut,ts leads to a sufficient condition for the global optimality of an admissible process (i.e., axt,ut pair which satisfies the dynamical equation x˙t=fxt,ut,t and the input/state constraints).Theorem 2.
(Krotov sufficient conditions). Ifx∗t,u∗t is an admissible process such that(20)sx∗t,u∗t,t=minx∈X,u∈Usxt,ut,t,∀t∈t0,tf,sfx∗tf=minx∈Xfsfx,then x∗t,u∗t is an optimal process. Here, Xf is the terminal set for admissible xt of x˙t=fxt,ut,t; that is, if xt is admissible, then xtf∈Xf.Proof.
See Section 2.3 in [2] for the proof.Remarks.
Some remarks which follow from Theorem 1 and Theorem 2 are now briefed as follows:(1)
The functionalJeq is the equivalent functional of the original functional J in (1). Specifically, Jeq=J∀xt,ut:x˙t=fxt,ut,t.(2)
The functionqxt,t, known as the Krotov function, can be any continuously differentiable function, and each qxt,t leads to a different equivalent functional Jeq. A general way to choose this function is not known, and the selection is usually done according to the specifics of the problem in hand. Moreover, it is possible that the same solution results for different selections of qxt,t [2]. While this ad hocness in choosing the Krotov function may seem burdensome, it provides enough freedom in making the selection according to the specifications of the problem in hand [11, 20]. In essence, this work exploits the ad hocness in selecting the Krotov function.(3)
The original GOCP is transformed to an equivalent optimization problem of the functionss and sf over the sets X,U, and Xf. Thus, the dynamical equation constraint x˙=fxt,ut,t is excluded from the optimization problem by defining Jeqxt,ut,t.(4)
If the optimization problem formulated in Theorem 2 is feasible (i.e., an admissible process satisfying the sufficient conditions in Theorem 2 can be found), then the functionqxt,t is called the solving function.(5)
Clearly, different selections ofqxt,t result in different optimization problems in Theorem 2, and thus, the selection of qxt,t is crucial to effectively solve the optimization problems in Theorem 2. Furthermore, the necessary conditions for existence of the optima (optimum) of the functions s and sf coincide with the popular Pontryagin’s minimum principle [20]. Moreover, a specific setting of the Krotov function qxt,t leads to the popular Hamilton–Jacobi–Bellman equation [2, 20]. These two observations lead to the conclusion that Krotov sufficient conditions are in fact the most general sufficient conditions for global results in optimal control theory.
The solution to the equivalent optimization problem given in Theorem 2 is generally computed using sequential methods because the resulting optimization problem is nonconvex. Such a sequence of processes is called anoptimizing sequence [10]. One such iterative method is the Krotov method in which an improving function is chosen at each iteration. To avoid iterative methods, we convexify the equivalent problems in Theorem 2 via a suitable selection of the Krotov function. The idea is illustrated in Figure 1 where the set Qc—the set of solving functions which result in the convexity of equivalent optimization problems in Theorem 2—is the set of interest. In the next section, the suitable Krotov functions for LQR and LQT problems are proposed. In the following, we use 2Jxt,ut,t as the cost functional instead of Jxt,ut,t, and the time variable t is dropped wherever it is required for the sake of simplicity.


### 3.2. Solution of OCP 1 (LQR Problem)

The equivalent optimization problem for OCP 1, as per Theorem 2, is given as follows.Equivalent OCP 1: compute an admissible pairx∗,u∗ which is as follows:(i)
minx,u∈ℝn×ℝmsx,u,t,∀t∈t0,tf, where(21)s=∂q∂t+∂q∂xAx+Bu+xTQx+uTRu.(ii)
minx∈ℝnsfx, where(22)sf=xTtfFtfxtf−qxtf,tf.The next proposition is one of the main results of this paper, where we propose a suitable Krotov function which will be useful in computing a direct solution to OCP 1.Proposition 1.
For Equivalent OCP 1, let the Krotov function be chosen as(23)qx,t=xTPx,and every element of the matrix P is differentiable ∀t∈t0,tf. Then, the following statements are equivalent:(a)
sx,u,t and sfxtf are convex functions in x,u and xtf, respectively.(b)
P satisfies the following matrix inequalities:(i)
P˙+PA+ATP+Q−1/2PBR−1BTP−1/4PBR−1BTPT−1/4PTBR−1BTP≽0,∀t∈t0,tf(ii)
Ftf−Ptf≽0Proof.
Withq selected as in (23), the function sx,u,t becomes(24)s=xTP˙x+xTP+PTAx+Bu+xTQx+uTRu=xTP˙x+xTPAx+xTPTAx+xTPTBu+xTPBu+xTQx+uTRu.
Adding and subtracting the termxT1/2PBR−1BTP+1/4PBR−1BTPT+1/4PTBR−1BTPx, we get(25)s=xTP˙+PA+ATP+Q−12PBR−1BTP−14PBR−1BTP−14PBR−1BTPx+xTPBu+xTPTBTx+uTRu+12xTPBR−1BTPx+14xTPBR−1BTPTx+14xTPTBR−1BTPx.
SinceR≻0, there exists a unique positive definite matrix [21], say R˜, such that R˜2=R and R˜−1R˜−1=R−1. Now rearranging the terms in (25), we get(26)s=xTP˙+PA+ATP−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTPx+R˜u+12R˜−1BTPT+BTPxTR˜u+12R˜−1BTPT+BTPx.
Clearly the second term in (26) is strictly convex. Now, s is convex iff the following condition is satisfied:(27)P˙+PA+ATP−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP≽0,∀t∈t0,tf.
Moreover, withq as in (23), sfxtf is given as(28)sf=xTtfFtfxtf−xTtfPtfxtf=xTtfFtf−Ptfxtf.
Finally,sf is convex iff(29)Ftf−Ptf≽0.Corollary 1.
Withqx,t selected as in Proposition 1, the following statements are true:(1)
The functionq=xTPx is a solving function for OCP 1.(2)
The global optimal control law for OCP 1 is given by(30)u∗=−12R−1BTPT+Px∗,where P is the solution of the matrix differential equation(31)P˙+PA+ATP+Q−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP=0,with the final value Ptf=Ftf.Proof.
Clearly, ifP satisfies (27), then s is independent of x and the obtained control law results in an admissible process. Hence, the selected q is the solving function. Furthermore, for minimization, it is required that the second term in (26) is zero which gives the optimal control law as(32)u∗=−12R−1BTPT+Px∗.


#### 3.2.1. Solution of OCP 1 with Final Timetf⟶∞ (Infinite Horizon LQR)

Next, the infinite-final time LQR problem for a linear time invariant (LTI) is considered. For this case, the terminal state weighing matrixFtf=0. The problem statement now is as follows.Problem 2.
(infinite horizon LQR problem). Compute an optimal control lawu∗t which minimizes the quadratic performance index/cost functional(33)Jxt,ut,t=0.5∫t0∞xTtQxt+uTtRutdt,subject to the system dynamics x˙t=Axt+But and drives the states of the system to zero (regulation). Here, xt0=x0 is given, and x∞ is free. Also, Q≽0andR≻0. The quantities A, B, Q, and R are time-independent.Solution.
For this problem, the matrix differential equation (27) needs to be solved with the boundary condition P∞=0. Computing this solution is equivalent to solving the following algebraic equation:(34)PA+ATP+Q−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP=0,and the resulting optimal control law is given as(35)u∗=−12R−1BTPT+Px∗.
Similar to the finite-final time case, it is easy to verify that the functionq=xTPx (where P is a time-invariant (constant) matrix) is a solving function. Note that since the final time tf⟶∞, it is necessary to ensure the stability of the closed-loop system. The next lemma provides a proof of the closed-loop stability under another condition on the P matrix.Lemma 1.
For the systemx˙=Ax+Bu, the control input (35) ensures the stability of the closed loop if P satisfies (34) with P+PT≻0.Proof.
Let the Lyapunov function beVxt=xTP+PTx. Then,(36)V˙=xTP+PTx˙+x˙TP+PTx=xTATP+ATPT+ATP+ATPTx−2xTPBR−1BTPx−xTPTBR−1BTPx−xTPBR−1BTPTx=2xT−Q−14PBR−1BTPT−14PTBR−1BTP−12PBR−1BTPx.
It is easy to verify the quantity−Q−1/4PBR−1BTPT−1/4PTBR−1BTP−1/2PBR−1BTP is negative semidefinite if Q≽0 and negative definite if Q≻0. Thus, using Lyapunov theory [22], the closed-loop system is stable if Q≽0 and asymptotically stable if Q≻0.Remarks.
The matrix differential equation (31) reduces to the popular matrix differential Riccati equation (MDRE) for a symmetric P matrix. Also, the algebraic equation (34) admits more numbers of solutions than the MDRE as demonstrated in Example 3 in Section 5. A more rigorous analysis and application of these solutions are the subject of future research.


### 3.3. Solution of OCP 2 (LQT Problem)

The equivalent optimization problem for OCP 2 is given as follows.Equivalent OCP 2: compute an optimal control lawu∗t which is as follows:(i)
minx,u∈ℝn×ℝmsx,u,t, ∀t∈t0,tf, where(37)s=∂q∂t+∂q∂xAx+Bu+eTQe+uTRu.(ii)
minx∈ℝnsfxtf, where(38)sf=eTtfFtfetf−qxtf,tf.Another main result of this paper is given in the next proposition, which will be useful in computing the direct solution to OCP 2.Proposition 2.
For Equivalent OCP 2, let the Krotov function be chosen as(39)q≜xTPx−2gTx,and every element of the matrix P and the vector g is differentiable ∀t∈t0,tf. Then, the following statements are equivalent:(a)
sx,u,t and sfxtf are bounded below and convex in x,u and xtf, respectively.(b)
(i)P satisfies the following matrix inequalities:(1)
P˙+PA+ATP−1/2PBR−1BTP−1/4PBR−1BTPT−1/4PTBR−1BTP+CTQC≽0∀t∈t0,tf(2)
CTtfFtfCtf−Ptf≽0(ii)
g satisfies the vector differential equation(40)g˙+ATg+CTQz−12P+PTBR−1BTg=0,
with the boundary conditiongtf=CTtfFtfztf.Proof.
Withq chosen as in (39), the function sx,u,t is given as(41)s=xTP˙x−2g˙Tx+xTP+PT−2gTAx+Bu+eTQe+uTRu=xTP˙+PA+PTA+CTQCx+xTPBu+xTPTBu−2g˙Tx−2gTAx−2gTBu+zTQz−2xTCTQz+uTRu.
Adding and subtracting the termsxT1/2PBR−1BTP+1/4PBR−1BTPT+1/4PTBR−1BTPx, gTBR−1BTg, and xTP+PTBR−1BTg, we get(42)s=xTP˙+PA+ATP−12PBR−1BTP+CTQC−14PBR−1BTPT−14PTBR−1BTPx−2xTg˙+ATg+CTQz−12P+PTBR−1BTg+12uTBTPx−uTBTg+12uTBTPTx−12xTPBR−1BTg−12xTPTR−1BTg+12xTPBu+14xTPBR−1BTPx+12xTPTBu+14xTPTBR−1BTPx+14xTPTBR−1BTPx+14xTPTBR−1BTPTx−12gBR−1BTPx−12gBR−1BTPTx+gTBR−1BTg.
SinceR≻0, there exists a unique positive definite matrix [21], say R˜, such that R˜2=RandR˜−1R˜−1=R−1. Now rearranging the terms in (43), we get(43)s=xTP˙+PA+ATP−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP+CTQCx+R˜u+12R˜−1BTP+PTx−R˜−1BTgTR˜u+12R˜−1BTP+PTx−R˜−1BTg−2xTg˙+ATg+CTQz−12P+PTBR−1BTg+zTQz−gTBR−1BTg.
The second term in (44) is positive definite and strictly convex. The third term is linear in x, and the fourth term is independent of x and u. The first term in (44) is quadratic in x, and it is convex iff it is positive semidefinite. Hence, the function sx,u,t is convex iff(44)P˙+PA+ATP−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP+CTQC≽0,∀t∈t0,tf.
Since the second term in (44) is linear in x, it is bounded below if(45)g˙+ATg+CTQz−12P+PTBR−1BTg=0.
Finally, the term independent ofx and u, i.e., zTQz−gTBR−1BTg, is bounded because zt and gt are bounded [23]. Next, with qxt,t as in (39), sfxtf is given as(46)sf=xTtfCTtfFtfCtf−Ptfxtf+2gTtf−2zTtfFtfCtfxtf+zTtfFtfztf.
Similar to the case ofs, sf is convex iff(47)CTtfFtfCtf−Ptf≽0.
Finally,sf is bounded below if(48)gtf=CTtfFtfztf.
The following corollary computes the optimal control law for OCP 2.Corollary 2.
Withqx,t selected as in Proposition 1, the following statements are true:(1)
The functionq=xTPx−2gTx is a solving function for OCP 2.(2)
The global optimal control law for OCP 2 is given by(49)u∗=−12R−1BTPT+Px∗+R−1BTg,where P is the solution of the matrix differential equation(50)P˙+PA+ATP−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP+CTQC=0,with the final value Ptf=CTtfFtfCtf, and gt satisfies the differential equation g˙+ATg+CTQz−1/2P+PTBR−1BTg=0 with the final value gtf=CTtfFtfztf.Proof.
Clearly, ifP satisfies (50) and g satisfies (45), then s is independent of x and the obtained control law yields an admissible process. Hence, qxt,t is the solving function. The third term in (44) is strictly convex and attains a minimum value when(51)R˜u+12R˜−1BTP+PTx−R˜−1BTg=0⟹u∗=−12R−1BTPT+Px∗+R−1BTg.


#### 3.3.1. Solution of OCP 2 with Final Timetf⟶∞ (Infinite Horizon LQT)

Next, the infinite-final time LQT problem for a linear time invariant (LTI) is considered. For this case, the terminal costFtf=0. The problem statement now reads as follows.Problem 3.
(infinite horizon LQT problem). Compute an optimal control lawu∗t which minimizes the quadratic performance index/cost functional(52)Jet,ut,t=0.5∫t0∞eTtQet+uTtRutdt,subject to the system dynamics x˙t=Axt+But and yt=Cxt and drives the states of the system to the desired trajectory zt. Here, et≜yt−zt is the error vector, xt0=x0 is given, xtf is free, tf is fixed, and x∞ is free. Also, Q≽0andR≻0.Solution.
For this problem, similar to the case of regulation, the matrix differential equation (50) needs to be solved with the boundary condition P∞=0. Similarly, the differential equation (45) needs to be solved with the boundary condition g∞=0. Computing these solutions is equivalent to solving(53)PA+ATP+Q−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP+CTQC=0,gt=−∫t∞expAT−1/2P+PtBR−1BTτ−1CTQzτdτ,and the resulting optimal control law is given as(54)u∗=−12R−1BTPT+Px∗+R−1BTg.
Similar to the finite-final time case, it is easy to verify that the functionq=xTPx−2gTx (where P is a time-invariant (constant) matrix) is a solving function. In this case too, similar to the case of regulation, the stability of the closed loop is ensured if P+PT≻0.Remarks.
As is clear from the solution methodologies proposed above, the solution of the considered problems does not require the notion of costates and any a priori information regarding the optimal cost function. Rather, the whole solution technique depends upon the selection of the Krotov functionqxt,t which directly affects the nature of equivalent optimization problems.


## 3.1. Krotov Sufficient Conditions in Optimal Control

The underlying idea behind Krotov sufficient conditions for global optimality of control processes is the total decomposition of the original OCP with respect to time using the so-called extension principle [2].


### 3.1.1. Extension Principle

The essence of the extension principle is to replace the original optimization problem with complex relations and/or constraints with a simpler one such that they are excluded in the new problem definition but the solution of the new problem still satisfies the discarded relations [11].Consider a scalar-valued functionalIv defined over a set M (i.e., v∈M) and the optimization problem as follows.Problem 1.
Findv¯ such that d=infv∈MIv where d≜Iv¯. Instead of solving Problem 1, another equivalent optimization problem is solved. Let L denote the equivalent representation of the original cost functional. Then, a new problem is formulated over N, a superset of M, as follows.Equivalent Problem 1. Findv¯ such that e=infv∈NLv where e≜Lv¯.
The equivalent problem is also called the extension of the original problem. The method of choosing of theequivalent functional L is not unique, and the selection is generally made according to specifications of the problem under consideration. This freedom in the selection of the equivalent functional can be exploited to tackle the generic nonconvex optimization problems. Also, it is necessary to ensure that Iv=Lv,∀v∈M, so that the optimizer v¯ is actually the optimizer of original Problem 1 [2]. The idea behind the extension principle is illustrated in Figure 1. Clearly, the application of the extension principle requires appropriate selection of the equivalent functional L and the set N.Figure 1
(a) Extension Principle. (b) Idea behind selection of solving function.
(a)
(b)


### 3.1.2. Application of Extension Principle to Optimal Control Problems

The equivalent problem of the GOCP which is obtained by the application of the extension principle is given in the following theorem which provides a concrete definition of the equivalent functionalL and the set N.Theorem 1.
For the GOCP, letqxt,t be a continuously differentiable function. Then, there is an equivalent representation of (1) given as(18)Jeqxt,ut,ts=sfxtf+qxt0,t0+∫t0tfsxt,ut,tdt,where(19)sxt,ut,t≜∂q∂t+∂q∂xfxt,ut,t+lxt,ut,t,sfxtf≜lfxtf−qxtf,tf.Proof.
See Section 2.3 in [2] for the proof.
The equivalent functionalJeqxt,ut,ts leads to a sufficient condition for the global optimality of an admissible process (i.e., axt,ut pair which satisfies the dynamical equation x˙t=fxt,ut,t and the input/state constraints).Theorem 2.
(Krotov sufficient conditions). Ifx∗t,u∗t is an admissible process such that(20)sx∗t,u∗t,t=minx∈X,u∈Usxt,ut,t,∀t∈t0,tf,sfx∗tf=minx∈Xfsfx,then x∗t,u∗t is an optimal process. Here, Xf is the terminal set for admissible xt of x˙t=fxt,ut,t; that is, if xt is admissible, then xtf∈Xf.Proof.
See Section 2.3 in [2] for the proof.Remarks.
Some remarks which follow from Theorem 1 and Theorem 2 are now briefed as follows:(1)
The functionalJeq is the equivalent functional of the original functional J in (1). Specifically, Jeq=J∀xt,ut:x˙t=fxt,ut,t.(2)
The functionqxt,t, known as the Krotov function, can be any continuously differentiable function, and each qxt,t leads to a different equivalent functional Jeq. A general way to choose this function is not known, and the selection is usually done according to the specifics of the problem in hand. Moreover, it is possible that the same solution results for different selections of qxt,t [2]. While this ad hocness in choosing the Krotov function may seem burdensome, it provides enough freedom in making the selection according to the specifications of the problem in hand [11, 20]. In essence, this work exploits the ad hocness in selecting the Krotov function.(3)
The original GOCP is transformed to an equivalent optimization problem of the functionss and sf over the sets X,U, and Xf. Thus, the dynamical equation constraint x˙=fxt,ut,t is excluded from the optimization problem by defining Jeqxt,ut,t.(4)
If the optimization problem formulated in Theorem 2 is feasible (i.e., an admissible process satisfying the sufficient conditions in Theorem 2 can be found), then the functionqxt,t is called the solving function.(5)
Clearly, different selections ofqxt,t result in different optimization problems in Theorem 2, and thus, the selection of qxt,t is crucial to effectively solve the optimization problems in Theorem 2. Furthermore, the necessary conditions for existence of the optima (optimum) of the functions s and sf coincide with the popular Pontryagin’s minimum principle [20]. Moreover, a specific setting of the Krotov function qxt,t leads to the popular Hamilton–Jacobi–Bellman equation [2, 20]. These two observations lead to the conclusion that Krotov sufficient conditions are in fact the most general sufficient conditions for global results in optimal control theory.
The solution to the equivalent optimization problem given in Theorem 2 is generally computed using sequential methods because the resulting optimization problem is nonconvex. Such a sequence of processes is called anoptimizing sequence [10]. One such iterative method is the Krotov method in which an improving function is chosen at each iteration. To avoid iterative methods, we convexify the equivalent problems in Theorem 2 via a suitable selection of the Krotov function. The idea is illustrated in Figure 1 where the set Qc—the set of solving functions which result in the convexity of equivalent optimization problems in Theorem 2—is the set of interest. In the next section, the suitable Krotov functions for LQR and LQT problems are proposed. In the following, we use 2Jxt,ut,t as the cost functional instead of Jxt,ut,t, and the time variable t is dropped wherever it is required for the sake of simplicity.


## 3.1.1. Extension Principle

The essence of the extension principle is to replace the original optimization problem with complex relations and/or constraints with a simpler one such that they are excluded in the new problem definition but the solution of the new problem still satisfies the discarded relations [11].Consider a scalar-valued functionalIv defined over a set M (i.e., v∈M) and the optimization problem as follows.Problem 1.
Findv¯ such that d=infv∈MIv where d≜Iv¯. Instead of solving Problem 1, another equivalent optimization problem is solved. Let L denote the equivalent representation of the original cost functional. Then, a new problem is formulated over N, a superset of M, as follows.Equivalent Problem 1. Findv¯ such that e=infv∈NLv where e≜Lv¯.
The equivalent problem is also called the extension of the original problem. The method of choosing of theequivalent functional L is not unique, and the selection is generally made according to specifications of the problem under consideration. This freedom in the selection of the equivalent functional can be exploited to tackle the generic nonconvex optimization problems. Also, it is necessary to ensure that Iv=Lv,∀v∈M, so that the optimizer v¯ is actually the optimizer of original Problem 1 [2]. The idea behind the extension principle is illustrated in Figure 1. Clearly, the application of the extension principle requires appropriate selection of the equivalent functional L and the set N.Figure 1
(a) Extension Principle. (b) Idea behind selection of solving function.
(a)
(b)


## 3.1.2. Application of Extension Principle to Optimal Control Problems

The equivalent problem of the GOCP which is obtained by the application of the extension principle is given in the following theorem which provides a concrete definition of the equivalent functionalL and the set N.Theorem 1.
For the GOCP, letqxt,t be a continuously differentiable function. Then, there is an equivalent representation of (1) given as(18)Jeqxt,ut,ts=sfxtf+qxt0,t0+∫t0tfsxt,ut,tdt,where(19)sxt,ut,t≜∂q∂t+∂q∂xfxt,ut,t+lxt,ut,t,sfxtf≜lfxtf−qxtf,tf.Proof.
See Section 2.3 in [2] for the proof.
The equivalent functionalJeqxt,ut,ts leads to a sufficient condition for the global optimality of an admissible process (i.e., axt,ut pair which satisfies the dynamical equation x˙t=fxt,ut,t and the input/state constraints).Theorem 2.
(Krotov sufficient conditions). Ifx∗t,u∗t is an admissible process such that(20)sx∗t,u∗t,t=minx∈X,u∈Usxt,ut,t,∀t∈t0,tf,sfx∗tf=minx∈Xfsfx,then x∗t,u∗t is an optimal process. Here, Xf is the terminal set for admissible xt of x˙t=fxt,ut,t; that is, if xt is admissible, then xtf∈Xf.Proof.
See Section 2.3 in [2] for the proof.Remarks.
Some remarks which follow from Theorem 1 and Theorem 2 are now briefed as follows:(1)
The functionalJeq is the equivalent functional of the original functional J in (1). Specifically, Jeq=J∀xt,ut:x˙t=fxt,ut,t.(2)
The functionqxt,t, known as the Krotov function, can be any continuously differentiable function, and each qxt,t leads to a different equivalent functional Jeq. A general way to choose this function is not known, and the selection is usually done according to the specifics of the problem in hand. Moreover, it is possible that the same solution results for different selections of qxt,t [2]. While this ad hocness in choosing the Krotov function may seem burdensome, it provides enough freedom in making the selection according to the specifications of the problem in hand [11, 20]. In essence, this work exploits the ad hocness in selecting the Krotov function.(3)
The original GOCP is transformed to an equivalent optimization problem of the functionss and sf over the sets X,U, and Xf. Thus, the dynamical equation constraint x˙=fxt,ut,t is excluded from the optimization problem by defining Jeqxt,ut,t.(4)
If the optimization problem formulated in Theorem 2 is feasible (i.e., an admissible process satisfying the sufficient conditions in Theorem 2 can be found), then the functionqxt,t is called the solving function.(5)
Clearly, different selections ofqxt,t result in different optimization problems in Theorem 2, and thus, the selection of qxt,t is crucial to effectively solve the optimization problems in Theorem 2. Furthermore, the necessary conditions for existence of the optima (optimum) of the functions s and sf coincide with the popular Pontryagin’s minimum principle [20]. Moreover, a specific setting of the Krotov function qxt,t leads to the popular Hamilton–Jacobi–Bellman equation [2, 20]. These two observations lead to the conclusion that Krotov sufficient conditions are in fact the most general sufficient conditions for global results in optimal control theory.
The solution to the equivalent optimization problem given in Theorem 2 is generally computed using sequential methods because the resulting optimization problem is nonconvex. Such a sequence of processes is called anoptimizing sequence [10]. One such iterative method is the Krotov method in which an improving function is chosen at each iteration. To avoid iterative methods, we convexify the equivalent problems in Theorem 2 via a suitable selection of the Krotov function. The idea is illustrated in Figure 1 where the set Qc—the set of solving functions which result in the convexity of equivalent optimization problems in Theorem 2—is the set of interest. In the next section, the suitable Krotov functions for LQR and LQT problems are proposed. In the following, we use 2Jxt,ut,t as the cost functional instead of Jxt,ut,t, and the time variable t is dropped wherever it is required for the sake of simplicity.


## 3.2. Solution of OCP 1 (LQR Problem)

The equivalent optimization problem for OCP 1, as per Theorem 2, is given as follows.Equivalent OCP 1: compute an admissible pairx∗,u∗ which is as follows:(i)
minx,u∈ℝn×ℝmsx,u,t,∀t∈t0,tf, where(21)s=∂q∂t+∂q∂xAx+Bu+xTQx+uTRu.(ii)
minx∈ℝnsfx, where(22)sf=xTtfFtfxtf−qxtf,tf.The next proposition is one of the main results of this paper, where we propose a suitable Krotov function which will be useful in computing a direct solution to OCP 1.Proposition 1.
For Equivalent OCP 1, let the Krotov function be chosen as(23)qx,t=xTPx,and every element of the matrix P is differentiable ∀t∈t0,tf. Then, the following statements are equivalent:(a)
sx,u,t and sfxtf are convex functions in x,u and xtf, respectively.(b)
P satisfies the following matrix inequalities:(i)
P˙+PA+ATP+Q−1/2PBR−1BTP−1/4PBR−1BTPT−1/4PTBR−1BTP≽0,∀t∈t0,tf(ii)
Ftf−Ptf≽0Proof.
Withq selected as in (23), the function sx,u,t becomes(24)s=xTP˙x+xTP+PTAx+Bu+xTQx+uTRu=xTP˙x+xTPAx+xTPTAx+xTPTBu+xTPBu+xTQx+uTRu.
Adding and subtracting the termxT1/2PBR−1BTP+1/4PBR−1BTPT+1/4PTBR−1BTPx, we get(25)s=xTP˙+PA+ATP+Q−12PBR−1BTP−14PBR−1BTP−14PBR−1BTPx+xTPBu+xTPTBTx+uTRu+12xTPBR−1BTPx+14xTPBR−1BTPTx+14xTPTBR−1BTPx.
SinceR≻0, there exists a unique positive definite matrix [21], say R˜, such that R˜2=R and R˜−1R˜−1=R−1. Now rearranging the terms in (25), we get(26)s=xTP˙+PA+ATP−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTPx+R˜u+12R˜−1BTPT+BTPxTR˜u+12R˜−1BTPT+BTPx.
Clearly the second term in (26) is strictly convex. Now, s is convex iff the following condition is satisfied:(27)P˙+PA+ATP−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP≽0,∀t∈t0,tf.
Moreover, withq as in (23), sfxtf is given as(28)sf=xTtfFtfxtf−xTtfPtfxtf=xTtfFtf−Ptfxtf.
Finally,sf is convex iff(29)Ftf−Ptf≽0.Corollary 1.
Withqx,t selected as in Proposition 1, the following statements are true:(1)
The functionq=xTPx is a solving function for OCP 1.(2)
The global optimal control law for OCP 1 is given by(30)u∗=−12R−1BTPT+Px∗,where P is the solution of the matrix differential equation(31)P˙+PA+ATP+Q−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP=0,with the final value Ptf=Ftf.Proof.
Clearly, ifP satisfies (27), then s is independent of x and the obtained control law results in an admissible process. Hence, the selected q is the solving function. Furthermore, for minimization, it is required that the second term in (26) is zero which gives the optimal control law as(32)u∗=−12R−1BTPT+Px∗.


### 3.2.1. Solution of OCP 1 with Final Timetf⟶∞ (Infinite Horizon LQR)

Next, the infinite-final time LQR problem for a linear time invariant (LTI) is considered. For this case, the terminal state weighing matrixFtf=0. The problem statement now is as follows.Problem 2.
(infinite horizon LQR problem). Compute an optimal control lawu∗t which minimizes the quadratic performance index/cost functional(33)Jxt,ut,t=0.5∫t0∞xTtQxt+uTtRutdt,subject to the system dynamics x˙t=Axt+But and drives the states of the system to zero (regulation). Here, xt0=x0 is given, and x∞ is free. Also, Q≽0andR≻0. The quantities A, B, Q, and R are time-independent.Solution.
For this problem, the matrix differential equation (27) needs to be solved with the boundary condition P∞=0. Computing this solution is equivalent to solving the following algebraic equation:(34)PA+ATP+Q−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP=0,and the resulting optimal control law is given as(35)u∗=−12R−1BTPT+Px∗.
Similar to the finite-final time case, it is easy to verify that the functionq=xTPx (where P is a time-invariant (constant) matrix) is a solving function. Note that since the final time tf⟶∞, it is necessary to ensure the stability of the closed-loop system. The next lemma provides a proof of the closed-loop stability under another condition on the P matrix.Lemma 1.
For the systemx˙=Ax+Bu, the control input (35) ensures the stability of the closed loop if P satisfies (34) with P+PT≻0.Proof.
Let the Lyapunov function beVxt=xTP+PTx. Then,(36)V˙=xTP+PTx˙+x˙TP+PTx=xTATP+ATPT+ATP+ATPTx−2xTPBR−1BTPx−xTPTBR−1BTPx−xTPBR−1BTPTx=2xT−Q−14PBR−1BTPT−14PTBR−1BTP−12PBR−1BTPx.
It is easy to verify the quantity−Q−1/4PBR−1BTPT−1/4PTBR−1BTP−1/2PBR−1BTP is negative semidefinite if Q≽0 and negative definite if Q≻0. Thus, using Lyapunov theory [22], the closed-loop system is stable if Q≽0 and asymptotically stable if Q≻0.Remarks.
The matrix differential equation (31) reduces to the popular matrix differential Riccati equation (MDRE) for a symmetric P matrix. Also, the algebraic equation (34) admits more numbers of solutions than the MDRE as demonstrated in Example 3 in Section 5. A more rigorous analysis and application of these solutions are the subject of future research.


## 3.2.1. Solution of OCP 1 with Final Timetf⟶∞ (Infinite Horizon LQR)

Next, the infinite-final time LQR problem for a linear time invariant (LTI) is considered. For this case, the terminal state weighing matrixFtf=0. The problem statement now is as follows.Problem 2.
(infinite horizon LQR problem). Compute an optimal control lawu∗t which minimizes the quadratic performance index/cost functional(33)Jxt,ut,t=0.5∫t0∞xTtQxt+uTtRutdt,subject to the system dynamics x˙t=Axt+But and drives the states of the system to zero (regulation). Here, xt0=x0 is given, and x∞ is free. Also, Q≽0andR≻0. The quantities A, B, Q, and R are time-independent.Solution.
For this problem, the matrix differential equation (27) needs to be solved with the boundary condition P∞=0. Computing this solution is equivalent to solving the following algebraic equation:(34)PA+ATP+Q−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP=0,and the resulting optimal control law is given as(35)u∗=−12R−1BTPT+Px∗.
Similar to the finite-final time case, it is easy to verify that the functionq=xTPx (where P is a time-invariant (constant) matrix) is a solving function. Note that since the final time tf⟶∞, it is necessary to ensure the stability of the closed-loop system. The next lemma provides a proof of the closed-loop stability under another condition on the P matrix.Lemma 1.
For the systemx˙=Ax+Bu, the control input (35) ensures the stability of the closed loop if P satisfies (34) with P+PT≻0.Proof.
Let the Lyapunov function beVxt=xTP+PTx. Then,(36)V˙=xTP+PTx˙+x˙TP+PTx=xTATP+ATPT+ATP+ATPTx−2xTPBR−1BTPx−xTPTBR−1BTPx−xTPBR−1BTPTx=2xT−Q−14PBR−1BTPT−14PTBR−1BTP−12PBR−1BTPx.
It is easy to verify the quantity−Q−1/4PBR−1BTPT−1/4PTBR−1BTP−1/2PBR−1BTP is negative semidefinite if Q≽0 and negative definite if Q≻0. Thus, using Lyapunov theory [22], the closed-loop system is stable if Q≽0 and asymptotically stable if Q≻0.Remarks.
The matrix differential equation (31) reduces to the popular matrix differential Riccati equation (MDRE) for a symmetric P matrix. Also, the algebraic equation (34) admits more numbers of solutions than the MDRE as demonstrated in Example 3 in Section 5. A more rigorous analysis and application of these solutions are the subject of future research.


## 3.3. Solution of OCP 2 (LQT Problem)

The equivalent optimization problem for OCP 2 is given as follows.Equivalent OCP 2: compute an optimal control lawu∗t which is as follows:(i)
minx,u∈ℝn×ℝmsx,u,t, ∀t∈t0,tf, where(37)s=∂q∂t+∂q∂xAx+Bu+eTQe+uTRu.(ii)
minx∈ℝnsfxtf, where(38)sf=eTtfFtfetf−qxtf,tf.Another main result of this paper is given in the next proposition, which will be useful in computing the direct solution to OCP 2.Proposition 2.
For Equivalent OCP 2, let the Krotov function be chosen as(39)q≜xTPx−2gTx,and every element of the matrix P and the vector g is differentiable ∀t∈t0,tf. Then, the following statements are equivalent:(a)
sx,u,t and sfxtf are bounded below and convex in x,u and xtf, respectively.(b)
(i)P satisfies the following matrix inequalities:(1)
P˙+PA+ATP−1/2PBR−1BTP−1/4PBR−1BTPT−1/4PTBR−1BTP+CTQC≽0∀t∈t0,tf(2)
CTtfFtfCtf−Ptf≽0(ii)
g satisfies the vector differential equation(40)g˙+ATg+CTQz−12P+PTBR−1BTg=0,
with the boundary conditiongtf=CTtfFtfztf.Proof.
Withq chosen as in (39), the function sx,u,t is given as(41)s=xTP˙x−2g˙Tx+xTP+PT−2gTAx+Bu+eTQe+uTRu=xTP˙+PA+PTA+CTQCx+xTPBu+xTPTBu−2g˙Tx−2gTAx−2gTBu+zTQz−2xTCTQz+uTRu.
Adding and subtracting the termsxT1/2PBR−1BTP+1/4PBR−1BTPT+1/4PTBR−1BTPx, gTBR−1BTg, and xTP+PTBR−1BTg, we get(42)s=xTP˙+PA+ATP−12PBR−1BTP+CTQC−14PBR−1BTPT−14PTBR−1BTPx−2xTg˙+ATg+CTQz−12P+PTBR−1BTg+12uTBTPx−uTBTg+12uTBTPTx−12xTPBR−1BTg−12xTPTR−1BTg+12xTPBu+14xTPBR−1BTPx+12xTPTBu+14xTPTBR−1BTPx+14xTPTBR−1BTPx+14xTPTBR−1BTPTx−12gBR−1BTPx−12gBR−1BTPTx+gTBR−1BTg.
SinceR≻0, there exists a unique positive definite matrix [21], say R˜, such that R˜2=RandR˜−1R˜−1=R−1. Now rearranging the terms in (43), we get(43)s=xTP˙+PA+ATP−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP+CTQCx+R˜u+12R˜−1BTP+PTx−R˜−1BTgTR˜u+12R˜−1BTP+PTx−R˜−1BTg−2xTg˙+ATg+CTQz−12P+PTBR−1BTg+zTQz−gTBR−1BTg.
The second term in (44) is positive definite and strictly convex. The third term is linear in x, and the fourth term is independent of x and u. The first term in (44) is quadratic in x, and it is convex iff it is positive semidefinite. Hence, the function sx,u,t is convex iff(44)P˙+PA+ATP−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP+CTQC≽0,∀t∈t0,tf.
Since the second term in (44) is linear in x, it is bounded below if(45)g˙+ATg+CTQz−12P+PTBR−1BTg=0.
Finally, the term independent ofx and u, i.e., zTQz−gTBR−1BTg, is bounded because zt and gt are bounded [23]. Next, with qxt,t as in (39), sfxtf is given as(46)sf=xTtfCTtfFtfCtf−Ptfxtf+2gTtf−2zTtfFtfCtfxtf+zTtfFtfztf.
Similar to the case ofs, sf is convex iff(47)CTtfFtfCtf−Ptf≽0.
Finally,sf is bounded below if(48)gtf=CTtfFtfztf.
The following corollary computes the optimal control law for OCP 2.Corollary 2.
Withqx,t selected as in Proposition 1, the following statements are true:(1)
The functionq=xTPx−2gTx is a solving function for OCP 2.(2)
The global optimal control law for OCP 2 is given by(49)u∗=−12R−1BTPT+Px∗+R−1BTg,where P is the solution of the matrix differential equation(50)P˙+PA+ATP−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP+CTQC=0,with the final value Ptf=CTtfFtfCtf, and gt satisfies the differential equation g˙+ATg+CTQz−1/2P+PTBR−1BTg=0 with the final value gtf=CTtfFtfztf.Proof.
Clearly, ifP satisfies (50) and g satisfies (45), then s is independent of x and the obtained control law yields an admissible process. Hence, qxt,t is the solving function. The third term in (44) is strictly convex and attains a minimum value when(51)R˜u+12R˜−1BTP+PTx−R˜−1BTg=0⟹u∗=−12R−1BTPT+Px∗+R−1BTg.


### 3.3.1. Solution of OCP 2 with Final Timetf⟶∞ (Infinite Horizon LQT)

Next, the infinite-final time LQT problem for a linear time invariant (LTI) is considered. For this case, the terminal costFtf=0. The problem statement now reads as follows.Problem 3.
(infinite horizon LQT problem). Compute an optimal control lawu∗t which minimizes the quadratic performance index/cost functional(52)Jet,ut,t=0.5∫t0∞eTtQet+uTtRutdt,subject to the system dynamics x˙t=Axt+But and yt=Cxt and drives the states of the system to the desired trajectory zt. Here, et≜yt−zt is the error vector, xt0=x0 is given, xtf is free, tf is fixed, and x∞ is free. Also, Q≽0andR≻0.Solution.
For this problem, similar to the case of regulation, the matrix differential equation (50) needs to be solved with the boundary condition P∞=0. Similarly, the differential equation (45) needs to be solved with the boundary condition g∞=0. Computing these solutions is equivalent to solving(53)PA+ATP+Q−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP+CTQC=0,gt=−∫t∞expAT−1/2P+PtBR−1BTτ−1CTQzτdτ,and the resulting optimal control law is given as(54)u∗=−12R−1BTPT+Px∗+R−1BTg.
Similar to the finite-final time case, it is easy to verify that the functionq=xTPx−2gTx (where P is a time-invariant (constant) matrix) is a solving function. In this case too, similar to the case of regulation, the stability of the closed loop is ensured if P+PT≻0.Remarks.
As is clear from the solution methodologies proposed above, the solution of the considered problems does not require the notion of costates and any a priori information regarding the optimal cost function. Rather, the whole solution technique depends upon the selection of the Krotov functionqxt,t which directly affects the nature of equivalent optimization problems.


## 3.3.1. Solution of OCP 2 with Final Timetf⟶∞ (Infinite Horizon LQT)

Next, the infinite-final time LQT problem for a linear time invariant (LTI) is considered. For this case, the terminal costFtf=0. The problem statement now reads as follows.Problem 3.
(infinite horizon LQT problem). Compute an optimal control lawu∗t which minimizes the quadratic performance index/cost functional(52)Jet,ut,t=0.5∫t0∞eTtQet+uTtRutdt,subject to the system dynamics x˙t=Axt+But and yt=Cxt and drives the states of the system to the desired trajectory zt. Here, et≜yt−zt is the error vector, xt0=x0 is given, xtf is free, tf is fixed, and x∞ is free. Also, Q≽0andR≻0.Solution.
For this problem, similar to the case of regulation, the matrix differential equation (50) needs to be solved with the boundary condition P∞=0. Similarly, the differential equation (45) needs to be solved with the boundary condition g∞=0. Computing these solutions is equivalent to solving(53)PA+ATP+Q−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP+CTQC=0,gt=−∫t∞expAT−1/2P+PtBR−1BTτ−1CTQzτdτ,and the resulting optimal control law is given as(54)u∗=−12R−1BTPT+Px∗+R−1BTg.
Similar to the finite-final time case, it is easy to verify that the functionq=xTPx−2gTx (where P is a time-invariant (constant) matrix) is a solving function. In this case too, similar to the case of regulation, the stability of the closed loop is ensured if P+PT≻0.Remarks.
As is clear from the solution methodologies proposed above, the solution of the considered problems does not require the notion of costates and any a priori information regarding the optimal cost function. Rather, the whole solution technique depends upon the selection of the Krotov functionqxt,t which directly affects the nature of equivalent optimization problems.


## 4. Analysis of the LMI Obtained upon Convexity Imposition

As mentioned earlier in Introduction, it was asserted in [18] that the origin of Riccati-type linear matrix inequalities (LMIs) is not much appreciated in the theory. This section tries to address this issue. Specifically, note that the convexity imposition for EOCP 1 for the infinite horizon case (in Section 3.2) results in the following matrix inequality:(55)PA+ATP+Q−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP≻0.Thus, this inequality preserves the strict convexity of EOCP 1 for the infinite horizon case. Furthermore, for a symmetricP, this inequality reduces to(56)ATP+PA+Q−PBR−1BTP≻0.Using Schur’s complement [24], (56) can be written as the following linear matrix inequality (LMI):(57)ATP+PA+QBTPPBR≻0.Thus, (56), which is essentially Riccati LMI, preserves the convexity of EOCP 1. This complete explanation for the origin of (57) fills the gap in the literature (regarding the origin of this LMI as stated in [18]).Next, an analysis of this inequality is presented. Specifically, the following question is being addressed:Is it possible to solve (56) to obtain stabilizing (and of course suboptimal) control laws (of the formu=−R−1BTPx) for the LQR problem? The motivation behind this query is the fact that once a stabilizing solution using (56) is found, it is straightforward to find how suboptimal the obtained solution is with respect to the optimal solution obtained using equation (27). This is true because it is known that if P satisfies (56), then the equivalent optimization problem is convex, and thus, it is straightforward to find how far is the obtained suboptimal solution from the global optimum. Thus, the problem of suboptimal control of linear systems with quadratic cost is also addressed upon solving (56). Note, moreover, that another LMI-based approach for the suboptimal control of linear systems with quadratic cost was utilized in [25] to design suboptimal distributed control laws for multiagent systems. With this motivation, the analysis of (56) is presented next. Specifically, sufficient conditions for the existence of a matrix, say, P˜, which satisfy this inequality and result in the stabilizing control laws of the form u=−R−1BTP˜x are derived in the next proposition.Proposition 3.
Define(i)
P∈ℝn×n as the solution of the algebraic Riccati equation, i.e., ATP+PA+Q−PBR−1BTP=0, P≻0(ii)
β=mineigP, β>0If∃ positive constants α, δ, and γ and a positive definite matrix P0 such that(1)
ATP0+P0A≺2αI (note that this condition implies that the eigenvalues of A lie to the left of the line s=α in the complex plane)(2)
γI≺P0≺δI with δ=β−α(3)
BR−1BT≻2α/βγ−δ2Ithen the following hold:(i)
ATP˜+P˜A+Q−P˜BR−1BTP˜≻0(ii)
A−BR−1BTP˜ is Hurwitz, where P˜=P−P0Proof.
SinceBR−1BT≻2α/βγ−δ2I, it follows that BR−1BT≻2α/2βγ−δ2I because βγ>0. Consider(58)BR−1BT≻2α2βγ−δ2I⟹2βγ−δ2BR−1BT≻2αI⟹2βγBR−1BT−δ2BR−1BT−2αI≻0⟹βBR−1BTγ+γBR−1BTβ−δBR−1BTδ−2αI≻0⟹PBR−1BTP0+P0BR−1BTP−P0BR−1BTP0−2αI≻0becauseP≻βIandγI≺P0≺δI⟹PBR−1BTP0+P0BR−1BTP−P0BR−1BTP0−ATP0−P0A≻0becauseATP0+P0A≺2αI⟹PBR−1BTP0+P0BR−1BTP−P0BR−1BTP0−ATP0−P0A+ATP+PA+Q−PBR−1BTP≻0Ricatti equation⟹ATP−P0+P−P0A+Q−P−P0BR−1BTP−P0≻0⟹ATP˜+P˜A+Q−P˜BR−1BTP˜≻0.
This completes the proof of parti.
Again,BR−1BT≻2α/βγ−δ2I implies that BR−1BT≻α/βγ−δ2I because α≻0. Next, consider(59)BR−1BT≻αIβγ−δ2⟹2βγ−δ2BR−1BT≻2αI⟹2βγ−δ2BR−1BT−2αI≻0⟹−2βγ−δ2BR−1BT+2αI≺0⟹−2βγBR−1BT+2δ2BR−1BT+2αI≺0⟹2αI−βγBR−1BT−βγBR−1BT+δ2BR−1BT+δ2BR−1BT≺0⟹2αI−PBR−1BTP0−P0BR−1BTP+P0BR−1BTP0+P0BR−1BTP0≺0⟹ATP0+P0A−PBR−1BTP0−P0BR−1BTP+P0BR−1BTP0+P0BR−1BTP0≺0⟹ATP0+P0A−P−P0BR−1BTP0−P0BR−1BTP−P0≺0⟹ATP0+P0A−P˜BR−1BTP0−P0BR−1BTP˜≺0⟹A−BR−1BTPP0+P0A−BR−1BTP˜≺0.
This completes the proof of partii.
Proposition 3 provides a result for suboptimal linear quadratic optimal control design. Further exploration and application of this result are the topic of future research.


### 4.1. Krotov Method

The Krotov method is one of the iterative methods to solve the equivalent optimization problems of Theorem 2. The steps of this method are as follows:(1)
Choose any admissible controlu0t and compute the corresponding admissible process v0=u0t,x0t using the dynamical equation x˙=fxt,ut,t of the system. Also, compute the cost J0.(2)
Construct a functionϕxt,t such that(a)
st,x0t,u0t=maxx∈ℝn,st,xt,u0t(b)
sfx0tf=maxx∈Xf,sfxtf(3)
Construct the control inputu˜t such that(60)u˜t=argminust,x,u.(4)
Compute the cost of the resulting admissible process and compare it with the cost obtained in step 1 to determine the extent of improvement achieved, viz., compare the difference between the two costs with a predefined tolerance, say ϵ. If the difference is less than ϵ, go back to step 2 with the improved process obtained in step 3; otherwise, stop.This method is demonstrated in Figure2. Now, the optimal control problem for a scalar system is solved to clarify the approach of the Krotov method.Figure 2
Krotov method.Problem 4.
For the systemx˙t=−xt+ut with x0=5, compute the optimal control law so as to minimize J=∫010x2t+u2tdt.Solution.
(i)
Choose an admissible process asu0t=0 which gives x0t=5. The corresponding cost is computed to be J0=25.
(ii)
Next, a function ϕ is chosen so as to maximizest,x,u0=∂ϕ/∂t+∂q/∂x−x+u+x2+u2 over x. Upon a quadratic selection of ϕ as ϕ=px2, the function s is given as st,x,u0=x2p˙−2p+1 and gx10=−p10x210. Choosing p such that p˙−2p+1=0 and p10=0 clearly turns out to be appropriate selection for this case. Then, the improved process is given as v1=x1,u1 where x˙1=−1−px1 and p10=0. The corresponding cost is computed to be J1=10.42.
(iii)
Next, the improved function is again selected to be a quadratic functionq=px2, and the corresponding equation for p is given as
(61)
p
˙
−
2
p
+
1
−
p
2
=
0
,with p10=0. The corresponding improved process is given by x˙2=−1−px2 with u2=−px2, and the corresponding cost is computed as J2=10.36. It was found that, upon further iterations, no improvement of cost was achieved, leading to the conclusion that this process is the globally optimal process.


## 4.1. Krotov Method

The Krotov method is one of the iterative methods to solve the equivalent optimization problems of Theorem 2. The steps of this method are as follows:(1)
Choose any admissible controlu0t and compute the corresponding admissible process v0=u0t,x0t using the dynamical equation x˙=fxt,ut,t of the system. Also, compute the cost J0.(2)
Construct a functionϕxt,t such that(a)
st,x0t,u0t=maxx∈ℝn,st,xt,u0t(b)
sfx0tf=maxx∈Xf,sfxtf(3)
Construct the control inputu˜t such that(60)u˜t=argminust,x,u.(4)
Compute the cost of the resulting admissible process and compare it with the cost obtained in step 1 to determine the extent of improvement achieved, viz., compare the difference between the two costs with a predefined tolerance, say ϵ. If the difference is less than ϵ, go back to step 2 with the improved process obtained in step 3; otherwise, stop.This method is demonstrated in Figure2. Now, the optimal control problem for a scalar system is solved to clarify the approach of the Krotov method.Figure 2
Krotov method.Problem 4.
For the systemx˙t=−xt+ut with x0=5, compute the optimal control law so as to minimize J=∫010x2t+u2tdt.Solution.
(i)
Choose an admissible process asu0t=0 which gives x0t=5. The corresponding cost is computed to be J0=25.
(ii)
Next, a function ϕ is chosen so as to maximizest,x,u0=∂ϕ/∂t+∂q/∂x−x+u+x2+u2 over x. Upon a quadratic selection of ϕ as ϕ=px2, the function s is given as st,x,u0=x2p˙−2p+1 and gx10=−p10x210. Choosing p such that p˙−2p+1=0 and p10=0 clearly turns out to be appropriate selection for this case. Then, the improved process is given as v1=x1,u1 where x˙1=−1−px1 and p10=0. The corresponding cost is computed to be J1=10.42.
(iii)
Next, the improved function is again selected to be a quadratic functionq=px2, and the corresponding equation for p is given as
(61)
p
˙
−
2
p
+
1
−
p
2
=
0
,with p10=0. The corresponding improved process is given by x˙2=−1−px2 with u2=−px2, and the corresponding cost is computed as J2=10.36. It was found that, upon further iterations, no improvement of cost was achieved, leading to the conclusion that this process is the globally optimal process.


## 5. Numerical Examples

In this section, the proposed solution methodology is demonstrated via numerical examples. Firstly, the finite-time optimal control laws for a scalar linear time-varying system are computed. Then, infinite-time optimal control laws for a linear time-invariant (LTI) multi-input multioutput (MIMO) system are computed.Example 1.
Compute the optimal control law for the systemx˙=−1/t+1x+u which minimizes the performance index J=0.5x2tf+0.5∫0tfx2+u2dt with tf=5 and x0=20.Solution.
Following Theorem 2, the equivalent optimization problem to be solved is given as(62)minx,u∈ℝ×ℝsx,u,t,(63)minxtf∈ℝsfxtf,tf,where s=∂q/∂t+∂q/∂x−1/t+1x+u+x2+u2 and sf=x2tf−qxtf,tf. According to Proposition 1, the Krotov function is chosen as(64)q=ptx2,and then the functions s and sf are read as(65)s=p˙tx2−2ptx2t+1+2ptxu+x2+u2,sf=x2tf−ptfx2tf.
Clearly, the functions is nonconvex. Because of this, the equivalent problems (62) and (63) are solved using an iterative method, such as the Krotov method in which q is chosen appropriately at each iteration. Instead, using Proposition 1, the direct solution can be computed by choosing pt such that(66)p˙t−2t+1pt−p2t+1≥0,∀t,1−p5≥0.
Finally, using Corollary 1,pt is chosen such that p˙t−2/t+1pt−p2t+1=0 and p5=1. The solution of the above differential equation is given as(67)pt=2.423∗105t−te2t+4.846∗1052.423∗105t+e2t+te2t+2.423∗105,and the optimal control law is computed as(68)u∗=−ptx∗.
The plots ofpt and the state response of the closed-loop system and the optimal control input are shown in Figures 3(a) and 3(b), respectively.Figure 3
Example 1: (a) plot ofp(t); (b) optimal control input and closed-loop response.
(a)
(b)Example 2.
Compute the optimal control law for the system given asx˙=−1/t+1x+u which minimizes the performance index J=0.510e2tf+∫0tf1000e2+u2dt with et≜zt−yt, the reference trajectory zt=t, tf=5, and x0=10.Solution.
The optimization problem to be solved is given as(69)minx,u∈ℝ×ℝsx,u,t,minxtf∈ℝsfxtf,tf,s=∂q∂t+∂q∂x−1t+1x+u+1000e2+u2,sf=10e2tf−qxtf,tf.
Let the Krotov function be chosen as(70)q=ptx2−2gtx,and then the functions s and sf are given as(71)s=p˙tx2−2g˙tx−2ptx2t+1+2ptxu+2gtxt+1−2gtu+1000t2+1000x2−2000zx,sf=10x2tf+10z2tf−2xtfztf−ptfx2tf+2gtfxtf.
Clearly, the functionss and sf are nonconvex. Again, using Proposition 2, the direct solution can be computed by choosing pt and gt such that(72)p˙−2t+1p−p2+1000≥0,∀t,10−p5≥0,g˙t−1t+1gt+1000t−pg=0withg5=50.
Finally, using Corollary 1,p is chosen such that p˙−2/t+1p−p2+1000=0,10−p5=0, and g˙t−1/t+1gt+1000t−pg=0withg5=50. The obtained pt and gt are as shown in Figure 4.
Finally, the optimal control law is computed as(73)u∗=−ptx∗+gt.
The closed-loop response is as shown in Figure4.
Next, infinite horizon optimal control problems for the LTI MIMO system are considered.Figure 4
Example 2: (a) plots ofp(t) and gt; (b) closed-loop response.
(a)
(b)Example 3.
For the MIMO systemx˙=Ax+Bu, compute an optimal control law to minimize the performance index J=0.5∫0∞xTQx+uTRudt where A=0111, B=1101, Q=2004, and R=0.5000.25. Also, x0=105T.Solution.
The optimization problem to be solved is given as(74)minx,u∈ℝ2×ℝ2sx,u,t,where(75)s=∂q∂t+∂q∂x0111x+1101u+xT2004x+uT0.5000.25u.
This function is nonconvex. However, as shall be demonstrated, the function can be convexified ifqx,t is chosen as per Proposition 1.
Let the Krotov function be(76)qx=xTPx,P=p11p12p21p22.
Then, the functionsx,u,t is given as(77)sx,u,t=xTP+PT0111x+1101u+2x12+4x22+0.5u12+0.25u22=x12p12+p21+2+x22p12+p21+2p22+4+x1x22p11+p12+p21+2p22+2u1x1p11+u2x12p11+p12+p21+u1x2p12+p21+u2x2p12+p21+2p22+0.5u12+0.25u22.
Next, using Proposition 1 and Corollary 1, the direct solution can be obtained by choosingP so as to satisfy(78)PA+ATP−12PBR−1BTP−14PBR−1BTPT−14PTBR−1BTP=0.
This selection results in a convexs. This fact is illustrated by the plots in Figure 5 where s is plotted against x and u for x and u along the lines x2=αx1 and u2=αu1 with α=1.
Equation (78) admits the four solutions P1=9.41726.06716.8083−15.095, P2=−5.755912.75611.318−6.5319, P3=5.7575−10.427−11.6545.0354, and P4=8.6906−5.9759−5.264715.05.
Finally, to ensure the stability of the closed loop,P is chosen such that Lemma 1 is satisfied, i.e., P+PT≻0. It can be easily verified that only P4 satisfies this requirement. Thus, P4 is the required value of the matrix P in this case. Finally, the optimal control law is given as(79)u∗=−12R−1BTP+PTx∗=−1.2887x1∗+0.4267x2∗−1.7240x1∗+−5.0787x2∗.
The closed-loop response is shown in Figure6(a).Figure 5
Plots ofs for α=1. (a) Example 3. (b) Example 4.
(a)
(b)Figure 6
Closed-loop response for (a) Example 3 (regulation) and (b) Example 4 (tracking).
(a)
(b)Example 4.
For the MIMO systemx˙=Ax+Bu and y=Cx, compute an optimal control law to minimize the performance index J=0.5∫0∞eTQe+uTRudt where A=0111, B=1101, C=1001 Q=20000400, R=0.5000.25, e is the error defined as e≜y−z, and z is the reference defined as z=0sinωt with ω=0.01π. Also, x0=52T.Solution.
The optimization problem to be solved is given as(80)minx,u∈ℝ2×ℝ2sx,u,t,where(81)s=∂q∂t+∂q∂x0111x+1101u+eT20000400e+uT0.5100.25u.
We use the Krotov function as(82)qx,t=xTPx−2gTx;P=p11p12p21p22;gt=g1tg2t.
Then, the functionsx,u,t is given by(83)s=x12p12+p21+200+x22p12+p21+2p22+400+x1x22p11+p12+p21+2p22+2u1x1p11+u2x12p11+p12+p21+u1x2p12+p21+u2x2p12+p21+2p22−2g1x2−2g1u1−2g1u2−2g2x1−2g2x2−2g2u2+400sin2ωt.
Again, it is easily verified that the functionsx,u,t is indeed convex for the selection of P and gt as in Corollary 2. P is computed using (78) similar to the case of regulation, and gt is a time-varying function which can be computed as the steady-state solution as in (53):(84)gt=−∫t∞expAT−1/2P+PTBR−1BTτ−tCTQzτdτ.
The convexity ofs by these selections of P and gt is illustrated by the plots in Figure 5 for α=1 similar to the case of regulation.
TheP matrix is chosen to be (similar to the case of regulation) P=P4, and gt is calculated to be(85)gt=−2.857sin0.0314t+0.003cos0.0314t0.0039cos0.031t−5.68sin0.031t.
The plots ofg1t and g2t are shown in Figure 7 which clearly show their boundedness. Finally, the optimal control law is calculated using Corollary 2, and the closed-loop response is given in Figure 6(b).Figure 7
Plots ofg1t and g2t for Example 4.


## 6. Conclusion

In this paper, we propose a novel method to compute globally optimal control laws for linear quadratic regulation and tracking problems based on Krotov sufficient conditions. The solution to the linear optimal control problem has been widely addressed in the literature using the celebrated CoV/HJB methods. These methods synthesize the global optimal control law, which is unique and requires some forced assumptions. In order to address this issue, we solved the optimal control problem using Krotov sufficient conditions, which do not require the notion of costates (and hence the related assumptions) and the existence of the continuously differentiable optimal cost function. The idea behind Krotov formulation is that the original optimal control problem is translated into another equivalent optimization problem utilizing the so-called extension principle. The resulting optimization problem is nonconvex, which is generally solved using iterative methods to yield the globally optimal solution. The angle of our attack is to compute a noniterative solution, which is achieved by imposing convexity conditions on the equivalent optimization problem. This convexity imposition lets us compute the optimal control laws without introducing any assumption or using anya priori information and also explains the origin of a Riccati LMI in the literature. To accomplish convexity imposition, the selection of the Krotov function becomes very crucial, which shall be addressed in the future specifically for nonlinear optimal control and distributed optimal control problems.



---

*Source: 9490512-2019-10-13.xml*