## 3 Discrete-Time Inverse Optimal Control  
Consider the following cost function related to trajectory tracking of the nonlinear system in (1)  
\[J(\delta_{k})=\sum\limits_{n=k}^{\infty}\left(l(\delta_{n})+u_{n}^{T}R_{z}u_{ n}\right) \tag{8}\]  
where \(\delta=z_{k}-z_{r,k}\); \(z_{r,k}\) is the desired trajectory for \(z_{k}\), \(J(\delta_{k}):R^{n}\to R^{+}\); \(l(\delta_{k}):R^{n}\to R^{+}\) is a positive semidefinite function, and \(R_{z}:R^{n}\to R^{+}\) is a real weighting matrix, symmetric and positive definite for control efforts. The cost function in (8) is a performance measurement. Availability of system states is necessary for the state feedback control design. Using the optimal value function \(J^{*}(\delta_{k})\) for (8) as a Lyapunov function \(V_{L}(\delta_{k})\), the cost function can be rewritten as [32]  
\[\begin{array}{ll}V_{L}(\delta_{k})&=l(\delta_{k})+u_{k}^{T}R_{z}u_{k}+\sum \limits_{n=k+1}^{\infty}l(\delta_{n})+u_{k}^{T}R_{z}u_{k}\\ &=l(\delta_{k})+u_{k}^{T}R_{z}u_{k}+V_{L}(\delta_{k+1})\end{array} \tag{9}\]  
From [PERSON]'s principle of optimality [33], for the case of infinite time horizon optimization, the value function \(V_{L}^{*}(\delta_{k})\) becomes time invariant and satisfies the discrete-time Bellman equation. The boundary condition \(V(0)=0\) must be satisfied so that \(V_{L}(\delta_{k})\) becomes a Lyapunov function. The following discrete-time Hamiltonian [32] is defined  
\[H(\delta_{k},u_{k})=l(\delta_{k})+u_{k}^{T}R_{z}u_{k}+V_{L}(\delta_{k+1})-V_{L }(\delta_{k+1})-V_{L}(\delta_{k}) \tag{10}\]  
A necessary condition for the optimal control law \(u_{k}\) is \(\partial H(\delta_{k},u_{k})/\partial u_{k}=0\). Then,  
\[0=2 Ru_{k}+\frac{\partial V_{L}(\delta_{k+1})}{\partial u_{k}}=2 Ru_{k}+g_{s}^{T }(z_{k})\frac{\partial V_{L}(\delta_{k+1})}{\partial\delta_{k}} \tag{11}\]  
The optimal control law to achieve trajectory tracking is therefore formulated as follows  
\[u_{k}^{*}=\frac{1}{2}R^{-1}g_{s}^{T}(z_{k})\frac{\partial V_{L}(\delta_{k+1})} {\partial\delta_{k+1}} \tag{12}\]  
By substituting (12) in (10), the Hamilton-Jacobi-Bellman (HJB) equation is obtained. Solving this equation is a challenging task, and to address this problem, inverse optimal control based on Lyapunov stability is proposed [32].  
_Definition: Tracking Inverse Optimal Control_  
For the control law in (12) and tracking error \(\delta_{k}=z_{k}-z_{r,k}\), inverse optimal stabilization along the desired trajectory is guaranteed if:(i) (global) asymptotic stability of \(z_{k}\) along reference \(z_{r,k}\) is achieved  
(ii) \(V_{L}(\delta_{k})\) is a (radially unbounded) positive definite function and satisfies the inequality  
\[\overline{V}(\delta_{k}):V_{L}(\delta_{k+1})-V_{L}(\delta_{k})+u_{k}^{T*}R_{z}u_ {k}^{*}\leq 0 \tag{13}\]  
The inverse optimal control law for trajectory tracking is based on a knowledge of \(V_{L}(\delta_{k})\), and a Control Lyapunov Function (CLF) \(V_{L}(\delta_{k})\) is proposed to ensure that requirements (i) and (ii) are satisfied. Hence, instead of solving the HJB equation for \(V_{L}(\delta_{k})\) a CLF is proposed as follows:  
\[V_{L}(\delta_{k})=\frac{1}{2}\delta_{k}^{T}P_{L}\delta_{k},\,P_{L}=P^{T}>0 \tag{14}\]  
By substituting (14) in (12), the following control law is obtained:  
\[u_{k}^{*}=\frac{1}{2}\Big{[}R_{z}+\frac{1}{2}\delta_{s}^{T}(z_{k})P_{L}g_{s}( z_{k})\Big{]}^{-1}\,g_{s}^{T}(z_{k})P_{L}\Big{(}f(z_{k})-z_{r,k+1}\Big{)} \tag{15}\]  
where \(P_{L}\) and \(R_{Z}\) are positive definite symmetric matrices that guarantee the existence of the inverse function in (15). The proof that the control law in (15) ensures stability and optimality for (1) is given in [32].