## 3 Performance Optimization
### Optimal Policy  
Based on the performance difference Formula (24), we have the following optimality condition. **Theorem 1** (Optimality Condition). : _A policy \(\mathbf{u}^{*}\) in \(\mathcal{C}\) is optimal if, and only if,_  
\[\mathcal{A}^{\mathbf{u}}\eta^{\mathbf{u}^{*}}+f^{\mathbf{u}}\geq 0=\mathcal{A}^{ \mathbf{u}^{*}}\eta^{\mathbf{u}^{*}}+f^{\mathbf{u}^{*}},\forall\mathbf{u}\in \mathcal{C}. \tag{25}\]  
_From (25), the optimality equation is:_  
\[\min_{\mathbf{u}\in\mathcal{C}}\{\mathcal{A}^{\mathbf{u}}\eta^{\mathbf{u}^{*} }+f^{\mathbf{u}}\}=0. \tag{26}\]  
Proof. : First, the \"if\" part follows from the performance difference Formula (24) and the Poisson Equation (21). Next, we prove the \"only if\" part: Let \(\mathbf{u}^{*}\) be an optimal policy. We need to prove that (25) holds. Suppose that this is not true. Then, there must exist one policy, denoted as \(\mathbf{u}^{\prime}\), such that (25) does not hold. That is, there must be at least one state, denoted as \(y\), such that  
\[P^{\mathbf{u}^{*}}\eta^{\mathbf{u}^{*}}(y)+f^{\mathbf{u}^{*}}(y)>P^{\mathbf{ u}^{\prime}}\eta^{\mathbf{u}^{*}}(y)+f^{\mathbf{u}^{\prime}}(y).\]Then we can create a policy \(\tilde{\mathbf{u}}\) by setting \(\tilde{\mathbf{u}}=\mathbf{u}^{\prime}\) when \(x=y\), and \(\tilde{\mathbf{u}}=\mathbf{u}^{*}\) when \(x\
eq y\). We have \(\eta^{\mathbf{u}^{*}}>\eta^{\mathbf{u}^{\prime}}\). This contradicts to the fact that \(\mathbf{u}^{*}\) is an optimal policy. Based on the optimality condition, the optimal control \(\mathbf{u}^{*}\) can be obtained by developing policy iteration algorithms. Roughly speaking, we start with any policy \(\mathbf{u}_{0}\). At the \(k\)th step, \(k=0,1,\cdots\), given a piecewise linear policy \(\mathbf{u}_{k}(x)=\tilde{\mathbf{K}}x\mathbf{1}_{x\geq 0}-\tilde{\mathbf{K}}x \mathbf{1}_{x<0}\), where \(\tilde{\mathbf{K}},\tilde{\mathbf{K}}\in\mathcal{K}\), we want to find a better policy by (26). We consider any policy \(\mathbf{u}(x)\). Setting \(h(x)=\eta^{\mathbf{u}_{k}}(x)=\tilde{\mathbf{G}}x^{2}\mathbf{1}_{x\geq 0}+ \tilde{\mathbf{G}}x^{2}\mathbf{1}_{x<0}\), by (5), (12), and (14), we have  
\[(P^{\mathbf{u}}\eta^{\mathbf{u}_{k}})(x)= (a_{1}\hat{\mathbf{G}}+a_{2}\tilde{\mathbf{G}})(A+\mathbf{B} \hat{\mathbf{K}})^{2}x^{2}\mathbf{1}_{x\geq 0}\] \[+(a_{1}\hat{\mathbf{G}}+a_{2}\tilde{\mathbf{G}})(A-\mathbf{B} \tilde{\mathbf{K}})^{2}x^{2}\mathbf{1}_{x<0}. \tag{27}\]  
where \(a_{1}\) and \(a_{2}\) satisfy Equations (15) and (16), respectively. Then, from (4) and (27), we have  
\[\mathbf{u}_{k+1}(x) =\arg\{\min_{\mathbf{u}\in\mathcal{C}}[(P^{\mathbf{u}}\eta^{ \mathbf{u}_{k}})(x)+f^{\mathbf{u}}(x)]\}\] \[=\hat{\mathbf{K}}_{k+1}x\mathbf{1}_{x\geq 0}-\bar{\mathbf{K}}_{k+1}x \mathbf{1}_{x<0},\]  
with  
\[\hat{\mathbf{K}}_{k+1} =\arg\min_{\mathbf{K}\in\mathcal{K}}[a_{1}\hat{C}^{2}\hat{ \mathbf{G}}+a_{2}\hat{C}^{2}\bar{\mathbf{G}}+Q+\mathbf{K}^{T}\mathbf{R} \mathbf{K}],\] \[\bar{\mathbf{K}}_{k+1} =\arg\min_{\mathbf{K}\in\mathcal{K}}[a_{1}\hat{C}^{2}\hat{ \mathbf{G}}+a_{2}\hat{C}^{2}\bar{\mathbf{G}}+Q+\mathbf{K}^{T}\mathbf{R} \mathbf{K}],\]  
where \(\hat{\mathcal{C}}=A+\mathbf{B}\mathbf{K}\), and \(\bar{\mathcal{C}}=A-\mathbf{B}\mathbf{K}\). It can be seen that if the policy \(\mathbf{u}_{k}(x)\) is a piecewise linear control, then we can find an improved policy \(\mathbf{u}_{k+1}(x)\), which is also piecewise linear. Moreover, if \(\hat{\mathbf{K}}_{k+1}=\hat{\mathbf{K}}\) and \(\mathbf{K}_{k+1}=\mathbf{K}\), i.e., \(\mathbf{u}_{k+1}=\mathbf{u}_{k}\), then the iteration stops. The policy \(\mathbf{u}_{k}\) satisfies the optimal condition (26) in Theorem 1, and therefore is an optimal control.