## 3 The OPTCON Algorithm
### Computational Details for the AL Procedure  
This subsection includes the technical computations for the AL procedure of the OPTCON3 algorithm. The basic tasks are applying [PERSON]'s principle of optimality, linearization of the nonlinear system in perturbation form, minimizing the objective function, and inserting the obtained feedback rule for controls back into the objective function. In this way the (temporary) components of the optimal objective function can be derived. After that, due to the stochastic nature of the parameters (\(\theta\)), all the components have to be adjusted to the extended system \(\left[ \begin{array}{c}x_{t}\\........\\ \theta\end{array} \right]\). Start with [PERSON]'s principle of optimality:  
\[J_{T-t}^{*}=\min_{u_{t}}E\{L_{t}(x_{t},u_{t})+J_{T-(t+1)}^{*}(x_{t})\}. \tag{19}\]  
The last term of (19) is  
\[J_{T-(t+1)}^{*}(x_{t})=\min_{u_{t+1}}E\Bigg{\{}...\min_{u_{T-1}}E\{\sum_{j=t+1 }^{T}L_{j}(x_{j},u_{j})/P^{T-1}\}.../P^{t+1}\}\Bigg{\}},\]  
where \(P^{t}=(\hat{x}_{t/t},\Sigma_{t/t})\). For \(\sum_{j=t+1}^{T}L_{j}(x_{j},u_{j})\) use the (Taylor) series expansion:  
\[ \begin{array}{c}\sum_{j=t+1}^{T}L_{j}(x_{j},u_{j})=\sum_{j=t+1}^{T}L_{j}(x_ {oj},u_{oj})+\sum_{j=t+1}^{T}(L_{jx}\delta x_{j}+\frac{1}{2}\delta\dot{x}_{j} ^{\prime}L_{jxx}\delta x_{j}\\ +\delta x_{j}^{\prime}L_{jxu}\delta u_{j}+L_{ju}\delta u_{j}+\frac{1}{2}\delta u _{j}^{\prime}L_{juu}\delta u_{j}),\end{array} \]  
where \(L_{jx}\), \(L_{ju}\) are the gradients and \(L_{jxx}\), \(L_{jxu}\), and \(L_{juu}\) are the Hessian matrices; \(\delta x_{j}=x_{j}-x_{oj}\), \(\delta u_{j}=u_{j}-u_{oj}\), and \((x_{oj},u_{oj})\) is a nominal path. Thus,  
\[J_{T-(t+1)}^{*}(x_{t})=J_{o,T-(t+1)}^{*}(x_{t})+\Delta J_{T-(t+1)}^{*}(x_{t}) \tag{20}\]  
and  
\[ \begin{array}{c}\Delta J_{T-(t+1)}^{*}(x_{t})=\min_{u_{t+1}}E\{...\min_{u_ {T-1}}E\{\sum_{j=t+1}^{T}(L_{jx}\delta x_{j}+\frac{1}{2}\delta x_{j}^{\prime} L_{jxx}\delta x_{j}\\ +\delta x_{j}^{\prime}L_{jxu}\delta u_{j}+L_{ju}\delta u_{j}+\frac{1}{2}\delta u _{j}^{\prime}L_{juu}\delta u_{j}/P^{T-1}\}.../P^{t+1})\}.\end{array}  \tag{21}\]  
Next, apply the Taylor expansion to the nonlinear system, i.e., linearize the system function \(f\) in (2) around the reference values:  
\[x_{t}\approx f_{t}(\hat{x}_{t-1/t-1},u_{t}^{*},x_{t})+F_{x_{t-1}}(x_{t-1}-\hat {x}_{t-1/t-1})+F_{u}(u_{t}-u_{t}^{*})+F_{x_{t}}(x_{t}-\hat{x}_{t/t})+\varepsilon _{t}.\]  
This can be rewritten in perturbation form:  
\[\delta x_{t}=F_{x_{t-1}}\delta x_{t-1}+F_{u}\delta u_{t}+F_{x_{t}}\delta x_{t} +\varepsilon_{t}\]  
After transformation  
\[\delta x_{t}=(I-F_{x_{t}})^{-1}F_{x_{t-1}}\delta x_{t-1}+(I-F_{x_{t}})^{-1}F_{ u_{t}}\delta u_{t}+(I-F_{x_{t}})^{-1}\varepsilon_{t}. \tag{22}\]  
Thus, (21) and (22) constitute the problem in perturbation form. Next, we have to prove that the term (21) can be expressed in the quadratic form as a function of \(\delta x_{t}\) only. Assume  
\[\Delta J^{*}_{T-(t+1)}(x_{t})=g_{t+1}+E\{h^{x}_{t+1}\delta x_{t}+\frac{1}{2} \delta x^{{}^{\prime}}_{t}H_{t+1}\delta x_{t}/P^{t+1}\} \tag{23}\]  
and apply the method of induction. By doing so the parameters \(g\), \(h^{x}\) and \(H\) are determined. The next step is to prove that rule (23) is also true for \(\Delta J^{*}_{T-j}(x_{t})\)\(\forall j\).