## 3 The OPTCON Algorithm
### The OPTCON3 Algorithm
The OPTCON3 algorithm basically uses the technique presented in [13; 15] but is augmented by approximating, in each step, the nonlinear system in a series of linear systems (iterative procedure as described in Section 3.1). The following steps (I-IV) of the OPTCON3 algorithm describe how to obtain an approximately optimal dual control solution to a stochastic problem. The algorithm needs the following input values (A legitimate question would be how to obtain the required (tentative) inputs. It obviously depends on the research question. In the case of some numerical experiments, one can resort to Monte Carlo simulations. In the case of a real-world application, one can estimate system parameters by an appropriate method or simply guess some values):  
\(f\) system function  
\(x_{0}=\overset{\circ}{x}_{0}\) initial values of state variables  
\((\overset{\circ}{u}_{t})_{t=1}^{T}\) tentative path of control variables  
\(\overset{\circ}{\theta}_{0}=\theta\) expected values of system parameters  
\(\overset{\circ}{\Sigma}_{0}^{\theta}=\Sigma^{\theta\theta}\) covariance matrix of system parameters  
\(\overset{\circ}{\Sigma}^{\alpha}\) covariance matrix of system noise  
\((\overset{\circ}{t}_{t})_{t=1}^{T}\) system noises  
\((\overset{\circ}{t}_{t})_{t=1}^{T}\) path of exogenous variables  
\((\overset{\circ}{t}_{t})_{t=1}^{T}\) target path for state variables  
\((\overset{\circ}{t}_{t})_{t=1}^{T}\) target path for control variables  
\(W^{\pi\pi}\), \(W^{\mu\pi}\) weighting matrices of objective function  
\(w_{t}^{u}\), \(w_{t}^{x}\) weighting vectors of objective function  
\(\alpha\) discount factor of objective function. As a result of the algorithm, an approximately optimal solution \((x_{t}^{*\pi})_{t=1}^{T}\), \((u_{t}^{*})_{t=1}^{T}\) and the corresponding value of the objective function \(J^{*}\) are obtained. **Step I:** Do the following search steps [1], [2] and [3] for each \(S=1,...,T\):  
**Step I-[1]:** Find the OL solution for the subproblem \((S,...,T)\): apply the procedure already used in OPTCON1. Fix \((x_{S}^{*},u_{S}^{*})\). **Step I-[2]:** Run a grid search of size \(\Pi\) around \((x_{S}^{*},u_{S}^{*})\), i.e., perform steps (A)-(G) for each \(\pi=1,...,\Pi\):  
**Step I-2A:** Find the OL solution \((x_{t}^{*\pi},u_{t}^{*\pi})_{S+1}^{T}\) for the subproblem \((S+1,...,T)\). **Step I-[2]:** The nonlinearity loop is run until the stop criterion is fulfilled when the difference between the values of the current and the previous iteration is smaller than a pre-specified number or the maximum number of iterations is achieved). As a result the approximately optimal solution \((x_{t}^{*\pi},u_{t}^{*\pi})_{S+1}^{T}\) has been found. Then go to the next step I-2B. Notes:After several runs of the nonlinearity loop, only the solution \((x_{S+1}^{s\pi},u_{S+1}^{s\pi})\) for the time period \(S+1\) will be taken as the optimal (nominal) solution. The calculations of the pairs \((x_{t^{\prime}}^{s\pi},u_{t^{\prime}}^{s\pi})\) for other periods \((t^{\prime}>S+1)\) have to be done again, taking into account the re-estimated parameters for all periods. After linearization the parameter matrices for the linearized system are obtained: \(A_{t}=(I-F_{x_{t}}^{x})^{-1}F_{x_{t-1}}^{x}\) and \(B_{t}=(I-F_{x_{t}}^{x})^{-1}F_{u_{t^{\prime}}}^{x}\), where \(F_{x_{t^{\prime}}}^{x}\), \(F_{x_{t-1}}^{x}\), and \(F_{u_{t}}^{x}\) are the derivatives of the system function \(f\) with respect to \(x_{t}\), \(x_{t-1}\), and \(u_{t}\) respectively. **Step I-2B:** Do a backward recursion to obtain auxiliary matrices for calculating components of the objective function.