## 3 The OPTCON Algorithm
### The OPTCON2 Algorithm  
The second version of the algorithm (called OPTCON2, see [10; 23] for more details) extends the first version with a passive learning strategy (open-loop feedback (OLF)). OLF uses the idea of re-estimating the model at the end of each time period \(t=1,...,T\). For that purpose, the model builder (policy maker) observes the current situation and uses the new information, namely the current values of the state variables, to improve his/her knowledge of the system. Recall that the stochastic system includes two kinds of uncertainties, namely additive (random system errors) and multiplicative ('structural' errors in parameters). We assume that there is some true law of motion in the background which is unknown to the policy maker. The passive learning strategy deals with the 'true' parameters \(\overline{\theta}\) which generate the model. However, the policy maker does not know these true parameters \(\overline{\theta}\) and works with the 'wrong' estimated parameters \(\hat{\theta}\). The idea of the OLF strategy is to observe the outcome of the model in each time period and use this information to bring the estimated parameters \(\hat{\theta}\) closer to the true values \(\overline{\theta}\). The OLF strategy of the OPTCON2 algorithm consists of the following (rough) steps running in a forward loop. In each time period \(S\) (\(S=1,\ldots,T\)) do the following:  
* Find an (approximately) optimal open-loop solution for the remaining subproblem (for the time periods from \(S\) to \(T\)) using the OPTCON1 algorithm. * Fix the predicted solution for time period \(S\), namely \(x_{S}^{*}\) and \(u_{S}^{*}\). * Observe the realized state variables \(x_{S}^{*\ast}\) which result from the chosen control variables (\(u_{S}^{*}\)), the true law of motion with parameter vector \(\overline{\theta}\), and the realization of the random disturbances \(\varepsilon_{S}\). The main difference between \(x_{S}^{\ast\ast}\) and \(x_{S}^{*}\) is that the former is driven by the true dynamic process with parameter vector \(\overline{\theta}\) and the latter by the estimated dynamic system with \(\hat{\theta}\). * Update the parameter estimate \(\hat{\theta}\) (via the Kalman filter and using the difference between \(x_{S}^{\ast\ast}\) and \(x_{S}^{*}\)) and use it in the next iteration step \(S+1\). The last step is carried out using the idea of the Kalman filter (see, e.g., [28] or [29]). The updating procedure according to the Kalman filter consists of two distinct phases, prediction and correction. First (the prediction phase), the predicted values of the state variables \(\hat{x}_{S/S-1}\), the vector of parameters \(\hat{\theta}_{S/S-1}\), and the covariances of the parameters  
Figure 1: Flow chart of OPTCON1. are calculated using the estimates from the previous time period. Next (the correction phase), these 'a priori' values of the state variables, the vector of the parameters, and the values of the covariances are corrected using the current observations of the state variables. The phase of calculating the predicted values of \(\hat{x}_{S/S-1}\) and \(\hat{\theta}_{S/S-1}\) is integrated in the previous steps of the OPTCON2 algorithm. At the end of time period \(S\) the predicted values of \(\hat{x}_{S/S-1}=x_{S}^{*}\) are calculated and \(\hat{\theta}_{S/S-1}=\hat{\theta}_{S-1}\) is known. The correction phase is reduced to the calculation of the parameter estimates \(\hat{\theta}_{S/S}\) and the parameter variances \(\Sigma_{S/S}^{\theta\theta}\) because the 'corrected' values of the state variables \(x_{S}^{\alpha*}=f(x_{S-1}^{\alpha*},x_{S}^{\alpha*},\mu_{S}^{*},\overline{ \theta},z_{S})+\varepsilon_{S}\) are calculated in the previous step of the algorithm or simply observed. As explained before, the updating procedure is based on the difference between \(x_{S}^{*}=f(x_{S-1}^{\alpha*},x_{S}^{*},\theta,z_{S})\) and \(x_{S}^{\alpha*}\) at the end of each time period. The OPTCON2 algorithm allows us to find an approximate solution of the optimal control problem using a passive learning strategy.