## 3 The OPTCON Algorithm
### The OPTCON3 Algorithm  
In this section we present the detailed description of the OPTCON3 algorithm. As mentioned above, the latest version of the OPTCON algorithm includes an active learning strategy. In the literature this strategy is also called closed-loop, adaptive, or dual control. The passive learning method in the OPTCON2 algorithm uses current observations to update the information about the dynamic system. The active learning strategy in OPTCON3 actively uses system perturbations to improve the optimal control performance by reducing the uncertainty in the future. The procedure for finding the closed-loop solution in the OPTCON3 algorithm is based on the linear active learning control method in [15]. Similar to the iterative structure in the OPTCON2 algorithm, the optimization runs in a forward loop (\(S=1,\dots,T\)). However, in each time period, more information about future measurements is used. To this end the objective function \(J\) (as defined in Equations (3) and (4)) is extended for stochastic components. We define in each time period \(J_{d}\) as the approximate total cost-to-go with \(T-S\) periods remaining. The approximate cost-to-go is broken down into three terms: \(J_{d}=J_{D}+J_{C}+J_{P}\). \(J_{D}\) is called deterministic term and incorporates only non-stochastic components. \(J_{C}\) is the cautionary factor that includes the stochastic elements in the current period. \(J_{P}\) is called probing term and captures the effect of dual learning on the uncertainty in the future periods. \(J_{C}\) and \(J_{P}\) constitute a separate quadratic minimization problem constrained by the nonlinear system. The original system needs to be expanded to the perturbation form \(\delta x_{t}\). The optimization problem has to be solved for the perturbed system, where \(\Delta J_{t}^{*}\) is the perturbed objective function. Using [PERSON] expansion of nonlinear system and [PERSON] optimization technique, we obtain the solution \(\Delta J_{t}^{*}\) as a quadratic function of \(\delta x_{t-1}\). The original \(J_{t}^{*}\) can be derived from \(\Delta J_{t}^{*}\). To take uncertain parameters into account, all the terms and formulas need to be adjusted to the augmented system \(\left[ \begin{array}{c}x_{t}\\........\\ \theta\end{array} \right]\). Next, a schematic structure of the OPTCON3 algorithm is presented. This structure is visualized in Figure 2.  
method is used (instead of grid search many other methods can be applied, such as gradient optimization or a more advanced heuristic approach). In the so-called \"\(\pi\)-loop\" we create a grid of possible solutions around the existing path \((x_{S}^{*},u_{S}^{*})\). In each iteration \((\pi=1,...,\Pi)\), i.e., in each grid point, the optimal control solution is obtained and evaluated using the objective function. Inside the \(\pi\)-loop (for each \(\pi\)), the following steps are carried out. An optimal open-loop solution for the subproblem from \(S+1\) to \(T\) is determined. Then the OL solution \((x_{S+1}^{\pi\pi},u_{S+1}^{\pi\pi})\) for the time period \(S+1\) is fixed. Next, after some auxiliary calculations (including, among other, Riccati matrices), the deterministic \(J_{D}\), cautionary \(J_{C}\), and probing \(J_{P}\) terms of the cost-to-go are determined in a forward loop from \(S+1\) to \(T\). Once the iteration of the \(\pi\)-loop has been completed, the total approximate objective function \(J_{d}=J_{D}+J_{C}+J_{P}\) can be obtained. When the search is completed, i.e., the approximately optimal path with \(minJ_{d}\) has been found, the obtained optimal control in time period \(S+1\) is carried out by the policy maker. As a result of these controls and the true dynamic systems the actual values of the state variables can be obtained. This new information is used by the policy maker to update and to adjust the parameter estimate \(\hat{\theta}\), whereby the Kalman filter is used. Using the updated stochastic parameters, the same active learning procedure is applied for the remaining time periods from \(S+2\) to \(T\).