# Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics +
## 4 Validation Case Studies
### Learning State-Feedback Controllers with Nonlinearly Parameterized IMF-AVI Using NNs
At each iteration, DDPG1 makes one update step of the Q-NN weights in the negative direction of the gradient of the MSE w.r.t. \(\pi_{j}\) with step size \(\alpha=0.001\) and one update step of the C-NN weights in the negative direction of the gradient of the Q-NN's output w.r.t. \(\theta_{j}\) with step size \(\alpha=0.001\). While DDPG2 differs in that the Q-NN training on each minibatch of each iteration is left to the same settings used for NP-IMF-AVI training (scaled conjugate gradient for maximum 500 epochs), only one gradient descent step is used to update the C-NN weights with the same \(\alpha=0.001\). The step-sizes were selected to ensure learning convergence. It was observed that DDPG1 has the slowest convergence (convergence appears after more than 20,000 iterations) since it performs only one gradient update step per iteration, DDPG2 has faster convergence speed (convergence appears after 5000 iterations) since it allows more gradient steps for Q-NN training, while NP-IMF-AVI has the highest convergence speed (convergence appears after 10 iterations), allowing more training in terms of gradient descent steps (with scaled conjugate gradient direction) for both Q-NN and for C-NN, at each iteration. This proves that, given the high-dimensional process, it is better to use the entire dataset \(D\) for offline training, as it was done with NP-IMF-AVI. On the other hand, the best performance with DDPG1 and DDPG2 is 0.003, not as good as the best one with the more computationally demanding NP-IF-AVI (0.0017), suggesting that minimizing the Q-NN by enumerating discrete actions to calculate the C-NN targets may actually escape local minima. The total learning time to convergence with DDPG1 and DDPG2 is about the same as with NP-IMF-AVI, which is to be expected since less calculations for DDPG1 takes more iterations until convergence appears. Notice that NP-IMF-AVI does not use soft target updates for its two NNs. The additional NN controller is not mandatory and the NP-IMF-AVI can be made similar to the LP-IMF-AVI case. In this case, the minimization of the Q-function NN estimate is to be performed by enumerating the discrete set of control actions \(\Lambda\subset\Omega_{U}\) and the targets calculation for the Q-function NN will use \(\{v(\mathbf{x}_{k}^{E},\mathbf{u}_{k})+\gamma Q(x_{k+1}^{E},argmin_{\mathbf{u }\in\Lambda}Q(\mathbf{x}_{k+1}^{E},\mathbf{u},\pi_{j}),\pi_{j-1})\}\). This approach merges the controller improvement step and the Q-function improvement step. However, for real-time control implementation after NP-IMF-AVI convergence, it is more expensive to find \(\mathbf{u}_{k}^{*}=argmin_{\mathbf{u}\in\Lambda}Q^{*}(\mathbf{x}_{k}^{E}, \mathbf{u},\pi_{j})\), since it requires evaluating the Q-function NN for a number of times proportional to the number of combinations of discrete control actions. Then only slower processes can be accommodated with this implementation. Whereas in the case when a dedicated controller NN is used, after NP-IMF-AVI convergence, the optimal control \(\mathbf{u}_{k}^{*}=\mathbf{C}^{*}(\mathbf{x}_{k}^{E},\mathbf{\theta}_{j})\) is calculated at once, through a single NN evaluation. This dedicated NN controller can also be obtained (trained) as a final step after the NP-IMF-AVI has converged to the optimal Q-function \(Q^{*}(\mathbf{x}_{k}^{E},\mathbf{u},\pi_{j})\) and the targets for the controller output are calculated as \(\{\mathbf{u}_{k}^{*}=argmin_{\mathbf{u}\in\Lambda}Q^{*}(\mathbf{x}_{k}^{E}, \mathbf{u},\pi_{j})\}\). Another original solution that uses a single NN Q-function approximator was proposed in [53], such that a quadratic approximation of the NN-fitted Q-function is used to directly derive a linear state-feedback controller with each iteration.