# Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics +
## 4 Validation Case Studies
### Learning State-Feedback Controllers with Nonlinearly Parameterized IMF-AVI Using NNs  
The previous LP-IMF-AVI for ORM tracking control learning scheme is next challenged by a NP-IMF-AVI implemented with NNs. In this case, two NNs are used to approximate the Q-function and the controller (the latter is sometimes avoidable, see the comments later on in this sub-section). The procedure follows the NP-IMF-AVI implementation described in [51; 24]. The same dataset of transition samples is used as was previously used for the LP-IMF-AVI. Notice that the NN-based implementation is widely used in the reinforcement learning-based approach of ADP and is generally more scalable to problems of high dimension. The controller NN (C-NN) estimate is a 10-3-2 (10 inputs because \(\mathbf{x}_{k}^{E}\in\mathbb{R}^{10}\), 3 neurons in the hidden layer, and 2 outputs corresponding to \(u_{k,1},u_{k,2}\)) with \(tanh\) activation function in the hidden layer and linear output activation. The Q-function NN (Q-NN) estimate is 12-25-1 with the same parameters as C-NN. Initial weights of both NNs are uniform random numbers with zero-mean and variance 0.3. Both NNs are to be trained using scaled conjugate gradient for a maximum of 500 epochs. The available dataset is randomly divided into training (80%) and validation data (20%). Early stopping during training is enforced after 10 increases of the training c.f. mean sum of squared errors (MSE) evaluated on the validation data. MSE is herein, for all networks, the default performance function used in training (c 2019 IEEE [12]). Figure 8: The LP-IMF-AVI parameters convergence. Figure 7: The IMF-AVI convergence on TITOAS: \(y_{k,1}^{m},y_{k,2}^{m}\), (red); \(u_{k,1},u_{k,2},y_{k,1},y_{k,2}\) for LP-IMF-AVI (black), for NP-IMF-AVI with NNs (blue), for the initial VRFT controller used for transitions collection (green) (c 2019 IEEE [12]). The NP-IMF-AVI proposed herein consists of two steps for each iteration \(j\). The first one calculates the targets for the NN \(Q(\mathbf{x}_{k}^{E},\mathbf{u}_{k},\boldsymbol{\pi}_{j})\) (having inputs \([(\mathbf{x}_{k}^{E})^{\top},(\mathbf{u}_{k})^{\top}]^{\top}\) and current iteration weights \(\boldsymbol{\pi}_{j}\)) as \(\{v(\mathbf{x}_{k}^{E},\mathbf{u}_{k})+\gamma Q(\mathbf{x}_{k+1}^{E},\mathbf{C }(\mathbf{x}_{k+1}^{E},\boldsymbol{\theta}_{j-1}),\boldsymbol{\pi}_{j-1})\}\), for all transitions in the dataset. Resulting in the trained Q-function estimator NN \(Q(\mathbf{x}_{k}^{E},\mathbf{u}_{k},\boldsymbol{\pi}_{j})\) with parameter weights \(\boldsymbol{\pi}_{j}\). The second step (the controller improvement) first calculates the targets for the controller \(\mathbf{C}(\mathbf{x}_{k}^{E},\boldsymbol{\theta}_{j})\) (with inputs \((\mathbf{x}_{k}^{E})^{\top}\)) as \(\{\mathbf{u}_{k}=argmin_{\mathbf{u}\in\Lambda}Q(\mathbf{x}_{k}^{E},\mathbf{u },\boldsymbol{\pi}_{j})\}\). Note that additional parameterization for the controller NN weights \(\boldsymbol{\theta}_{j}\) is needed. Training produces the improved controller characterized by the new weights \(\boldsymbol{\theta}_{j}\). Here, the discrete set of control actions \(\Lambda\subset\Omega_{U}\) used to minimize the Q-NN estimate for computing the controller targets is the Cartesian product of two identical sets of control actions, each containing 21 equally spaced values in [-1;1], i.e., \(\{-1,-0.9,...,0.9,1\}\). A discount \(\gamma=0.95\) will be used and each iteration of the NP-IMF-AVI produces a C-NN that is tested on the standard test scenario shown in Figure 6 by measuring the same normalized c.f. \(I_{test}\) for \(N=2000\), on the same test scenario that was used in the case of the LP-IMF-AVI. The NP-IMF-AVI is iterated 50 times and all the stabilizing controllers that are better than the VRFT multivariable controller running on the standard test scenario described in Figure 7 (in terms of smaller \(J_{test}\)) are stored. The best C-NN across 50 iterations renders \(J_{test}=0.0025\).