# Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics +
## 4 Validation Case Studies
### Learning State-Feedback Controllers with Nonlinearly Parameterized IMF-AVI Using NNs
The tracking performance for the best NN controller found with the NP-IMF-AVI is shown in blue lines in Figure 7. The convergence process is depicted in Figure 9.  
A gridsearch is next performed for the NP-IMF-AVI training process, by changing the dataset size from 30,000 to 50,000 to 70,000, combined with 17, 19, and 21 discrete values used for minimizing the Q-function over the two control inputs. For the case of 50,000 data with 17 uniform discrete possible values for each control input, \(J_{test}=0.0017\) which is the same with the best performance of the LP-IMF-AVI. Notice that neither the nonlinear state-feedback controller of the NP-IMF-AVI nor the linear state-feedback controller of the LP-IMF-AVI have integral component, while the linear output feedback controller tuned by VRFT and used for exploration has integrators. Two additional approaches exist for dealing with a NP-IMF-AVI using two NNs, for each of the Q-NN approximator and for the C-NN. For example, [32] used to cascade the C-NN and the Q-NN. After training the Q-NN and producing the new weights \(\boldsymbol{\pi}_{j}\), the weights of the Q-NN are fixed and only the weights \(\boldsymbol{\theta}_{j}\) of the C-NN are trained, with all the targets equal to \(\{0\}\) for all the inputs \(\mathbf{x}_{k}^{E}\) of the cascaded NN \(Q(\mathbf{x}_{k}^{E},\mathbf{C}(\mathbf{x}_{k}^{E},\boldsymbol{\theta}_{j}), \boldsymbol{\pi}_{j})\). In this way, the C-NN is forced to minimize the Q-NN. The disadvantage is the vanishing gradient problem of the resulted cascaded network that deepens through more hidden layers, therefore only small corrections are brought to the C-NN part that is further away from the Q-NN's output. Yet another solution [14] uses, for the controller improvement step, a single/several gradient descent step/steps \(\boldsymbol{\theta}_{j}=\boldsymbol{\theta}_{j-1}-\alpha\frac{1}{N}\sum_{k=1} ^{N}\frac{dQ}{4\boldsymbol{\theta}}\frac{d\boldsymbol{\theta}}{d\boldsymbol{ \theta}}|(\boldsymbol{\theta}_{j-1}\mathbf{x}_{k}^{E})\) with each iteration of NP-IMF-AVI, with step size \(\alpha>0\) and with gradient \(\frac{1}{N}\sum_{k=1}^{N}\frac{dQ}{4\boldsymbol{\theta}}\frac{d\boldsymbol{ \theta}}{d\boldsymbol{\theta}}|(\boldsymbol{\theta}_{j-1}\mathbf{x}_{k}^{E})\) accumulated over all inputs \(\mathbf{x}_{k}^{E}\) of the cascaded NN \(Q(\mathbf{x}_{k}^{E},\mathbf{C}(\mathbf{x}_{k}^{E},\boldsymbol{\theta}_{j-1} ),\boldsymbol{\pi}_{j})\), over fixed Q-NN weights. Essentially, the two approaches described above are equivalent and the number of gradient descent steps at each iteration is user-selectable. Also, no minimization by enumerating a finite set of control actions needs to be performed in either of the two above approaches. The above two equivalent  
Figure 9: The nonlinearly parameterized IMF-AVI (NP-IMF-AVI) convergence. approaches are effectively a particular case of the Neural-Fitted Q-iteration with Continuous Actions (NFQCA) approach [14], more recently to be updated with some changes to Deep Deterministic Policy Gradient (DDPG) [52]. DDPG uses two NNs as well, for the Q-NN and for the C-NN. It was originally developed to work in online off-policy mode, hence the need to update the Q-NN and the C-NN in a faster way on a relatively small number of transition samples (called minibatch) randomly extracted from a replay buffer equivalent to the dataset \(D\), in order to break the time correlation of consecutive samples. The effectiveness of DDPG in real-time online control has yet to be proven. Two variants of _offline_ off-policy DDPG called DDPG1 and DDPG2 are run for comparisons purposes. Both use minibatches of 128 transitions from the dataset \(D\) at each training iteration. While both use soft target updates of the Q-NN weights \(\pi^{\prime}_{j}=\tau\pi_{j}+(1-\tau)\pi^{\prime}_{j-1}\) and of the C-NN weights \(\theta^{\prime}_{j}=\tau\theta_{j}+(1-\tau)\theta^{\prime}_{j-1}\), with \(\tau=0.005\). \(\pi^{\prime}_{j}\) and \(\theta^{\prime}_{j}\) are used to calculate the targets for the Q-NN training.