## 3 Fully Decoupled Extended Kalman Filter
### Fully Decoupled Extended Kalman Filter
The behavior of a neural network can be represented by the following discrete-time nonlinear dynamic system:  
\[\mathbf{\theta}_{k+1}=\mathbf{\theta}_{k}+\mathbf{w}_{k}, \tag{13}\]  
\[\mathbf{y}_{k}=\mathbf{h}_{k}(\mathbf{\theta}_{k},\mathbf{u}_{k})+\mathbf{v}_{k}, \tag{14}\]  
where \(\mathbf{\theta}_{k}\) is the system state vector, which contains the synaptic weights of the neural network, \(\mathbf{u}_{k}\) is the neural network input, \(\mathbf{y}_{k}\) is the neural network output, \(\mathbf{h}_{k}(\cdot)\) is a nonlinear function of synaptic weights and input, \(\mathbf{w}_{k}\) and \(\mathbf{v}_{k}\) are additive Gaussian white noise in the process and the measurement, respectively. The process equation depicts that the state of the ideal neural network is characterized as a steady state corrupted by noise. The observation or measurement equation represents the desired response vector of the network as a nonlinear function of the input vector and the state vector, plus the measurement noise. This noise and that of the process are characterized by a zero mean with covariance specification, thus:  
\[E\Big{[}\mathbf{w}_{k}\mathbf{w}_{l}^{T}\Big{]}= \begin{cases}\mathbf{Q}_{k},& k=l\\ \mathbf{0},&k\
eq l\end{cases} , \tag{15}\]  
\[E\Big{[}\mathbf{v}_{k}\mathbf{v}_{l}^{T}\Big{]}= \begin{cases}\mathbf{R}_{k},& k=l\\ \mathbf{0},&k\
eq l\end{cases} . \tag{16}\]  
In [15], the introduction of process noise is justified as a simple and easily controlled mechanism that helps ensure that the error covariance approximation matrix can retain the necessary property of being non-negatively definite and, in this way, avoid divergence  
Figure 2: Schematic illustration of the different levels of decoupling. problems. These authors also found that, if chosen carefully, it helps speed up the training process and, more importantly, produces a better performance of the trained network, besides referring to the other three research studies that also found benefits from introducing process noise. The authors of reference [35] state that it is often advantageous to add noise to the model to force the filter to continually adjust the weights and prevent the Kalman gain from going to zero. In some training algorithms, gradient-based learning also exploits the benefits of process noise. In [21], to efficiently implement the filter, noise is artificially introduced into the process, with a diagonal matrix as covariance. Many prefer to use the identity matrix multiplied by a very small value [19], as an example, in [36] the noise covariance is \(I\cdot 10^{-6}\), where \(I\) is the identity matrix. Instead, few researchers [13; 37] do so without the introduction of noise in the process. However, one can also think of the neural model of optimal weights without noise in both the process and the measurement [38]. Regarding the noise in the output, although it is common to use constant diagonal covariance, it seems convenient to update during the training. Chapter 7 of [12] states that measurement noise in the EKF affects convergence speed and performance. The authors recommend using a diagonal covariance matrix and gradually decreasing the values as the training process progresses, being able to update based on the error covariance. A different way to update them is based on a weighted sum of the previous value and another value dependent on the magnitude of the weight update. In [13] it uses an initial value \(I\) (identity), and the magnitude of the noise covariance of the measurement is exponentially reduced through the function \(I\cdot e^{-k/50}\), with \(k\) denoting the epoch number in the course of training. In [21] is similar, but with an exponential decay factor determined based on the total number of epochs, to conclude with a very low power noise; for [21] this cancels out the noisy behavior of the filter as it approaches the minimum.