# Deep Data Assimilation: Integrating Deep Learning with Data Assimilation
## 3 The Deep Data Assimilation (DDA) Model  
Data assimilation (DA) methodologies improve the levels of confidence in computational predictions (i.e., improve numerical forecasted results) by incorporating observational data into a prediction model. However, the error propagation into the forecasting model is not improved by DA, so that, at each step, correction have to be based from scratch without learning from previous experience of error correction. The strongly nonlinear character of many physical processes of interest can result in the dramatic amplification of even small uncertainties in the input, so that they produce large uncertainties in the system behavior [5]. Because this instability, as many observations are assimilated as possible to the point where a strong requirement to DA is to enable real-time utilization of data to improve predictions. Therefore, it is desirable to use machine learning method in order to learn a function which accumulates the process of previous assimilation process. We use NN to model this process. The key concept is in recording each step of state correction during an assimilation period and then learn a function in order to capture this updating mechanism. The system model is then revised by composing this learned updating  
Figure 3: The Deep Data Assimilation (DDA) process. Figure 2: The Data Assimilation (DA) process. function with the current system model. Such a process continues by further learning the assimilation process with the updated system model. The resulting NN-forecasting model is then a forecasting model with an intrinsic assimilation process. We introduce a neural network \(\mathcal{G}_{\omega}\), starting by a composition with the model \(M\), as described in Figure 4. The training target is:  
\[u_{k}=\mathcal{G}_{\omega}\big{(}Mu_{k-1}\big{)}. \tag{16}\]  
We train \(\mathcal{G}_{\omega}\) by exploiting the results of DA, as described in Algorithm 2. The \(\mathcal{G}_{\omega}\) model can be the iterative form \(\mathcal{G}_{\omega,i}\circ\mathcal{G}_{\omega,i-1}\circ...\circ\mathcal{G}_{ \omega,1}\circ M\), where \(\mathcal{G}_{\omega,i}\) is defined as the network that was trained in \(i\)th iteration and \(\circ\) denotes the composition function. Let \(\mathcal{M}_{i}\) be the model that was implemented in the \(i\)th iteration, it is:  
\[\mathcal{M}_{i}=\mathcal{G}_{\omega,i}\circ\mathcal{G}_{\omega,i-1}\circ... \circ\mathcal{G}_{\omega,1}\circ M, \tag{17}\]  
from which \(\mathcal{M}_{i}=\mathcal{G}_{\omega,i}\circ\mathcal{M}_{i-1}\), and \(\mathcal{M}_{0}=M\). For iteration \(i>1\), the training set for \(\mathcal{G}_{\omega,i}\) is \(\{(u_{k},u_{k}^{DDA})\}\), which exploits the \(\{(u_{k-1}^{DDA},u_{k}^{DDA})\}\) from the last updated model \(\mathcal{M}_{i}\). In this paper, we implement a Recurrent Neural Network (RNN) based on Elman networks [41]. It is a basic RNN structure that loops through the hidden layer output as an internal variable (that corresponds to the Jordan network [42] that outputs \(u_{t+-}\) as a variable). As shown in Figure 5, the RNN is specifically expressed as  
\[ \begin{split} h_{t}&=tanh(w_{1}[u_{k},h_{t-1}]+b_{1 })\\ u_{k}^{DDA}&=w_{2}(tanh(w_{1}[u_{k},h_{t-1}]+b_{1 }))+b_{2}\end{split}  \tag{18}\]  
Figure 4: The schematic diagram of DDA. This choice to train the NN using data from DA introduces a reduction of the model's error at each time step that can be quantified as reduction of the background error covariance matrix, as proved in next theorem. **Theorem 1**. : _Let \(B_{k-1}\) and \(B_{k}\) be the error covariance matrices at step \(k-1\) and step \(k\) in Algorithm 2, respectively. We prove that_  
\[\|B_{k}\|_{\infty}\leq\|B_{k-1}\|_{\infty} \tag{20}\]  
_i.e., the error in background data is reduced at each iteration._  
Proof. : We prove the thesis by a reductio ad absurdum.