## 3 Methods
### VS-GRU: Variable Sensitive GRU
#### 3.4.2 Univariate Missing Pattern Extraction  
The weight matrices in the update function of standard GRU are not diagonal matrices so that a variety of variable interaction patterns can be learned from the multivariate time series. However, given that most data are missing in clinical time series, this design combines variables with a low missing rate and variables with a high missing rate and it might be harmful for the model to extract the useful information of the real observations. It goes without saying that the real observations are more critical than the imputations. Because they are mixed together, GRU cannot distinguish them well. Although we add the mask indicator \(m\) and missing factor \(\beta\) into the update functions, after the transformation, GRU cannot recognize which input \(x_{t}^{d}\) matches its mask indicator \(m_{t}^{d}\) and missing factor \(\beta^{d}\). As a result, the information of time series \(x\) and its missing pattern \(m\) and missing factor \(\beta\) are learned separately. To fully learn the information of the real observations, we use the weight vector in the update gate, the reset gate and the candidate hidden state update formulas in GRU, and calculate the vector dot product to perform training. It is the same that we change the matrices into diagonal matrices in order to learn the information of each variable itself. Even though there are some variables completely missing because variable features are learned independently, GRU can still exploit useful information from variables with low missing rates. The update functions are as follows:  
\[z_{t}^{1}=\sigma\left(W_{z}^{1}\cdot\hat{x}_{t}+U_{z}^{1}\cdot \hat{h}_{t-1}^{1}+V_{z}^{1}\cdot m_{t}+P_{z}^{1}\cdot\beta^{1}+b_{z}^{1} \right), \tag{12}\] \[r_{t}^{1}=\sigma\left(W_{r}^{1}\cdot\hat{x}_{t}+U_{r}^{1}\cdot \hat{h}_{t-1}^{1}+V_{r}^{1}\cdot m_{t}+P_{r}^{1}\cdot\beta^{1}+b_{r}^{1}\right),\] (13) \[\bar{h}_{t}^{1}=\tanh\left(W_{h}^{1}\cdot\hat{x}_{t}+U_{h}^{1} \cdot\left(r_{1}^{1}\odot\bar{h}_{t-1}^{1}\right)+V_{h}^{1}\cdot m_{t}+P_{h}^ {1}\cdot\beta^{1}+b_{h}^{1}\right), \tag{14}\]\[h_{t}^{1}=\left(1-z_{t}^{1}\right)\odot\hat{h}_{t-1}^{1}+z_{t}^{1}\odot\tilde{h}_ {t}^{1}. \tag{15}\]  
In Equations (12)-(14), \(W^{1}\), \(U^{1}\), \(V^{1}\) and \(P^{1}\) are vectors with the same dimension as variables. \(\beta^{1}\) belonging to one variable in the same time series remains the same in all time steps. We adopt decay mechanism of GRU-D to decay input \(h_{t}\) and dynamic imputation mechanism to impute \(x_{t}\), but we use the same form in Equation (11) to calculate \(\gamma_{x}\) and \(\gamma_{h}\). Each input variable has its independent learning procedure, as shown in Figure 3. A fully connected layer with a sigmoid activation function is added at the output of the GRU layer for the classification problem. This layer can integrate all variables and outputs the probability. We call this GRU framework variable sensitive GRU (VS-GRU), which exploits variables with different missing rate independently and is sensitive to variables with low missing rates. On the other hand, we can see from Equations (12)-(14) that VS-GRU uses a weight vector rather than a weight matrix and updates itself through a vector dot product rather than matrix multiplication. Thus, another advantage of this change is that it can be computed much faster than standard GRU using much less parameters even with the addition of \(V^{1}\) and \(P^{1}\). For example, if we use a standard GRU to process multivariate time series with 10 variables and use 64 hidden units, the total number of parameters is 4190. However, VS-GRU with the decay mechanism and dynamic imputation mechanism only has 230 parameters. However, VS-GRU only relies on the last fully connected layer to perform variable integration and its simple structure may not be enough to address complex problems such as multi-task classification problems. To address this problem, we propose VS-GRU integration (VS-GRU-i), which consists of two layers of GRU.