# Hierarchical Gated Recurrent Unit with Semantic Attention for Event Prediction
## 4 Experiments
### Evaluation  
There is no well-established method for event prediction, and human-based evaluation is expensive. Referring to [PERSON] et al. [29], error rate, the probability of word misclassification, was used to evaluate our model. It is defined as the proportion of the number of error words in the prediction result to the total number of words in the sentence. \[ErrorRate=\frac{1}{N_{w}}\sum_{m=1}^{M^{t}}\sum_{i=1}^{N_{w}^{e}}I\left(w_{m, i}^{e}\
eq w_{m,i}^{e}\right) \tag{20}\]  
where \(I(\cdot)\) is an indicator function; when the result of a word prediction is wrong, i.e., \(w_{m,i}^{e}\
eq w_{m,i}^{e}\), the function is equal to 1. \(w_{m,i}^{e}\) indicate the \(i\)-th word predicted in the event \(e_{m}\). \(w_{m,i}^{e}\) stands for the real \(i\)th word in \(e_{m}\). \(N_{m}^{e}\) is the number of words of \(e_{m}\). \(M^{t}\) is the number of events in the event link \(L\). \(N_{w}\) denotes the total number of words in the event link \(L\). Notice that, when predicting a word, only when the word and the position are both correct, the result is correct. A low error rate for the model indicates that the prediction model is more accurate. \begin{table}
\begin{tabular}{l r r} \hline \hline  & **International News** & **China News** \\ \hline Events & 120,879 & 259,327 \\ \hline Event links & 114,785 & 237,480 \\ \hline Words & 2521 & 3538 \\ \hline Training & 91,824 & 189,984 \\ \hline Validation & 11,480 & 23,748 \\ \hline Test & 11,481 & 23,748 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Statistics of datasets. \begin{table}
\begin{tabular}{l r r} \hline \hline  & **Batches** & **Training** \\ \hline International News & 837,800 & 71 h 43 min \\ \hline China News & 894,300 & 75 h 23 min \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Training detail. Our model was compared to following baseline methods:  
1. **N-Gram** is a language model which assumes that the occurrence of the \(N\)th word is only related to the previous \(N-1\) words. It uses probabilistic methods to reveal the statistical laws inherent in language units. Here, we used two n-gram with different smoothing techniques, Backoff N-Gram and Modified Kneser-Ney [30], to compare with our model. 2. **GRU**[10] is a variant of RNN that has a simpler structure for the long-term dependencies of RNN labeling. It can do well with sequence problems. 3. **HRED**[27] is a hierarchical encoder-decoder suggestion model that encodes the context information of the query layer and the session layer, respectively, and finally provides context-aware query suggestions for users. It is similar to the task of our model. Here, the parameter n of n-gram is 3. For neural network models, their parameters are the same. Experimental results are given in Table 3.  
In general, because international news has fewer words than China news, error rates of international news are lower than China news. Our results demonstrate that the accuracy of language models is extremely low. Error rates of two n-gram models are much higher than deep learning methods. It demonstrates that the traditional language model, which relies only on the probability of occurrence between n-words, has difficulty meeting the needs of the task. Compared with the language model, deep learning methods can better adapt to the prediction task, and their accuracy are greatly improved. For the deep learning model, the hierarchical RNN model, HRED, is more accurate than the simple single-layer GRU model in our dataset. This manifests that, in the event link-based prediction task, the hierarchical model that processes the event information and event link information hierarchically can extract the sequence information of the event better than the single-layer model. However, both models only use RNN to obtain the time-based information of events for prediction. For our model, compared with the HRED, error rates decrease by 2.82% and 2.24%. It shows that adding the prior knowledge to time-based event information can help to improve the accuracy of event prediction task.