# Hierarchical Gated Recurrent Unit with Semantic Attention for Event Prediction
## 3 Our Proposal
### Hierarchical Semantic Attention
#### 3.3.1 Encoding Layer  
The encoding layer mainly uses GRU and ALN to calculate time-based event information and prior knowledge of the current event link, respectively. It is composed of two parts: event encoder and semantic encoder. Event encoder maps the word sequence of events in the input event link to event vectors by GRU, without using manually extracted features. To obtain a more comprehensive event embedding, we use bidirectional GRU to obtain event embeddings through the structured self-attentive mechanism [24]. For an event \(E=\left\{w_{1},\cdots,w_{N}\right\}\), we encode an event as follows:  
\[\overrightarrow{h_{n}^{t}}=\overrightarrow{GRU}\left(w_{n}, \overrightarrow{h_{n-1}^{e}}\right);\overleftarrow{h_{n}^{e}}= \overleftarrow{GRU}\left(w_{n},\overleftarrow{h_{n+1}^{e}}\right) \tag{2}\]  
\[h_{n}^{e}=\left[\overrightarrow{h_{n}^{t}},\overleftarrow{h_{n}^{e}} \right], \tag{3}\]  
\[H^{e}=(h_{1}^{e},h_{2}^{e},\cdots,h_{N}^{e}) \tag{4}\]  
Figure 2: Illustration of our proposed HS-GRU model for event prediction. The encoding layer obtains the time-based event information and prior knowledge of current event link. The attention layer fuses time-based event information and prior knowledge by semantic selective attention mechanism. The prediction layer is used to generate the next event text description. \[A=\text{softmax}\left(W_{s2}\tanh\left(W_{s1}H^{e^{T}}\right)\right) \tag{5}\]  
\[e=\tanh\left(w_{f}\left(AH^{e}\right)+b_{f}\right) \tag{6}\]  
where \(\overrightarrow{GRU}\) is a function of encoding word sequences with unidirectional GRU, \(\overrightarrow{h_{n}^{t}}\in\mathbb{R}^{d_{e}}\) denotes the unidirectional hidden state, and \(\overrightarrow{h_{m,0}^{e}}=0\), \(\mathbf{h_{n}^{e}}\) is the hidden state at the nth position connected by \(\overrightarrow{h_{n}^{t}}\), \(\overleftarrow{h_{n}^{e}}\), \(A\) is a weight matrix of the structured self-attention mechanism, and \(W_{s1}\) and \(W_{s2}\) are parameter matrices. \(tanh()\) is an activation function. \(softmax()\) ensures that the sum of weights equals 1. Finally, the embedding \(e\) of the event \(E\) is obtained from the product of the weight matrix and the hidden vector by a fully connected layer whose parameters are \(w_{f}\) and \(b_{f}\). The semantic encoder mainly encodes the prior knowledge of the current event link, which is the semantic background. We extract the semantic background in event links by ALN, which provides prior knowledge for prediction. We build an ALN with historical event links, which are all event links in the training dataset. The ALN contains all semantics of historical event links, thus the network is large. However, the prior knowledge associated with the current event link is only a subgraph in ALN. Therefore, for the current event link, we extract a sub-ALN related to the current event link in ALN as the prior knowledge of event prediction. The sub-ALN contains words in the current event link, first neighbors of these words in ALN, and the edges connected to these words. However, sub-ALN is a network structure based on word association, which cannot be directly used for calculation, thus we need to encode the network as a high-dimensional vector. To encode a network, we employ Graph Neural Networks (GNN), a graph encoding technique, to obtain an embedding of sub-ALN associated with the current event link. [PERSON] et al. (2018) first proposed the basic graph neural network (GNN), extending neural network for computing graph-structured data. [PERSON] et al. (2019) introduced gated units on GNN and proposed gated GNN, which is the structure we used in our model. GNN encodes the current node by adjacent nodes, so that it automatically extracts features of graphs with rich node connections. GNN takes account of the characteristics of graphs, which is also suitable for sub-ALN. We present the learning process of node vectors in a sub-ALN.