## 3 Deep Neural Networks
### Hierarchical Attention Network  
Hierarchical Attention Network (HAN) [21] is a deep neural network model used to represent documents employing the hierarchical structure of documents. Due to the varying amount of information that words or phrases carry depending on the context, two levels of attention mechanism were used--first at the word level and second at the sentence level (Figure 3). The attention mechanism additionally carries the ability to monitor which word or phrase contributed the most to a particular decision made by the model, which helps us to understand decisions made by the model. The implementation used in our experiments was based on the publicly available implementation repository [22].  
Figure 2: Distributed Bag of Words Paragraph Vector [18].  
The HAN model consists of a word encoder, a word attention layer, a sentence encoder and a sentence attention layer. Suppose a document contains \(L\) sentences denoted by \(s_{i}\), and each sentence contains \(T_{i}\) words. The \(w_{it}\) of the range \(t\) in \([1,T]\) represents the words in the \(i\)th sentence. Given a sequence of word vectors \(w_{it}\), \(t\) in \([1,T]\), we perform their encoding by the matrix \(W_{e}\), \(x_{ij}=W_{e}w_{ij}\). Using bidirectional GRU networks [23], we obtain the encoded vectors by summing the output information from both directions, thus including the context information. Latent states of the model are calculated according to Formula (8). The bidirectional GRU layer contains a forward part \(\vec{f}\), which, based on the sentence \(s_{i}\), processes words from \(w_{i1}\) to \(w_{iT}\) and a backward part \(\overleftarrow{f}\), which processes words from \(w_{iT}\) to \(w_{i1}\).  
\[\begin{split} x_{it}&=W_{e}w_{it},t\in[1,T]\\ \vec{h_{it}}&=\overleftarrow{GRU_{x_{it}}},t\in[1,T ]\\ \vec{h_{it}}&=\overleftarrow{GRU_{x_{it}}},t\in[T,1 ].\end{split} \tag{8}\]  
The encoded expressions are obtained as a concatenation of the hidden states of the network in both directions according to Formula (9).  
\[h_{it}=[\vec{h_{it}},\overleftarrow{h_{it}}]. \tag{9}\]  
Because words affect the meaning of a sentence to varying degrees, an attention mechanism was introduced to extract keywords and aggregate them to form a sentence vector. Formula (10) presents HAN attention, where \(u\) is the attention state for each word, \(\alpha\) is its weight, and \(s\) is the initial state of the encoded sentence obtained by summing the weighted states of the individual words.  
\[\begin{split} u_{it}&=\tanh(W_{w}h_{it}+b_{w})\\ \alpha_{it}&=softmax(u_{it}^{T}u_{w})\\ s_{i}&=\sum_{t=1}^{T}\alpha_{it}h_{it}.\end{split} \tag{10}\]  
Initially, we feed the word representation \(h_{it}\), obtained in the previous layer, into a 1-layer neural network to obtain its hidden state \(u_{it}\). We then measure the relevance of the  
Figure 3: Hierarchical Attention Network [21].  
words as the similarity of \(u_{it}\) to the context of the words \(u_{w}\) and obtain the weights \(\alpha_{it}\) with the function \(softmax\). We obtain the sentence representation \(s_{i}\) as a weighted sum of the word representations \(h_{it}\).  
We obtain the encoded document vector in a similar way, using bidirectional GRU networks as presented in Formula (11).  
\[\begin{array}{l}\vec{h_{i}}=\overrightarrow{GRU_{s_{i}}},i\in[1,L]\\ \overleftarrow{h_{i}}=\overleftarrow{GRU_{s_{i}}},i\in[L,1].\end{array} \tag{11}\]  
We obtain an analogously encoded document as a concatenation of hidden network states in both directions according to Formula (12)  
\[h_{i}=[\vec{h_{i}},\overleftarrow{h_{i}}]. \tag{12}\]  
Once again, we apply the attention mechanism, this time to encoded sentences (Formula (13)), where \(u_{s}\) is the document context and \(v\) is the resulting document vector. The only difference is the replacement of word representations with sentence representations.  
\[\begin{array}{l}u_{i}=\tanh(W_{s}h_{i}+b_{s})\\ \alpha_{i}=softmax(u_{i}^{T}u_{s})\\ v=\sum_{t=1}^{T}\alpha_{i}h_{i}.\end{array} \tag{13}\]