## 7 Our Approach--Hierarchical Weighted Attention Network  
By analysing the fine-tuned models using the Local Interpretable Model-agnostic Explanations (LIME) algorithm [49], insights were gained into how the best methods work. Using the LIME algorithm, it was observed that a hierarchical attention network performs well with texts whose topic is not sewn into the statistical features of the text itself but is derived from the meaning of the text. However, there are cases where the HAN model assigns too much attention to certain words, as a result of which it incorrectly guesses the topic, while the statistical TFIDF approach performs correctly. To overcome the weaknesses of the hierarchical attention network and to enrich them with statistical features, we propose the Hierarchical Weighted Attention Network (HWAN). The architecture of this neural network model is based on a hierarchical attention network enriched with the statistical values of a text's features. Additionally, for each input token, the value of the statistical feature of a given word is passed as an input parameter of the model. For each word, its attention value is calculated, then, together with the statistical features (BoW, TF or TFIDF value for given words), a selected operation is performed on them (as shown in Figure 18 in the section \"Word attention\"). In the image, the statistical features are successively called \(w_{21}\), \(w_{22}\), up to \(w_{27}\), and the operation is denoted by the tensor product symbol. The result of this operation is a latent variable of a single sentence and it is passed to the document level bidirectional Long-Short Term Memory [30] layer. After that, there is the \(softmax\) layer, which calculates \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Reuters** & **Ohsumed** & **20 Newsgroups** & **BBC** & **BBCSport** \\ \hline BoW & 83.42, \(\sigma=0.0\) & **60.25**, \(\sigma=0.0\) & 71.72, \(\sigma=0.0042\) & 97.03, \(\sigma=0.0077\) & 96.88, \(\sigma=0.0127\) \\ TFIDF & 75.58, \(\sigma=0.0\) & 56.44, \(\sigma=0.0\) & **85.55**, \(\sigma=0.0008\) & **97.84**, \(\sigma=0.0037\) & **98.51**, \(\sigma=0.0117\) \\ \hline LSA & 64.12, \(\sigma=0.0017\) & 35.72, \(\sigma=0.0012\) & 71.76, \(\sigma=0.0053\) & 96.90, \(\sigma=0.0030\) & 97.83, \(\sigma=0.0117\) \\ LDA & 52.91, \(\sigma=0.0093\) & 20.07, \(\sigma=0.0036\) & 60.90, \(\sigma=0.0102\) & 93.08, \(\sigma=0.0056\) & 92.80, \(\sigma=0.0419\) \\ P-SIF & 62.61, \(\sigma=0.0016\) & 34.33, \(\sigma=0.0015\) & 63.41, \(\sigma=0.0057\) & 95.33, \(\sigma=0.0063\) & 94.70, \(\sigma=0.0267\) \\ Doc2 Vec-DM & 79.81, \(\sigma=0.0013\) & 58.66, \(\sigma=0.0021\) & 75.25, \(\sigma=0.0075\) & 96.63, \(\sigma=0.0071\) & 97.29, \(\sigma=0.0096\) \\ Doc2 Vec-DBoW & 82.65, \(\sigma=0.0006\) & 59.48, \(\sigma=0.0006\) & 82.52, \(\sigma=0.0040\) & 97.35, \(\sigma=0.0046\) & 97.96, \(\sigma=0.0096\) \\ HAN & 81.33, \(\sigma=0.0046\) & 54.39, \(\sigma=0.0029\) & **89.57**, \(\sigma=0.0066\) & **99.24**, \(\sigma=0.0046\) & **100**, \(\sigma=0.0\) \\ Longformer base & 79.90, \(\sigma=0.0042\) & 53.88, \(\sigma=0.0077\) & 81.33, \(\sigma=0.0054\) & **98.20**, \(\sigma=0.0028\) & **99.32**, \(\sigma=0.0061\) \\ \hline SOTA & 97.44 [37] & 75.86 [47] & 88.6 [48] & & 99.59 [37] \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Averaged overall classification accuracy (in percent); cross-validation for Ohsumed, BBC and BBCSport, test–train split for Reuters and 20 Newsgroups. The maximum length of input data for HAN was set at 30 utterances 300 words each. Numbers in bold denotes results better than BoW and TFIDF or the best overall result.