# Integrating PubMed Label Hierarchy Knowledge into a Complex Hierarchical Deep Neural Network +
## 4 Experimental Assessment and Discussion
### Evaluation Metrics  
Different evaluation measures for multilabel text classification were proposed in the literature. These measures can be grouped into two classes: _flat_ and _hierarchical_[37]. The flat measures base evaluates each single result obtained for each label in terms of Precision \(P_{i}\), Recall \(R_{i}\), and F1 score \(F_{i}\) for each class \(c_{i}\), which equals  
\[P_{i}=\frac{tp_{c_{i}}}{tp_{c_{i}}+fp_{c_{i}}}; \tag{1}\]  
\[R_{i}=\frac{tp_{c_{i}}}{tp_{c_{i}}+fn_{c_{i}}} \tag{2}\]  
\[F_{i}=\frac{2\cdot P_{i}\cdot R_{i}}{P_{i}+R_{i}} \tag{3}\]  
where \(tp_{c_{i}}\), \(fp_{c_{i}}\), and \(fn_{c_{i}}\) are, respectively, the true positives, false positives, and false negatives of the class \(c_{i}\). These values are then averaged, obtaining the measures for the text classification task. The microaverage, which gives equal weight to each per-document classification result [38], is obtained by averaging over all labels each member of the fraction \begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{**Vector Size**} & **Training** & \multirow{2}{*}{**Window Size**} & **Training Set** & \multirow{2}{*}{**Vocabulary Size**} \\  & **Algorithm** & & & **Size** & \\ \hline
200 & CBOW & 5 & 10,876,004 & 1,701,632 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: BioASQ word embedding features. \begin{table}
\begin{tabular}{l c c} \hline \hline
**BERT Model** & **Training Corpus Word Count** & **Biomedical Domain Corpus Word Count** \\ \hline BERT-large-cased & \(\approx\)21.3 Billion words & \(\approx\)18 Billion words \\ \hline \hline \end{tabular}
\end{table} 
Table 3: BioBERT features. of the previous equations, respectively, obtaining the micro-Precision \(MiP\), micro-Recall \(MiR\), and micro-F1 \(MiF\), defined as  
\[MiP = \frac{\sum_{i=1}^{L}tp_{c_{i}}}{\sum_{i=1}^{L}(tp_{c_{i}}+fp_{c_{i}})} \tag{4}\] \[MiR = \frac{\sum_{i=1}^{L}tp_{c_{i}}}{\sum_{i=1}^{L}(tp_{c_{i}}+fn_{c_{i }})}\] (5) \[MiF = \frac{2\cdot MiP\cdot MiR}{MiP+MiR} \tag{6}\]  
with \(L\) equal to the total class number. To complete the flat-measure-based evaluation, Accuracy \(Acc\) must be considered, which is calculated as  
\[Acc=\frac{\sum\limits_{i=i}^{M}\frac{t_{p_{i}}+t_{p_{i}}}{t_{p_{i}}+t_{f_{i}}+ f_{f_{i}}+f_{f_{i}}}}{M} \tag{7}\]  
where \(t_{n_{i}}\) are the true negatives for the class \(c_{i}\). Another flat measure is the example-based metrics, which evaluate bipartitions calculated on the average differences in the true label and the predicted label sets over all examples of the evaluation dataset. If we have a dataset with \(M\) multilabel examples and if \(Y_{i}\), \(i=1\dots M\) is the set of true labels for each example and \(Z_{i}\) is the set of predicted labels for \(i\)th example, the example-based Precision, Recall, and F1 score, respectively, indicated as \(EBP\), \(EBR\), and \(EBF\), are defined as  
\[EBP = \frac{1}{M}\sum_{i=i}^{M}\frac{|Y_{i}\cap Z_{i}|}{|Z_{i}|} \tag{8}\] \[EBR = \frac{1}{M}\sum_{i=i}^{M}\frac{|Y_{i}\cap Z_{i}|}{|Y_{i}|}\] (9) \[EBF = \frac{1}{M}\sum_{i=i}^{M}\frac{|Y_{i}\cap Z_{i}|}{|Y_{i}|+|Z_{i}|} \tag{10}\]  
The \(EBP\) metric estimates how many predicted labels for the \(i\)th example are correct, while the \(EBR\) estimates the number of assigned labels correctly retrieved; \(EBF\) combines both measures for a global evaluation. Furthermore, when the classification problem has a hierarchically organized label set, it is also possible to adopt the hierarchical measures that consider the label hierarchy in the classification results, taking into account the full path of the exact concepts in the class hierarchy and measuring whether parts of the path have been correctly classified.