# Integrating PubMed Label Hierarchy Knowledge into a Complex Hierarchical Deep Neural Network +
## 4 Experimental Assessment and Discussion
### Results and Discussion  
In order to investigate the capability of the HDNN in incorporating the hierarchical label knowledge into the neural network, we compared it with a classic CNN architecture, such as the one described in [40], where the same CNN layers of the HDNN are implemented but organized in a flat topology. In detail, considering that we simplified the problem using at least the fifth level of the MeSH tree in our experimental assessment, as described in Section 4.1, each internal node corresponding to an element of the hierarchy is formed by a cascade of two DNN layers (CNN and dense layers) for the two preprocessing CNN layers in the _Initial Feature Processing_ block (see Figure 1). In this case, it is possible to estimate 17 layers for the deepest tree branch and 4 layers for the flat CNN architecture. Tables 4 and 5, respectively, show the flat and hierarchical measures obtained with a simple CNN architecture and the proposed HDNN approach, using static (WE) or dynamic word embeddings (BioBERT). As expected, it is possible to note a performance improvement in the case of the HDNN, in both terms of flat and hierarchical measures. Moreover, the use of dynamic BioBERT-based embedding provides further improvement compared with the classic static word embedding approach in the first layer of the neural network. In more detail, the MiF and the EBF scores obtained by HDNN are, respectively, improved by 24.7% and 30.8% in the case of static WE and by 23.7% and 35.8% using BioBERT embeddings. Moreover, the increment in the HiF score is equal to 19.8% for HDNN WE and 35.9% for HDNN BioBERT. The higher impact of the HDNN is obtained in terms of LCA-based hierarchical measures: the LCA-F improves by 58.6% when static WEs are used and by 65.2% when exploiting the BioBERT model to encode input text. In summary, the HDNN obtains improvements in overall performances in terms of all the considered measures (with the slight exception of HiP for HDNN WE, probably caused by the overestimation of the error provided by this measure in the case of nodes with many ancestors). More importantly, the results show a significant increment in the case of LCA-based hierarchical measures. The information related to this score considers the correct classification of the label hierarchy, giving a higher weight in the case of the assignment of a label that lies in the same hierarchy path. This behavior demonstrates that the HDNN is able to assign more correct labels with respect to a flat CNN architecture, but in many cases, other labels, although not originally included in the manual annotation, could be a reasonable assignment. The next examples can better clarify this result. Referring to Figure 3, if a document related to cardiac edema is tagged with the label _Heart Failure_ (the higher-level label in the hierarchy path), a flat measure will consider this result completely wrong, while the hierarchical measure will suggest that the result is not perfect but a good result. For our purposes, this information is very important because an improvement in the hierarchical measures shows that the neural network has improved the correct classification of the labels along the hierarchy path, demonstrating that some information about the hierarchy has been embedded within the HDNN. More in detail, considering as an example a paper from the dataset, the article _Myocardial Edema: A Translational View_ has been manually tagged with several MeSHs by NLM human experts (for simplicity, we focus on a single example). Among them, there is the label _Cell Enlargement_ [G07.345.249.410.500]. The flat DNN architectures were not able to correctly predict it. On the other hand, the HDNN BioBERT, although it also did not find this label, identified a set of ancestors of this MeSH, namely _Physiological Phenomena_ [G07], _Growth and Development_ [G07.345], _Growth_ [G07.345.249], and _Cell Growth Processes_ [G07.345.249.410], which is the father node of the correct label.