# Integrating PubMed Label Hierarchy Knowledge into a Complex Hierarchical Deep Neural Network +
## 3 Methodology
### Hierarchical Deep Neural Network  
The main purpose of the proposed methodology is the automatic creation of a DNN topology whose structure and interconnections reproduce the label hierarchy. To this end, a recursive algorithm is defined. This algorithm iterates basic neural network blocks, connecting them to each other following the label hierarchy, creating, as a result, the schema depicted in Figure 1. As shown in the figure, the nodes of the HDNN can be classified into three main groups:* _Preprocessing node:_ Where textual data in the input are converted into a word-embeddings-based representation (the green ones in the upper left part of the figure);
* _Internal node_: A DNN basic block whose inputs are word-embeddings-based and the classification result is obtained as the output of its parent node. In the same way, its output is connected in input to the corresponding children node;
* _Root Node_: A special internal node that corresponds to the root of the hierarchy. It differs from the other internal nodes because its input is directly and only connected to the output of the preprocessing node. The preprocessing nodes create an encoded-word representation, exploiting static or dynamic word embedding models [2; 33], exploiting in both cases models pretrained on biomedical document collections, to improve the final results [18; 34]. The word vectors that represent the words of a document in input are extracted and then concatenated and convoluted in the block named _Initial Feature Processing_, producing a set of features extracted from the text representation, and sending it to each node of the HDNNs. Figure 2a,b shows the details of the latter two aforementioned node types. These nodes have two main purposes: (i) specializing the features extracted for the corresponding level of labels in the hierarchy tree and (ii) predicting the classes and providing this information to their children nodes, in addition to the final complete prediction. The internal blocks of these nodes include a CNN layer, whose inputs are the results of the preprocessing block, followed by max-pooling and flattened layers; only in the case of internal nodes, the flattened output is concatenated with the output coming from its father node. Finally, the last layer is a dense fully connected layer devoted to label classification. As mentioned, the prediction is also sent to the children of that node in the topology. In summary, the nodes of one level specialize and extract the features for the same level of the hierarchy and provide as output the classifications, which are used as input by their corresponding children nodes. Figure 1: Schema of the internal architecture of an HDNN: The green blocks represent the mapping of the text into its own encoded version, provided to the root and all internal nodes (red arrows). The black arrows represent the outputs of each node, connected to the internal nodes of the next hierarchy level. The outputs of these nodes are also used to build the final classification output classes (depicted as colored arrows). The only structural difference between the root node and the internal nodes is that the latter concatenates the output of other directly connected internal nodes with the output of their own convolution layer. Furthermore, the convolution layer of the root node has a higher number of filters in order to capture more information at the top level of the topology. The topology of the HDNN can be fully automatically generated by iterating the aforementioned basic blocks, reproducing the same exact structure of the hierarchical graph of the labels. To this end, we specifically defined a recursive algorithm, whose details are shown in the next pseudocode in Algorithm 1.