# On the Privacy-Utility Trade-Off in Differentially Private Hierarchical Text Classification
## 4 Quantifying Utility and Privacy in HTC
### HTC Model Architectures  
Architectures for HTC include training a single classifier predicting classes in the flattened hierarchy (_flat_), training multiple classifiers predicting classes for a given level or node (_local_) and training a single classifier that respects the class hierarchy (_global_). We chose a global HTC approach that features a single ANN with one output layer per level \(L_{n}\) in the tree hierarchy of the given HTC task. Each output layer is a fully connected layer with a _softmax_ activation which uses the output of the last hidden layer to predict the class for the input text on \(L_{n}\). The architecture thus consists of only one model and does not ignore the class hierarchy. This yields two main advantages. First, our HTC classifier exhibits a reduced training time and therefore also a lower overall privacy cost in comparison to local HTC approaches. Secondly, the data analyst can still retrieve the prediction probabilities per level in contrast to flat HTC approaches that only provide prediction probabilities of the individual nodes [34]. Our HTC approach, however, has the disadvantage that a post-processing step is needed to obtain predictions consistent with the hierarchy, since each output layer makes predictions independently for its hierarchy level. We show an example in Figure 2a where the prediction for \(L_{2}\) does not coincide with the prediction for \(L_{1}\) and \(L_{3}\), resulting in an undefined assignment. We suggest to resolve such inconsistencies by multiplying the softmax probabilities along the path from the root to each possible node. After comparing the multiplied probabilities, we output the path with the highest probability as prediction, leading to only consistent predictions. Even though we defined the output layer architecture for our HTC beforehand, multiple options for the architecture of the input and intermediary layers exist. We consider architectures widely used in state-of-the-art text classification as the basis for formulating a DP hierarchical text classifier. In the sequel, we briefly describe the three architectures employed in our methodology. **Bag-of-Words. ** BoW models represent ANNs that ignore the word order within a text. BoW models comprise a single embedding layer in which the word vectors are added or averaged, and which is followed by one or more feed-forward layers with a softmax activation in the last layer. The BoW classifier used in this paper is based on the architecture of _fastText_[35], which achieves high accuracy and is computationally efficient due to the usage of a single feed-forward layer for classification. Each token is first embedded and the mean of all embeddings before the output layer is computed afterwards. Prior to training, the embedding layer is initialized with the widely used GloVe (v.1.2) embeddings ([[https://nlp.stanford.edu/projects/glove/](https://nlp.stanford.edu/projects/glove/)]([https://nlp.stanford.edu/projects/glove/](https://nlp.stanford.edu/projects/glove/))) that are pre-trained on the \"Wikipedia 2014\" and \"Gigaword 5\" corpora [26; 36]. For training, we use the Adam optimizer. The classifier architecture is visualized in Figure 3a. **Convolutional Neural Networks. ** A CNN contains one or more convolutional layers that convolve the input with a filter of a given width to extract and detect patterns. Although CNNs are widely used for detecting patterns in images, CNNs have also been effectively used for detecting patterns in a text [37; 38] and do not ignore the word order. In this paper, we use the original CNN classifier for text classification proposed by [PERSON] [39]. Their architecture first applies three convolutional layers to the embeddings, which are then concatenated and passed through a dropout layer to foster generalization. The architecture of the resulting HTC model is provided in Figure 3a. For training, we use the Adam optimizer and Glove embeddings.