# On the Privacy-Utility Trade-Off in Differentially Private Hierarchical Text Classification
## 4 Quantifying Utility and Privacy in HTC
### Utility Metrics  
There are two approaches for evaluating the utility provided of the described model architectures, namely, flat and hierarchical evaluation metrics [42; 43]. To illustrate the difference between flat and hierarchical classification metrics, consider the tree hierarchy depicted in Figure 2b. Assume that the true category for a given test example \(x\) is 3.2.1 (_green_) and that two different classifiers output 3.2.2 and 3.1 as the predicted categories (_red_). When flat evaluation metrics are used, both systems are penalized equally since both predictions are counted as false negatives for the true category 3.2.1. However, the second classifier's error is more severe since its prediction is in an unrelated sub-tree of node 3, which is considered in hierarchical evaluation metrics.  
In this work, we assess the utility of HTC based on a mix of flat and hierarchical metrics: _accuracy_ and the hierarchical and lowest common ancestor (LCA) _F-measure_. We report the (flat) accuracy _Acc_ due to its wide use in machine learning. We calculate the _hierarchical and LCA F-measure_ since they are considered the state of the art in the field of HTC. The hierarchical _F-_measure \(F_{H}\) is calculated from _hierarchical precision_\(P_{H}\) and _hierarchical recall_\(R_{H}\), and it is defined as follows [42]:  
Figure 3: Architecture of target models and attack model. (**a**) Architecture of the HTC models used in the experiments. Pre-trained layers are marked _gray_; (**b**) Attack model architecture and observed attack features from an HTC target model.  
Figure 2: Visualization for obtaining and evaluating HTC predictions. (**a**) HTC predictions before postprocessing (_green_); (**b**) Two incorrect predictions (_red_) for a fictional input belonging to another category (_green_).  
\[P_{H} =\frac{\sum_{i}|Anc_{i}\cap\hat{Anc_{i}}|}{\sum_{i}|\hat{Anc_{i}}|},\] \[R_{H} =\frac{\sum_{i}|Anc_{i}\cap\hat{Anc_{i}}|}{\sum_{i}|Anc_{i}|},\] \[F_{H} =\frac{2P_{H}R_{H}}{P_{H}+R_{H}}.\]  
For a record \(i\), \(Anc_{i}\) is the set consisting of the _true_ class and all ancestors (except the root). Analogously, \(\hat{Anc_{i}}\) is the set consisting of the _predicted_ class and all ancestors (except the root). In Figure 2b, \(Anc_{i}=\{3.2.1,3.2,3\}\) for the true class 3.2.1. \(\hat{Anc_{i}}\) is \(\{3.2.2,3.2,3\}\) and \(\{3.1,3\}\) with \(R_{H}=\frac{2}{3}\) and \(R_{H}=\frac{1}{3}\), respectively.  
The LCA metrics \(P_{LCA}\), \(R_{LCA}\), \(F_{LCA}\) are differing from the previous hierarchical metrics only by not considering the nodes above the lowest common ancestor in \(Anc_{i}\), and thus voiding the overpenalization of errors for nodes with many ancestors [43]. Again, in the example shown in Figure 2b, for the prediction 3.2.2, we would have \(LCA=3.2\) with \(Anc_{i}=\{3.2.1,3.2\}\) and \(A\hat{Anc_{i}}=\{3.2.2,3.2\}\) and therefore, \(R_{LCA}=\frac{1}{2}\). For the prediction 3.1, we would have \(LCA=3\) with \(Anc_{i}=\{3.2.1,3.2,3\}\) and \(A\hat{Anc_{i}}=\{3.1,3\}\) and therefore \(R_{LCA}=\frac{1}{3}\).