# On the Privacy-Utility Trade-Off in Differentially Private Hierarchical Text Classification
## 7 Evaluation
### Empirical Privacy and Utility  
A theoretic comparison of the CNN, BoW and Transformer models with respect to their robustness toward noise that is introduced by DP is only insightful to a limited extend, since their architectures and pre-training paradigms vary. However, in general, the bias-variance trade-off for ANNs allows us to formulate high-level expectations. Simple ANNs will likely be prone to high bias and thus underfit in comparison to larger ANNs. Thus, the BoW model will potentially perform poorer on test data than the CNN or Transformer architecture, even in the presence of pre-training [53]. In contrast, large ANNs will have high variance and thus require larger amounts of training data to generalize well. Thus, the Transformer model will likely perform poorer on small datasets. In general, the bias decreases and the variance increases with the ANN size [54]. In combination with DP, we expect high bias models such as the BoW to be less affected by the introduced noise. Additionally, Transformer models may be negatively affected by gradient explosion when using relu activation functions in combination with DP [55]. Figure 4 states the utility and Figure 5 states the privacy scores over \(\epsilon\) for the three datasets and model architectures. Furthermore, we additionally report the theoretical bound on \(Adv\) by [PERSON] et al. [32] to allow a comparison of the theoretical and the empirical MI advantage. Notably, even if two classifiers were trained with the same noise multiplier \(z\), they do not necessarily yield the same DP privacy parameter \(\epsilon\) due to differing training epochs until convergence. All corresponding \(\epsilon\) values per model and dataset were calculated for \(\delta=\frac{1}{|D_{target}^{train}|}\) per dataset and are stated in Table 2.  
\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline  & & \multicolumn{4}{c}{**BestBuy**} & \multicolumn{4}{c}{**Reuters**} & \multicolumn{4}{c}{**DBPedia**} \\ \cline{3-10}  & & **BoW** & **CNN** & **Transformer** & **BoW** & **CNN** & **Transformer** & **BoW** & **CNN** & **Transformer** \\ \hline \multirow{2}{*}{learning rate} & Orig. & 0.001 & 0.001 & 0.005 & 0.001 & 0.001 & 0.005 & 0.001 & 0.001 & 0.005 \\ \cline{2-10}  & DP & 0.01 & 0.001 & 0.015 & 0.008 & 0.001 & 0.005 & 0.016 & 0.001 & 0.01 \\ \hline \multirow{2}{*}{batch size} & Orig. & 32 & 32 & 32 & 32 & 32 & 32 & 32 & 32 & 32 \\ \cline{2-10}  & DP & 64 & 64 & 64 & 64 & 64 & 32 & 64 & 64 & 32 \\ \hline \(\mathcal{C}\) & DP & 0.19 & 1.48 & 2.07 & 0.33 & 6.28 & 12.86 & 0.03 & 0.21 & 1.6 \\ \hline \multirow{2}{*}{microbatch size} & DP & 1 & 1 & 1 & 1 & 1 & 4 & 1 & 1 & 4 \\ \hline \multirow{4}{*}{\(\epsilon\)} & \(z=0.1\) & 30,253 & 33,731 & 5902 & 2048 & 1091 & 21,597 & 4317 & 6414 & 24,741 \\ \cline{2-10}  & \(z=0.5\) & 11.5 & 11.1 & 6.58 & 4.19 & 4.11 & 4.4 & 5.1 & 6.29 & 4.88 \\ \cline{2-10}  & \(z=1  
As expected, the model utility and adversary's success consistently decrease with stronger DP parameters for all models and all datasets. Figure 4a shows that for BestBuy, the BoW model's utility is the most robust to the introduced noise, while the Transformer model's utility is most sensitive to the introduced noise. This observation becomes most evident when considering the flat accuracy _Acc_ (_blue_), and it is in line with our expectation for small datasets formulated at the beginning of this section. The hierarchical utility metrics \(F_{H}\) and \(F_{LCA}\) do not decrease as strongly as _Acc_, since they also account for partially correct predictions. Interestingly, for BestBuy, the CNN model's MI metrics in Figure 5a already reach the baseline level at \(\epsilon=33\),731 (\(z=0.1\)). The large \(\epsilon\) points out that with respect to the upper bound, a huge privacy loss is occurring (i.e., \(e^{\epsilon}\)) and the advantage should also be maximal (i.e., \(e^{\epsilon}-1\)[32]). However, the empirical membership advantage lies far below this theoretical bound.