# On the Privacy-Utility Trade-Off in Differentially Private Hierarchical Text Classification
## 7 Evaluation
### Drivers for Attack Performance
#### 7.2.2 Reduced Target Model Generalization
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{2}{c}{\begin{tabular}{c} _n = 41,625_, \\ **14 epochs** \\ \end{tabular} } & \begin{tabular}{c} _n = 41,625_, \\ **50 epochs** \\ \end{tabular} & \begin{tabular}{c} _n = 41,625_, \\ **100 epochs** \\ \end{tabular} & \
\begin{tabular}{c} _n = 4000_, \\ **30 epochs** \\ \end{tabular} &
\begin{tabular}{c} _n = 400,_ \\ **30 epochs** \\ \end{tabular} \\ \hline \multirow{4}{*}{\(\mathcal{D}_{target}^{train}\)} & \(L_{1}\) & 99.71\% & 99.94\% & 99.94\% & 99.94\% & 98.44\% \\ \cline{2-7}  & \(L_{2}\) & 99.20\% & 99.86\% & 99.92\% & 99.44\% & 85.16\% \\ \cline{2-7}  & \(L_{3}\) & 96.91\% & 99.74\% & 99.81\% & 96.07\% & 63.28\% \\ \cline{2-7}  & Loss & 0.18 & 0.01 & 0.01 & 0.30 & 3.15 \\ \hline \multirow{4}{*}{\(\mathcal{D}_{target}^{test}\)} & \(L_{1}\) & 97.24\% & 96.93\% & 97.06\% & 93.47\% & 84.31\% \\ \cline{2-7}  & \(L_{2}\) & 95.00\% & 94.79\% & 94.73\% & 87.89\% & 59.23\% \\ \cline{2-7}  & \(L_{3}\) & 89.32\% & 91.11\% & 91.35\% & 78.53\% & 37.76\% \\ \cline{2-7}  & Loss & 0.89 & 1.60 & 1.87 & 2.08 & 3.97 \\ \hline \(L_{3}\) Gap & 7.60\% & 8.63\% & 8.47\% & 17.54\% & 25.52\% \\ \hline Loss Ratio & 5.2 & 160 & 187 & 6.93 & 1.26 \\ \hline \(Acc_{MI}\) & 53.06\% & 53.92\% & 54.01\% & 64.62\% & 75.00\% \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Per-level accuracies and summarized loss for BestBuy Transformer network without DP. Figure 6: Loss distribution of members and non-members for BestBuy. Each boxplot is on a log scale depicting outliers (black), median (green) and mean (red) of the respective distribution. (**a**) Loss on original model after 14 epochs; (**b**) Loss on overfit model after 100 epochs. we also reduced the number of levels to one, resulting in an attack accuracy of 51.38%. This shows that an increased (decreased) number of hierarchy levels leads to higher (lower) attack performances. Interestingly, when combining the effects of changing the number of hierarchy levels and decreasing the number of training examples, we made the observation that the ratio of the number of training examples to levels in the HTC task has a direct influence on the MI attack performance. We first noted this effect for BestBuy and then validated if we can also see it for the other datasets. The results confirm our expectations and are shown in Figure 7.  
Finally, we train the hierarchical classifier from scratch, without leveraging pre-trained weights to initialize the model. We hypothesize that this classifier variation might be more vulnerable to MI attacks, since a model without pre-training might tend to memorize more information about \(\mathcal{D}_{target}^{train}\). Training the original BestBuy Transformer classifier from scratch did not converge to a useful HTC model, with only 18% accuracy on the first level. This effect can be explained by the relatively small amount of training data compared to the large corpora the Transformer model is usually pre-trained on. The issue can be overcome by replacing the BERT-Base layers with BERT-Tiny layers, since a tiny layer contains fewer weights to train. The hierarchical BERT-Tiny classifier trained from scratch yields a model with 75.17% flat _Acc_. The trained attack model for this variation converged at 55.04% accuracy, which is \(\approx\)2% higher than the attack on the original target model. This relatively small increase reveals that the use of pre-trained weights for the target models is not the reason for the relatively poor attack performance.