# On the Privacy-Utility Trade-Off in Differentially Private Hierarchical Text Classification
## 7 Evaluation
### Drivers for Attack Performance
#### 7.2.2 Reduced Target Model Generalization  
Next, we describe four approaches that we formulated to increase the attack model performance by reducing target model generalization in HTC. Each approach is first motivated and then evaluated based on experimental results. First, we provoke an overfitted target model by training without early stopping for a fixed number of epochs, which is chosen significantly higher than the original number of epochs obtained with early stopping. In doing so, we deliberately force the model to overfit, i.e., adapt to the few samples in \(\mathcal{D}^{train}_{target}\) instead of approximating the underlying distribution. We evaluated this approach based on the BestBuy Transformer classifier. Table 3 shows the metrics of the original model in the first column, which converged after 14 epochs. The second and third column reveal the metrics for overfit models, which are trained for 50 and 100 epochs, respectively. As expected, the overfit models achieve a smaller training loss and a higher test loss. However, surprisingly, the achieved test accuracy does not drop compared to the original model, while the training accuracy on \(L_{3}\) increases to over 99%. The corresponding attack model accuracies rise from 53.06% to 53.92% and 54.01%, respectively. This insignificant change may appear counter-intuitive given the increased loss on \(\mathcal{D}^{test}_{target}\). However, when analyzing the loss distribution of \(\mathcal{D}^{train}_{target}\) and \(\mathcal{D}^{test}_{target}\), we observe that the median losses decrease similarly as depicted in Figure 6a,b. The reason for the high average loss on \(\mathcal{D}^{test}_{target}\) is due to the high loss value of a few outliers. Therefore, the loss ratio is not a consistently good indicator for MI attack effectiveness in practice, and rather, the accuracy gap should be taken into account. Second, we reduce the number of training examples in \(\mathcal{D}^{train}_{target}\). With this adaption, the hierarchical classifier should not generalize as well as the original classifier due to two reasons. First, the training dataset is less representative of the problem domain, and second, under-represented classes contain even fewer examples. We again evaluate this approach based on the BestBuy transformer classifier, which originally contains \(n=41\),625 training examples. Training the classifier with only 10% of the training examples indeed leads to worse generalization with a maximum train-test gap of 17.54% on the third level as shown in Table 3. The trained attack model for this variation converged at 64.62% accuracy, which is a significant increase compared to the original target model. Further reducing the training data to \(n=400\) examples reduces the target model performance even more, with a maximum train-test gap of 25.52% on the third level, as evident from Table 3. For this target model with \(n=400\), we observe an attack accuracy of 75%. In conclusion, reducing the number of training examples results in a significant MI attack improvement in comparison to changing the attack model features. Third, we increase the number of hierarchy levels in the data, resulting in a more complex HTC task, which we again hypothesize to lead to worse generalization. Moreover, additional hierarchy levels lead to additional classification outputs and therefore additional attack features, which might further boost attack performance. Indeed, when increasing the number of levels from three to seven for the BestBuy BoW classifier, the corresponding MI attack accuracy rises from 56.15% to 57.07%. To ensure that this effect was caused by noise,