# On the Privacy-Utility Trade-Off in Differentially Private Hierarchical Text Classification
###### Abstract  
Hierarchical text classification consists of classifying text documents into a hierarchy of classes and sub-classes. Although Artificial Neural Networks have proved useful to perform this task, unfortunately, they can leak training data information to adversaries due to training data memorization. Using differential privacy during model training can mitigate leakage attacks against trained models, enabling the models to be shared safely at the cost of reduced model accuracy. This work investigates the privacy-utility trade-off in hierarchical text classification with differential privacy guarantees, and it identifies neural network architectures that offer superior trade-offs. To this end, we use a white-box membership inference attack to empirically assess the information leakage of three widely used neural network architectures. We show that large differential privacy parameters already suffice to completely mitigate membership inference attacks, thus resulting only in a moderate decrease in model utility. More specifically, for large datasets with long texts, we observed Transformer-based models to achieve an overall favorable privacy-utility trade-off, while for smaller datasets with shorter texts, convolutional neural networks are preferable.  
text classification; differential privacy; membership inference +
Footnote †: journal: Medical  
single instances of the training data by sole access to the trained model [8; 15]. A rather large gap has been observed between the high theoretical bound on information leakage under MI attacks that can be derived from DP guarantees and the empirical information leakage posed by MI attacks [16]. Consequently, choosing the anonymization strength via privacy parameter \(\varepsilon\) remains a challenging problem, since a data owner can either choose to consider the theoretical or empirical information leakage.  
Our work compares the empirical privacy-utility trade-off of multiple HTC models by quantifying privacy under MI. We hypothesize that there are preferable ANN architectures with respect to the privacy-accuracy trade-off when applying DP to HTC. Unlike previous studies on the privacy-accuracy trade-off for numerical or image data [15; 16; 17], our work focuses on textual data which requires different ANN architectures, such as Transformers [18]. The main contributions of this paper are:  
* Empirically quantifying and comparing the privacy-utility trade-off for three widely used HTC ANN architectures on three reference datasets. In particular, we consider Bag of Words (BoW), Convolutional Neural Networks (CNNs) and Transformer-based architectures.
* Connecting DP privacy guarantees to MI attack performance for HTC ANNs. In contrast to the adversary considered by DP, the MI adversary represents an ML specific threat model.
* Recommending HTC model architectures and privacy parameters for the practitioner based on the privacy-utility trade-off under DP and MI.  
This paper is structured as follows. Section 2 recalls key aspects of DP, MI attacks and HTC. Section 4 formulates our approach for modeling the privacy-utility trade-off in HTC. Section 5 introduces reference datasets and Section 6 presents the experiment setup. Experiment results results are presented in Section 7 and subsequently discussed in Section 8. Section 3 introduces related work. Conclusions are drawn in Section 9.