## 6 Results and Discussion  
As introduced above, we trained the SVM with 488 manually tagged sentences. Additionally, we also tested the model by using 366 sentences for training (75%) and the remaining 122 for testing (25%). The average F1 score among the annotation labels considered in the study was 71%. On the one hand, this accuracy is at a similar level to the best accuracy achieved by related works also conducting annotations of privacy policies (albeit with different classes) [8; 10]. On the other hand, accuracy is limited by (i) the relatively large number of tags involved in the multi-label classification, which are also not clearly separated; and (ii) the fact that the inter-annotator agreement was not very high, which indicates that the interpretation of the statements in the policies may be subjected to discrepancies even among human experts. To show the behavior of the system in a real setting, we applied the trained model to both the complete policies from which we extracted the sentences used for training (Google, Mozilla, Netflix and Twitter) and a fresh set of policies of six well-known services and companies: LinkedIn ([[https://www.linkedin.com/legal/privacy-policy](https://www.linkedin.com/legal/privacy-policy)]([https://www.linkedin.com/legal/privacy-policy](https://www.linkedin.com/legal/privacy-policy)) (accessed on 16 February 2021)), Instagram ([[https://help.instagram.com/519522125107875](https://help.instagram.com/519522125107875)]([https://help.instagram.com/519522125107875](https://help.instagram.com/519522125107875)) (accessed on 16 February 2021)), AliExpress ([[https://rule.alibaba.com/rule/detail/2034.htm](https://rule.alibaba.com/rule/detail/2034.htm)]([https://rule.alibaba.com/rule/detail/2034.htm](https://rule.alibaba.com/rule/detail/2034.htm)) (accessed on 16 February 2021)), Telegram ([[https://telegram.org/privacy](https://telegram.org/privacy)]([https://telegram.org/privacy](https://telegram.org/privacy)) (accessed on 16 February 2021)), Appl ([[https://www.apple.com/legal/privacy/en-ww/](https://www.apple.com/legal/privacy/en-ww/)]([https://www.apple.com/legal/privacy/en-ww/](https://www.apple.com/legal/privacy/en-ww/)) (accessed on 16 February 2021)) and Huawei([[https://www.huawei.com/en/privacy-policy](https://www.huawei.com/en/privacy-policy)]([https://www.huawei.com/en/privacy-policy](https://www.huawei.com/en/privacy-policy)) (accessed on 16 February 2021)). When selecting the policies, we avoided picking more than one service offered by the same company (e.g., Instagram/Facebook, YouTube/Google), because the content of their policies may overlap. Moreover, as done for the training data, the fresh set of policies was chosen trying to cover significantly different perspectives regarding privacy protection (generally speaking, Telegram and Instagram have very different insights regarding personal data and privacy). Figure 1 reports the scores resulting from the automatic multi-label annotation of each policy, both per-goal and in global. Results are sorted from best to worst by their global score. Intuitively, the ranking seems a good match with the public opinion on how these companies deal with customers' privacy [15]. To conduct a more formal evaluation, we compared our results with the relative ratings provided by Guard ([[https://useguard.com/](https://useguard.com/)]([https://useguard.com/](https://useguard.com/)) (accessed on 16 February 2021)) for the seven policies that the latter application has already rated, which are, Instagram, Twitter, AliExpress, Netflix, LinkedIn, Mozilla and Telegram. We chose this application because it is the only related work that provides ratings of policies that have been updated according to the legal framework of the GDPR, and that also considers a similar set of services. As mentioned previously, the ratings provided by Guard for each policy consist of a single mark between A+ and F, which aggregates heterogeneous criteria such as generic privacy threats and scandals starred by the company.