# AI-Based Analysis of Policies and Images for Privacy-Conscious Content Sharing
## 5 Experimental Evaluation
### Information Retrieval from Privacy Policies
#### 5.1.4 Using Vocabulary of Key Terms for Privacy Practices  
Table 3 shows the results achieved using the vocabulary of key terms for privacy practices. Overall, with respect to the use of bi-grams, this method achieves higher accuracy (0.991 vs. 0.979) and lower F1 score (0.683 vs. 0.89). As we can also see with this method, precision is larger than recall. Based on the F1 score, the best results are achieved for some identifier sub-tags, while the lowest results are obtained by _aggregate SSO_, _SSO_, and _contact_ data types.  
The accuracy achieved by this method is impressive. However, since the classes are strongly unbalanced, accuracy is not a particularly reliable metric to evaluate these results. Indeed, from the results, we can observe that the classifiers using the vocabulary of key terms are biased towards the most common class 'false'. As a consequence, they achieve high accuracy, but their results in terms of F1 score are much lower. In contrast, the results achieved using bi-grams are less biased and achieve a good F1 score.  
\begin{table}
\begin{tabular}{l c c c c} \hline
**Data Type** & **Precision** & **Recall** & **F1** & **Accuracy** \\ \hline Contact & 0.891 & 0.871 & 0.881 & 0.975 \\ Contact address book & 0.925 & 0.867 & 0.894 & 0.979 \\ Contact city & 0.973 & 0.894 & 0.93 & 0.996 \\ Contact email address & 0.932 & 0.924 & 0.928 & 0.941 \\ Contact password & 0.922 & 0.936 & 0.929 & 0.985 \\ Contact phone number & 0.959 & 0.901 & 0.927 & 0.965 \\ Contact postal address & 0.916 & 0.814 & 0.856 & 0.954 \\ Contact ZIP & 0.971 & 0.857 & 0.905 & 0.992 \\ Demographic & 0.931 & 0.92 & 0.925 & 0.987 \\ Demographic age & 0.959 & 0.917 & 0.937 & 0.984 \\ Demographic gender & 0.958 & 0.919 & 0.937 & 0.988 \\ Identifier & 0.857 & 0.626 & 0.683 & 0.973 \\ Identifier ad ID & 0.984 & 0.87 & 0.918 & 0.985 \\ Identifier cookie or similar tech & 0.967 & 0.958 & 0.962 & 0.97 \\ Identifier device ID & 0.931 & 0.885 & 0.906 & 0.958 \\ Identifier IMEI & 0.999 & 0.954 & 0.975 & 0.998 \\ Identifier IMSI & 1 & 0.893 & 0.94 & 0.999 \\ Identifier IP address & 0.978 & 0.943 & 0.96 & 0.977 \\ Identifier MAC & 0.976 & 0.917 & 0.944 & 0.993 \\ Identifier mobile carrier & 0.885 & 0.844 & 0.863 & 0.986 \\ Identifier SIM serial & 0.874 & 0.799 & 0.832 & 0.996 \\ Identifier SSID BSSID & 0.998 & 0.639 & 0.717 & 0.997 \\ Location & 0.942 & 0.923 & 0.932 & 0.953 \\ Location Bluetooth & 0.916 & 0.842 & 0.875 & 0.984 \\ Location cell tower & 0.936 & 0.795 & 0.851 & 0.983 \\ Location GPS & 0.956 & 0.867 & 0.906 & 0.981 \\ Location IP address & 0.93 & 0.731 & 0.799 & 0.977 \\ Location WiFi & 0.923 & 0.782 & 0.837 & 0.975 \\ SSO & 0.883 & 0.806 & 0.839 & 0.964 \\ Facebook SSO & 0.887 & 0.931 & 0.908 & 0.982 \\ Aggregate contact & 0.922 & 0.917 & 0.919 & 0.921 \\ Aggregate location & 0.953 & 0.927 & 0.94 & 0.957 \\ Aggregate demographic & 0.951 & 0.911 & 0.93 & 0.972 \\ Aggregate identifier & 0.944 & 0.942 & 0.942 & 0.942 \\ Aggregate SSO & 0.891 & 0.872 & 0.881 & 0.965 \\ Aggregate all & 0.939 & 0.861 & 0.89 & 0.979 \\ \hline \end{tabular}
\end{table}
Table 2: Classification performance using the bi-gram method.