## 4 Research Approach Vulnerabilities
### Weak Datasets  
We identified several studies that used weak, outdated, or unrepresentative datasets for developing and testing their model. As discussed in the previous section, these datasets validate the model and become the ultimate frame of reference for understanding the model's performance. If a researcher evaluates his/her model based on an overly simplistic and unrepresentative dataset, the validity of the model becomes questionable when it comes to applying the identity deception detection approach in a real-world scenario. This is a common practice in identity deception research. One reason is because of the lack of recent or publicly available datasets that represent diverse OSN or network traffic. Another major contributing factor to the weakening of datasets relates to pre-processing due to the desire to have two distinct classes of \"normal\" and \"malicious\" users in the dataset. Every member of the two classes should display consistent behavior that is in line with what the model's approach to detection posits these classes as. Naturally, human behavior is non-deterministic, and trying to abstract the plethora of possible human activity on OSNs into two categories of \"normal\" and \"irregular\" behavior runs the risk of eliminating much nuanced contextual information from real-world data in order to condense the domain on which the model operates. A direct consequence of this categorization is the fact that users in the dataset who display an ambiguous set of features are removed from the dataset entirely, so that their ambiguity does not interfere with the \"cleanliness\" of the dataset. This decision to remove users with ambiguous social context (in the case of a social feature model [2]) or confusing individual behaviors (in the case of an atomic feature model [8]) is effective at removing the inherent \"noise\" to any dataset at the expense of weakening the dataset by removing real-world observations from it. Developing a model against an overly sanitized dataset can weaken its external validity. Furthermore, the limitation of available datasets can often lead to studies utilizing simulated data that are based on real-world observations and assumptions. The effect of these practices is similar to data sanitization [34] with the additional risk that the assumptions about user behavior that deception detection studies make to generate datasets can also influence both internal and external validity.  
Finally, another dataset weakness we found in recent literature is the fact that all studies we referenced in this work have exclusively used a single OSN dataset; as such, the identity deception detection models may be highly customized for a specific OSN, and such models are unlikely to be able to generalize to other OSNs.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Faulty Feature Selection** & **Weak Datasets** & **Precision Bias** & **Data Selection Bias** \\ \hline \multirow{4}{*}{Social feature models} & Select features that relate to the social context of an OSN (e.g., following relationships); the ground truth comes from a seed of pre-identified malicious users [2] & Datasets tend to contain easily separable clusters of users, users with an ambiguous social context are removed from the dataset [3] & Prioritize identifying components of the graph that are homogeneous rather than seeking out moments that are more ambiguous [4] \\ \hline \multirow{4}{*}{Atomic feature models} & Selecting features relating to a user’s online behavior (e.g., clickstream); the ground truth comes from a pre-existing \"profile\" for malicious behavior [6] & Datasets tend to contain manually selected users from real-world traffic, easily separable and free of much “noise” [8] & Tend to allow a few \"hard to classify\", but malicious users in order to boost the model’s performance with regards to “hard to classify” data points [3] \\ \hline \hline \end{tabular}
\end{table}
Table 1: Primary types of models, along with their common research vulnerabilities.