# Identifying Mis-Configured Author Profiles on Google Scholar Using Deep Learning
## 6 Results
### Performance Comparison
#### 6.2.2 Performance Comparison  
The comparative results are shown in Table 2. We can find that our proposed model achieves the best performance with the assist of the NLP features of paper lists and the co-author-based social features. We use [PERSON]'s test [34] to calculate the statistical significance between our method and the baselines. The result shows that our proposed method is significantly different from each of the baseline methods (\(p\)-value \(<0.001\)). By comparing different baselines, it is easy to find that the machine learning models outperform the graph-based sybil detection method (i.e., SybilSCAR) and document classification method (i.e., HAN), which reflects that the features we extract and select are more effective to identify mis-configured author profiles. In particular, the performance of SybilSCAR is not very good, which indicates that the similarity between scholars who have co-author relationship does not play a crucial role in the detection and semi-supervised learning based on the co-author graph cannot detect mis-configuration effectively. The NLP features and the co-author-based social features can also contribute to the detection; however, the improvement is not as great as expected. We believe that there are two reasons that may explain it: Firstly, some mis-configured author profiles only contains a small number of wrongly claimed papers, which makes it difficult to detect. Secondly, interdisciplinary research is a hot spot. Some scholars may collaborate with scholars from different fields to do research on various topics. Therefore, papers in one scholar's list may be diverse, which makes it hard to judge without other references.  
In conclusion, our model synthesizes the NLP feature of each paper, co-author-based social relationship and the selected features on the author profile page to justify the reliability of the author profile, which is more comprehensive and accurate compared with the baseline methods. The paper title encoding module can effectively characterize the NLP feature of the title content. The paper attention-based aggregator can integrate the features of each paper and capture the sequential feature of the whole paper list. Node2 vec can obtain the social feature based on the co-author relationship. Finally, the tree-structure classifier is effective to predict the reliability of the author profile based on all the features. To further demonstrate the contributions of these modules, we conduct more experiments in the next subsection.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Methods** & **Precision** & **Recall** & **F1-Score** & **AUC** &
\begin{tabular}{c} \(p\)**-Value** \\ **(McNeemar’s Test)** \\ \end{tabular} \\ \hline Linear Regression & 0.9220 & 0.9912 & 0.9553 & 0.5913 & \(2.921\times 10^{-9}\) \\ Decision Tree & 0.9237 & 0.9890 & 0.9552 & 0.7664 & \(3.283\times 10^{-8}\) \\ Random Forest & 0.9313 & 0.9735 & 0.9520 & 0.7570 & \(5.702\times 10^{-7}\) \\ XGBoost & 0.9296 & 0.9768 & 0.9526 & 0.8062 & \(2.899\times 10^{-7}\) \\ LightGBM & 0.9306 & 0.9768 & 0.9532 & 0.7835 & \(1.439\times 10^{-7}\) \\ SybilSCAR & 0.4803 & 0.9946 & 0.6477 & 0.6148 & \(1.623\times 10^{-28}\) \\ HAN & 0.9227 & 0.9095 & 0.9161 & 0.6301 & \(2.896\times 10^{-4}\) \\ Our Proposed Model & 0.9307 & 0.9934 & 0.9610 & 0.8200 & / \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance comparison with baselines.