# Identifying Mis-Configured Author Profiles on Google Scholar Using Deep Learning
## 6 Results
### Component-Wise Evaluation  
To evaluate and quantify the contribution of each component of our proposed model, we compare them with some other alternative modules.  
Paper Attention-Based Aggregator: First, we focus on the paper attention-based aggregator module. We compare the \"CW-RNN w / Attention\" with sole \"CW-RNN\" and sole \"Attention\". Then we replace the CW-RNN model with two other RNN-based methods which are commonly used, i.e., LSTM [35] and Dilated RNN [36], to obtain the sequential features. LSTM uses a specialized architecture which contains an input gate, an output gate and a forget gate to mitigate the problem of vanishing and exploding gradients. Thus, it handles both long-term and short-term characteristics. Dilated RNN utilizes a multi-layer and cell-independent architecture characterized by multi-resolution dilated recurrent skip connections to learn sequential dependencies of different scales at different layers. The results are shown in Table 3. We only compare their performance with a fully-connected neural network classifier here.  
With the assistance of model ablation, we validate the contributions of sequential features and attention mechanism. Attention mechanism can effectively select the more critical features in detecting the mis-configured author profiles. Additionally, we can find that CW-RNN outperforms LSTM and Dilated RNN in capturing the sequential features of the paper list, which indicates that a scholar may focus on several projects at the same time and the sequential features can be extracted from the papers with some intervals after sorting.  
However, our aggregator still has some deficiencies. The publication time in the list is only accurate to year which is coarse-grained. Moreover, the publication information of some papers are incomplete. These issues may account for the inaccuracy of the paper sorting. The third-party information might be able to make up these deficiencies and we leave it for the future work.  
Classifier: Then we also compare different classifiers to show the effectiveness of LightGBM classifier from the experimental perspective. The algorithms we compare here can be divided into single learning models including LR and DT, and ensemble learning models such as RF and XGBoost. The results are shown in Table 4. All the non-neural network classifiers outperform the fully-connected classifier. Our proposed model introduces the advantages of these classic models into the neural network model and build an integrated model. LightGBM achieves the best performance.  
Ablation Study: To further evaluate the contributions of social graph embedding and manually selected features, we conduct an ablation study. For the sake of convenience, we only compare their performance with a full-connected neural network classifier. In Table 5,  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Methods** & **Precision** & **Recall** & **F1-Score** \\ \hline FC & 0.9508 & 0.8513 & 0.8983 \\ LR & 0.9261 & 0.9823 & 0.9534 \\ DT & 0.9292 & 0.9845 & 0.9561 \\ RF & 0.9216 & 0.9989 & 0.9587 \\ XGBoost & 0.9330 & 0.9834 & 0.9575 \\ LightGBM & 0.9307 & 0.9934 & 0.9610 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Results of prediction using different classifiers.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Methods** & **Precision** & **Recall** & **F1-Score** \\ \hline CW-RNN & 0.9689 & 0.7544 & 0.8482 \\ Attention & 0.9711 & 0.6667 & 0.7906 \\ LSTM w / Attention & 0.9689 & 0.7572 & 0.8501 \\ Dilated RNN w / Attention & 0.9612 & 0.7387 & 0.8334 \\ CW-RNN w / Attention & 0.9508 & 0.8513 & 0.8983 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance of different aggregators.  
\"Model w/o Social\" means the neural network without the social relationship embedding and \"Model w/o Manual\" means the model without the manually selected features. We can find that the performance will be degraded if either of these components is removed.