Spam detection on digital repositories using deep neural networks: Zenodo's use case
Creators: name: Panero, Pablo
Publisher: name: Zenodo
Year: 2020
Subjects: Spam Detection, Classifier, Machine Learning, Deep Learning, Neural Networks, Digital Repository, Zenodo

description: <strong>Important note</strong>: The main file to which this work refers to is <em>0-zenodo-spam-classifier.pdf</em>. <strong>Abstract</strong> Nobody wants to get something they do not want, and that is spam. Spam content has become a big problem in our digital era, and therefore it also affects digital repositories. Hosting spam can have a big impact on a service. From the actual hardware costs of storing it, getting skewed usage statistics, including distribution of material that violates copyright, to finally and most important serving undesired advertisement to users. Zenodo is a catch-all open digital repository, and as such it is a target for spam. Zenodo's current spam classifier is not performant enough in terms of accuracy, and requires human intervention to classify between 30 and 500 entries per day. Moreover, the workflow set to both run and train the classifier is not optimal: content is not classified in real time and the model is retrained in an ad-hoc manner. This situation translates in many hours that the support team has to spend manually classifying content. In order to solve these problems this thesis proposes a deep neural network based classifier along with practical guidelines to set it up in a production environment and improve the workflow. <strong>Description</strong> This record contains the Jupyter notebooks (in <em>.ipynb</em> and <em>.html</em> formats) used for the processing, analysis and generation of neural network models for spam detection on Zenodo metadata dump. The conclusions of said work are stated below. <strong>Discussion</strong><br> <br> The work done in this master thesis studies the suitability of neural networks to tackle the spam problem in general purpose institutional repositories, specifically Zenodo. As a first step, the usage of Random Forest based on the already existing spam classifier is tested. Achieving an almost perfect score of 99.98\% accuracy for ham content, but a lower 91.90\% for spam content. Taking around 45 minutes to train, and 25 seconds to predict one third (533k) of the records. Apart from setting the base ground for accuracy and time performance for the neural networks, these classifiers were trained also using a reduced text corpus, which proved that 4000 words is enough text to obtain an accurate classification. Nonetheless, it is possible that this number can be reduced, further work on this aspect would result in smaller vectors and faster training and predicting times, especially in the neural networks. Finally, features based on Zenodo's supporters experience and literature were extracted generating a new dataset of categorical data. Another Random Forest based classifier was trained using this dataset. Although the time performance was 7.5 times faster and the accuracy on the \textit{ham} class remained the same, it decreased almost a 3\% the accuracy for spam records. Concerning the neural networks, the state of the art showed that similar methods to those used successfully in computer vision could achieve good results also for natural language processing tasks, such as text classification. Many studies managed to predict spam on SMS, Twitter and other social networks and news content with high accuracy. Particularly, the work of Jain et al., who studied the use convolutional neural networks \cite{jain2018spam}, recurrent neural networks \cite{jain2019optimizing} and a combination of both of them \cite{jain2019spam}, achieving a 99.01\% and 95.48\% accuracy on SMS and Twitter datasets respectively. The models proposed by Jain et al. were tested on balanced subset of Zenodo's data, obtaining high results (99\% precision) for the \textit{spam} class and a bit lower (93 to 95\% precision) for the \textit{ham} class. In addition, the influence of language (e.g. English vs other languages) and stopwords was tested, and proved to not have a significant effect on the prediction performance. It could be observed that the model configurations proposed by Jain et al. contain a few layers with a big amount of units or filters. Therefore, models with fewer units or filters but deeper in layers were trained. Testing also different architectures (e.g. VGG like), optimizers (e.g. Adam, Adagrad) and embedding sizes (e.g. 200, 300). As a result, 30 configurations were trained. Obtaining similar results than Jain et al. being the only difference a 1\% on the F1 Score of the \textit{spam} class. However, this network trains significantly faster, with an average of 210s per epoch against the 250s of the network proposed in the literature. Finally, the literature models and the best deeper models were tested against the whole dataset. All of them showed poor performance. While the \textit{ham} class achieved high precision and in some cases also high recall, the \textit{spam} class precision was very low, in some cases close to 0. This means, that the amount of false positives is very high. Once again this is a consequence of the imbalanced nature of the dataset and the technique used to balance it. It is suspected that not all record clusters were represented in the undersampled set. There are several potential solutions: One would be to perform a clustering of the ham records and perform a uniformly distributed sampling of those (note that it was preferred, but due to time constraints it was not possible). Another option would be to train the models with more subsets of the ham records until a suitable result is obtained upon prediction. Yet another option would be to oversample the spam class to reach a balanced 1.6M records. However, this might result in an overfit of said class since the amount of original spam records is much smaller (45 times). <strong>Conclusion</strong> Neural networks have proven to achieve good results (around 99\% accuracy and precision) on a small subset of the dataset. However, they need to be trained with more data in order to be appropriate for a production environment. This comes with the challenge of obtaining a sensible balanced dataset with all types of records represented. Therefore, the production setup should still be the Random Forest. However, certain improvements could be made such as reduction of the number of words or the treatment of those (removal of punctuation marks, etc.). <strong>Future work</strong> There are several lines of work that can get out of this work. In terms of data processing and features extraction, it would be interesting to be able to correlate the data with user data such as: How old is the user in the system? How many records has it published? How may communities does it belong to? Where do the spammers come from (via e.g. IP geolocalisation)?. Moreover, the creation time of the record would allow a time series analysis which could profile the time, and from that the location, of the spammers. Finally, the data could be enriched with checks for the veracity of external identifiers, such as ORCID, although this might be useful to the repository itself and should be done by it. Concerning the multi-language nature of the data, it would be interesting to analyze the presence of stopwords in the data itself. Answering to the question: Do spammers use more (or less) stopwords than legitimate publishers? This could become a new extracted features (e.g. number of stopwords present in the record). Regarding the suitability of the model for a production environment, as stated before the model is not ready since it gives a big amount of false positives. However, this effect could be minimized if used as part of an ensemble with a meta classifier taking the decision from the output of several classifiers. In times also getting back the correct result to train the NN model. Another alternative would be the used of a pipeline, using first the NN model and passing to the RF those who are positive; considering them as spam if both model give a result above a certain threshold (to be defined). Then again retro-feeding the model training with the correct result. Finally, external tools such as Google BERT\footnote{https://github.com/google-research/bert} could be tested on the data; and even add it to the meta ensemble or the pipeline. Note: All the tested models are in the <em>models.zip</em> file, all the datasets used in the <em>dataset.zip</em> and the results of all the interactions are available in image format in the <em>images.zip</em> file. In addition, some intermediate vectors (like the predictions of the full dataset) are available in <em>vectors.zip.</em>