# Email Campaign Evaluation Based on User and Mail Server Response
###### Abstract  
The goal of an email service provider company is to send out a large number of emails to help its clients realise successful email marketing activities. Thousands of emails sent every minute need to be analysed in real time to reduce spam or phishing. The paper describes a method that uses real-time tracking of key campaign metrics such as the opens count, clicks count, hard bounces count, etc., to identify campaigns that should be stopped because they can be classified as spam or phishing. The key point of this solution is that we do not analyse email content. Nevertheless, the proposed neural networks are highly effective--the F1-score is above 0.95 for any used sample. Furthermore, the approach allows us to use the same model regardless of the language of an email. The method was developed and verified in collaboration with Freshmail, a leading provider of email marketing services in Poland. Validation of the method on real data provided by the company confirmed its high effectiveness.  
spam and phishing detection; artificial neural networks; email campaign metrics; transaction emails +
Footnote †: journal: Medical  
optical recognition of the text embedded in the graphics is required [10; 11]. Moreover, links placed inside an email are analysed to detect those that may be dangerous to the user if clicked [12; 13; 14]. We can apply a shallow links analysis [15; 16], which is largely concerned with the statistical characteristics of links, such as the number of letters, digits, words, underscores, etc., as well as a deep analysis, which allows us to see where the link leads, even if there are redirects along the way. An additional problem is the language version of the email being sent. If the client sends emails in different languages, the problem of analysing the content for spam recognition is even more complex [17; 18]. All of these elements make the analysis of thousands of emails sent every minute extremely complex and require significant hardware resources on the ESP's side. The goal of the work presented in this article was to attempt to develop a method that does not directly use the content of an email. The features used in the machine learning process relate to the technical aspects of the campaign that is being sent such as the opens count, clicks count, hard bounces count, etc. It turns out that an ESP company can very successfully using real-time tracking of key campaign metrics to identify campaigns that should be stopped. Since this method does not analyse the content of the email, it works regardless of the language of the emails sent.  
The presented solution was developed as part of the SendGuard project led by the FreshMail company. The project is founded by The National Centre for Research and Development, Poland. All computational experiments were performed on real data collected by FreshMail, and the developed solutions will be deployed in the company.  
The main contributions of the paper can be summarised as follows:  
* Based on the analysis of data collected by Freshmail, we proposed sets of features that are sufficient to analyse the technical aspects of sending campaign and transactional emails;
* We developed methods for labelling large data sets automatically;
* We defined artificial neural networks models and proved that they are highly effective (the F1-score was above 0.95 for any used sample) using real data collected by the company.  
The paper is organised as follows. In Section 2, we describe the overall schema of the data processing system in the company. Section 3 describes the data sets with which we work. Artificial Neural Networks (ANN) models and the results of their application are described in Section 4. Some conclusions are provided in Section 5.