# Email Campaign Evaluation Based on User and Mail Server Response
## 3 Data Sets
### Campaigns  
In the case of email campaigns, the statistics shown in Table 1 are available for each minute of sending. From the point of view of machine learning, we treat each minute as a separate object. For the presented solution, each object is described by 24 conditional attributes:  
* Eight statistics for the current minute (as shown in Table 1);
* Eight statistics for the previous minute (for the first minute copy of the same values was used);
* Eight statistics for historical campaigns of the same client, e.g., the average number of soft bounced per minute for previous campaigns.  
For the first two groups, the values were divided by the number of subscribers for the current campaign. For the last group, the values were divided by the total number of subscribers for all previous campaigns.  
Due to the huge volume of the data (millions of records), it was necessary to develop an automated method of labelling. The starting point for this procedure was the attribute that describes the total number of bad events for a given campaign defined as the sum of the soft bounced count, hard bounced count, resigned count, and complaint count (divided by the number of subscribers). Data were divided into two categories, good and bad. As a result of the experiments, five data-labelling methods were proposed:  
* label1--The campaign should be stopped (label bad) if the number of bad events exceeds 5%.
* label2--The campaign should be stopped if the hard bounced count exceeds 10% or the resign count exceeds 1.7% and the unique click count does not exceed 1.5%.
* label3--The campaign should be stopped if the standardised value (\(z\)-score) of bad events is greater than 2.
* label4--The campaign should be stopped if the unique open count does not exceed 2%.
* label5--A method based on examples of bad clients identified by FreshMail.  
Let us focus on the last method (label5). The company provided identifiers of clients who have sent bad campaigns in the past. A preliminary analysis of the campaigns sent by the clients showed that not all of these campaigns could be considered bad. To identify bad campaigns, a clustering of this subset was performed. Using the elbow method, it was determined that a division into eight clusters is optimal. The centres of these clusters are shown in Table 2.  
Based on the analysis of the centres, clusters 0, 2, 3, and 6 are considered to be clusters containing correct campaigns (due to the high values of opens and clicks), while clusters 1, 4, 5, and 7 are considered to be clusters containing campaigns that should be stopped (due to the low values of opens and clicks or the high values of hard bounces). All campaigns were labelled according to which of the designated centres they were closest to.  
\begin{table}
\begin{tabular}{c c} \hline
**Attribute (Metric)** & **Description** \\ \hline opens count & number of opens \\ unique opens count & number of unique opens \\ clicks count & number of clicks (at least one link) \\ unique clicks count & number of unique clicks \\ soft bounced count & number of soft bounced \\ hard bounced count & number of hard bounced \\ resigned count & number of resigns \\ complaint count & number of complaints \\ \hline \end{tabular}
\end{table}
Table 1: Technical attributes (metrics) collected by ESP servers.  
Based on an analysis of the labelling results of methods 1 through 5, it was indicated that it would be optimal to use the superior method (label6), which takes advantage of those previously used. Method 6 is based on three rules:  
\begin{tabular}{l l l l l l l l}
1. & If label3 = 1, then label6 = 1; & & & & & \\
2. & If label3 = 0, and label1 = label2 = label4 = label5 = 1, then label6 = 1; & & & \\
3. & Otherwise, label6 = 0. & & & & & \\ \end{tabular}