# Framework Based on Simulation of Real-World Message Streams to Evaluate Classification Solutions
## 3 Case Study: Spam Email Detection
### Materials and Methods  
In order to perform an analysis of the features included in the evaluation framework, the Spam Email Detection case study was performed in the Spam Email Detection scenario. The case study focuses on running simulations to analyse the performance behaviour with respect to the factors involved in the generation of the email stream. The LearningAntiSpamServer classifier is initially trained on 500 spam and 500 ham mails from the SpamAssassin corpus [25] in all simulation runs. In the case study the factors considered for the simulation are Message Send Rate (MSR), Message Batch Size (MBS), Message Size (MS), Message Obfuscation (MO) and Latency (LAT). The levels for each of these factors are MSR: 1-constant or 2-random; MBS: 1-constant or 2-random; MS: 1-Without limit or 2-With lower limit; MO: 1-Without obfuscation or 2-With obfuscation; and LAT: 1-Null, 2-Intermediate or 3-Extreme. Spammer (SP), Social Network (SNP), Personal (PP), Marketing (MP) and Informational (IP) profiles are modelled for the spam detection environment; however, the proposed solution allows the modelling of other profiles for this and other message analysis contexts. These profiles form clusters in which you can group different types of emails that occur in real-world scenarios. The characterisation of the email corpus used by each profile modelled for the spam detection environment is shown in Table 1.  
In order to determine the constant values of the message send rate and message batch size factors to be used in the experiment, a small study was conducted with 30 university students. These students were asked how many emails they had received per hour during the day from sources associated with the defined profiles. In addition, the category other sources (Others) were included for those emails that were not included in the profiles. From the data, we obtained the average number of emails received per hour for each of the sources, which is shown in Table 2.  
As can be seen, social networks and news media have the highest average email traffic. Taking into account the data collected, the following values of the levels were tested in the simulation for the experiment with the highest number of emails in the shortest time:  
* Message send rate constant: 5 min. \begin{table}
\begin{tabular}{l l c c c} \hline \hline
**Profile** & **Source** & **Instances** & **Spam** & **Ham** \\ \hline Spammer & Personal email accounts & 150 & 150 & 0 \\ Social Network & Personal email accounts & 300 & 300 & 0 \\ Personal & Personal email accounts & 450 & 425 & 25 \\ Marketing & Personal email accounts & 500 & 500 & 0 \\ Informational & Personal email accounts & 500 & 240 & 260 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Characterisation of email records by profile. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **SNP** & **IP** & **MP** & **PP** & **SP** & **Others** \\ \hline Mean & 30 & 14 & 36 & 2 & 6 & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Average number of emails received per hour. * Message batch size constant: The average number of emails received per hour from each source. In the case of source Others it was not taken into account as it was not covered by the profiles. In addition, the lower limit value of the MS factor was chosen to be 10 KB. Obfuscation, which is applied to the body of the mail, models one of the behaviours of a spammer and is only applied to mails with a real spam class. The time variation in the sending of training instances aims to evaluate the quality of the response as the arrival time of the [instance, class] pair increases. In the experiment, the time variation has two possible values: higher frequency (sending training instances with a random frequency between [0:1] min) and lower frequency (sending training instances with a random frequency between [5:20] min). This variable is a fundamental aspect for semi-supervised learning solutions.