# Framework Based on Simulation of Real-World Message Streams to Evaluate Classification Solutions
###### Abstract
However, some proposals have presented forms of evaluation that take these aspects into account [6; 11; 12; 13; 14; 15]. Proposals for evaluating adversarial data manipulation by spammers in the message stream context have not been found. However, proposals have been identified in other contexts that attempt to measure classifier stability in the face of adversarial attacks [12; 13]. In [12], the stability of the classifier under adversarial contamination of the training data is quantified by introducing a metric to classify its robustness. In [13], frameworks for security analysis and evaluation of classification algorithms are proposed by simulating attack scenarios. Regarding the evaluation of concept drift in the message stream context, two were found, specifically in spam detection [6; 11]. In [11], the SDAI methodology is proposed, based on the measurement of classifier accuracy in four different but complementary scenarios: static, dynamic operations, adaptive capabilities and internationalisation. The static scenario evaluates the overall performance of the classifier in a controlled environment, while the dynamic operation scenario measures its behaviour under automatic updating schemes. The adaptive capabilities scenario simulates the operation of the classifier in a server context, categorising messages from multiple senders and covering different subject areas, and the internationalisation scenario evaluates the ability of the classifier to classify incoming messages in different languages, whether in standalone or server mode. In [6], a strategy very similar to the adaptive scenario of [11] is proposed, where the classifier is trained on a corpus (SpamAssassin corpus) and tested on another corpus (Ling-Spam corpus), which covers different topics than the training corpus. The main differences of the variant in [6] are the algorithms used to process and classify the messages and the corpora used for training and testing. In [6], four spam filters trained on five email datasets collected from different sources at different times are evaluated to see if the spam filters maintain their generalisation performance. Both proposals essentially try to deal with concept drift and spammer influence, ref. [11] in the dynamic scenario and both in the adaptive scenario. However, in the dynamic scenario, the following perspective is orientedtowards the action of the classifier rather than the generation of the stream. The stream is generated using cross-validation by segmenting the base corpus into 10 partitions and selecting 9 for training and 1 for testing. This procedure does not allow adequately modelling environments with intermediate and extreme latency; the behaviour of the different spammers and other message sources that may exist is limited to what is reflected in the messages of the corpus used and does not allow adapting the behaviour by adjusting parameters such as spammer strategies, message sending ratio, message batch size and other parameters that make the stream more realistic; and the configuration to evaluate different types of message processing services, classifiers or a combination of both can become complex. In addition, several frameworks have been proposed to assist users in carrying out specific aspects of the stream classification process [16]. These frameworks allow the integration of different stream classification tasks, improve interoperability and include all necessary components for algorithm development [17]. A stream classification framework may include data generators or real-world datasets for benchmarking, data processing, classifier algorithms, and testing of classification results [16]. Existing frameworks that have been used to classify text messages include Jubatus [14] and Massive Online Analysis (MOA) [15]. Jubatus, a framework that emerged from a Japanese research project [14], emphasises distributed processing and features a model-sharing architecture to support practical training and collaboration of classification models.