# Process-Oriented Stream Classification Pipeline: A Literature Review
## 3 Data Sources and Benchmarking
### Benchmarking Frameworks
streamDM only incorporates different approaches to read the data stream but does not offer explicit preprocessing steps. However, it offers more available learners than Jubatus, such as various stochastic gradient descent approaches, bagging, different Hoeffding trees, and ensemble learners. Likewise, it offers a broader evaluation and monitoring functionality, such as prequential evaluation and metrics based on a contingency table. River is the recent result of the fusion of the sci-kit-multiflow and the creme Python libraries [96]. It grants access to the whole scikit-learn (see www.scikit-learn.org/stable/index.html (accessed on 5 September 2022)) library and the streaming background of the python package creme (see www.pypi.org/project/creme/ (accessed on 5 September 2022)). River includes nearly all benchmark datasets in Table 2 and various preprocessing approaches such as feature selection and normalization. Additionally, numerous classification algorithms and a broad range of monitoring and evaluation methods were incorporated, e.g., for concept drift handling. It is faster than the classical machine learning libraries PyTorch and Tensorflow if one observation at a time must be labeled, but ref. [88] claim that it is slower than MOA, since Java code is expected to run faster than Python code. Finally, we discuss the most frequently cited framework, Massive Online Analysis (MOA) [97]. It merges numerous functionalities of the previously mentioned frameworks, especially related to River and streamDM. It is based on WEKA, the Waikato Environment for Knowledge Analysis framework, a toolkit for batch machine learning algorithms [98]. The framework is written in Java and can be executed via a built-in graphical user interface (GUI) or the command line. Additionally, it holds interfaces to the statistical programming language R (see www.jwijffels.github.io/RMOA/, www.cran.r-project.org/web/packages/streamMOA/index.html (accessed on 5 September 2022)). \begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & **VFML** & **Jubatus** & **streamDM** & **River** & **MOA** \\ \hline Artificial datasets & ✓ & ✓ & ✓ & ✓ & ✓ \\ Real-world datasets & & ✓ & ✓ & ✓ & ✓ \\ Preprocessing & ✓ & ✓ & ✓ & ✓ & ✓ \\ Data Segmentation & ✓ & ✓ & ✓ & ✓ & ✓ \\ Labeling & & & & ✓ & ✓ \\ Trees & ✓ & ✓ & ✓ & ✓ & ✓ \\ Neural networks & ✓ & ✓ & ✓ & ✓ & ✓ \\ Neighborhood-based & & & ✓ & ✓ & ✓ \\ Frequency-based & ✓ & ✓ & ✓ & ✓ & ✓ \\ Rule-based & ✓ & ✓ & ✓ & ✓ & ✓ \\ SVM & & & ✓ & ✓ & ✓ \\ Ensemble & & ✓ & ✓ & ✓ & ✓ \\ Concept drift detection & & & ✓ & ✓ & ✓ \\ Update Mechanisms & & & & ✓ & ✓ \\ Evaluation & ✓ & ✓ & ✓ & ✓ & ✓ \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Overview of relevant stream classification frameworks. MOA was initially developed to facilitate stream classification performance measurement to manage the stream algorithm's speed, memory usage, and accuracy. Thus, the maximum amount of available memory space in MOA is fixed, while the other two dimensions can be adjusted to the user's needs. Currently, its limits are handling up to ten different class labels. It includes many datasets we identified as relevant benchmark datasets in Table 2 and some from Table 3. It holds many preprocessing approaches such as feature selection and discretization. MOA includes many stream classification-related algorithms, such as decision trees and Naive Bayes classifiers, as well as several evaluation methods such as the prequential holdout. Advanced application scenarios include extensions such as a Tweet-reader to collect Twitter tweets directly, a framework for sentiment analysis, and data reduction techniques on streams without drift. Additionally, it contains all the functionalities of the Scalable Advanced Massive Online Analysis (SAMOA) framework, which is no longer maintained but has been integrated into MOA. SAMOA was an Apache project to create a new platform that performs stream mining in a distributed environment. Lastly, different applications and platforms, such as OpenML [32], provide easy access to MOA.