# Two-Step Classification with SVD Preprocessing of Distributed Massive Datasets in Apache Spark
)1 Computer Engineering and Informatics Department, University of Patras, 26504 Patras, Hellas; [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (G.V. )1 Computer Engineering and Informatics Department, University of Patras, 26504 Patras, Hellas; [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (G.V. )1 Computer Engineering and Informatics Department, University of Patras, 26504 Patras, Hellas; [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (G.V. )1 Computer Engineering and Informatics Department, University of Patras, 26504 Patras, Hellas; [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (G.V. )1 Computer Engineering and Informatics Department, University of Patras, 26504 Patras, Hellas; [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (G.V. )1 Computer Engineering and Informatics Department, University of Patras, 26504 Patras, Hellas; [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (G.V. )1 Computer Engineering and Informatics Department, University of Patras, 26504 Patras, Hellas; [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (G.V. )1 Computer Engineering and Informatics Department, University of Patras, 26504 Patras, Hellas; [EMAIL_ADDRESS] (A.A.A. ); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (G.V. )1 Computer Engineering and Informatics Department, University of Patras, 26504 Patras, Hellas; [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (G.V. )1 Computer Engineering and Informatics Department, University of Patras, 26504 Patras, Hellas; [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); [EMAIL_ADDRESS] (A.A.); mvonisanos@ceid.upatras. with a strong static type system running on top of the JVM ecosystem. Moreover, PySpark, namely the Spark Python API, has remarkable success since it provides Python functionality for Spark Resilient Distributed Datasets (\(RDDs\)), the main Spark data abstraction type. Among machine learning (\(ML\)) tasks, classification stands out as one of the most computationally intensive ones. This can be attributed not only to the massive data volume driving the advanced ML models but also to the time required to fetch a specific batch with dynamically formed or even ad hoc queries involving thousands of rows. Moreover, in the big data era, even elementary operations, such as a single inner product or a hash mapping, may require additional time because of the number of operands involved and their distribution across the shared file system. In addition, the iterative training process of most classifiers results in a high number of computations. Even evaluating a loss function may take a significant portion of the total computations in some very rare cases. In order to improve performance, feature selection or transformation can take place before the actual classification. Although this preprocessing step requires time, the resulting feature set may contain considerably less attributes which may further represent the various classes in an easily separable way. In turn, this leads to a more efficient classification process in terms of both accuracy and total computation time. The primary research objective of this article is twofold. Firstly, the two-step feature selection and classification methodology is described. As a preprocessing step, the singular value decomposition (\(SVD\)) has been selected as it efficiently identifies eigenfeatures hidden in massive datasets. As stated in our previous work, learning new data features while preserving old data features can be considered as one of the most important goal of incremental learning methods [1].