# Two-Step Classification with SVD Preprocessing of Distributed Massive Datasets in Apache Spark
## 6 Conclusions and Future Work  
This article focuses on two topics directly related to distributed ML, namely on the performance of one- vs two-step classification in terms of precision, recall, and the \(F1\) metric as well as on the relationships of the dataset size with these metrics and with the total computation time. The two-step classification as a methodological framework is rooted in the engineering approach of dividing a complex task into simpler ones. So far the framework has yielded a number of important techniques such as the Gauss-Dantzig model selector and the EM algorithms for parameter estimation. The proposed architecture has the following general form:  
* SVD has been initially applied to the original data matrix in order to obtain a low dimensional representation with a smaller attribute set.
* The classifier has been subsequently applied to the new attribute set in order to obtain the final labellings.  
In order to test our approach, we examined two different datasets, one binary and one multiclass, and we recorded the performance of various classification algorithms in a distributed environment. Each classification method had strengths and limitations, depending on the dataset. For example, in Higgs dataset, decision tree was mediocre, but in PAMAP, it outperformed all the other classification methods.  
From this work, certain conclusions can be drawn. First and foremost, it shows the potential of Spark to apply well-known ML operations to big data seamlessly without knowing how the dataset is distributed across HDFS. Additionally, it should be noted that it is not always apparent how an ML algorithm, especially an iterative one, can be implemented in the Spark DAG paradigm. The latter demonstrates the important contribution of MLlib as it provides a number of important classification algorithms. Third, from an algorithmic perspective it shows that modular architectures based on preprocessing can be more efficient, especially in a distributed environment, and thus the computational resources given to the preprocessing step are well spent. This is especially true for the SVD, which is typically considered expensive in traditional computing systems for large datasets. Fourth, sophisticated algorithms add significantly to the performance of a system. The wallclock time required to complete classification has been used as the complexity benchmark for reasons stated in Section 4.  
As for future work, more datasets can serve as performance benchmarks of the proposed classification method. More tests will yield a better understanding of the optimal combinations of feature set size and classifier. Of course, the cluster on which we run the classification methods is another key aspect of the cloud computing in general, so checking the same or similar algorithms and dataset on different clusters may prove fundamental and reveal latent knowledge.  
Conceptualization, A.A., G.D., A.K., P.M and G.V.; methodology, A.A., G.D., A.K., P.M and G.V.; software, A.A., G.D., A.K., P.M and G.V.; validation, A.A., G.D., A.K., P.M and G.V.; formal analysis, A.A., G.D., A.K., P.M and G.V.; investigation, A.A., G.D., A.K., P.M and G.V.; resources, A.A., G.D., A.K., P.M and G.V.; data curation, A.A., G.D., A.K., P.M and G.V.; writing--original draft preparation, A.A., G.D., A.K., P.M and G.V.; writing--review and editing, A.A., G.D., A.K., P.M and G.V.; visualization, A.A., G.D., A.K., P.M and G.V.; supervision, A.A., G.D., A.K., P.M and G.V.; project administration, A.A., G.D., A.K., P.M and G.V.; funding acquisition, A.A., G.D., A.K., P.M and G.V. All authors have read and agreed to the published version of the manuscript.  
This article is part of Project 451, a long term research initiative whose primary objective is the development of novel, scalable, numerically stable, and interpretable tensor analytics.  
**Acknowledgments:** The authors gratefully acknowledge the support of NVIDIA corporation with the donation of the Titan Xp GPU.  
**Conflicts of Interest:** Authors declare no conflict of interest.