title: ÐÐµÑÐ¾Ð´Ð¸ÐºÐ° Ð½Ð°ÑÑÑÐ¾Ð¹ÐºÐ¸ Ð¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð² ÐºÐ¾Ð½ÑÐ¸Ð³ÑÑÐ°ÑÐ¸Ð¸ Apache Spark Ð´Ð»Ñ Ð¿Ð¾Ð²ÑÑÐµÐ½Ð¸Ñ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÐ¸ ETL-Ð¿ÑÐ¾ÑÐµÑÑÐ¾Ð²
title: Ð²ÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¼Ð°Ð³Ð¸ÑÑÑÐ°
titleType: Subtitle
title: The methodology for customizing of configuration parameters of Apache Spark to improve the performance of ETL processes
Creators: name: Ð¨Ð°ÑÐ¼Ð¸Ð½, ÐÐ³Ð¾Ñ
Publisher: name: Ð¡Ð°Ð½ÐºÑ-ÐÐµÑÐµÑÐ±ÑÑÐ³ÑÐºÐ¸Ð¹ Ð¿Ð¾Ð»Ð¸ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ð¹ ÑÐ½Ð¸Ð²ÐµÑÑÐ¸ÑÐµÑ ÐÐµÑÑÐ° ÐÐµÐ»Ð¸ÐºÐ¾Ð³Ð¾
Year: 2022
Subjects: ÐÑÑÐ¸ÑÐ»Ð¸ÑÐµÐ»ÑÐ½ÑÐµ Ð¼Ð°ÑÐ¸Ð½Ñ ÑÐ»ÐµÐºÑÑÐ¾Ð½Ð½ÑÐµ
Ð±Ð¾Ð»ÑÑÐ¸Ðµ Ð´Ð°Ð½Ð½ÑÐµ
Ð²Ð·ÑÑÐ² ÐºÐ¾Ð½ÑÐ¸Ð³ÑÑÐ°ÑÐ¸Ð¾Ð½Ð½Ð¾Ð³Ð¾ Ð¿ÑÐ¾ÑÑÑÐ°Ð½ÑÑÐ²Ð°
Ð¿Ð¾Ð´Ñ Ð¾Ð´ Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ Ð¾Ð±ÑÑÐµÐ½Ð¸Ñ
big data
configuration space explosion
ML-based approach

description: ÐÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¿Ð¾ÑÐ²ÑÑÐµÐ½Ð° ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐºÐµ Ð¼ÐµÑÐ¾Ð´Ð¸ÐºÐ¸ Ð½Ð°ÑÑÑÐ¾Ð¹ÐºÐ¸ Ð¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð² ÐºÐ¾Ð½ÑÐ¸Ð³ÑÑÐ°ÑÐ¸Ð¸ Big Data ÑÑÐµÐ¹Ð¼Ð²Ð¾ÑÐºÐ° Apache Spark Ð´Ð»Ñ Ð¿Ð¾Ð²ÑÑÐµÐ½Ð¸Ñ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÐ¸ ETL-Ð¿ÑÐ¾ÑÐµÑÑÐ¾Ð². ÐÐ°Ð½Ð½Ð°Ñ Ð¼ÐµÑÐ¾Ð´Ð¸ÐºÐ° Ð¿Ð¾Ð·Ð²Ð¾Ð»ÑÐµÑ Ð¿Ð¾Ð»ÑÑÐ¸ÑÑ Ð¾Ð¿ÑÐ¸Ð¼Ð°Ð»ÑÐ½ÑÐµ ÐºÐ¾Ð½ÑÐ¸Ð³ÑÑÐ°ÑÐ¸Ð¸ Ð´Ð»Ñ Ð¿Ð°ÑÐ°Ð»Ð»ÐµÐ»ÑÐ½Ð¾ ÑÐ°Ð±Ð¾ÑÐ°ÑÑÐ¸Ñ ETL-Ð¿ÑÐ¾ÑÐµÑÑÐ¾Ð². Ð Ð´Ð°Ð½Ð½Ð¾Ð¹ ÑÐ°Ð±Ð¾ÑÐµ Ð¾Ð¿Ð¸ÑÐ°Ð½Ñ Ð¾ÑÐ½Ð¾Ð²Ð½ÑÐµ Ð¿Ð¾Ð´ÑÐ¾Ð´Ñ ÑÐµÑÐµÐ½Ð¸Ñ Ð¿ÑÐ¾Ð±Ð»ÐµÐ¼Ñ Ð½Ð°ÑÑÑÐ¾Ð¹ÐºÐ¸ Big Data ÑÑÐµÐ¹Ð¼Ð²Ð¾ÑÐºÐ¾Ð² Ñ Ð±Ð¾Ð»ÑÑÐ¸Ð¼ ÐºÐ¾Ð½ÑÐ¸Ð³ÑÑÐ°ÑÐ¸Ð¾Ð½Ð½ÑÐ¼ Ð¿ÑÐ¾ÑÑÑÐ°Ð½ÑÑÐ²Ð¾Ð¼: Ð»ÑÑÑÐ¸Ðµ Ð¿ÑÐ°ÐºÑÐ¸ÐºÐ¸, Ð¿Ð¾Ð´ÑÐ¾Ð´Ñ Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ Ð°Ð½Ð°Ð»Ð¸Ð·Ð° Ð¿ÑÐ¾Ð³ÑÐ°Ð¼Ð¼, Ð¿Ð¾Ð´ÑÐ¾Ð´Ñ Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ Ð¿Ð¾Ð¸ÑÐºÐ°, Ð¿Ð¾Ð´ÑÐ¾Ð´Ñ Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ Ð¾Ð±ÑÑÐµÐ½Ð¸Ñ. ÐÑÐ¾Ð°Ð½Ð°Ð»Ð¸Ð·Ð¸ÑÐ¾Ð²Ð°Ð½Ñ Ð¸Ñ Ð´Ð¾ÑÑÐ¾Ð¸Ð½ÑÑÐ²Ð° Ð¸ Ð½ÐµÐ´Ð¾ÑÑÐ°ÑÐºÐ¸. ÐÑÐµÐ´Ð»Ð°Ð³Ð°ÐµÑÑÑ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸Ðµ Ð¿Ð¾Ð´ÑÐ¾Ð´Ð° Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ Ð¾Ð±ÑÑÐµÐ½Ð¸Ñ, Ð¿ÑÐ¾ÑÐµÑÑ ÐºÐ¾ÑÐ¾ÑÐ¾Ð³Ð¾ Ð·Ð°ÐºÐ»ÑÑÐ°ÐµÑÑÑ Ð² Ð¾Ð±ÑÑÐµÐ½Ð¸Ð¸ Ð¼Ð¾Ð´ÐµÐ»ÐµÐ¹ Ð¿ÑÐ¾Ð³Ð½Ð¾Ð·Ð¸ÑÐ¾Ð²Ð°Ð½Ð¸Ñ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÐ¸ Ð²ÑÐµÐ¼ÐµÐ½Ð¸ Ð²ÑÐ¿Ð¾Ð»Ð½ÐµÐ½Ð¸Ñ, Ð¿Ð°ÑÐ°Ð»Ð»ÐµÐ»ÑÐ½Ð¾ ÑÐ°Ð±Ð¾ÑÐ°ÑÑÐ¸Ñ Ð½Ð° Ð¾Ð´Ð½Ð¾Ð¼ ÐºÐ»Ð°ÑÑÐµÑÐµ, ETL-Ð¿ÑÐ¾ÑÐµÑÑÐ¾Ð² Ñ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸ÐµÐ¼ Ð¾Ð±ÑÑÐ°ÑÑÐ¸Ñ Ð´Ð°Ð½Ð½ÑÑ, ÑÐ¾Ð±ÑÐ°Ð½Ð½ÑÑ ÐºÐ¾Ð¼Ð¿Ð¾Ð½ÐµÐ½ÑÐ¾Ð¼ ÑÐ±Ð¾ÑÐ°. ÐÐ±ÑÑÐµÐ½Ð½ÑÐµ Ð¼Ð¾Ð´ÐµÐ»Ð¸ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·ÑÑÑÑÑ Ð² Ð¸ÑÐµÑÐ°ÑÐ¸Ð²Ð½Ð¾Ð¼ Ð¿ÑÐ¾ÑÐµÑÑÐµ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ð° Ð¿Ð¾Ð¸ÑÐºÐ° Ð´Ð»Ñ Ð¿Ð¾Ð»ÑÑÐµÐ½Ð¸Ñ Ð¾Ð¿ÑÐ¸Ð¼Ð°Ð»ÑÐ½ÑÑ ÐºÐ¾Ð½ÑÐ¸Ð³ÑÑÐ°ÑÐ¸Ð¹ ÐºÐ°Ð¶Ð´Ð¾Ð³Ð¾ ETL-Ð¿ÑÐ¾ÑÐµÑÑÐ°. Ð ÑÐ¾Ð´Ðµ ÑÐµÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ð¸ Ð¿Ð¾Ð´ÑÐ¾Ð´Ð° Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¾ Ð²Ð»Ð¸ÑÐ½Ð¸Ðµ ÑÐ°ÑÑÐ¼Ð°ÑÑÐ¸Ð²Ð°ÐµÐ¼ÑÑ Spark Ð¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð² Ð½Ð° Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÑ ETL-Ð¿ÑÐ¾ÑÐµÑÑÐ¾Ð² Ð¸ Ð¿ÑÐ¾Ð²ÐµÐ´ÐµÐ½Ð° Ð¾ÑÐµÐ½ÐºÐ° Ð¸Ñ Ð²ÑÐµÐ¼ÐµÐ½Ð¸ ÑÐ°Ð±Ð¾ÑÑ Ð½Ð° Ð½Ð°Ð¹Ð´ÐµÐ½Ð½ÑÑ Ð¾Ð¿ÑÐ¸Ð¼Ð°Ð»ÑÐ½ÑÑ ÐºÐ¾Ð½ÑÐ¸Ð³ÑÑÐ°ÑÐ¸ÑÑ, Ñ Ð¿Ð¾Ð¼Ð¾ÑÑÑ ÐºÐ¾ÑÐ¾ÑÑÑ ÑÐ´Ð°Ð»Ð¾ÑÑ Ð¿Ð¾Ð²ÑÑÐ¸ÑÑ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÑ Ð¿Ð°ÑÐ°Ð»Ð»ÐµÐ»ÑÐ½Ð¾ ÑÐ°Ð±Ð¾ÑÐ°ÑÑÐ¸Ñ ETL-Ð¿ÑÐ¾ÑÐµÑÑÐ¾Ð² Ð·Ð° ÑÑÐµÑ ÑÐ°Ð²Ð½Ð¾Ð¼ÐµÑÐ½Ð¾Ð³Ð¾ ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð¸Ñ ÑÐµÑÑÑÑÐ¾Ð² ÐºÐ»Ð°ÑÑÐµÑÐ° Ð¼ÐµÐ¶Ð´Ñ Ð½Ð¸Ð¼Ð¸.
description: The graduate qualification work is devoted to the development of the methodology for customizing of configuration parameters of the Apache Spark framework for Big Data architectures to improve the performance of ETL processes. This methodology allows to obtain optimal configurations for parallel running ETL processes. In this work the basic approaches solving the problem of configuring Big Data frameworks with large configuration space are described, namely the best practices, program analysis-based approaches, search-based approaches, ML-based approaches. Also, their advantages and disadvantages are analysed. Itâs proposed to use an ML-based approach, the process of which is to train the run-time performance prediction models of ETL processes running in parallel on the same cluster using training data collected by the acquisition component. The trained models are used in the iterative search algorithm process to obtain the optimal configurations of each ETL process. During the implementation of the approach the influence of the considered Spark parameters on the performance of ETL processes is investigated and the estimation of their operating time on the found optimal configurations is carried out, through which it is possible to improve the performance of parallel running ETL processes by evenly distributing the cluster resources between them.