title: Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° ÑÑÐµÐ¹Ð¼Ð²Ð¾ÑÐºÐ°, ÑÐ°ÑÑÐ¸ÑÑÑÑÐµÐ³Ð¾ Ð²Ð¾Ð·Ð¼Ð¾Ð¶Ð½Ð¾ÑÑÐ¸ Ð³ÑÑÐ¿Ð¿Ð¸ÑÐ¾Ð²ÐºÐ¸ Ð´Ð°Ð½Ð½ÑÑ Apache Spark
title: Ð²ÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¼Ð°Ð³Ð¸ÑÑÑÐ°
titleType: Subtitle
title: Development of a framework that extends Apache Sparkâs capabilities to group data
Creators: name: ÐÐ¾Ð¿Ð¾Ð², ÐÐ°ÐºÑÐ¸Ð¼
Publisher: name: Ð¡Ð°Ð½ÐºÑ-ÐÐµÑÐµÑÐ±ÑÑÐ³ÑÐºÐ¸Ð¹ Ð¿Ð¾Ð»Ð¸ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ð¹ ÑÐ½Ð¸Ð²ÐµÑÑÐ¸ÑÐµÑ ÐÐµÑÑÐ° ÐÐµÐ»Ð¸ÐºÐ¾Ð³Ð¾
Year: 2020
Subjects: Ð±Ð¾Ð»ÑÑÐ¸Ðµ Ð´Ð°Ð½Ð½ÑÐµ
Apache Spark
Scala
Hadoop
Apache Hive
Ð¾Ð¿ÑÐ¸Ð¼Ð¸Ð·Ð°ÑÐ¸Ñ
Ð°ÑÑ Ð¸ÑÐµÐºÑÑÑÐ° ÑÐ¸ÑÑÐµÐ¼Ñ
Big Data
optimization
systems architecture

description: ÐÐ°Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¿Ð¾ÑÐ²ÑÑÐµÐ½Ð° ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐºÐµ ÑÑÐµÐ¹Ð¼Ð²Ð¾ÑÐºÐ°, ÑÐ°ÑÑÐ¸ÑÑÑÑÐµÐ³Ð¾ Ð²Ð¾Ð·Ð¼Ð¾Ð¶Ð½Ð¾ÑÑÐ¸ Apache Spark Ð·Ð° ÑÑÐµÑ ÑÐ¾ÐºÑÐ°ÑÐµÐ½Ð¸Ñ ÐºÐ¾Ð»Ð¸ÑÐµÑÑÐ²Ð° Ð¿ÑÐµÐ¾Ð±ÑÐ°Ð·Ð¾Ð²Ð°Ð½Ð¸Ð¹ Ñ ÑÐ¸ÑÐ¾ÐºÐ¸Ð¼Ð¸ Ð·Ð°Ð²Ð¸ÑÐ¸Ð¼Ð¾ÑÑÑÐ¼Ð¸ Ð¸, ÐºÐ°Ðº ÑÐ»ÐµÐ´ÑÑÐ²Ð¸Ðµ, ÐºÐ¾Ð»Ð¸ÑÐµÑÑÐ²Ð° Ð¿ÐµÑÐµÑÐ°ÑÐ¾Ð²Ð¾Ðº Ð´Ð°Ð½Ð½ÑÑ. Ð­ÑÐ¾ Ð´Ð¾ÑÑÐ¸Ð³Ð°ÐµÑÑÑ Ð·Ð° ÑÑÐµÑ Ð¾Ð±ÑÐµÐ´Ð¸Ð½ÐµÐ½Ð¸Ñ Ð¿Ð¾ÑÐ»ÐµÐ´Ð¾Ð²Ð°ÑÐµÐ»ÑÐ½ÑÑ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ð¾Ð² Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸ Ð´Ð°Ð½Ð½ÑÑ Ð² ÑÐµÐ¿Ð¾ÑÐºÐ¸ Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ Ð¾Ð±ÑÐ¸Ñ ÐºÐ»ÑÑÐµÐ²ÑÑ Ð¿Ð¾Ð»ÐµÐ¹, Ð° ÑÐ°ÐºÐ¶Ðµ Ð³ÑÑÐ¿Ð¿Ð¸ÑÐ¾Ð²ÐºÐ¸ ÑÐ°Ð¼Ð¸Ñ Ð´Ð°Ð½Ð½ÑÑ Ð¿Ð¾ ÐºÐ»ÑÑÐ°Ð¼, Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ ÐºÐ¾ÑÐ¾ÑÑÑ Ð¸ Ð¿ÑÐ¾Ð¸ÑÑÐ¾Ð´Ð¸Ñ Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ°. ÐÐ°Ð´Ð°ÑÐ¸, ÑÐµÑÐµÐ½Ð½ÑÐµ Ð² ÑÐ¾Ð´Ðµ ÑÐ°Ð±Ð¾ÑÑ:
1. Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° Ð¼ÐµÑÐ¾Ð´Ð° Ð¾Ð¿ÑÐ¸Ð¼Ð¸Ð·Ð°ÑÐ¸Ð¸ Ð¿ÑÐ¾Ð³ÑÐ°Ð¼Ð¼ Ñ Ð±Ð¾Ð»ÑÑÐ¸Ð¼ ÐºÐ¾Ð»Ð¸ÑÐµÑÑÐ²Ð¾Ð¼ Ð¿ÑÐµÐ¾Ð±ÑÐ°Ð·Ð¾Ð²Ð°Ð½Ð¸Ð¹ Ñ ÑÐ¸ÑÐ¾ÐºÐ¸Ð¼Ð¸ Ð·Ð°Ð²Ð¸ÑÐ¸Ð¼Ð¾ÑÑÑÐ¼Ð¸.
2. Ð ÐµÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ñ Ñ Ð¿Ð¾Ð¼Ð¾ÑÑÑ Ð´Ð°Ð½Ð½Ð¾Ð³Ð¾ Ð¼ÐµÑÐ¾Ð´Ð° ÑÑÐµÐ¹Ð¼Ð²Ð¾ÑÐºÐ°, Ð¿ÑÐ¸Ð½Ð¸Ð¼Ð°ÑÑÐ¸Ð¹ Ð½Ð° Ð²ÑÐ¾Ð´ ÑÐµÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ð¸ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ð¾Ð², Ð¿Ð¾Ð´Ð»ÐµÐ¶Ð°ÑÐ¸Ñ Ð¾Ð±ÑÐµÐ´Ð¸Ð½ÐµÐ½Ð¸Ñ Ð² ÑÐµÐ¿Ð¾ÑÐºÐ¸.
Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐ°Ð½Ð½ÑÐ¹ ÑÑÐµÐ¹Ð¼Ð²Ð¾ÑÐº Ð±ÑÐ» Ð¿ÑÐ¸Ð¼ÐµÐ½ÐµÐ½ Ð² ÑÐ¸ÑÑÐµÐ¼Ðµ ÑÐ¾Ð¿Ð¾ÑÑÐ°Ð²Ð»ÐµÐ½Ð¸Ñ, ÑÐ¾Ð´ÐµÑÐ¶Ð°ÑÐµÐ¹ Ð±Ð¾Ð»ÑÑÐ¾Ðµ ÐºÐ¾Ð»Ð¸ÑÐµÑÑÐ²Ð¾ Ð¿Ð¾ÑÐ»ÐµÐ´Ð¾Ð²Ð°ÑÐµÐ»ÑÐ½ÑÑ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ð¾Ð² Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸ Ð´Ð°Ð½Ð½ÑÑ, ÑÑÐµÐ±ÑÑÑÐ¸Ñ Ð³ÑÑÐ¿Ð¿Ð¸ÑÐ¾Ð²Ð¾Ðº Ð´Ð°Ð½Ð½ÑÑ. ÐÐ¾ÑÐ»Ðµ ÑÑÐ¾Ð³Ð¾ Ð±ÑÐ»Ð¾ Ð¿ÑÐ¾Ð²ÐµÐ´ÐµÐ½Ð¾ ÑÑÐ°Ð²Ð½ÐµÐ½Ð¸Ðµ ÑÑÑÐµÐºÑÐ¸Ð²Ð½Ð¾ÑÑÐ¸ Ð²ÑÐ¿Ð¾Ð»Ð½ÐµÐ½Ð¸Ñ Ð¿ÑÐ¾Ð³ÑÐ°Ð¼Ð¼Ñ Ð´Ð¾ Ð¸ Ð¿Ð¾ÑÐ»Ðµ Ð¾Ð¿ÑÐ¸Ð¼Ð¸Ð·Ð°ÑÐ¸Ð¸.
Ð ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÐµ Ð¿ÑÐ¸Ð¼ÐµÐ½ÐµÐ½Ð¸Ðµ ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐ°Ð½Ð½Ð¾Ð³Ð¾ ÑÑÐµÐ¹Ð¼Ð²Ð¾ÑÐºÐ° Ð¿Ð¾Ð·Ð²Ð¾Ð»Ð¸Ð»Ð¾ Ð¸Ð·Ð±ÐµÐ¶Ð°ÑÑ Ð»Ð¸ÑÐ½Ð¸Ñ Ð¿ÐµÑÐµÐ¼ÐµÑÐµÐ½Ð¸Ð¹ Ð´Ð°Ð½Ð½ÑÑ Ð¼ÐµÐ¶Ð´Ñ ÑÐ·Ð»Ð°Ð¼Ð¸ ÐºÐ»Ð°ÑÑÐµÑÐ° Ð¸ ÑÐ²ÐµÐ»Ð¸ÑÐ¸ÑÑ Ð¼Ð°ÐºÑÐ¸Ð¼Ð°Ð»ÑÐ½Ð¾ Ð²Ð¾Ð·Ð¼Ð¾Ð¶Ð½ÑÑ Ð½Ð°Ð³ÑÑÐ·ÐºÑ Ð½Ð° Ð¾Ð¿ÐµÑÐ°ÑÐ¸Ð²Ð½ÑÑ Ð¿Ð°Ð¼ÑÑÑ. ÐÐ»Ð°Ð³Ð¾Ð´Ð°ÑÑ ÑÑÐ¾Ð¼Ñ Ñ ÑÐ¸ÑÑÐµÐ¼Ñ Ð¿Ð¾Ð²ÑÑÐ¸Ð»Ð°ÑÑ ÑÐºÐ¾ÑÐ¾ÑÑÑ ÑÐ°Ð±Ð¾ÑÑ Ð¸ ÑÑÑÐµÐºÑÐ¸Ð²Ð½Ð¾ÑÑÑ ÑÐ°ÑÑÐ¾Ð´Ð° ÑÐµÑÑÑÑÐ¾Ð².
description: This study is devoted to the development of a framework that extends the capabilities of Apache Spark by reducing the number of transformations with wide dependencies and, as a result, the number of data shuffles. This achieved by combining sequential data processing algorithms in chains based on common key fields, as well as grouping the data itself according to the keys by which the processing takes place. Tasks solved in the course of work:
1. Development of an optimization method for programs with many transformations with wide dependencies.
2. Implementation of the framework using this method, which takes as input implementations of the algorithms to be combined into chains.
The developed framework was applied to a matching system containing many sequential data processing algorithms requiring data groupings. After that, a comparison of the effectiveness of the program before and after optimization was made.
As a result, the application of the developed framework allowed avoiding unnecessary data movement between cluster nodes and increasing the maximum possible load on the RAM. Thanks to this, the system has increased operating speed and resource efficiency.