title: Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° Ð°Ð²ÑÐ¾Ð½Ð¾Ð¼Ð½Ð¾Ð³Ð¾ Ñ ÑÐ°Ð½Ð¸Ð»Ð¸ÑÐ° Ð´Ð°Ð½Ð½ÑÑ Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ Ð¸Ð½ÑÑÑÑÐ¼ÐµÐ½ÑÐ° Apache Spark
title: Ð²ÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¼Ð°Ð³Ð¸ÑÑÑÐ°
titleType: Subtitle
title: Development of an offline data warehouse based on the Apache Spark
Creators: name: ÐÐ¾ ÐÐ¸Ð½ÑÑÐ¾
Publisher: name: Ð¡Ð°Ð½ÐºÑ-ÐÐµÑÐµÑÐ±ÑÑÐ³ÑÐºÐ¸Ð¹ Ð¿Ð¾Ð»Ð¸ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ð¹ ÑÐ½Ð¸Ð²ÐµÑÑÐ¸ÑÐµÑ ÐÐµÑÑÐ° ÐÐµÐ»Ð¸ÐºÐ¾Ð³Ð¾
Year: 2023
Subjects: Ð°Ð²ÑÐ¾Ð½Ð¾Ð¼Ð½Ð¾Ðµ Ñ ÑÐ°Ð½Ð¸Ð»Ð¸ÑÐµ Ð´Ð°Ð½Ð½ÑÑ
Apache Spark
Apache Hive
Apache Superset
Ð±Ð¾Ð»ÑÑÐ¸Ðµ Ð´Ð°Ð½Ð½ÑÐµ
Ð¾Ð¿ÑÐ¸Ð¼Ð¸Ð·Ð°ÑÐ¸Ñ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÐ¸ Ð²ÑÑÐ¸ÑÐ»Ð¸ÑÐµÐ»ÑÐ½ÑÑ ÑÐ¸ÑÑÐµÐ¼ Ñ ÑÐ°Ð½ÐµÐ½Ð¸Ñ Ð´Ð°Ð½Ð½ÑÑ
offline data warehouse
optimizing the performance of computational storage systems

description: ÐÐ°Ð³Ð¸ÑÑÐµÑÑÐºÐ°Ñ Ð´Ð¸Ð¿Ð»Ð¾Ð¼Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ°Â  Ð¿Ð¾ÑÐ²ÑÑÐµÐ½Ð° ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐºÐµ Ð°Ð²ÑÐ¾Ð½Ð¾Ð¼Ð½Ð¾Ð³Ð¾ ÑÑÐ°Ð½Ð¸Ð»Ð¸ÑÐ° Ð´Ð°Ð½Ð½ÑÑ Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ apache Spark Ð¸ ÑÑÐ°Ð²Ð½ÐµÐ½Ð¸Ñ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÐ¸ ÑÐ¸ÑÑÐµÐ¼Ñ ÑÑÐµÐ½Ð¸Ñ Ð¸ Ð·Ð°Ð¿Ð¸ÑÐ¸ Ð´Ð°Ð½Ð½ÑÑ Ñ ÑÑÐ°Ð´Ð¸ÑÐ¸Ð¾Ð½Ð½ÑÐ¼ ÑÑÐ°Ð½Ð¸Ð»Ð¸ÑÐµÐ¼ Ð´Ð°Ð½Ð½ÑÑ, Ð° ÑÐ°ÐºÐ¶Ðµ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸Ñ Ð¿Ð»Ð°Ð½Ð¸ÑÐ¾Ð²ÑÐ¸ÐºÐ° Ð·Ð°Ð´Ð°Ñ Ð´Ð»Ñ Ð¿Ð»Ð°Ð½Ð¸ÑÐ¾Ð²Ð°Ð½Ð¸Ñ ÑÐ°Ð±Ð¾ÑÑ ÑÐ¸ÑÑÐµÐ¼Ñ ÑÑÐ°Ð½Ð¸Ð»Ð¸ÑÐ° Ð´Ð°Ð½Ð½ÑÑ ÑÐ°ÐºÐ¸Ð¼ Ð¾Ð±ÑÐ°Ð·Ð¾Ð¼, ÑÑÐ¾Ð±Ñ Ð²Ð¸Ð·ÑÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ñ Ð´Ð°Ð½Ð½ÑÑ Ð¾ÑÑÐ°Ð¶Ð°Ð»Ð° ÐºÐ¾ÑÑÐµÐ»ÑÑÐ¸Ñ Ð¼ÐµÐ¶Ð´Ñ Ð½Ð¸Ð¼Ð¸. ÐÑÐµÐ´Ð¼ÐµÑÐ¾Ð¼ Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ ÑÐ²Ð»ÑÐµÑÑÑ: Ð¿Ð¾Ð²ÑÑÐµÐ½Ð¸Ðµ ÑÑÑÐµÐºÑÐ¸Ð²Ð½Ð¾ÑÑÐ¸ Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸ Ð¸ ÑÑÐ°Ð½ÐµÐ½Ð¸Ñ Ð´Ð°Ð½Ð½ÑÑ Ð² Ð°Ð²ÑÐ¾Ð½Ð¾Ð¼Ð½ÑÑ ÑÑÐ°Ð½Ð¸Ð»Ð¸ÑÐ° Ð´Ð°Ð½Ð½ÑÑ Ð·Ð° ÑÑÐµÑ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸Ñ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ð¾Ð² Ð²ÑÑÐ¾ÐºÐ¾Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½ÑÑ, ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð½ÑÑ Ð²ÑÑÐ¸ÑÐ»ÐµÐ½Ð¸Ð¹ Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ Ð¸Ð½ÑÑÑÑÐ¼ÐµÐ½ÑÐ° Apache Spark. ÐÐ¾ ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÐ°Ð¼ ÑÑÐ°Ð²Ð½Ð¸ÑÐµÐ»ÑÐ½Ð¾Ð³Ð¾ Ð°Ð½Ð°Ð»Ð¸Ð·Ð° Ð¿ÑÐµÐ´Ð»Ð°Ð³Ð°ÐµÑÑÑ ÑÐ¸ÑÑÐµÐ¼Ð° Ð°Ð²ÑÐ¾Ð½Ð¾Ð¼Ð½Ð¾Ð³Ð¾ ÑÑÐ°Ð½Ð¸Ð»Ð¸ÑÐ° Ð´Ð°Ð½Ð½ÑÑ Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ ÑÑÐ°Ð½Ð¸Ð»Ð¸ÑÐ° Ð´Ð°Ð½Ð½ÑÑ Hive Ð¸ Spark Ð² ÐºÐ°ÑÐµÑÑÐ²Ðµ Ð²ÑÑÐ¸ÑÐ»Ð¸ÑÐµÐ»ÑÐ½Ð¾Ð³Ð¾ Ð¼ÐµÑÐ°Ð½Ð¸Ð·Ð¼Ð°. ÐÐ¿Ð¸ÑÐ°Ð½Ð½Ð°Ñ ÑÐ¸ÑÑÐµÐ¼Ð° ÑÑÐ°Ð½Ð¸Ð»Ð¸ÑÐ° Ð´Ð°Ð½Ð½ÑÑ Ð¿ÑÐµÐ´Ð½Ð°Ð·Ð½Ð°ÑÐµÐ½Ð° Ð´Ð»Ñ ÑÐ±Ð¾ÑÐ° Ð¸ Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸ Ð´Ð°Ð½Ð½ÑÑ, Ð¿Ð¾ÑÑÑÐ¿Ð°ÑÑÐ¸Ñ Ñ Ð²Ð½ÐµÑÐ½ÐµÐ³Ð¾ Ð¸Ð½ÑÐµÑÑÐµÐ¹ÑÐ°, ÑÐµÑÐµÐ· ÑÑÐ°Ð½Ð¸Ð»Ð¸ÑÐµ Ð´Ð°Ð½Ð½ÑÑ Ð´Ð»Ñ Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸ Ð¸ Ð°Ð½Ð°Ð»Ð¸Ð·Ð° Ð´Ð°Ð½Ð½ÑÑ Ð¸, Ð½Ð°ÐºÐ¾Ð½ÐµÑ, Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ Ð¿Ð¾ÑÑÐµÐ±Ð½Ð¾ÑÑÐµÐ¹ Ð¿ÑÐµÐ´Ð¿ÑÐ¸ÑÑÐ¸Ñ Ð´Ð»Ñ Ð²Ð¸Ð·ÑÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ð¸ ÑÐ¾Ð¾ÑÐ²ÐµÑÑÑÐ²ÑÑÑÐ¸Ñ Ð´Ð°Ð½Ð½ÑÑ. Ð ÑÐ°Ð±Ð¾ÑÐµ Ð¾Ð¿Ð¸ÑÐ°Ð½Ñ Ð´ÐµÑÐ°Ð»Ð¸ ÑÐµÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ð¸ Ð¿ÑÐ¾Ð³ÑÐ°Ð¼Ð¼Ð½Ð¾Ð³Ð¾ Ð¾Ð±ÐµÑÐ¿ÐµÑÐµÐ½Ð¸Ñ, Ð° ÑÐ°ÐºÐ¶Ðµ Ð¿Ð¾ÑÐ¾Ðº Ð´Ð°Ð½Ð½ÑÑ Ð¸ Ð¼ÐµÑÐ¾Ð´Ñ, Ð¸ Ð¼ÐµÑÐ¾Ð´Ñ ÑÐµÑÑÐ¸ÑÐ¾Ð²Ð°Ð½Ð¸Ñ ÐºÐ¾Ð½ÐµÑÐ½Ð¾Ð¹ ÑÐ¸ÑÑÐµÐ¼Ñ. Ð¡ÑÐ°Ð²Ð½Ð¸Ð²Ð°Ñ ÑÑÑÐµÐºÑÐ¸Ð²Ð½Ð¾ÑÑÑ ÑÐ°Ð·Ð»Ð¸ÑÐ½ÑÑ Ð¿ÑÐ¾ÑÐµÑÑÐ¸ÑÐ¾Ð²Ð°Ð½Ð½ÑÑ Ð½Ð°Ð±Ð¾ÑÐ¾Ð² Ð´Ð°Ð½Ð½ÑÑ, ÑÑÐ¾ ÑÑÐ°Ð½Ð¸Ð»Ð¸ÑÐ° Ð´Ð°Ð½Ð½ÑÑ Ð½Ð° Ð±Ð°Ð·Ðµ Spark Ð±Ð¾Ð»ÐµÐµ ÑÑÑÐµÐºÑÐ¸Ð²Ð½Ñ Ð² Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐµ Ð¸ ÑÑÐ°Ð½ÐµÐ½Ð¸Ð¸ Ð´Ð°Ð½Ð½ÑÑ, ÑÐµÐ¼ ÑÑÐ°Ð´Ð¸ÑÐ¸Ð¾Ð½Ð½ÑÐµ ÑÑÐ°Ð½Ð¸Ð»Ð¸ÑÐ° Ð´Ð°Ð½Ð½ÑÑ.
description: The masters degree work is devoted to the development of an offline data warehouse based on Apache Spark and a comparison of the performance of a data reading and writing system with a traditional data warehouse, and the use of a task scheduler to schedule the data warehouse system so that data visualization reflects the correlation between them. The subject of the research is: improving the efficiency of processing and storing data in offline data warehouses using algorithms for high-performance, distributed computing based on the tool Apache Spark. Based on the results of the comparative analysis, a stand-alone data warehouse system based on the Hive data warehouse and Spark as a computing engine is proposed. The described data warehouse system is designed to collect and process data coming from external interface, through data warehouse for data processing and analysis and finally based on enterprise needs for visualization of relevant data. The paper describes the details of the software implementation as well as the data flow and the methods and techniques for testing the final system. By comparing the performance of the different data sets tested, that Spark-based data warehouses are more efficient in processing and storing data than traditional data warehouses.