title: Ð ÐµÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ñ Ð¿ÑÐ¾Ð³ÑÐ°Ð¼Ð¼Ð½Ð¾Ð¹ ÑÐ¸ÑÑÐµÐ¼Ñ Ð°Ð½Ð°Ð»Ð¸Ð·Ð° Ð´Ð°Ð½Ð½ÑÑ Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ ELK-ÑÑÐµÐºÐ°
title: Ð²ÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¼Ð°Ð³Ð¸ÑÑÑÐ°
titleType: Subtitle
title: Implementation of ELK-stack based data analysis software system
Creators: name: ÐÐ°Ð½ÑÑÑÐ¸Ð½, ÐÐ½Ð´ÑÐµÐ¹
Publisher: name: Ð¡Ð°Ð½ÐºÑ-ÐÐµÑÐµÑÐ±ÑÑÐ³ÑÐºÐ¸Ð¹ Ð¿Ð¾Ð»Ð¸ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ð¹ ÑÐ½Ð¸Ð²ÐµÑÑÐ¸ÑÐµÑ ÐÐµÑÑÐ° ÐÐµÐ»Ð¸ÐºÐ¾Ð³Ð¾
Year: 2024
Subjects: Ð°Ð½Ð°Ð»Ð¸Ð· Ð´Ð°Ð½Ð½ÑÑ
ÑÑÐµÐº
BI ÑÐ¸ÑÑÐµÐ¼Ð°
data analysis
stack
BI system

description: ÐÑÐµÐ´ÑÑÐ°Ð²Ð»ÐµÐ½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¿Ð¾ÑÐ²ÑÑÐµÐ½Ð° ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐºÐµ Ð¸ Ð²Ð½ÐµÐ´ÑÐµÐ½Ð¸Ñ Ð¿ÑÐ¾Ð³ÑÐ°Ð¼Ð¼Ð½Ð¾Ð¹ ÑÐ¸ÑÑÐµÐ¼Ñ Ð´Ð»Ñ Ð°Ð½Ð°Ð»Ð¸Ð·Ð° Ð´Ð°Ð½Ð½ÑÑ Ð¾ Ð¿ÑÐ¾Ð´Ð°Ð¶Ðµ ÑÐ¾Ð²Ð°ÑÐ¾Ð² Ñ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸ÐµÐ¼ ELK-ÑÑÐµÐºÐ° (Elasticsearch, Logstash, Kibana). Ð ÑÐ°Ð¼ÐºÐ°Ñ ÑÐ°Ð±Ð¾ÑÑ ÑÐµÑÐ°Ð»Ð¸ÑÑ ÑÐ»ÐµÐ´ÑÑÑÐ¸Ðµ Ð·Ð°Ð´Ð°ÑÐ¸: 1. Ð¡Ð¾Ð·Ð´Ð°Ð½Ð¸Ðµ Ð¸ Ð½Ð°ÑÑÑÐ¾Ð¹ÐºÐ° ÐºÐ¾Ð¼Ð¿Ð¾Ð½ÐµÐ½ÑÐ¾Ð² ÑÐ¸ÑÑÐµÐ¼Ñ: 1. Ð ÐµÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ñ Ð·Ð°Ð³ÑÑÐ·ÐºÐ¸ ÐºÐ¾Ð½ÑÐ¸Ð³ÑÑÐ°ÑÐ¸Ð¸ Ð¸ ÑÑÐµÐ½Ð¸Ñ Ð¿ÐµÑÐµÐ¼ÐµÐ½Ð½ÑÑ Ð¾ÐºÑÑÐ¶ÐµÐ½Ð¸Ñ. 2. ÐÐ°ÑÑÑÐ¾Ð¹ÐºÐ° Kafka Ð´Ð»Ñ ÑÐ¾Ð·Ð´Ð°Ð½Ð¸Ñ Ð¸ ÑÐ¿ÑÐ°Ð²Ð»ÐµÐ½Ð¸Ñ ÑÐ¾Ð¿Ð¸ÐºÐ°Ð¼Ð¸. 3. Ð¡Ð¾Ð·Ð´Ð°Ð½Ð¸Ðµ ÑÐ¸Ð»ÑÑÑÐ¾Ð² Ð´Ð»Ñ Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸ Ð´Ð°Ð½Ð½ÑÑ Ð² Logstash. 4. ÐÐ±ÑÐ°Ð±Ð¾ÑÐºÐ° ÑÐ°Ð·Ð»Ð¸ÑÐ½ÑÑ ÑÑÐµÐ½Ð°ÑÐ¸ÐµÐ², Ð²ÐºÐ»ÑÑÐ°Ñ Ð½ÐµÐ³Ð°ÑÐ¸Ð²Ð½ÑÐµ, Ð´Ð»Ñ ÐºÐ¾ÑÑÐµÐºÑÐ½Ð¾Ð¹ ÑÐ°Ð±Ð¾ÑÑ ÑÐ¸ÑÑÐµÐ¼Ñ. 2. Ð ÐµÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ñ Ð¼Ð¾Ð´ÑÐ»Ñ Ð¿ÑÐµÐ´Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸ Ð´Ð°Ð½Ð½ÑÑ: 1. Ð Ð°ÑÐ¿Ð¾Ð·Ð½Ð°Ð²Ð°Ð½Ð¸Ðµ Ð¸ Ð·Ð°Ð¿Ð¾Ð»Ð½ÐµÐ½Ð¸Ðµ Ð½ÐµÐ´Ð¾ÑÑÐ°ÑÑÐ¸Ñ Ð´Ð°Ð½Ð½ÑÑ Ð¸Ð· Ð¾Ð¿Ð¸ÑÐ°Ð½Ð¸Ð¹ Ð¾Ð±ÑÑÐ²Ð»ÐµÐ½Ð¸Ð¹. 2. ÐÐ±ÑÐ°Ð±Ð¾ÑÐºÐ° ÑÐ»ÑÑÐ°ÐµÐ² Ñ Ð¾ÑÑÑÑÑÑÐ²ÑÑÑÐ¸Ð¼Ð¸ Ð¸Ð»Ð¸ Ð½ÐµÐºÐ¾ÑÑÐµÐºÑÐ½ÑÐ¼Ð¸ Ð¿Ð°ÑÐ°Ð¼ÐµÑÑÐ°Ð¼Ð¸. 3. ÐÐ±ÑÐ°Ð±Ð¾ÑÐºÐ° Ð¿ÑÑÑÑÑ Ð¸ Ð½ÐµÐ²Ð°Ð»Ð¸Ð´Ð½ÑÑ Ð·Ð°Ð¿Ð¸ÑÐµÐ¹. 3. Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° Ð¼Ð¾Ð´ÑÐ»Ñ Ð°Ð½Ð°Ð»Ð¸Ð·Ð° Ð´Ð°Ð½Ð½ÑÑ: 1. ÐÐ±ÑÐ°Ð±Ð¾ÑÐºÐ° ÐºÐ¾ÑÑÐµÐºÑÐ½ÑÑ Ð·Ð°Ð¿ÑÐ¾ÑÐ¾Ð² Ð½Ð° Ð°Ð½Ð°Ð»Ð¸Ð· Ð¸ Ð²Ð·Ð°Ð¸Ð¼Ð¾Ð´ÐµÐ¹ÑÑÐ²Ð¸Ðµ Ñ Elasticsearch Ð¸ Spark. 2. Ð£Ð¿ÑÐ°Ð²Ð»ÐµÐ½Ð¸Ðµ Ð¾ÑÐ¸Ð±Ð¾ÑÐ½ÑÐ¼Ð¸ Ð·Ð°Ð¿ÑÐ¾ÑÐ°Ð¼Ð¸ Ð¸ Ð¾ÑÐ¿ÑÐ°Ð²ÐºÐ° ÑÐ¾Ð¾ÑÐ²ÐµÑÑÑÐ²ÑÑÑÐ¸Ñ ÐºÐ¾Ð´Ð¾Ð² Ð¾ÑÐ¸Ð±Ð¾Ðº. 3. ÐÐ°Ð³ÑÑÐ·ÐºÐ° Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐ°Ð½Ð½ÑÑ Ð´Ð°Ð½Ð½ÑÑ Ð¾Ð±ÑÐ°ÑÐ½Ð¾ Ð² Elasticsearch Ð´Ð»Ñ Ð´Ð°Ð»ÑÐ½ÐµÐ¹ÑÐµÐ³Ð¾ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸Ñ. Ð¢Ð°ÐºÐ¸Ð¼ Ð¾Ð±ÑÐ°Ð·Ð¾Ð¼, Ð±ÑÐ»Ð° ÑÐ¾Ð·Ð´Ð°Ð½Ð° Ð¸ Ð²Ð½ÐµÐ´ÑÐµÐ½Ð° ÑÐ°Ð±Ð¾ÑÐ¾ÑÐ¿Ð¾ÑÐ¾Ð±Ð½Ð°Ñ BI-ÑÐ¸ÑÑÐµÐ¼Ð°, Ð¿ÑÐµÐ´Ð¾ÑÑÐ°Ð²Ð»ÑÑÑÐ°Ñ Ð¼Ð¾ÑÐ½ÑÐµ Ð¸Ð½ÑÑÑÑÐ¼ÐµÐ½ÑÑ Ð´Ð»Ñ ÑÐ±Ð¾ÑÐ°, Ð¿ÑÐµÐ´Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸, Ð°Ð½Ð°Ð»Ð¸Ð·Ð° Ð¸ Ð²Ð¸Ð·ÑÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ð¸ Ð´Ð°Ð½Ð½ÑÑ Ð¾ Ð¿ÑÐ¾Ð´Ð°Ð¶Ðµ ÑÐ¾Ð²Ð°ÑÐ¾Ð². Ð¡Ð¸ÑÑÐµÐ¼Ð° Ð¿Ð¾Ð·Ð²Ð¾Ð»ÑÐµÑ Ð¾ÑÐ³Ð°Ð½Ð¸Ð·Ð°ÑÐ¸ÑÐ¼ Ð±Ð¾Ð»ÐµÐµ ÑÑÑÐµÐºÑÐ¸Ð²Ð½Ð¾ Ð°Ð½Ð°Ð»Ð¸Ð·Ð¸ÑÐ¾Ð²Ð°ÑÑ ÑÑÐ½Ð¾ÑÐ½ÑÐµ Ð´Ð°Ð½Ð½ÑÐµ, Ð²ÑÑÐ²Ð»ÑÑÑ ÑÐµÐ½Ð´ÐµÐ½ÑÐ¸Ð¸ Ð¸ Ð¿ÑÐ¸Ð½Ð¸Ð¼Ð°ÑÑ Ð¾Ð±Ð¾ÑÐ½Ð¾Ð²Ð°Ð½Ð½ÑÐµ ÑÐ¿ÑÐ°Ð²Ð»ÐµÐ½ÑÐµÑÐºÐ¸Ðµ ÑÐµÑÐµÐ½Ð¸Ñ, ÑÑÐ¾ ÑÐ²Ð»ÑÐµÑÑÑ Ð²Ð°Ð¶Ð½ÑÐ¼ ÑÑÐ»Ð¾Ð²Ð¸ÐµÐ¼ Ð´Ð»Ñ ÑÑÐ¿ÐµÑÐ½Ð¾Ð³Ð¾ ÑÑÐ½ÐºÑÐ¸Ð¾Ð½Ð¸ÑÐ¾Ð²Ð°Ð½Ð¸Ñ Ð¸ ÑÐ°Ð·Ð²Ð¸ÑÐ¸Ñ Ð² ÑÑÐ»Ð¾Ð²Ð¸ÑÑ ÑÐ¾Ð²ÑÐµÐ¼ÐµÐ½Ð½Ð¾Ð¹ ÑÐ¸ÑÑÐ¾Ð²Ð¾Ð¹ ÑÐºÐ¾Ð½Ð¾Ð¼Ð¸ÐºÐ¸.
description: This work is dedicated to the development and implementation of a software system for analyzing sales data using the ELK stack (Elasticsearch, Logstash, Kibana). The following tasks were addressed within the scope of this work: 1. Creation and configuration of system components: 1. Implementation of configuration loading and reading of environment variables. 2. Setup of Kafka for creating and managing topics. 3. Creation of filters for data processing in Logstash. 4. Handling various scenarios, including negative ones, for the correct operation of the system. 2. Implementation of the data preprocessing module: 1. Recognition and filling of missing data from advertisement descriptions. 2. Handling cases with missing or incorrect parameters. 3. Processing of empty and invalid records. 3. Development of the data analysis module: 1. Processing of correct analysis requests and interaction with Elasticsearch and Spark. 2. Handling erroneous requests and sending appropriate error codes. 3. Uploading processed data back to Elasticsearch for further use. Thus, a functional BI system was created and implemented, providing powerful tools for collecting, preprocessing, analyzing, and visualizing sales data. The system enables organizations to more effectively analyze market data, identify trends, and make informed managerial decisions, which is a crucial condition for successful operation and development in the modern digital economy.