title: Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° Ð¸ Ð¾Ð¿ÑÐ¸Ð¼Ð¸Ð·Ð°ÑÐ¸Ñ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ð¾Ð² ÑÐ°Ð±Ð¾ÑÑ Ñ Ð±Ð¾Ð»ÑÑÐ¸Ð¼Ð¸ Ð´Ð°Ð½Ð½ÑÐ¼Ð¸ Ñ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸ÐµÐ¼ ÐºÐ»Ð°ÑÑÐµÑÐ° Hadoop Ð¸ ÑÑÐµÐ¹Ð¼Ð²Ð¾ÑÐºÐ° Java Spark
title: Ð²ÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð±Ð°ÐºÐ°Ð»Ð°Ð²ÑÐ°
titleType: Subtitle
title: Development and optimization of algorithms for processing data using the Hadoop cluster and the Java Spark framework
Creators: name: ÐÐ¸ÐºÐ¸ÑÐ¸Ð½, ÐÐ½Ð´ÑÐµÐ¹
Publisher: name: Ð¡Ð°Ð½ÐºÑ-ÐÐµÑÐµÑÐ±ÑÑÐ³ÑÐºÐ¸Ð¹ Ð¿Ð¾Ð»Ð¸ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ð¹ ÑÐ½Ð¸Ð²ÐµÑÑÐ¸ÑÐµÑ ÐÐµÑÑÐ° ÐÐµÐ»Ð¸ÐºÐ¾Ð³Ð¾
Year: 2024
Subjects: PL/SQL
Apache Hadoop
Apache Spark
Java
Ð±Ð¾Ð»ÑÑÐ¸Ðµ Ð´Ð°Ð½Ð½ÑÐµ
ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð½ÑÐµ Ð±Ð°Ð·Ñ Ð´Ð°Ð½Ð½ÑÑ
Ð¾Ð¿ÑÐ¸Ð¼Ð¸Ð·Ð°ÑÐ¸Ñ ÑÐ°Ð±Ð¾ÑÑ Ñ Ð±Ð¾Ð»ÑÑÐ¸Ð¼Ð¸ Ð´Ð°Ð½Ð½ÑÐ¼Ð¸
Ð¼Ð¸Ð³ÑÐ°ÑÐ¸Ñ
java
big data
distributed databases
optimization of working with big data
migration

description: ÐÐ°Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¿Ð¾ÑÐ²ÑÑÐµÐ½Ð° ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐºÐµ Ð¼ÐµÑÐ¾Ð´Ð¾Ð² Ð¿ÐµÑÐµÐ½Ð¾ÑÐ° Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ð¾Ð², Ð½Ð°Ð¿Ð¸ÑÐ°Ð½Ð½ÑÑ Ð½Ð° Ð¿ÑÐ¾ÑÐµÐ´ÑÑÐ½Ð¾Ð¼ ÑÐ·ÑÐºÐµ PL/SQL Ð½Ð° ÑÑÐµÐ¹Ð¼Ð²Ð¾ÑÐº Ð´Ð»Ñ Java Apache Spark, Ð° ÑÐ°ÐºÐ¶Ðµ Ð¾Ð¿ÑÐ¸Ð¼Ð¸Ð·Ð°ÑÐ¸Ñ Ð´Ð°Ð½Ð½ÑÑ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ð¾Ð² Ð´Ð»Ñ ÑÐ°Ð±Ð¾ÑÑ Ñ Ð±Ð¾Ð»ÑÑÐ¸Ð¼Ð¸ Ð´Ð°Ð½Ð½ÑÐ¼Ð¸. ÐÐ°Ð´Ð°ÑÐ¸, ÐºÐ¾ÑÐ¾ÑÑÐµ ÑÐµÑÐ°Ð»Ð¸ÑÑ Ð² ÑÐ¾Ð´Ðµ Ð²ÑÐ¿Ð¾Ð»Ð½ÐµÐ½Ð¸Ñ ÑÐ°Ð±Ð¾ÑÑ: 1. ÐÑÐ¾Ð²ÐµÐ´ÐµÐ½Ð¸Ðµ Ð¾Ð±Ð·Ð¾ÑÐ° ÑÑÐµÐ´ÑÑÐ² ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ¸ 2. ÐÐ½Ð°Ð»Ð¸Ð· Ð¸ÑÑÐ¾Ð´Ð½ÑÑ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ð¾Ð², Ð¸ ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° Ð¼ÐµÑÐ¾Ð´Ð¸ÐºÐ¸ Ð¿ÐµÑÐµÐ½Ð¾ÑÐ° Ð¸ÑÑÐ¾Ð´Ð½ÑÑ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ð¾Ð² Ð½Ð° Apache Spark 3. Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° Ð¼ÐµÑÐ¾Ð´Ð¾Ð² Ð¾Ð¿ÑÐ¸Ð¼Ð¸Ð·Ð°ÑÐ¸Ð¸ ÑÐ°Ð±Ð¾ÑÑ Ñ Ð±Ð¾Ð»ÑÑÐ¸Ð¼Ð¸ Ð´Ð°Ð½Ð½ÑÐ¼Ð¸ 4. ÐÐ½Ð°Ð»Ð¸Ð· Ð¼ÐµÑÐ¾Ð´Ð¾Ð² Ð¾Ð¿ÑÐ¸Ð¼Ð¸Ð·Ð°ÑÐ¸Ð¸ Ñ ÑÐ°Ð·Ð»Ð¸ÑÐ½ÑÐ¼ Ð¾Ð±ÑÑÐ¼Ð¾Ð¼ Ð´Ð°Ð½Ð½ÑÑ 5. ÐÐµÐ¼Ð¾Ð½ÑÑÑÐ°ÑÐ¸Ñ ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÐ¾Ð² ÑÐ°Ð±Ð¾ÑÑ Ð¿ÑÐ¾Ð³ÑÐ°Ð¼Ð¼ Ð±ÐµÐ· Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸Ñ ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐ°Ð½Ð½ÑÑ Ð¼ÐµÑÐ¾Ð´Ð¾Ð² Ð¾Ð¿ÑÐ¸Ð¼Ð¸Ð·Ð°ÑÐ¸Ð¹ Ð¸ ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÐ¾Ð² ÑÐ°Ð±Ð¾ÑÑ Ð¿ÑÐ¾Ð³ÑÐ°Ð¼Ð¼Ñ Ñ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸ÐµÐ¼ Ð´Ð°Ð½Ð½ÑÑ Ð¼ÐµÑÐ¾Ð´Ð¾Ð². Ð ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÐµ Ð²ÑÐ¿Ð¾Ð»Ð½ÐµÐ½Ð¸Ñ ÑÐ°Ð±Ð¾ÑÑ Ð±ÑÐ»Ð¾ ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐ°Ð½Ð¾ Ð¸ Ð¾Ð¿ÑÐ¸Ð¼Ð¸Ð·Ð¸ÑÐ¾Ð²Ð°Ð½Ð½Ð¾ Ð¿ÑÐ¾Ð³ÑÐ°Ð¼Ð¼Ð½Ð¾Ðµ Ð¾Ð±ÐµÑÐ¿ÐµÑÐµÐ½Ð¸Ðµ Ð´Ð»Ñ ÐÐÐ Â«Ð¡Ð±ÐµÑÐ±Ð°Ð½ÐºÂ», Ð¿Ð¾Ð»Ð½Ð¾ÑÑÑÑ Ð¿Ð¾Ð²ÑÐ¾ÑÑÑÑÐµÐµ ÑÑÐ½ÐºÑÐ¸Ð¾Ð½Ð°Ð»ÑÐ½Ð¾ÑÑÑ Ð½Ð°Ð¿Ð¸ÑÐ°Ð½Ð½ÑÑ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ð¾Ð² Ð½Ð° PL/SQL, Ð½Ð¾ ÑÐ°Ð±Ð¾ÑÐ°ÑÑÐµÐµ Ð±ÑÑÑÑÐµÐµ.
description: This work is devoted to the development of methods for transferring algorithms written in the procedural language PL/SQL to the Java framework Apache Spark, as well as optimizing these algorithms for working with big data. Tasks that were solved during the work: 1. Review of development tools 2. Analysis of source algorithms, and development of a methodology for migrating source algorithms to Apache Spark 3. Development of methods for optimizing work with big data 4. Analysis of optimization methods with different amounts of data 5. Demonstration of the results of the programs without using the developed optimization methods and the results of the program using these methods. As a result of the work, software was developed and optimized for Sberbank PJSC, completely migrating the functionality of the algorithms written in PL/SQL to Apache Spark with enhanced time performance.