title: ÐÐ¾Ð²ÑÑÐµÐ½Ð¸Ðµ ÑÐµÐ°Ð»ÑÐ½Ð¾Ð¹ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÐ¸ Ð²ÑÑÐ¾ÐºÐ¾Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½ÑÑ ÐºÐ»Ð°ÑÑÐµÑÐ¾Ð² Ñ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸ÐµÐ¼ Ð¼ÐµÑÐ¾Ð´Ð¾Ð² Ð¼Ð°ÑÐ¸Ð½Ð½Ð¾Ð³Ð¾ Ð¾Ð±ÑÑÐµÐ½Ð¸Ñ
title: Ð²ÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð±Ð°ÐºÐ°Ð»Ð°Ð²ÑÐ°
titleType: Subtitle
title: Increasing real productivity in high-performance clusters using machine learning
Creators: name: ÐÐ¾Ð¹ÑÐ¾Ð²Ð°, ÐÐ°ÑÐ°Ð»ÑÑ
Publisher: name: Ð¡Ð°Ð½ÐºÑ-ÐÐµÑÐµÑÐ±ÑÑÐ³ÑÐºÐ¸Ð¹ Ð¿Ð¾Ð»Ð¸ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ð¹ ÑÐ½Ð¸Ð²ÐµÑÑÐ¸ÑÐµÑ ÐÐµÑÑÐ° ÐÐµÐ»Ð¸ÐºÐ¾Ð³Ð¾
Year: 2024
Subjects: Ð¼Ð¾Ð´ÐµÐ»Ñ Ð²ÑÐ¶Ð¸Ð²Ð°ÐµÐ¼Ð¾ÑÑÐ¸
Ð¼Ð°ÑÐ¸Ð½Ð½Ð¾Ðµ Ð¾Ð±ÑÑÐµÐ½Ð¸Ðµ
ÑÑÐ¿ÐµÑÐºÐ¾Ð¼Ð¿ÑÑÑÐµÑÐ½ÑÐ¹ ÐºÐ»Ð°ÑÑÐµÑ
ÑÐµÐ½Ð·ÑÑÐ¸ÑÐ¾Ð²Ð°Ð½Ð½ÑÐµ Ð´Ð°Ð½Ð½ÑÐµ
survival model
machine learning
supercomputer cluster
censored data

description: Ð¢ÐµÐ¼Ð° Ð²ÑÐ¿ÑÑÐºÐ½Ð¾Ð¹ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð¾Ð¹ ÑÐ°Ð±Ð¾ÑÑ: Â«ÐÐ¾Ð²ÑÑÐµÐ½Ð¸Ðµ ÑÐµÐ°Ð»ÑÐ½Ð¾Ð¹ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÐ¸ Ð²ÑÑÐ¾ÐºÐ¾Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½ÑÑ ÐºÐ»Ð°ÑÑÐµÑÐ¾Ð² Ñ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸ÐµÐ¼ Ð¼ÐµÑÐ¾Ð´Ð¾Ð² Ð¼Ð°ÑÐ¸Ð½Ð½Ð¾Ð³Ð¾ Ð¾Ð±ÑÑÐµÐ½Ð¸ÑÂ». Ð Ð°Ð±Ð¾ÑÐ° Ð¿Ð¾ÑÐ²ÑÑÐµÐ½Ð° ÑÐµÑÐµÐ½Ð¸Ñ Ð·Ð°Ð´Ð°ÑÐµ Ð¿Ð¾Ð²ÑÑÐµÐ½Ð¸Ñ ÑÐµÐ°Ð»ÑÐ½Ð¾Ð¹ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÐ¸ Ð³Ð¸Ð±ÑÐ¸Ð´Ð½ÑÑ ÑÑÐ¿ÐµÑÐºÐ¾Ð¼Ð¿ÑÑÑÐµÑÐ½ÑÑ Ð¿Ð»Ð°ÑÑÐ¾ÑÐ¼ (ÐÐ¡ÐÐ), ÑÐ¾ÑÑÐ¾ÑÑÐ¸Ñ Ð¸Ð· ÑÐ°Ð·Ð»Ð¸ÑÐ½ÑÑ ÑÐ¸Ð¿Ð¾Ð² Ð²ÑÑÐ¸ÑÐ»Ð¸ÑÐµÐ»ÑÐ½ÑÑ ÑÐ·Ð»Ð¾Ð² (CPU, GPU, FPGA), ÑÐ°Ð±Ð¾ÑÐ°ÑÑÐ¸Ñ Ð¿Ð¾Ð´ ÑÐ¿ÑÐ°Ð²Ð»ÐµÐ½Ð¸ÐµÐ¼ Ð´Ð¸ÑÐ¿ÐµÑÑÐµÑÐ° Ð²ÑÑÐ¸ÑÐ»Ð¸ÑÐµÐ»ÑÐ½ÑÑ ÑÐµÑÑÑÑÐ¾Ð² ÐºÐ°Ðº ÑÐµÐ½ÑÑÑ ÐºÐ¾Ð»Ð»ÐµÐºÑÐ¸Ð²Ð½Ð¾Ð³Ð¾ Ð¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸Ñ (Ð¦ÐÐ). Ð Ð°Ð·ÑÐ°Ð±Ð°ÑÑÐ²Ð°ÐµÐ¼ÑÐ¹ Ð¿Ð¾Ð´ÑÐ¾Ð´ Ð¾ÑÐ»Ð¸ÑÐ°ÐµÑÑÑ Ð¾Ñ Ð¸Ð·Ð²ÐµÑÑÐ½ÑÑ ÑÐµÑÐµÐ½Ð¸Ð¹, Ð½Ð°Ð»Ð¸ÑÐ¸ÐµÐ¼ ÑÐ¸ÑÑÐµÐ¼Ñ Ð¼Ð°ÑÐ¸Ð½Ð½Ð¾Ð³Ð¾ Ð¾Ð±ÑÑÐµÐ½Ð¸Ñ Ð´Ð¸ÑÐ¿ÐµÑÑÐµÑÐ° SLURM, Ð¿Ð»Ð°Ð½Ð¸ÑÑÑÑÐµÐ³Ð¾ ÑÐµÐºÑÑÑÑ Ð·Ð°Ð³ÑÑÐ·ÐºÑ Ð²ÑÑÐ¸ÑÐ»Ð¸ÑÐµÐ»ÑÐ½ÑÑ ÑÐµÑÑÑÑÐ¾Ð² Ñ ÑÑÐµÑÐ¾Ð¼ Ð½Ðµ ÑÐ¾Ð»ÑÐºÐ¾ Ð·Ð°Ð¿ÑÐ¾ÑÐ¾Ð² Ð¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°ÑÐµÐ»ÐµÐ¹, Ð½Ð¾ Ð¸ Ð¸Ð½ÑÐ¾ÑÐ¼Ð°ÑÐ¸Ð¸ Ð¾Ð± ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÐ¸Ð²Ð½Ð¾ÑÑÐ¸ ÑÐ°Ð±Ð¾ÑÑ ÐÐ¡ÐÐ Ð² Ð¿ÑÐµÐ´ÑÐ´ÑÑÐ¸Ðµ ÑÑÐ°Ð¿Ñ ÑÑÐ½ÐºÑÐ¸Ð¾Ð½Ð¸ÑÐ¾Ð²Ð°Ð½Ð¸Ñ. ÐÐ±ÑÐµÐºÑÐ¾Ð¼ Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ ÑÐ²Ð»ÑÐµÑÑÑ ÑÑÐ¿ÐµÑÐºÐ¾Ð¼Ð¿ÑÑÑÐµÑÐ½ÑÐ¹ ÑÐµÐ½ÑÑ Â«ÐÐ¾Ð»Ð¸ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ð¹Â», ÑÐ°Ð±Ð¾ÑÐ°ÑÑÐµÐ³Ð¾ Ð² ÑÐµÐ¶Ð¸Ð¼Ðµ ÐºÐ¾Ð»Ð»ÐµÐºÑÐ¸Ð²Ð½Ð¾Ð³Ð¾ Ð¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸Ñ. ÐÑÐµÐ´Ð¼ÐµÑÐ¾Ð¼ Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ ÑÐ²Ð»ÑÐµÑÑÑ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼ Ð¾ÑÐµÐ½ÐºÐ¸ Ð½ÐµÐ¾Ð±ÑÐ¾Ð´Ð¸Ð¼Ð¾Ð³Ð¾ Ð²ÑÐµÐ¼ÐµÐ½Ð¸ Ð²ÑÐ¿Ð¾Ð»Ð½ÐµÐ½Ð¸Ñ Ð¿ÑÐ¸ÐºÐ»Ð°Ð´Ð½Ð¾Ð¹ Ð·Ð°Ð´Ð°Ñ Ð¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°ÑÐµÐ»ÐµÐ¹, Ñ ÑÑÐµÑÐ¾Ð¼ Ð¸Ñ ÑÑÐ¿ÐµÑÐ½Ð¾Ð³Ð¾ Ð·Ð°Ð²ÐµÑÑÐµÐ½Ð¸Ñ. Ð¦ÐµÐ»ÑÑ Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ ÑÐ²Ð»ÑÐµÑÑÑ Ð¾Ð¿ÑÐ¸Ð¼Ð¸Ð·Ð°ÑÐ¸Ñ Ð¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð² Ð·Ð°Ð´Ð°Ñ ÑÐ°Ðº, ÑÑÐ¾Ð±Ñ Ð¾Ð±ÐµÑÐ¿ÐµÑÐ¸ÑÑ ÑÑÐ¿ÐµÑÐ½Ð¾Ðµ Ð²ÑÐ¿Ð¾Ð»Ð½ÐµÐ½Ð¸Ðµ Ð½Ð°Ð¸Ð±Ð¾Ð»ÑÑÐµÐ³Ð¾ ÑÐ¸ÑÐ»Ð° Ð¿ÑÐ¸ÐºÐ»Ð°Ð´Ð½ÑÑ Ð·Ð°Ð´Ð°Ñ Ð·Ð° Ð¸Ð½ÑÐµÑÐ²Ð°Ð» Ð²ÑÐµÐ¼ÐµÐ½Ð¸, Ð²ÑÐ´ÐµÐ»ÐµÐ½Ð½ÑÐ¹ Ð´Ð»Ñ ÐºÐ¾Ð½ÐºÑÐµÑÐ½Ð¾Ð¹ Ð·Ð°Ð´Ð°ÑÐ¸. ÐÐ²Ð¾Ð´Ð¸ÑÑÑ Ð°Ð±ÑÑÑÐ°ÐºÑÐ¸Ñ Â«Ð²ÑÐµÐ¼Ñ Ð²ÑÐ¶Ð¸Ð²Ð°Ð½Ð¸Ñ Ð¿ÑÐ¸ÐºÐ»Ð°Ð´Ð½ÑÑ Ð·Ð°Ð´Ð°Ð½Ð¸Ð¹Â», ÐºÐ¾ÑÐ¾ÑÐ°Ñ Ð¿Ð¾Ð·Ð²Ð¾Ð»ÑÐµÑ Ð¾ÑÐµÐ½Ð¸Ð²Ð°ÑÑ ÑÑÑÐµÐºÑÐ¸Ð²Ð½Ð¾ÑÑÑ ÑÐ°Ð±Ð¾ÑÑ Ð´Ð¸ÑÐ¿ÐµÑÑÐµÑÐ° ÐºÐ°Ðº ÑÐ¸ÑÐ»Ð¾ ÑÑÐ¿ÐµÑÐ½Ð¾ Ð·Ð°Ð²ÐµÑÑÐµÐ½Ð½ÑÑ Ð¿ÑÐ¸ÐºÐ»Ð°Ð´Ð½ÑÑ Ð·Ð°Ð´Ð°Ð½Ð¸Ð¹ Ðº Ð¾Ð±ÑÐµÐ¼Ñ ÑÐ¸ÑÐ»Ñ Ð·Ð°Ð´Ð°Ñ Ð¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°ÑÐµÐ»ÐµÐ¹, Ð²ÑÐ¿Ð¾Ð»Ð½ÑÐµÐ¼ÑÑ ÐºÐ»Ð°ÑÑÐµÑÐ¾Ð¼ Ð·Ð° Ð²ÑÐ±ÑÐ°Ð½Ð½ÑÐ¹ Ð¸Ð½ÑÐµÑÐ²Ð°Ð» Ð²ÑÐµÐ¼ÐµÐ½Ð¸.
description: The subject of the graduate qualification work is Â«Increasing real productivity in high-performance clusters using machine learningÂ». The work is dedicated to solving the problem of increasing real productivity of hybrid supercomputing platforms (HSPs), which consist of various types of computing nodes (CPU, GPU, FPGA) managed by a resource manager resource. The developed approach differs from known solutions by incorporating a machine learning system into the SLURM resource manager. This system schedules current resource loads taking into account not only user requests but also information about the performance of the HSP in previous operational stages. The research subject is an algorithm for estimating the required execution time of user task, considering their successful completion. The aim of the study is to optimize task parameters in a way that ensures successful execution of the highest number of task within a specific time interval allocated for each task. The efficiency of the resource manager is determined as the ratio of successfully completed user task to the total number of tasks executed by the cluster within a selected time interval. An abstraction of "survival time of user task"is introduced. To assess the survival function, it is proposed to utilize accumulated statistics from the operation of the HSP and machine learning methods to develop a prediction function for the execution time of new tasks.