Learning anomalies from graph: predicting compute node failures on HPC clusters

Venue: NLDL_org_2025_Conference
Published: 2024-09-09 11:11:51

Abstract: Today, high-performance computing (HPC) systems play a crucial role in advancing artificial intelligence. Nevertheless, the estimated global data center electricity consumption in 2022 was around 1\% of the final global electricity demand. Therefore, as HPC systems advance towards Exascale computing, research is required to ensure their growth is sustainable and environmentally friendly. Data from infrastructure monitoring can be leveraged to predict downtimes, ensure these are treated in time, and increase the overall system's utilization. In this paper, we compare four machine-learning approaches, three of them based on graph embeddings, to predict compute node downtimes. The experiments were performed with data from Marconi 100, a tier-0 production supercomputer at CINECA in Bologna, Italy. Our results show that the machine learning models can accurately predict downtime, surpassing current state-of-the-art models.

Main paper: 
 related work: 
 The size, complexity, and heterogeneous architecture of contemporary HPC systems require introducing machine learning methodologies that support the work of the system administrators [11] . As characterized by Netti et al. [11] , anomaly detection and prediction are among the applications with the most direct impact on the overall availability, and by extension, sustainability [12] , of the HPC system. Consequently, much effort has been invested in building data-driven models for anomaly detection and, later, anomaly prediction and anticipation.Based on the telemetry data, various node failure detection approaches, such as RUAD [13] or PROCTOR [14] , have been proposed. These approaches, based on self-supervised learning, are only capable of recognizing the anomalies and failures that are taking place. They cannot provide failure predictions, allowing the system administrators to manage the system proactively, including intelligent scheduling considering anticipated hardware failures and preventative maintenance. While there has been an effort to create an anomaly anticipation system for HPC compute node failures, such as the one proposed by Borghesi et al. [15] , such systems can anticipate the failure but provide no estimation about the time frame for it. Failure prediction approaches exist for components within the compute node, such as the work of Devesh Tiwari et al. [16] or Yu Liu et al. [17] focus on component failure prediction (disk failure specifically). Besides not being holistic and only covering a part of potential compute node failures, these approaches also have limited prediction windows [18] .The prediction model must include additional information beyond the telemetry data to go beyond component failure prediction or node-level failure detection. The current state-of-the-art approach, GRAAFE [12] , is based on the observation that the additional information about the physical layout of the compute nodes within a computing room aids in the ability to train the compute node anomaly prediction model. This information is encoded as a graph: each compute node is represented as a vertex connected to its nearest neighbors, and node telemetry data is represented for each compute node as a vertex attribute. While different graph topologies have been tested, the optimal graph topology uncovered is a line graph representing a single compute rack in a compute room. Different approaches to anomaly detection with graphs have been applied to other domains and could be considered in HPC environments. Among them we find anomaly detection with graphs approaches that aim to alleviate struc-152 tural distribution shifts and novel techniques for 153 posing the problem as a temporal graph clustering 154 problem [19, 20] . The source of our data is a collection of the Marconi 157 100 supercomputer sensory data, which was gathered 158 by Borghesi et al. and was made publicly accessible 159 at https://zenodo.org/records/7541722 [10] . More 160 specifically, we focused on a subset of the original 161 data stored in the 1.tar distribution file where in-162 formation about sixteen computing nodes in one 163 rack of the system are available. The data is a two-164 dimensional table, where rows represent timestamps 165 taken 15 minutes apart, ranging between March 9 th 166 2020 and September 28 th 2022. This means roughly 167 86 thousand rows for each compute node data. Ap-168 proximately 9 thousand rows have missing values. 169 Columns, on the other hand, contain different sen-170 sor measurements of the system aggregated over the 171 15 minute time interval, which include the power 172 consumption of the fans and CPUs, the tempera-173 ture of the GPUs cores and memory, the voltage of 174 the power supply and many more. To be more pre-175 cise, each sensor measurement is given in 4 columns, 176 which store the minimum and maximum values, the 177 average values, and the standard deviation of the 178 measurement. Two additional columns are provided, 179 one that stores the time information and the other 180 one that annotates anomalies in the data. The last 181 column has integer values, where zero means there 182 was no anomaly, and a value greater than zero sig-183 nals there was an anomaly, which means the system 184 was unavailable then. The timestamp column has 185 date and time as values, and the remaining sensor 186 columns have numerical values. ticated approach. Given a time series of length n, natural visibility heuristics can be applied applied to create a graph G = (V, E), where nodes V represent time series data points and edges E represent visibility relationships between those nodes. Then a deep learning model can be used to transform the graph G into a vector representation H = f embed (G). Choosing k=5 would encode information from each sensor with the same number of columns as the one-hot encoded approach while eventually conveying more information. To this end, we constructed natural visibility graphs for each sensor in a given state, considering the values of up to ten preceding states. In Fig. 2 we show how a sample drawing on how sensor signal was reduced into a sequence of unique values and later to a visibility graph. The Graph2Vec approach was used to transform the graphs into vector embedding representations.Finally, we also created a slightly different graph representation encoding information about the sensors and joined it to the abovementioned visibility graphs to the nodes representing the sensors from which the sensor readings originate (see Fig. 2 ). This way, the random walks would convey information about current and past states and how these relate to particular sensors and sensed aspects (e.g., temperature). These graphs were then turned into an embedding using the Graph2Vec approach.To create natural visibility graph embeddings, we employed a Graph2Vec model trained over all of the visibility graphs, regardless of the sensor. The embedding creation process began with generating tagged random walks over a graph, which served as documents for training the Graph2Vec model. Once the model was trained, new random walks over the graphs were conducted, and these were used to perform inference and obtain a vector from the previously trained model. The final embedding for each graph was obtained by averaging these vec- tors, providing a robust representation of the graph's structure. Since one state produced several natural visibility graphs (for each sensor, there was one natural visibility graph), the resulting embeddings were concatenated into one vector embedding for that state, following the same positional convention we used for the one-hot encoding schema. In addition, we also created the state graph embeddings following the same procedure as with the natural visibility graph embeddings, with the difference being that instead of using the natural visibility graphs, we used the state graphs. Because the state graphs were larger we decided to perform a random walk of length 20 per node.  
experiments: 
 We performed four experiments to understand how different features and graph representations affected the outcomes when predicting compute node downtimes. Our focus on graph representations is rooted in the fact that while much sensor data could be encoded by providing features that capture t n-1 values, graph representation allows the conveying of information regarding the sensor time series and 338 associating it to specific domain knowledge, such 339 as sensor ID and sensor types. Three forecasting 340 horizons were considered, predicting one to three 341 states ahead. We detail them below: 342 Experiment A The purpose of the first exper-343 iment was to establish a baseline, considering the 344 one hot encoded representation of the compute node 345 states. The feature vector size was 65.Experiment B The purpose of the second exper-347 iment was to determine whether a natural visibility 348 graph representation of past sensor values provides 349 additional information that could enhance the mod-350 els performance. The natural visibility graph rep-351 resentation was used to encode sensor data and a 352 Graph2Vec representation to turn those graphs into 353 embeddings. A single Graph2Vec model was used 354 to learn visibility graphs from all the sensors. The 355 decision was made under the assumption that the 356 embedding model would learn best from a higher 357 number and variety of samples than would have 358 learned if exposed only to visibility graphs of a sin-359 gle sensor. To make the results comparable against 360 Experiment A, we created embeddings for each sen-361 sor considering a vector size 5. Therefore, the final 362 feature vector size matched the original one with a 363 size of 65.Experiment C The purpose of the third exper-365 iment was to determine whether the information 366 encoded in experiments A and B was complemen-367 tary and, therefore, joining it would lead to better 368 results. The intuition behind the experiment was 369 that (i) the one-hot encoded representation only had 370 information about the current state but missed the 371 sensor values observed in the near past, and (ii) the 372 visibility graphs only encode the topology of a time 373 series but do not encode information regarding the 374 actual time series values. The feature vector was 375 created by concatenating the hot encoded features 376 with the natural visibility graph embeddings. The 377 feature vectors had a total length of 120 values.Experiment D The purpose of the fourth ex-379 periment was to determine whether the represen-380 tation from Experiment C could be enriched with 381 an additional embedding representation that not 382 only considered the visibility graphs but also encode 383 information about the aspects that they measured 384 and the particular sensor that provided the sensor 385 values. To that end, we created an additional graph 386 representing the sensors, their sensing domain, and 387 the associated natural visibility graphs. This exper-388 iment used the one-hot encoded data, the natural 389 visibility graph embeddings, and the state graph 390  
results and discussion: 
 Table 1 shows the results obtained across the com- We have no explanation to why the graph embeddings alone displayed poor performance and further research is required to reach a better understanding. Among possible reasons we consider the fact that the natural visibility graphs capture only the topological aspects of the time series and any information related to their particular values is lost. This could be amended by adding complementary graph representations such as quantile graphs or ordinal partition graphs. Another reason could be the quality of the Graph2Vec emdbeddings which may depend on how the graph was sampled. A more thorough evaluation of hyperparameters is required to understand how the number of paths and their length affects the outcomes. 
conclusion: 
 Enabling predictive maintenance at HPC and datacenters is critical to minimize downtimes and maximize their utilization. Machine learning models have shown great promise accurately forecasting such events ahead of time. In this research we propose a graph-based approach to predicting compute nodes downtime and evaluate it on public data from Marconi 100 -a tier-0 production supercomputer from CINECA. To that end, we first process the sensor reading data by treating missing values and discretizing them. We used natural visibility graphs to represent time series and include them into graph representations describing compute nodes and their sensors. Such graphs were converted into embed-  
021: 
 The increasing research and deployment of artifi-022 cial intelligence require massive hardware, which 023 has strong implications for HPC and data center 024 energy sustainability and makes efficient utilization 025 of the resources even more critical [1] . Maintaining 026 consistent availability of HPC resources is crucial 027 to avoid negative impact on research and minimize The fact that network architectures are governed by the same organizing principles regardless of the science domain and can provide a unified representation of heterogeneous data is a compelling reason driving research at the intersection of network science and machine learning [7] .The approaches described above leverage sensor readings or logs to predict compute node downtimes. Nevertheless, such approaches miss much contextual information by not being able to include additional points of view, such as the kind of information being monitored or the sensor placement. Such information can be included through a graph representation. E.g., Molan et al. [8, 9] joined sensor readings for each point in time considering the sensors' topological location within a particular rack. Nevertheless, such a representation provides no information on the types of sensors being used. Furthermore, the graph representations provide no context regarding past sensor readings -something that could be relevant to understanding whether we are heading to a particular state. To mitigate these issues, we propose a different graph representation with nodes representing specific sensor types and associated sensors and subgraphs resulting from translating time series into networks following specific heuristics. Such graphs are then encoded into embeddings and used for downstream compute node downtime prediction.In this research, we develop local and global machine-learning models that leverage graph embeddings summarizing domain knowledge regarding sensor types and readings to minimize system unavailability. We trained and evaluated the machine learning models on a subset of the publicly available data from the Marconi 100 supercomputer compiled by Borghesi et al. [10] . The models' performance was evaluated considering the AUC ROC scores. Our results show that the models trained with graph embeddings and sensor reading information performed best. Furthermore, global models exhibited a stronger performance than local ones. While sensor reading data and graph embeddings resulted in a stronger mean performance than using sensor reading data only, the difference remained consistently small across the forecasting horizons.The rest of this paper is structured as follows: Section 2 presents related work, Section 3 introduces the dataset we worked on. Section 4 describes the methodology we followed and Section 5 the exper- 
 
 Appendix: Accounting for all the sensors, there are 354columns, though we chose only a small subset ofthe sensors to focus on. Our point of interest isthe columns with the average values of the mea-191 surements. Furthermore, we selected some columns 192 from which to get our data. These columns include 193 ambient avg, dimm0 temp avg, fan disk power avg, 194 gpu0 core temp avg, gpu0 mem temp avg, 195 p0 io power avg, p0 mem power avg, p0 power avg, 196 ps0 input power avg, ps0 output curre avg, 197 ps0 output volta avg, fan0 0 avg and 198 p0 vdd temp avg. 199 4 Methodology 200 Data preprocessing To develop the features 201 needed for training our model, we followed a se-202 ries of steps to preprocess the data and generate 203 new representations of the information.204 


Decision: The paper presents a comparative analysis of various approaches for predicting issues in high-performance computing (HPC) systems. Strengths of the work include a detailed and insightful explanation of the dataset and a clear overview of the results. A key weakness, however, is the limited discussion on the experimental results, although the proposed methodology is sufficiently innovative. Additionally, the motivation behind the study is not sufficiently clear, and the introduction lacks background information, making it challenging for readers outside the domain to grasp the context and significance of the research.

Following the rebuttal, the authors have significantly improved the paper by adding background information and additional experimental insights. They have also refined their discussion of the results, making it more comprehensive and appropriately detailed.

Considering all comments from the reviewers, as well as the authors' constructive responses, I would recommend acceptance of the paper.

Reviews: review: title: title: The study focuses on predicting compute node failures in HPC clusters with learning form graph approach.
paper_summary: paper_summary: The study compares four machine learning approaches for predicting compute node downtime, three of which are based on graph embeddings, and compares the results.
main_review: main_review: The authors still could not provide the desired academic article summary, especially in the abstract. For example, the contribution of the work is not quantified. Success is expressed in vague words. Again, unfortunately, there is no clear statement in the introduction for a clear contribution to the literature. Despite all these, I can say that it is an acceptable study.
strength_weakness: strength_weakness: Poor writing of the study
The abstract does not reflect the study and does not contain the information required in an academic abstract.
The introduction is poorly written and does not include a paragraph about the contribution of the study.
Failure to explain all the steps taken in the study with good examples.
Poor placement of Figures and Tables in the work
title: title: Lack of novelty and mathematical justifications
paper_summary: paper_summary: The paper compares different approaches in predicting issues in high-performance computing systems. For this, the authors utilize the M100 dataset consisting of telemetry data with known anomalies. In the experiments, the authors compare 4 different approaches to predict future anomalies by using a discretized version of the time series and incorporating a natural visibility graph.
main_review: main_review: I want to thank the authors for their careful answers. While a few of my concerns are addressed, there is still a significant lack of theoretical justification, although I appreciate the empirical work and insights. My biggest concern about the justification of using visibility graphs remains, and I believe a more direct graphical representation (e.g., a (dynamic) Bayesian network or a graphical causal model) would be a much clearer formulation and a more direct modeling of domain knowledge. I strongly recommend looking into related works about graphical causal models as this seems to be a great application for it. While I am willing to increase my score, I cannot give a recommendation for acceptance due to the lack of theoretical work.
strength_weakness: strength_weakness: - Transformation of the data lacks justification (e.g., the discretization)
- Although interesting, it's unclear why natural visibility graphs are used instead of simple lagged versions of columns to incorporate time structures
- Seeing that the model is trained for predicting 1, 2, or 3 steps ahead, it is unclear how this is better than the works discussed in the related work section which seem to be applicable in a similar way
- Lack of comparison with existing approaches in the experiments
- No real novelty aside from the particular approach of data processing and chosen modeling approach