# Construction of a User-Oriented Usability Metric Model for High-Performance Computer Systems

Authors: Jian Sun; Jin Tong
Journal: Journal of Sensors
 (2022)

---

## Abstract

The peak performance of high-performance computers is affected by many factors in practical operation. Scientific programs in different application areas use different algorithms and have different requirements for HPC systems. For example, communication-intensive algorithms require higher network performance of the machine, and computation-intensive algorithms require higher processor performance. In the face of many HPC systems, how to evaluate and select the HPC system with high availability is a problem that must be solved. In this paper, we study how to construct a user-oriented usability metric model for high-performance computing systems, and the experimental results show that the proposed model can be good for the usability of high-performance computing systems and the performance indicators, and actual tests based on high-performance computing are the most authentic, reliable, authentic, and persuasive.


---

## Body

## 1. Introduction

Currently, in terms of computing power alone, high-performance computers are experiencing a relatively rapid development phase. From November 2010, when the TH-1A system topped the TOP500 for the first time on behalf of China, the peak performance was 25.7 trillion times/sec; to June 2016, when the Shenwei-TaihuLight system won the top of the TOP500 list, the peak performance had reached 1.25 trillion times, which is an increase of about 50 times in computational performance [1]. The increase in computing power of high-performance computers undoubtedly has an important impact on the performance of high-performance computing applications but at the same time some typical applications, which put higher demands on the I/O performance of high-performance computers, such as climate modeling research, oil seismic exploration data processing, and gene sequencing data analysis [2]. The increasing I/O demand for I/O-intensive applications represented by these applications has led to an increasingly prominent I/O bottleneck in the storage system, which has become one of the main bottlenecks of high-performance computers [3]. This is because external storage systems mainly consist of disk memory, which has been slow in developing its performance as a hybrid electrical, magnetic, and mechanical device [4].With the development of parallel computing as well as computer hardware and software, HPC is being used in more and more fields, methods of computation and simulation are flourishing, and many research groups, companies, and institutions have their HPC cluster systems [5]. Although these cluster systems are different in scale, their architecture is the same, that is, the whole system consists of several nodes, the simplest cluster can be composed of master nodes and computing nodes, for a slightly larger cluster, the master node may also be subdivided into landing nodes, storage nodes, etc., and the main task of this system is to undertake the computing nodes; each node can be a separate workstation with some independence, the systems of the compute nodes are mirrored to each other, all nodes are interconnected through high-speed networks, the tasks are distributed to each compute node through message passing communication, the advantage of this loosely coupled cluster system that is not a single system mirror is that it is low cost and easy to build and implement in a certain scale, and also, the disadvantage is that the loosely coupled whole system leads to more complex management [6]. There are already some studies on supercomputer performance evaluation [7] and operation monitoring [8], but for evaluation systems, individual vendors have some closed evaluation tools, but the implementation of these tools is mostly not publicly available and often developed only for their software and hardware characteristics, without universality [9].At present, the standard for judging the performance of supercomputers is Linpack (linear system software package). In short, it depends on who runs faster when solving a set of unaryn-th dense linear algebraic equations, so as to evaluate the floating-point performance of high-performance computers. It is only a way to test performance, similar to the familiar iometer and other software, but for different testing fields. The current supercomputer evaluation standard Linpack does not consider the power consumption of supercomputers, but only the running speed of supercomputers. However, now the power consumption of supercomputers has also appeared in the evaluation criteria. People have begun to pay attention to the performance of energy conservation and emission reduction. When we look at the comprehensive strength of a supercomputer, there is also a very important indicator, that is, power consumption.


## 2. Related Work

HPC, a cutting-edge branch of computer science [10], is currently in a rapid development phase. The most representative and convincing aspect of this is undoubtedly the computing performance of top supercomputers. The TOP500 organization selects the top 500 fastest-running supercomputers in the world at that time in June and November every year, and in the TOP500 list released in June 2016, China’s self-developed Shenwei-TaizhouLight supercomputer system [11] was ranked first with a peak performance of 125.436 PFlops, taking over the lead from the 3-year-old TH-2 supercomputer to become the fastest computer running in the world today. Table 1 shows the top 10 supercomputers in the latest TOP500 list, and it can be seen that the sustained performance of the top 10 supercomputers is all above 5 PetaFLOPS.Table 1
Top 10 of TOP500 supercomputer parameters.
NameComputational performance (Rmax/TFlop)Memory capacityFile systemStorage capacity (PB)Sunway TaihuLight93,014.61.3 PBSWGFS20Tianhe-233,862.71 PBLustre/H2FS12.4Titan17,590.0710 PBLustre28Sequoia17,173.21.5 PBLustre53K computer10,510.01.3 PBFEFS40Mira8,586.6786 PBGPFS7.6Trinity8,100.9/Lustre>80Piz Daint6,271.0/Lustre2.5Hazel Hen5,640.2/Lustre12Shaheen II5,537.0/Lustre17.6At the same time, various major HPC research countries have been fully engaged in the development of ExaFLOPS-scale HPCs [12]. Abroad, the United States established the National Strategic Computing Initiative (NSCI) [13] in July 2015 to consolidate the U.S. leadership in the field of high-performance computing and plans to be the first to develop the first tens of billions of supercomputers. The EUs launched at the end of 2013, includes “Towards 10 Gigabit HPC” in the strategic priority area of Future and Emerging Technologies (FET) [14] program. Japan has also accelerated the development of E-class computers, and RIKEN and Fujitsu have decided to develop a new generation of Post-K supercomputers [15]. In China, the national “HPC” key special project was officially launched in 2016, and its overall goal is to break through the core technology of E-class computers have taken the lead in starting the development of the first tens of billions of supercomputing prototype system in China [16]. Undoubtedly, the development level of high-performance computing technology has become an important symbol reflecting the scientific and technological strength and comprehensive national power of a country [17]. With high-performance computing technology and the continuous progress of high-performance computing application technology, the fields of high-performance computing applications have been expanded and the depth has been strengthened [18]. Taking the National Supercomputing Center in Tianjin as an example, the TH-1A high-performance computing system deployed in the center is not only widely used in many fields such as scientific numerical computation, climate modeling research, petroleum, seismic exploration, biomedical development, and industrial design simulation but also plays an increasingly important role in smart cities and intelligent transportation [19].


## 3. Methodology

### 3.1. Usability Evaluation Process Design

First, it is necessary to extract the most influential key factors from many factors as usability evaluation indexes for the existing numerical model’s computing demands on the HPC environment. Secondly, according to the business requirements of the numerical model, the performance usability evaluation procedure set of HPC system in the China bureau industry is established, constraints are set, test topics are formed, tests are conducted according to test methods, evaluation criteria of test results are formulated, qualitative and quantitative evaluations of test results (including physical significance rationality, performance, and function) are made according to the criteria, and thus, comprehensive usability evaluation conclusions are given. In the complete cycle of test evaluation, it is necessary to introduce quality control, life cycle, and version management of the information carriers involved and generated, such as programs, data, software, models, documents, and reports. The overall solution design is shown in Figure1.Figure 1
Performance availability evaluation scheme of HPC system.When evaluating HPC systems, there are four basic performance indicators: system size, system peak, network bandwidth, and network latency, and the basic performance parameters of HPC systems are given in Table2. For the performance evaluation of HPC applications, there are also evaluation indexes such as computational scale, computational time, computational efficiency, and parallel acceleration ratio.Table 2
Basic performance parameters of HPC system.
Performance parametersSymbolsMeaningSystem sizenNumber of processorsSystem peakRpeakn×singleprocessorpeak generally refers to the floating point peak, Gflops, TflopsNetwork bandwidthrLong message communication rate of the network, Mbit/s, Gbit/sNetwork latencyto,LTime to transmit 0 bytes or single words, us, msThe basic parameters of high-performance computer systems, often do not answer the user’s concerns. Moreover, due to the differences in processor structure, storage hierarchy, and interconnection network topology, it is really difficult to characterize the comprehensive performance of the system by a single performance index such as system size, system peak, network bandwidth, and latency, so it is necessary to further make a comprehensive availability metric of the HPC system. The logical relationship of the metric modules is shown in Figure2.Figure 2
Sequence of detection module.According to the definition of availability, it is known that availability can be calculated by the following formula.(1)Availability=MTTFMTTF+MTTR.HPC providers claim to provide high availability, high performance, and low-cost services, but users are unable to evaluate the availability of these services. Therefore, this paper proposes an end-user-oriented usability evaluation method for high-performance computing systems to address this problem. MTTF uses the parameters provided by high-performance computing service providers, and MTTR is replaced by the startup time of virtual machines.


### 3.2. Virtual Machine Startup Time Metric

Figure3 illustrates the boot process of a Linux physical machine. The boot process of a physical machine is divided into five main stages: (1) loading the BIOS. When the computer is powered on, the computer will first load the BIOS information, which contains CPU-related information, device boot sequence, hard disk and memory information, etc. (2) Read the MBR, which is the Master Boot Record (MBR), 512 bytes in size, and after the system finds the MBR of the hard disk specified by B10S, it will copy it to the physical memory where the 0x7c00 address is located. (3) Boot Loader. The Boot Loader is a small program that runs before the kernel is run. This small program initializes the hardware devices and establishes the memory space mapping map. (4) Load the kernel. According to the path set by grub where the kernel image is located, the system reads the memory image and performs the decompression operation. (5) Initialization. The system places the decompressed kernel in memory and calls the start_kemel() function to start a series of initialization functions and initialize various devices to complete the establishment of the Linux kernel environment.Figure 3
The boot process of a physical machine.The virtual machine does not have its various resources preallocated before the user makes a request, so the virtual machine startup process also includes the allocation of resources. The allocation of resources requires multiple modules of the HPC platform to work together, and the startup time varies depending on the platform. Take Openstack as an example, starting an instance in Openstack Nova requires multiple components such as API service, cloud controller, scheduling, compute service, and network controller to work together to complete. Nova is one of the components in OpenStack. As a computing service of the cloud platform, Nova provides a template. The Nova component is used to schedule and manage the computing resources of the entire cloud platform, that is, to provide cloud computing services. The importance of the Nova component is also destined to be the most complex component in OpenStack. The Nova service has multiple subservices, and the self-service communicates through RPC. Nova subservice concept is as follows:(1)
Nova API Service-Installed on the Controller Node. Accepting and corresponding requests sent by the client, Nova API is responsible for receiving and responding to end-user managed virtual machines and cloud hard disk requests. That is to say, I want to create a virtual machine in OpenStack (the creation of the virtual machine is finally completed in Nova), and the request I send is received by Nova API and sent to Nova, and then, the next specific operation is performed. Nova API is the entrance to the entire Nova. It receives user requests and sends instructions to the message queue, and the corresponding service executes related instruction messages. It provides OpenStack API, Amazon EC2 API, and administrator control API(2)
Nova API-Metadata Service. It accepts requests from instance metadata. This service usually runs in the host-to-host mode with the Nova-network service, which means that it will only be used by multiple Nova nodes(3)
Nova Compute-Installed on the Compute Node. It is the core service in the Nova component; it implements the function of managing virtual machines and realizes the creation, start, pause, shutdown, and deletion on the compute node virtual machine(4)
Nova Scheduler. It mainly plays a scheduling role. If there are multiple Nova computing nodes, when a user initiates a request to create a virtual machine, the Nova scheduler will decide to place the virtual machine creation on that computing node(5)
Nova Conductor. It mainly provides data query function and provides interactive data between Nova compute and database, just to prevent the database from being insecure after Nova compute is attacked, so Nova conductor is needed to scheduleAs shown in Figure4, the API service receives a run_instance command request and forwards it to the cloud controller, where it mainly performs authentication of permissions, etc. After the authentication is passed, the cloud controller sends the received command to the scheduler, which follows a certain scheduling policy and specifies a compute node to start an instance. The compute service node is mainly used to manage the instance startup and termination, etc. At this time, the virtual machine also needs a fixed IP for normal startup; therefore, the compute service component will send a message to the network controller requesting the assignment of a fixed IP. With these conditions in place, the process required for the traditional system startup will be executed within the compute service component to finally complete the startup of the virtual machine.Figure 4
Startup process of virtual machine.


### 3.3. Improved User-Oriented Availability Assessment Method for HPC Systems

Server manufacturers typically provide the Mean Time Between Failures (MTTFs) of servers in their product specifications. The failure of a single physical machine causes all its virtual machines to be unavailable, so the MTTF of a single virtual machine is related to the degree of virtualization of the physical machine, i.e., to the number of virtual machines running simultaneously on a single physical machine. Assume that each physical machine in the HPC platform runsk virtual machines on average. The MTTF of a virtual machine is 1/k times the MTTF of a physical machine, i.e.,
(2)MTTFvn=1k∗MTTFpn,whereMTTFpn is the MTTF of the physical server, MTTFvn is the MTTF of the virtual machine, and k is the average number of virtual machines started on a single physical machine. Then, the availability of a single virtual machine is calculated as follows.
(3)A=MTTFvnMTTFvn+MTTRvn.The user-oriented evaluation method of HPC system availability is based on the assumption that the number of virtual machines running on each physical machine in the cluster is the same. However, in reality, the virtual machines in the cluster are dynamically migrated to maintain load balancing, so the above end-user-oriented evaluation method cannot estimate the availability of high-performance computing systems well. Therefore, this paper proposes an HPC system availability evaluation model for HPC server providers based on observing the types and frequencies of server failures for a long time and then calculating them statistically.As shown in Figure5, the MTTF is defined as
(4)MTTF=∑downtime‐uptimefailuretimes.Figure 5
Definition of the average time before failure.Failure time is the time interval between the last time the device returns to normal (uptime) and the moment when the device fails this time (downtime). According to the definition of MTTF, the model of MTTF is calculated according to the following statistics. Assume that the data collection matrix of MTTF of the physical machine is(5)PM=n11n12⋯n1mn21n22⋯n2m⋮⋮⋮⋮nn1nn2⋯nm,wheren denotes that there are n physical machines in the HPC platform; m denotes that there are m time intervals when data are collected, i.e., t1,t2,⋯,tm,nij denotes the total number of failures of physical machine i in tj time periods. Physical machine is the name of physical computer relative to virtual machine. Physical machines provide a hardware environment for virtual machines, sometimes referred to as “host” or “hosts.” The computer we usually see is a physical machine. The physical server is also called an independent server. As the name suggests, it is a real physical server lying in the computer room. The independent server is selected to meet the needs of customers. Because the clients are independent on the server, they are granted a lot of disk space. The bandwidth speed depends on the way the client manages the server. Reloading or upgrading the server will be very slow, However, due to the need to provide highly reliable services, it requires high processing capacity, stability, reliability, security, scalability, manageability, and so on. Physical server is actually a high-performance, high availability computer that provides professional services for terminal clients on the network. Server is just a general term. There are different server leases according to different application fields. Advantages: (1) It has remote operation permission. (2) It has high stability and speed and is suitable for large websites. (3) Strong protection ability, sufficient bandwidth, and high configuration. Assume that the matrix N of the number of virtual machines started at different time intervals on each physical machine is
(6)N=k11k12⋯k1mk21k22⋯k2m⋮⋮⋮⋮kn1kn2⋯kmm,wherekij denotes the number of VMs started by physical machine i in tj time period. Then, the MTTF of the virtual machine is
(7)VM=PM.∗N=n11n12⋯n1mn21n22⋯n2m⋮⋮⋮⋮nn1nn2⋯nrm·k11k12⋯k1mk21k22⋯k2m⋮⋮⋮⋮kn1kn2⋯knm=n11k11n12k12⋯n1mk1mn21k21n22k22⋯n2mk2m⋮⋮⋮⋮nn1kn1nn2kn2⋯nrmkrm.Then, according to the definition of MTTF, i.e., equation (4), the MTTF of a single physical machine is
(8)MTTFpi=∑k=1mtk∑j=1mnij,wherek denotes the period number, from 1 to m; j denotes the number of times the physical machine i fails in the period j, from 1 to m; 11 denotes the MTTF of the physical machine i. The MTTF of the physical machine cluster is as follows:
(9)MTTFp=∑k=1mtk∑i=1n∑j=1mnij,wherei denotes the physical machine number; MTTFp denotes the MTTF of the physical machine cluster. The MTTF of the virtual machine cluster is as follows:
(10)MTTFv=∑k=1mtk∑i=1n∑j=1mnijkij,whereMTTFv denotes the MTTF of the virtual machine cluster. From equation (9), we have
(11)∑k=1mtk=∑i=1n∑j=1mnij∗MTTFp.Substituting equation (11) into equation (10) yields
(12)MTTFV=∑k=1mtk∑i=1n∑j=1mnijkij=∑i=1n∑j=1mnij∗MTTFP∑i=1n∑j=1mnijkij=∑i=1n∑j=1mnij∑i=1n∑j=1mnijkij∗MTTFP.Therefore, the availability of the virtual machine is calculated as follows.(13)A=MTTFvMTTFv+MTTRv.


## 3.1. Usability Evaluation Process Design

First, it is necessary to extract the most influential key factors from many factors as usability evaluation indexes for the existing numerical model’s computing demands on the HPC environment. Secondly, according to the business requirements of the numerical model, the performance usability evaluation procedure set of HPC system in the China bureau industry is established, constraints are set, test topics are formed, tests are conducted according to test methods, evaluation criteria of test results are formulated, qualitative and quantitative evaluations of test results (including physical significance rationality, performance, and function) are made according to the criteria, and thus, comprehensive usability evaluation conclusions are given. In the complete cycle of test evaluation, it is necessary to introduce quality control, life cycle, and version management of the information carriers involved and generated, such as programs, data, software, models, documents, and reports. The overall solution design is shown in Figure1.Figure 1
Performance availability evaluation scheme of HPC system.When evaluating HPC systems, there are four basic performance indicators: system size, system peak, network bandwidth, and network latency, and the basic performance parameters of HPC systems are given in Table2. For the performance evaluation of HPC applications, there are also evaluation indexes such as computational scale, computational time, computational efficiency, and parallel acceleration ratio.Table 2
Basic performance parameters of HPC system.
Performance parametersSymbolsMeaningSystem sizenNumber of processorsSystem peakRpeakn×singleprocessorpeak generally refers to the floating point peak, Gflops, TflopsNetwork bandwidthrLong message communication rate of the network, Mbit/s, Gbit/sNetwork latencyto,LTime to transmit 0 bytes or single words, us, msThe basic parameters of high-performance computer systems, often do not answer the user’s concerns. Moreover, due to the differences in processor structure, storage hierarchy, and interconnection network topology, it is really difficult to characterize the comprehensive performance of the system by a single performance index such as system size, system peak, network bandwidth, and latency, so it is necessary to further make a comprehensive availability metric of the HPC system. The logical relationship of the metric modules is shown in Figure2.Figure 2
Sequence of detection module.According to the definition of availability, it is known that availability can be calculated by the following formula.(1)Availability=MTTFMTTF+MTTR.HPC providers claim to provide high availability, high performance, and low-cost services, but users are unable to evaluate the availability of these services. Therefore, this paper proposes an end-user-oriented usability evaluation method for high-performance computing systems to address this problem. MTTF uses the parameters provided by high-performance computing service providers, and MTTR is replaced by the startup time of virtual machines.


## 3.2. Virtual Machine Startup Time Metric

Figure3 illustrates the boot process of a Linux physical machine. The boot process of a physical machine is divided into five main stages: (1) loading the BIOS. When the computer is powered on, the computer will first load the BIOS information, which contains CPU-related information, device boot sequence, hard disk and memory information, etc. (2) Read the MBR, which is the Master Boot Record (MBR), 512 bytes in size, and after the system finds the MBR of the hard disk specified by B10S, it will copy it to the physical memory where the 0x7c00 address is located. (3) Boot Loader. The Boot Loader is a small program that runs before the kernel is run. This small program initializes the hardware devices and establishes the memory space mapping map. (4) Load the kernel. According to the path set by grub where the kernel image is located, the system reads the memory image and performs the decompression operation. (5) Initialization. The system places the decompressed kernel in memory and calls the start_kemel() function to start a series of initialization functions and initialize various devices to complete the establishment of the Linux kernel environment.Figure 3
The boot process of a physical machine.The virtual machine does not have its various resources preallocated before the user makes a request, so the virtual machine startup process also includes the allocation of resources. The allocation of resources requires multiple modules of the HPC platform to work together, and the startup time varies depending on the platform. Take Openstack as an example, starting an instance in Openstack Nova requires multiple components such as API service, cloud controller, scheduling, compute service, and network controller to work together to complete. Nova is one of the components in OpenStack. As a computing service of the cloud platform, Nova provides a template. The Nova component is used to schedule and manage the computing resources of the entire cloud platform, that is, to provide cloud computing services. The importance of the Nova component is also destined to be the most complex component in OpenStack. The Nova service has multiple subservices, and the self-service communicates through RPC. Nova subservice concept is as follows:(1)
Nova API Service-Installed on the Controller Node. Accepting and corresponding requests sent by the client, Nova API is responsible for receiving and responding to end-user managed virtual machines and cloud hard disk requests. That is to say, I want to create a virtual machine in OpenStack (the creation of the virtual machine is finally completed in Nova), and the request I send is received by Nova API and sent to Nova, and then, the next specific operation is performed. Nova API is the entrance to the entire Nova. It receives user requests and sends instructions to the message queue, and the corresponding service executes related instruction messages. It provides OpenStack API, Amazon EC2 API, and administrator control API(2)
Nova API-Metadata Service. It accepts requests from instance metadata. This service usually runs in the host-to-host mode with the Nova-network service, which means that it will only be used by multiple Nova nodes(3)
Nova Compute-Installed on the Compute Node. It is the core service in the Nova component; it implements the function of managing virtual machines and realizes the creation, start, pause, shutdown, and deletion on the compute node virtual machine(4)
Nova Scheduler. It mainly plays a scheduling role. If there are multiple Nova computing nodes, when a user initiates a request to create a virtual machine, the Nova scheduler will decide to place the virtual machine creation on that computing node(5)
Nova Conductor. It mainly provides data query function and provides interactive data between Nova compute and database, just to prevent the database from being insecure after Nova compute is attacked, so Nova conductor is needed to scheduleAs shown in Figure4, the API service receives a run_instance command request and forwards it to the cloud controller, where it mainly performs authentication of permissions, etc. After the authentication is passed, the cloud controller sends the received command to the scheduler, which follows a certain scheduling policy and specifies a compute node to start an instance. The compute service node is mainly used to manage the instance startup and termination, etc. At this time, the virtual machine also needs a fixed IP for normal startup; therefore, the compute service component will send a message to the network controller requesting the assignment of a fixed IP. With these conditions in place, the process required for the traditional system startup will be executed within the compute service component to finally complete the startup of the virtual machine.Figure 4
Startup process of virtual machine.


## 3.3. Improved User-Oriented Availability Assessment Method for HPC Systems

Server manufacturers typically provide the Mean Time Between Failures (MTTFs) of servers in their product specifications. The failure of a single physical machine causes all its virtual machines to be unavailable, so the MTTF of a single virtual machine is related to the degree of virtualization of the physical machine, i.e., to the number of virtual machines running simultaneously on a single physical machine. Assume that each physical machine in the HPC platform runsk virtual machines on average. The MTTF of a virtual machine is 1/k times the MTTF of a physical machine, i.e.,
(2)MTTFvn=1k∗MTTFpn,whereMTTFpn is the MTTF of the physical server, MTTFvn is the MTTF of the virtual machine, and k is the average number of virtual machines started on a single physical machine. Then, the availability of a single virtual machine is calculated as follows.
(3)A=MTTFvnMTTFvn+MTTRvn.The user-oriented evaluation method of HPC system availability is based on the assumption that the number of virtual machines running on each physical machine in the cluster is the same. However, in reality, the virtual machines in the cluster are dynamically migrated to maintain load balancing, so the above end-user-oriented evaluation method cannot estimate the availability of high-performance computing systems well. Therefore, this paper proposes an HPC system availability evaluation model for HPC server providers based on observing the types and frequencies of server failures for a long time and then calculating them statistically.As shown in Figure5, the MTTF is defined as
(4)MTTF=∑downtime‐uptimefailuretimes.Figure 5
Definition of the average time before failure.Failure time is the time interval between the last time the device returns to normal (uptime) and the moment when the device fails this time (downtime). According to the definition of MTTF, the model of MTTF is calculated according to the following statistics. Assume that the data collection matrix of MTTF of the physical machine is(5)PM=n11n12⋯n1mn21n22⋯n2m⋮⋮⋮⋮nn1nn2⋯nm,wheren denotes that there are n physical machines in the HPC platform; m denotes that there are m time intervals when data are collected, i.e., t1,t2,⋯,tm,nij denotes the total number of failures of physical machine i in tj time periods. Physical machine is the name of physical computer relative to virtual machine. Physical machines provide a hardware environment for virtual machines, sometimes referred to as “host” or “hosts.” The computer we usually see is a physical machine. The physical server is also called an independent server. As the name suggests, it is a real physical server lying in the computer room. The independent server is selected to meet the needs of customers. Because the clients are independent on the server, they are granted a lot of disk space. The bandwidth speed depends on the way the client manages the server. Reloading or upgrading the server will be very slow, However, due to the need to provide highly reliable services, it requires high processing capacity, stability, reliability, security, scalability, manageability, and so on. Physical server is actually a high-performance, high availability computer that provides professional services for terminal clients on the network. Server is just a general term. There are different server leases according to different application fields. Advantages: (1) It has remote operation permission. (2) It has high stability and speed and is suitable for large websites. (3) Strong protection ability, sufficient bandwidth, and high configuration. Assume that the matrix N of the number of virtual machines started at different time intervals on each physical machine is
(6)N=k11k12⋯k1mk21k22⋯k2m⋮⋮⋮⋮kn1kn2⋯kmm,wherekij denotes the number of VMs started by physical machine i in tj time period. Then, the MTTF of the virtual machine is
(7)VM=PM.∗N=n11n12⋯n1mn21n22⋯n2m⋮⋮⋮⋮nn1nn2⋯nrm·k11k12⋯k1mk21k22⋯k2m⋮⋮⋮⋮kn1kn2⋯knm=n11k11n12k12⋯n1mk1mn21k21n22k22⋯n2mk2m⋮⋮⋮⋮nn1kn1nn2kn2⋯nrmkrm.Then, according to the definition of MTTF, i.e., equation (4), the MTTF of a single physical machine is
(8)MTTFpi=∑k=1mtk∑j=1mnij,wherek denotes the period number, from 1 to m; j denotes the number of times the physical machine i fails in the period j, from 1 to m; 11 denotes the MTTF of the physical machine i. The MTTF of the physical machine cluster is as follows:
(9)MTTFp=∑k=1mtk∑i=1n∑j=1mnij,wherei denotes the physical machine number; MTTFp denotes the MTTF of the physical machine cluster. The MTTF of the virtual machine cluster is as follows:
(10)MTTFv=∑k=1mtk∑i=1n∑j=1mnijkij,whereMTTFv denotes the MTTF of the virtual machine cluster. From equation (9), we have
(11)∑k=1mtk=∑i=1n∑j=1mnij∗MTTFp.Substituting equation (11) into equation (10) yields
(12)MTTFV=∑k=1mtk∑i=1n∑j=1mnijkij=∑i=1n∑j=1mnij∗MTTFP∑i=1n∑j=1mnijkij=∑i=1n∑j=1mnij∑i=1n∑j=1mnijkij∗MTTFP.Therefore, the availability of the virtual machine is calculated as follows.(13)A=MTTFvMTTFv+MTTRv.


## 4. Experiments

### 4.1. Measurement Time Analysis

As shown in Figure6, company B’s HPC system takes the least amount of wall clock time to run the actual business application model provided, but it does not have the highest theoretical peak speed. In contrast, company C has a high theoretical peak speed but takes longer to run the actual business application.Figure 6
Comparison of actual service runtime and peak speed.


### 4.2. Acceleration Ratio Analysis

The acceleration ratio is the main evaluation index of parallel processing system performance. The ideal speedup ratio is linear, but the actual speedup ratio is gradually lower than the linear value. The efficiency obtained by an ideal application is independent of the system size and remains constant. However, the efficiency of the actual application is gradually decreasing with the increase of the scale and cannot be kept constant. In this paper, we plot the acceleration ratio efficiency of the items involved in the scalability test item by item. From this, it can be visually found that there is a certain gap between HPC products in terms of HPC parallel performance. Figure7 shows the acceleration ratio of each company’s HPC system when testing the WRF mode quadruple nesting. It can be found that company A has the best scalability. Company C is less scalable, and the acceleration ratio does not improve when the process is above 320, which does not meet the test requirements. Therefore, the acceleration ratio performance index is still an important index for HPC system performance evaluation.Figure 7
Acceleration ratio comparison graph.To further investigate the system availability, this paper compares the user-oriented availability at different checkpoint intervals for the liquid-cooled case as shown in Figure8(a) and the air-cooled case as shown in Figure 8(b). The highest value of availability in the liquid-cooled case occurs between 2 and 3 hours of checkpoint interval, and the highest value of availability in the air-cooled case occurs between 1 and 2 hours. When the checkpoint retention interval is very small, frequent checkpoint settings will have a large overhead; when the checkpoint retention interval is too large, there will be a large job execution overhead after the failure of the checkpoint. For the above reasons, there is a better balance of checkpoint interval settings for systems of different sizes and working states.Figure 8
Variation of availability with checkpoint interval for systems with different node sizes.
(a)
The case of liquid cooling(b)
The case of air coolingThe user-oriented availability for different task lengths in the liquid-cooled case is shown in Figure9(a) and in the air-cooled case is shown in Figure 9(b). The results show that the availability decreases slightly with the growth of the number of nodes under the same conditions, but the magnitude is small. The different cooling methods have a greater impact on the system’s availability. In addition, the system availability decreases with the growth of task length because the strategy of example 2 is to abandon the running task after node failure, and the longer the task needs to be executed, the greater the loss of the interrupted task after a system failure.Figure 9
Variation of availability with task length for example 2 systems with different node sizes.
(a)
Case with liquid cooling(b)
Case with air coolingFurther experiments were conducted to validate the usability evaluation method for HPC systems, because the following limitations exist in the validation of the availability evaluation method for HPC server providers: (1) the number of failures of the server cluster and the number of running virtual machines on a single server at different periods need to be observed for a long time and counted; (2) the observation period is long, at least one year is needed; (3) if the experimental simulation is used, it is necessary to artificially inject failures regularly for observation and counting, which destroys the original environment of availability, and the observed data are not realistic. Therefore, this paper only validates the usability evaluation method for end-user-oriented high-performance computing systems, MTTF uses the parameters provided by the server provider, and MTTR uses the measured virtual start-up time instead.This verification experiment uses the Bootchart and BootTimeScript tools to measure the startup time of the virtual machine. The Bootchart tool records the startup time between the time when the virtual machine has allocated resources and after all services have started normally, while the BootTimeScript tool records the startup time between the time when the user requests resources and after the network services have started, and the relationship between these two tools is used to measure the startup time of the virtual machine. The experiments were tested five times with different methods, and the final results were averaged. The results of the startup time measurements for different configurations of virtual machines provided by the Amazon platform are listed in Table3.Table 3
Startup time of different configurations of Amazon virtual machines.
Testing toolsBootchart(s)BootTimeScript(s)Virtual machine nameaws1aws2aws3aws4aws1aws2aws3aws4Test 151.8851.9650.6650.4471.1568.7552.6872.23Test 252.0651.8750.7550.2865.5458.4155.7861.47Test 352.2251.9350.6751.2968.1464.5652.4165.86Average value52.0151.9150.7550.5568.5262.6156.7767.97From Table3, we can get that the VM startup time measured by the Bootchart tool gradually decreases as the VM configuration is upgraded, and the startup time measured by the BootTimeScript tool fluctuates more and is not regular. The main reason is that BootTimeScript records the time of resource allocation, and the amount of user access at a certain moment on the public high-performance computing platform is variable, and when users request a virtual machine, they first put the request into the queue for waiting, so the time of virtual machine resource allocation is related to the number of users waiting in the queue, so the virtual machine startup time measured by the BootTimeScript tool fluctuates greatly. Analyzing the Bootchart recording process, we can get the average time of network service opening is 6.8 s, and the fluctuation of resource allocation time ranges from 45 s to 70 s. Because it is not feasible to evaluate MTTF from the user’s perspective using the evaluation method for HPC service providers, the method of MTTF using the parameters provided by the server provider is adopted. Assuming a single server MTTF of 100,000 hours, the public HPC platform Amazon and the private HPC platform built in this lab run 1000 virtual machines per server on average, respectively, from equation (2).
(14)MTTFOpenStack=11000×10millionhours=100hours,(15)MTTFaus=11000×10millionhours=100hours.The usability evaluation model of the end-user-oriented HPC system is obtained from equation (3) as
(16)A=MTTFvmMTTFvm+MTTRvm.The availability of the HPC platform is 99.99%, and the Amazon EC2 service level agreement promises 99.95% availability for each Amazon EC2 region. The availability of both the platform and the Amazon HPC platform is evaluated using the end-user-oriented evaluation method to exceed 99.99%, which is higher than the standard of both the HPC platform and the Amazon HPC platform, respectively, and therefore, the commitment of the cloud provider and the Amazon cloud provider is up to standard.


## 4.1. Measurement Time Analysis

As shown in Figure6, company B’s HPC system takes the least amount of wall clock time to run the actual business application model provided, but it does not have the highest theoretical peak speed. In contrast, company C has a high theoretical peak speed but takes longer to run the actual business application.Figure 6
Comparison of actual service runtime and peak speed.


## 4.2. Acceleration Ratio Analysis

The acceleration ratio is the main evaluation index of parallel processing system performance. The ideal speedup ratio is linear, but the actual speedup ratio is gradually lower than the linear value. The efficiency obtained by an ideal application is independent of the system size and remains constant. However, the efficiency of the actual application is gradually decreasing with the increase of the scale and cannot be kept constant. In this paper, we plot the acceleration ratio efficiency of the items involved in the scalability test item by item. From this, it can be visually found that there is a certain gap between HPC products in terms of HPC parallel performance. Figure7 shows the acceleration ratio of each company’s HPC system when testing the WRF mode quadruple nesting. It can be found that company A has the best scalability. Company C is less scalable, and the acceleration ratio does not improve when the process is above 320, which does not meet the test requirements. Therefore, the acceleration ratio performance index is still an important index for HPC system performance evaluation.Figure 7
Acceleration ratio comparison graph.To further investigate the system availability, this paper compares the user-oriented availability at different checkpoint intervals for the liquid-cooled case as shown in Figure8(a) and the air-cooled case as shown in Figure 8(b). The highest value of availability in the liquid-cooled case occurs between 2 and 3 hours of checkpoint interval, and the highest value of availability in the air-cooled case occurs between 1 and 2 hours. When the checkpoint retention interval is very small, frequent checkpoint settings will have a large overhead; when the checkpoint retention interval is too large, there will be a large job execution overhead after the failure of the checkpoint. For the above reasons, there is a better balance of checkpoint interval settings for systems of different sizes and working states.Figure 8
Variation of availability with checkpoint interval for systems with different node sizes.
(a)
The case of liquid cooling(b)
The case of air coolingThe user-oriented availability for different task lengths in the liquid-cooled case is shown in Figure9(a) and in the air-cooled case is shown in Figure 9(b). The results show that the availability decreases slightly with the growth of the number of nodes under the same conditions, but the magnitude is small. The different cooling methods have a greater impact on the system’s availability. In addition, the system availability decreases with the growth of task length because the strategy of example 2 is to abandon the running task after node failure, and the longer the task needs to be executed, the greater the loss of the interrupted task after a system failure.Figure 9
Variation of availability with task length for example 2 systems with different node sizes.
(a)
Case with liquid cooling(b)
Case with air coolingFurther experiments were conducted to validate the usability evaluation method for HPC systems, because the following limitations exist in the validation of the availability evaluation method for HPC server providers: (1) the number of failures of the server cluster and the number of running virtual machines on a single server at different periods need to be observed for a long time and counted; (2) the observation period is long, at least one year is needed; (3) if the experimental simulation is used, it is necessary to artificially inject failures regularly for observation and counting, which destroys the original environment of availability, and the observed data are not realistic. Therefore, this paper only validates the usability evaluation method for end-user-oriented high-performance computing systems, MTTF uses the parameters provided by the server provider, and MTTR uses the measured virtual start-up time instead.This verification experiment uses the Bootchart and BootTimeScript tools to measure the startup time of the virtual machine. The Bootchart tool records the startup time between the time when the virtual machine has allocated resources and after all services have started normally, while the BootTimeScript tool records the startup time between the time when the user requests resources and after the network services have started, and the relationship between these two tools is used to measure the startup time of the virtual machine. The experiments were tested five times with different methods, and the final results were averaged. The results of the startup time measurements for different configurations of virtual machines provided by the Amazon platform are listed in Table3.Table 3
Startup time of different configurations of Amazon virtual machines.
Testing toolsBootchart(s)BootTimeScript(s)Virtual machine nameaws1aws2aws3aws4aws1aws2aws3aws4Test 151.8851.9650.6650.4471.1568.7552.6872.23Test 252.0651.8750.7550.2865.5458.4155.7861.47Test 352.2251.9350.6751.2968.1464.5652.4165.86Average value52.0151.9150.7550.5568.5262.6156.7767.97From Table3, we can get that the VM startup time measured by the Bootchart tool gradually decreases as the VM configuration is upgraded, and the startup time measured by the BootTimeScript tool fluctuates more and is not regular. The main reason is that BootTimeScript records the time of resource allocation, and the amount of user access at a certain moment on the public high-performance computing platform is variable, and when users request a virtual machine, they first put the request into the queue for waiting, so the time of virtual machine resource allocation is related to the number of users waiting in the queue, so the virtual machine startup time measured by the BootTimeScript tool fluctuates greatly. Analyzing the Bootchart recording process, we can get the average time of network service opening is 6.8 s, and the fluctuation of resource allocation time ranges from 45 s to 70 s. Because it is not feasible to evaluate MTTF from the user’s perspective using the evaluation method for HPC service providers, the method of MTTF using the parameters provided by the server provider is adopted. Assuming a single server MTTF of 100,000 hours, the public HPC platform Amazon and the private HPC platform built in this lab run 1000 virtual machines per server on average, respectively, from equation (2).
(14)MTTFOpenStack=11000×10millionhours=100hours,(15)MTTFaus=11000×10millionhours=100hours.The usability evaluation model of the end-user-oriented HPC system is obtained from equation (3) as
(16)A=MTTFvmMTTFvm+MTTRvm.The availability of the HPC platform is 99.99%, and the Amazon EC2 service level agreement promises 99.95% availability for each Amazon EC2 region. The availability of both the platform and the Amazon HPC platform is evaluated using the end-user-oriented evaluation method to exceed 99.99%, which is higher than the standard of both the HPC platform and the Amazon HPC platform, respectively, and therefore, the commitment of the cloud provider and the Amazon cloud provider is up to standard.


## 5. Conclusion

The analysis of applications can provide an understanding of the characteristics of the programs in the application area and the different requirements of the programs for the system components and provide a basis for users to evaluate and purchase high-performance computer systems. From an economic point of view, it can obtain the most appropriate system through the minimum funds. The practical experience shows that the fluctuation of resource allocation time ranges from 45 s to 70 s, the availability with checkpoint interval for systems with different node sizes improved by 12%, and the availability of the HPC platform is 99.99%. In the decision-making of purchasing efficient computing systems, the most important criterion is whether they meet the needs of users.



---

*Source: 4100639-2022-08-19.xml*