English:
[0001] High Performance Computing (HPC) and cluster computing involve connecting individual computing nodes to create a distributed system capable of solving complex problems. These nodes may be individual desktop computers, servers, processors or similar machines capable of hosting an individual instance of computation. More specifically, these nodes are constructed out of hardware components including, but not limited to, processors, volatile memory (RAM), magnetic storage drives, mainboards, network interface cards, and the like.
[0003] Features and advantages of embodiments of the present invention will become apparent from the appended claims, the following detailed description of one or more example embodiments, and the corresponding figures, in which:
[0004] Figure 1 includes a schematic diagram of a cluster for HPC in an embodiment of the invention.
[0005] Figure 2 includes a schematic diagram of a compute node in an embodiment of the invention.
[0006] Figure 3 includes a block diagram of volatile memory of a compute node in an embodiment of the invention.
[0007] Figures 4-5 includes flow diagrams for a first phase of checkpoint processing in embodiments of the invention.
[0008] Figures 6-7 include flow diagrams for a second phase of checkpoint processing in embodiments of the invention.
[0009] In the following description, numerous specific details are set forth but embodiments of the invention may be practiced without these specific details. Well-known circuits, structures and techniques have not been shown in detail to avoid obscuring an understanding of this description. "An embodiment", "various embodiments" and the like indicate embodiment(s) so described may include particular features, structures, or characteristics, but not every embodiment necessarily includes the particular features, structures, or characteristics. Some embodiments may have some, all, or none of the features described for other embodiments. "First", "second", "third" and the like describe a common object and indicate different instances of like objects are being referred to. Such adjectives do not imply objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner. "Connected" may indicate elements are in direct physical or electrical contact with each other and "coupled" may indicate elements co-operate or interact with each other, but they may or may not be in direct physical or electrical contact. Also, while similar or same numbers may be used to designate same or similar parts in different figures, doing so does not mean all figures including similar or same numbers constitute a single or same embodiment.
[0010] An embodiment of the invention includes a low-latency mechanism for performing a checkpoint on a distributed application. This includes a multi-step checkpoint process that minimizes the latency experienced by an application.
[0011] Figures 1 and 2 collectively address a cluster including compute nodes as well as a more detailed example of the compute nodes themselves. The following discusses both figures as needed to describe various embodiments of the invention.
[0012] Figure 1 includes a schematic diagram of a cluster for high performance computing in an embodiment. A distributed application is running on multiple compute nodes 110, 111, 112, 113, and 114, which are connected by remote direct memory access (RDMA) capable network 115. Input/output (IO) nodes 120, 121, 122 connect to compute nodes 110, 111, 112, 113, 114 over RDMA network 115 and persistent storage array 130 over storage network 125. Although shown separately, the compute, 10 nodes, and networks may share the same hardware.
[0013] Process manager 105 controls the overall flow of the application. More specifically, a "process manager" is used to control other nodes in the cluster. For example, process manager 105 may be used to start processes on multiple machines in a cluster remotely, set up the cluster environment and launch processes used in message passing interface (MPI) jobs, provide libraries of commands related to MPI jobs and distributed computing, initiate checkpoints at programmed intervals, and the like. MPI is an application program interface (API) specification that allows computers to communicate with one another. The specification defines the syntax and semantics of a core of library routines useful in cluster computing. In an embodiment, process manager 105 communicates with the compute and IO nodes to start a checkpoint, coordinate the activities of the nodes during the checkpoint, and receives indication that the checkpoint is done.
[0014] Figure 2 includes a schematic diagram of compute node 210 in an embodiment of the invention. For each compute node one or more processors 201, 202, 203, 204 may be used to process one or more application processes, such as processes of a distributed application. Processors 201, 202, 203, 204 may couple to volatile memory (e.g., RAM) 215 via RDMA network interface card (NIC) 220 or other RDMA hardware. Volatile memory 215 may further couple to non-volatile memory region (NMR) (e.g., flash memory, application optimized non- volatile memory, and the like) 225. Thus, various compute nodes provide applications with access to a low-latency NMR. The NMR may be included locally in the compute node (as shown in Figure 2) or accessible through the RDMA network. RDMA NIC 220 couples compute node 210 and IO nodes to RDMA network 115 and is capable of accessing NMR 225 and/or volatile memory 215 directly.
[0015] RDMA supports zero-copy networking by enabling the transfer of data directly to or from application memory, eliminating the need to copy data between application memory (e.g., memory 215) and the data buffers in the operating system. Such transfers require no work to be done by CPUs, caches, or context switches, and transfers continue in parallel with other system operations. When an application performs an RDMA Read or Write request, the application data is delivered directly to the network, reducing latency.
[0016] In an embodiment, process manager 105 signals applications included on processors 201, 202, 203, and 204 when a checkpoint is required. After receiving a signal from process manager 105, an application halts external communication and saves the state of all calculations to NMR 225. State data may be written to NMR 225 using bus transfers for local NMR such as NMR 225, or using RDMA NIC 220 for local or remote NMRs. Use of RDMA NIC 220 for local NMR 225 may free host processor (e.g., processor 201) from needing to control bus transfers. Once done, the application processes reply to process manager 105 that they have completed their checkpoint tasks and continue with further calculations. This completes a first phase of a checkpoint process. The CPU (201, 202, 203, and 204) or RDMA may also transfer process data, which is related to the applications being processed on compute node 210, from volatile memory 215 to NMR 225.
[0017] A second phase of the checkpoint process begins after the computational states and processed data have been saved to NMR 225. Then IO nodes 120, 121, 122 access NMR 225 across RDMA network 115. State information and process data are read out of the NMR or NMRs 225 and written to storage array 130. Process manager 105 is notified of the final completion of the checkpoint, which allows NMRs (e.g., 225) to be reused.
[0018] Although the use of NMR 225 provides for greater fault tolerance recovery, the use of non-volatile memory in the first phase of the checkpoint process may be replaced with volatile memory in order to reduce latency, but at greater costs.
[0019] Thus, conventional systems may save computational state to persistent storage. For distributed applications this usually means using a distributed file system to save state information to remotely located hard disk drives. As a result, the application is prevented from continuing calculations until the checkpoint data has been written to persistent storage across a latency inducing network. In contrast, an embodiment of the invention uses a multiphase checkpoint process and RDMA to reduce the latency (as seen from the perspective of the application) required to perform a checkpoint. This allows checkpoints to occur more often, which is essential for scaling up applications to large cluster sizes (e.g., exascale). By making use of RDMA technologies embodiments avoid competing with applications for processing power while copying the data from the compute nodes to the storage arrays.
[0020] Furthermore, conventional systems do not combine the use of fast, secondary memory regions (e.g., NMR 225) with RDMA protocols. Together, these features allow applications to quickly checkpoint data to smaller, affordable memory regions, with background RDMA transfers offloading the data to larger, cheaper storage units. Embodiments of the invention may be utilized in various products, including MPI products involved in clusters and HPC systems.
[0022] As mentioned above, multiple applications may run on each compute node 110, 111, 112, 113, and 114. Figure 3 includes an embodiment for volatile memory, such as memory 215 of Figure 2. Memory 315 is divided into workspaces 316, 336, 346, one workspace per application. Each workspace is divided into one or more sections of calculated process data (e.g., produced as a result of processing a distributed application), along with general state information regarding the progress of the calculation (e.g., contents of processor registers). For example, workspace 316 includes section 317 for state data related to a first application. Workspace 316 further includes sections 318 and 319 respectively for process data 321, 322, both of which are related to the first application. In similar fashion, a second application relates to state information 337 and section 338 for data 341. A third application relates to state information 347 and sections 348, 349 respectively for data 351, 352.
[0023] A conventional checkpoint operation can be described with the following sequence: (1) initiate a checkpoint so the application halts computations; (2) a compute node transfers workspace state information over a network to an IO node; (3) the IO node writes the workspace state data to non-volatile memory (e.g., hard drive); (4) the compute node transfers workspace processed data sections to the IO node; (5) the IO node writes each section to non-volatile memory; and then (6) the compute node continues with computation. This can be viewed as a push model, where the compute node (and its processor) pushes/writes the data to the 10 node and the processor is burdened with the data transfer all the way to the 10 node.
[0024] In contrast, embodiments in Figures 4-7 concern a pull model where state information and computed data are read (i.e., pulled) by an 10 node. Thus, an embodiment pulls the data over the network to alleviate network congestion at the 10 node. Using an RDMA NIC, an 10 node reads data from a compute node. To reduce checkpoint time as experienced by the application, data in the process' workspace is copied to an NMR on the compute node. The copying of the data is a joint effort between the compute node CPU and RDMA NIC. For example, an embodiment uses the following sequence: (1) initiate a checkpoint, (2) an application halts computations, (3) the compute node processor copies the workspace to local NMR, (4) the compute node continues with computation, (5) the 10 node reads the workspace into the 10 network buffer, and then (6) the 10 node writes the 10 network buffer to non-volatile memory (e.g., hard drive).
[0025] Specifically addressing Figure 4, in block 405 processor 201 determines whether a checkpoint exists. For example, processor 201 of compute node 210 determines whether process manager 105 has initiated a checkpoint. If not, in block 410 processor 201 processes the application. However, if a checkpoint has been initiated then in blocks 415, 420 each process is halted and, in one embodiment, state information is stored in volatile memory 215 (along with process data that is also stored in memory 215). In block 425 state information is then stored to NMR 225. However, in other embodiments the state information is stored directly to NMR 225 instead of being first located in memory 215.
[0026] In block 430 the "pending RDMA request" is set to 0. Then, in block 435 the workspace (e.g., including state information 317 (if not already located in NMR) and data 321, 322) is saved to NMR 225. Figure 5 illustrates a more detailed embodiment of block 435. In block 440, if there are no pending RDMA requests then the processor continues computing in block 410. However, if there are such requests then the RDMA requests are processed in block 445. This may be done simultaneously with the processor processing the application due to usage of RDMA and storage of state information and data in NMR 225. Embodiments for block 445 are discussed in greater detail in Figures 6 and 7.
[0027] In Figure 5 block 535 corresponds to block 435 of Figure 4. Process 500 includes alternative paths for saving workspaces 316, 336, 346 to NMR. In block 540, the system determines whether all sections have been processed. If yes, then no copying is needed (block 545). However, if sections still need to be processed then the process advances to block 550. In block 550 it is determined whether the pending number of RDMA requests satisfies a threshold. Such a threshold may be based on a capacity limitation for the device (e.g., for the RDMA NIC). For example, if the pending number of requests is less than the threshold then RDMA is an option. Thus, in block 555 the number of pending RDMA requests is incremented. In block 560 the RDMA write is submitted (e.g., submitted into a queue) in order to copy the pertinent section (e.g., workspace 316) to NMR 225 via, for example, RDMA NIC 220 included in compute node 210.
[0028] However, if in block 550 the RDMA requests exceed a threshold then in block 575 the section (e.g., workspace 316) may be copied to NMR 225 via a processor (e.g., processor 201). After copying the section in volatile memory is marked as available (block 580) and processor 201 may resume processing the application and storing other data into the volatile memory just released. In block 585 a pull request may be submitted to remote nodes (e.g., IO node 120) along with the section's NMR address and any needed cryptographic keys, hashes, and the like. In block 565 if the RDMA request is complete then the RDMA request may be processed in block 570 (discussed in greater detail below in passage related to Figure 6).
[0029] Thus, as seen in Figure 5 the copying or transferring of process data (and possibly state information in some embodiments) from a workspace in volatile memory 215 to NMR 225 is a joint effort between the compute node processor (e.g., CPU 201) and a RDMA utility (e.g., RDMA NIC 220). The host CPU may perform steps 575, 580, and 585 and the RDMA NIC may perform steps 555 and 560. The decision "pending RDMA requests < threshold" of block 550 is used to determine if RDMA NIC 220 copies a section to NMR 225 using RDMA writes (the "yes" path) or if host CPU 201 copies the section to NMR 225 (the "no" path). As an example, RDMA NIC 220 may copy section 1-1 (element 318) to a first portion of NMR 225, while CPU 201 copies section 1-2 (element 319) to a second portion of NMR 225. The second portion does not overlap the first portion of NMR 225. In other words, the first and second portions may share the same memory but are separate from another to allow simultaneous access to the first and second portions. The copies may be made in parallel
[0030] Figure 6 includes an embodiment showing greater detail of block 570 of Figure 5. In Figure 6, block 670 includes processing the RDMA completion. The RDMA completion may be a signal indicating the transfer or copying of information from the compute node's local volatile memory (e.g., 215) to local non-volatile memory (e.g., 225) is complete. Furthermore, after the data has been copied to NMR 225, compute node 210 may signal the control node that it is done. The control node may then signal the IO nodes when compute node 210 has completed copying its data to NMR 225.
[0032] Figure 7 concerns the pull operation as seen from the perspective of IO node 120, 121, 122. In block 705 the IO node (e.g., 120) receives a notification. The notification or signal may be the pull request that was the subject of block 690 in Figure 6. If so, in block 710 IO node 120 submits an RDMA read to the specified NMR 225 address along with needed cryptographic information (e.g., keys or information encrypted in a way that is compliant with a key on the compute node, etc.). The process then returns to block 705.
[0033] However, if the signal or notification of block 705 includes notification that the RDMA read (i.e., pull) is complete, IO node 120 may now write (i.e., push) the received information to other non-volatile storage such as array 130. In block 720 IO node 120 may signal to process manager 105 that the push (to storage array) and pull (from NMR) operations are complete. The process then returns to block 705. In the pull operation the data is accessed by RDMA NIC 220 directly without involving the host CPUs (201, 202, 203, 204) of the compute node 210 or nodes.
[0034] In various embodiments, RDMA hardware (e.g., RDMA NIC 220) may be located locally on compute node 210 or just accessible via RDMA network 115. Locating the hardware locally on each compute node allows both the compute node CPUs (201, 202, 203, 204) and RDMA NICs 220 to copy the data from RAM 215 to NMR 225, from where the IO nodes 120, 121, 122 can fetch the data.
[0035] Thus, various embodiments provide one or more features that, for example, help reduce checkpoint latency. For example, one embodiment calls for the combined use of a CPU 201 and RDMA NIC 220 to transfer workspace data to local NMR 225. The "combined use" is exemplified in the "yes" and "no" branches for block 550 of Figure 5. The combined use format helps offload transfer burdens from the compute node processor so the processor can more quickly return to processing the application.
[0036] An embodiment also helps reduce latency based on the use of the compute node's local NMR 225 and DMA assisted hardware (e.g., RDMA NIC 220) which help reduce the time required to complete the checkpoint from an application's (running on the compute node) perspective (e.g., by removing the traditional need to transfer the information from volatile memory 215 across a network link 115 to a remote IO 120, 121, 122).
[0037] Further, an embodiment using segmentation of workspace data into sections helps reduce latency. As seen in Figure 3, process data sections 321 and 322 are divided away from each other. Also state information 317 is divided away from sections 321, 322. As a result, as soon as a section has been saved to NMR 225, the compute process may continue calculations on data within that section. RDMA hardware can continue to copy sections (in the background) while CPU 201 is re-dedicated to performing calculations.
[0038] Also, in certain embodiments the use of RDMA hardware by 10 nodes 120, 121, 122 to pull saved sections from the compute node NMR 225 helps reduce latency. An IO node 120, 121, 122 may pull a section as soon it has been copied to the NMR 225, providing overlapping operation with new sections being saved to volatile memory 215 and even to other sections of NMR 225. This reduces the minimum time required between checkpoints. Where IO nodes 120, 121, 122 fetch the data across the network 115, the use of RDMA allows this to occur without using the processing capabilities on compute nodes 210. The RDMA devices may also be used when copying the data between RAM 215 and NMR 225. This allows the system to overlap processing with the copying of data between RAM 215 and NMR 225 (once some portion of RAM may be modified), and also allows overlapping processing with the data being transferred over the network 115 to the IO nodes 120, 121, 122.
[0039] Specifically, conventional methods focus on increasing the speed of a performing any of the traditional steps. That is, when a checkpoint is requested, all nodes cease computation, write their data over the network to permanent storage, and then resume computation. After all data has been collected at the IO node, the checkpoint is done from the viewpoint of the compute nodes. The IO nodes then copy the data stored in the NMR to cheaper disks. This is a "push" model with the checkpoint time limited by the speed of the network writing to the IO nodes. While the checkpoint network operations are in progress, computation is blocked. In contrast, various embodiments of the invention reduce latency by facilitating overlapping operations through the use of hardware assist (i.e., the ability to process an application on the compute node while an IO node pulls information from the compute node).
[0040] Embodiments, such as compute nodes 210 and/or IO nodes 120, 121, 122 may be implemented in many different system types. Referring now to Figure 8, shown is a block diagram of a system in accordance with an embodiment of the present invention. Multiprocessor system 500 is a point-to-point interconnect system, and includes a first processor 570 and a second processor 580 coupled via a point-to-point interconnect 550. Each of processors 570 and 580 may be multicore processors. First processor 570 may include a memory controller hub (MCH) and point-to-point (P-P) interfaces. Similarly, second processor 580 may include a MCH and P-P interfaces. The MCHs may couple the processors to respective memories, namely memory 532 and memory 534, which may be portions of main memory (e.g., a dynamic random access memory (DRAM)) locally attached to the respective processors. First processor 570 and second processor 580 may be coupled to a chipset 590 via P-P interconnects, respectively. Chipset 590 may include P-P interfaces. Furthermore, chipset 590 may be coupled to a first bus 516 via an interface. Various input/output (I/O) devices 514 may be coupled to first bus 516, along with a bus bridge 518, which couples first bus 516 to a second bus 520. Various devices may be coupled to second bus 520 including, for example, a keyboard/mouse 522, communication devices 526, and data storage unit 528 such as a disk drive or other mass storage device, which may include code 530, in one embodiment. Code may be included in one or more memories including memory 528, 532, 534, memory coupled to system 500 via a network, and the like. Further, an audio I/O 524 may be coupled to second bus 520.
[0041] The term "processor" may refer to any device or portion of a device that processes electronic data from registers and/or memory to transform that electronic data into other electronic data that may be stored in registers and/or memory. A processor or controller may include control logic intended to represent any of a wide variety of control logic known in the art and, as such, may well be implemented as a microprocessor, a micro-controller, a field- programmable gate array (FPGA), application specific integrated circuit (ASIC), programmable logic device (PLD) and the like.
[0042] Embodiments may be implemented in code and may be stored on a non-transitory storage medium having stored thereon instructions which can be used to program a system to perform the instructions. The storage medium may include, but is not limited to, any type of disk including floppy disks, optical disks, optical disks, solid state drives (SSDs), compact disk read-only memories (CD-ROMs), compact disk rewritables (CD-RWs), and magneto- optical disks, semiconductor devices such as read-only memories (ROMs), random access memories (RAMs) such as dynamic random access memories (DRAMs), static random access memories (SRAMs), erasable programmable read-only memories (EPROMs), flash memories, electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, or any other type of media suitable for storing electronic instructions.
[0043] Embodiments of the invention may be described herein with reference to data such as instructions, functions, procedures, data structures, application programs, configuration settings, code, and the like. When the data is accessed by a machine, the machine may respond by performing tasks, defining abstract data types, establishing low-level hardware contexts, and/or performing other operations, as described in greater detail herein. The data may be stored in volatile and/or non-volatile data storage. The terms "code" or "program" cover a broad range of components and constructs, including applications, drivers, processes, routines, methods, modules, and subprograms and may refer to any collection of instructions which, when executed by a processing system, performs a desired operation or operations. In addition, alternative embodiments may include processes that use fewer than all of the disclosed operations, processes that use additional operations, processes that use the same operations in a different sequence, and processes in which the individual operations disclosed herein are combined, subdivided, or otherwise altered. Components or modules may be combined or separated as desired, and may be positioned in one or more portions of a device.

中文:
[0001]高性能计算(HPC)和集群计算涉及到连接各计算节点以形成能够解决复杂问题的分布式系统。这些节点可以是单个的台式计算机、服务器、处理器或能够托管单个计算实例的类似机器。更具体地,这些节点是由包括但不限于处理器、易失性存储器(RAM)、磁存储驱动器、主板、网络接口卡等硬件部件构成的。
[0003]通过随附的权利要求书、一个或多个示例性实施例的如下详细说明以及对应的附图,本发明的实施例的多个特征和优点将变得清晰,在附图中:
[0004]图1包括本发明的实施例中的用于HPC的集群的示意图。
[0005]图2包括本发明的实施例中的计算节点的示意图。
[0006]图3包括本发明的实施例中的计算节点的易失性存储器的框图。
[0007]图4-5包括本发明的实施例中的检查点处理的第一阶段的流程图。
[0008]图6-7包括本发明的实施例中的检查点处理的第二阶段的流程图。
[0009]在下面的说明中,阐述了多方面具体的细节,但是本发明的实施例可以在不具有这些具体细节的情况下实施。为避免使对本说明书的理解模糊,未详细示出公知的电路、结构和技术。“实施例”、“各个实施例”等表示如此描述的实施例可以包括特定的特征、结构或特点,但是不是每一个实施例必然包括该所述特定的特征、结构或特点。一些实施例可以具有针对其他实施例所描述的特征中的一些、全部或不包括。“第一”、“第二”、“第三”等描述了共同的对象并且表示提及类似对象的不同实例。这些形容词不暗示如此描述的对象必须以时间上或空间上的给定序列中、按级别或以任何其他方式。“连接”可以表示元件彼此直接物理或电接触,“耦合”可表示元件彼此配合或相互作用,但是它们可以或者可以不直接物理或电接触。而且,虽然相似或相同的标记可以用来在不同图中指示相同或相似的部件,这样做不意味着包括相似或相同标记的所有图都构成单个或相同的实施例。
[0010]本发明的实施例包括用于对分布式应用执行检查点的低时延机制。这包括使得应用所经受的时延最小化的多步检查点过程。
[0011]图1和图2一起解决了包括计算节点以及计算节点本身的更详细的示例的集群。下面根据需要讨论两个图来描述本发明的各个实施例。
[0012]图1包括实施例中的用于高性能计算的集群的示意图。分布式应用正在通过远程直接存储器存取(RDMA)使能网络115连接的多个计算节点110、111、112、113和114上运行。输入/输出(IO)节点120、121、122通过RDMA网络115连接到计算节点110、111、112、113、114且经由存储网络125连接到永久存储阵列130。虽然单独地示出,但是计算节点、IO节点和网络可以共享相同的硬件。
[0013]过程管理器105控制应用的总流程。更具体地,“过程管理器”用来控制集群中的其他节点。例如,过程管理器105可用于远程启动在集群中多个机器上的过程,设置集群环境并且发起在消息传递接口(MIP)作业中使用的过程,提供与MPI任务和分布式计算有关的命令库,以规划的间隔启动检查点,等等。MPI是允许计算机彼此通信的应用程序接口(API)规范。该规范定义了可用于集群计算中的库例程的核心的语法和语义。在实施例中,过程管理器105与计算节点和IO节点通信以启动检查点,来协调在检查点期间的节点的活动,并且接收检查点完成的指示。
[0014]图2包括本发明的实施例中的计算节点210的示意图。对于每个计算节点,一个或多个处理器201、202、203、204可用于处理一个或多个应用过程,诸如分布式应用的过程。处理器201、202、203、204可以经由RDMA网络接口卡(NIC)220或其他RDMA硬件与易失性存储器(例如,RAM)215耦合。易失性存储器215还可以与非易失性存储器区域(NMR)(例如,闪存、应用优化非易失性存储器等)225耦合。因此,各计算节点为应用提供对低时延NMR的存取。NMR可本地包含在计算节点(如图2所示)内或可通过RDMA网络存取。RDMA NIC220将计算节点210和IO节点与RDMA网络115耦合并且能够直接存取NMR225和/或易失性存储器215。
[0015]RDMA通过实现与应用存储器直接进行数据传递来支持零复制网络化,消除了在应用存储器(例如,存储器215)与操作系统中的数据缓冲器之间复制数据的需要。这种传递不需要通过CPU、高速缓存或上下文开关来完成工作,并且传递与其他系统操作并行地继续。当应用执行RDMA读或写请求时,应用数据直接输送到网络,降低了时延。
[0016]在实施例中,当需要检查点时,过程管理器105对包括在处理器201、202、203和204中的应用发信号。在接收到来自过程管理器105的信号之后,应用暂停外部通信且将所有计算的状态保存到NMR225中。可以利用诸如NMR225的本地NMR的总线传递,或者利用本地或远程NMR的RDMA NIC220,将状态数据写入NMR225。本地NMR225的RDMA NIC220的使用可以免除主机处理器(例如,处理器201)控制总线传递的需要。一旦完成,应用过程应答过程管理器105:它们已经完成了它们的检查点任务并且继续进行进一步的计算。这完成了检查点过程的第一阶段。CPU(201、202、203和204)或RDMA还可以将与在计算节点210上处理的应用有关的过程数据从易失性存储器215传递到NMR225。
[0017]在计算状态和处理数据已经保存到NMR225之后,开始检查点过程的第二阶段。然后,IO节点120、121、122跨越RDMA网络115来访问NMR225。状态信息和过程数据从一个或多个NMR225中读出且写入存储阵列130。将检查点的最终完成通知过程管理器105,这允许重新使用NMR(例如,225)。
[0018]虽然NMR225的使用提供了较大的容错恢复,但是在检查点过程的第一阶段使用非易失性存储器可由易失性存储器替代,从而减少时延,但是付出更大的代价。
[0019]因此,常规的系统可以将计算状态保存到永久存储中。对于分布式应用,这通常意味着,利用分布式文件系统将状态信息保存到远程定位的硬盘驱动器。结果,防止应用继续计算,直到检查点数据已经跨越时延诱发网络而写入永久存储。作为对比,本发明的实施例使用多阶段检查点过程和RDMA来减少执行检查点所需的时延(从应用的视角看到的)。这允许更频繁地出现检查点,这对于将应用扩展到大的集群尺寸(例如,千兆兆级)是很重要的。通过利用RDMA技术,实施例避免了在将数据从计算节点复制到存储阵列的同时与用于处理电力的应用竞争。
[0020]此外,常规的系统不将快速的辅助存储器区域(例如,NMR225)的使用与RDMA协议结合。同时,这些特征允许应用将数据快速地检查点到较小的、可负担的存储器区域,后台RDMA传递将数据卸载给较大的、较廉价的存储单元。本发明的实施例可以用于各种产品,包括在集群和HPC系统中涉及到的MPI产品。
[0022]如上所示,多个应用可以在每个计算节点110、111、112、113和114上运行。图3包括易失性存储器的实施例,诸如图2的存储器215。存储器315被划分成多个工作空间316、336、346,每个应用一个工作空间。每个工作空间被划分成计算的过程数据(例如,由于处理分布式应用而产生的)的一个或多个区段,以及关于计算进程的普通状态信息(例如,处理器寄存器的内容)。例如,工作空间316包括用于与第一应用有关的状态数据的区段317。工作空间316还包括分别用于都与第一应用有关的过程数据321、322的区段318和319。以类似的方式,第二应用与状态信息337和数据341的区段338有关。第三应用与状态信息347和分别用于数据351、352的区段348、349有关。
[0023]常规的检查点操作可通过下面的序列来描述:(1)启动检查点,以使应用暂停计算;(2)计算节点通过网络将工作空间状态信息传递到IO节点;(3)IO节点将工作空间状态数据写入非易失性存储器(例如,硬盘驱动器);(4)计算节点将已处理的工作空间数据区段传递到IO节点;(5)IO节点将每个区段写入非易失性存储器;以及然后(6)计算节点继续进行计算。这能够视为推模型(push model),其中计算节点(及其处理器)将数据推入/写入IO节点,并且处理器负担所有到IO节点的数据传递。
[0024]相比而言,图4-7中的实施例涉及拉模型(pull model),其中通过IO节点来读取(即,拉出)状态信息和计算数据。因此,实施例通过网络拉出数据以缓解IO节点处的网络拥塞。利用RDMA NIC,IO节点从计算节点读取数据。为了减少应用所经过的检查点时间,过程的工作空间中的数据被复制到计算节点上的NMR。数据的复制是计算节点CPU与RDMA NIC之间的联合努力。例如,实施例使用以下序列:(1)启动检查点;(2)应用暂停计算;(3)计算节点处理器将工作空间复制到本地NMR;(4)计算节点继续进行计算;(5)IO节点将工作空间读入IO网络缓冲器;以及然后(6)IO节点将IO网络缓冲器写入非易失性存储器(例如,硬盘驱动器)。
[0025]具体地参考图4,在框405中,处理器210判定检查点是否存在。例如,计算节点210的处理器201判定过程管理器105是否已启动检查点。如果为否,则在框410中,处理器201处理应用。然而,如果检查点已经启动,则在框415、420中,每个过程暂停,且在一个实施例中,将状态信息存储到易失性存储器215中(连同也存储在存储器215中的过程数据一起)。在框425中,状态信息随后存储到NMR225中。然而,在其他实施例中,状态信息被直接存储到NMR225中,而不是首先定位在存储器215中。
[0026]在框430中,“未决的RDMA请求”被设定为0。然后,在框435中,工作空间(例如,包括状态信息317(如果不是已经位于NMR中)和数据321,322)保存到NMR225中。图5示出了框435的更详细的实施例。在框440中,如果不存在未决的RDMA请求,则在框410中处理器继续计算。但是,如果存在这样的请求,则在框445中处理RDMA请求。由于RDMA的使用以及状态信息和数据存储在NMR225中,这可与处理器处理应用同时完成。在图6和图7中更详细地论述了框445的实施例。
[0027]在图5中,框535对应于图4中的框435。过程500包括用于将工作空间316、336、346保存到NMR的可选路径。在框540中,系统判定是否已处理了所有的区段。如果为是,则无需复制(框545)。然而,如果仍需要处理区段,则过程前进到框550。在框550中,判定未决的RDMA请求的数量是否满足阈值。该阈值可以基于设备(例如,RDMA NIC)的容量限制。例如,如果未决的请求数量小于阈值,则RDMA是可选项。因此,在框555中,未决的RDMA请求的数量增加。在框560中,RDMA写被提交(例如,提交到队列中),从而通过例如计算节点210中所包含的RDMA NIC220将相关区段(例如,工作空间316)复制到NMR225。
[0028]然而,如果在框550中RDMA请求超过阈值,则在框575中区段(例如,工作空间316)可经由处理器(例如,处理器201)复制到NMR225中。在复制之后,易失性存储器中的区段被标记为可用(框580),并且处理器201可以恢复处理应用并且将其他数据存储到刚刚释放的易失性存储器中。在框585中,可以将拉出请求连同区段的NMR地址和任何所需的加密密钥、散列等一起提交给远程节点(例如,IO节点120)。在框565中,如果RDMA请求完成,则可以在框570中处理RDMA请求(下文结合图6进行更详细论述)。
[0029]因此,如图5中所看出的,过程数据(以及在一些实施例中可能的状态信息)从易失性存储器215中的工作空间复制或传递到NMR225是计算节点处理器(例如,CPU201)和RDMA实用程序(例如,RDMA NIC220)之间的联合努力。主机CPU可以执行步骤575、580和585,并且RDMA NIC可以执行步骤555和560。框550的决策“未决的RDMA请求<阈值”用来判定RDMA NIC220是否利用RDMA写将区段复制到NMR225(“是”路径)或者主机CPU201是否将区段复制到NMR225(“否”路径)。作为例子,RDMA NIC220可以将区段1-1(元素318)复制到NMR225的第一部分,而CPU201将区段1-2(元素319)复制到NMR225的第二部分。NMR225的该第二部分与第一部分不重叠。换言之,第一和第二部分可共享同一存储器,但是彼此分离,从而允许对第一和第二部分同时存取。复制可与CPU201和NIC220两者处理传递的不同部分并行地(即,同时地)进行。这与常规方法形成对比,常规方法利用更直接的方法,其中CPU处理所有的复制。在实施例中,一旦RAM215的区段已被复制到NMR225,就可以修改RAM215。例如,一开始修改RAM215,计算就可以在计算节点上继续进行。
[0030]图6包括示出了图5的框570的更多细节的实施例。在图6中,框670包括处理RDMA完成。RDMA完成可以是指示从计算节点的本地易失性存储器(例如,215)传递或复制信息到本地非易失性存储器(例如,225)完成的信号。此外,在数据已经复制到NMR225之后,计算节点210可以对控制节点发出其已完成的信号。当计算节点210已经完成将其数据复制到NMR225时,控制节点随后可以对IO节点发信号。
[0032]图7涉及拉出操作,如从IO节点120、121、122的视角所看到的。在框705中,IO节点(例如,120)接收通知。通知或信号可以是作为图6中的框690的主题的拉出请求。如果是,则在框710中,IO节点120将RDMA读连同所需的加密信息提交到指定的NMR225地址(例如,以与计算节点上的密钥兼容的方式加密的密钥或信息,等等)。然后,过程返回到框705。
[0033]然而,如果框705的信号或通知包括RDMA读(即,拉出)完成的通知,则IO节点120现在可以将接收到的信息写入(即,推入)诸如阵列130的其他非易失性存储设备中。在框720中,IO节点120可以向过程管理器105发信号:推入(到存储阵列)以及拉出(从NMR)操作完成。过程随后返回到框705。在拉出操作中,RDMA NIC220直接存取数据,而不涉及计算节点210或多个节点的主机CPU(201、202、203、204)。
[0034]在各个实施例中,RDMA硬件(例如,RDMA NIC220)可本地定位在计算节点210中或者仅可经由RDMA网络115访问。将硬件本地定位在每个计算节点上允许计算节点CPU(201、202、203、204)和RDMA NIC220将数据从RAM215复制到NMR225,从NMR225中IO节点120、121、122能够取得数据。
[0035]因此,各个实施例提供了一个或多个特征,这些特征例如帮助减少检查点时延。例如,一个实施例调用CPU201和RDMA NIC220的组合使用以将工作空间数据传递到本地NMR225。在图5的框550的“是”和“否”分支中例示出所述“组合使用”。组合使用格式帮助从计算节点处理器卸载传递负担,因此处理器能够更快速地返回以处理应用。
[0036]实施例还基于计算节点的本地NMR225和DMA辅助硬件(例如,RDMA NIC220)的使用来帮助减少时延,从(在计算节点上运行的)应用的视角看,这有助于减少完成检查点所需的时间(例如,通过免除将信息从易失性存储器215跨越网络链路115传递到远程IO120、121、122的传统需要)。
[0037]此外,利用工作空间数据分割成区段的实施例帮助减少时延。如图3所示,过程数据区段321和322被划分成彼此远离。而且,状态信息317被划分而远离区段321、322。结果,一旦区段已保存到NMR225,计算过程就可以继续对该区段内的数据进行计算。RDMA硬件能够继续复制区段(在后台),而CPU201再次专用于执行计算。
[0038]而且,在一些实施例中,IO节点120、121、122使用RDMA硬件从计算节点NMR225拉出保存的区段有助于减少时延。IO节点120、121、122可以在区段一旦复制到NMR225时就拉出区段,提供与新区段保存到易失性存储器215以及甚至保存到NMR225的其他区段相重叠的操作。这减少了检查点之间所需的最小时间。在IO节点120、121、122通过网络115取得数据的情况下,RDMA的使用允许这在不使用计算节点210上的处理能力的情况下发生。当在RAM215与NMR225之间复制数据时,也可以使用RDMA设备。这允许系统将处理与在RAM215与NMR225之间复制数据相重叠(一旦RAM的某部分可修改),还允许处理与通过网络115将数据传递到IO节点120、121、122相重叠。
[0039]具体地,常规方法集中于提高执行任何传统步骤的速度。也即,当检查点被请求时,所有节点停止计算,通过网络将它们的数据写入永久存储,然后恢复计算。在IO节点处已收集到所有数据之后,从计算节点的视角看,完成了检查点。IO节点随后将存储在NMR中的数据复制到更廉价的磁盘。这是“推”模型,其中通过写入IO节点的网络速度来限制检查点时间。在检查点网络操作正在进行的同时,计算受阻挡。相比而言,本发明的各个实施例通过利用硬件辅助便于重叠操作而减少了时延(即,在IO节点从计算节点拉出信息的同时在计算节点上处理应用的能力)。
[0040]诸如计算节点210和/或IO节点120、121、122的实施例可以多种不同的系统类型来实现。现在参考图8,显示了依照本发明的实施例的系统的框图。多处理器系统500是点对点互连系统,并且包括经由点对点互连550耦合的第一处理器570和第二处理器580。处理器570和580中的每一个可以是多核处理器。第一处理器570可以包括存储器控制器中心(MCH)和点对点(P-P)接口。类似地,第二处理器580可以包括MCH和P-P接口。MCH可以将处理器耦合到各存储器,即存储器532和存储器534,存储器532和存储器534可以是本地附接到相应处理器的主存储器(例如,动态随机存取存储器(DRAM))的部分。第一处理器570和第二处理器580可以分别经由P-P互连与芯片组590耦合。芯片组590可以包括P-P接口。此外,芯片组590可以经由接口与第一总线516耦合。各个输入/输出(I/O)设备514可以连同将第一总线516与第二总线520耦合的总线桥518一起与第一总线516耦合。各个设备可以与第二总线520耦合,包括例如键盘/鼠标522、通信设备526以及数据存储单元528,诸如磁盘驱动器或其他大容量存储设备,在一个实施例中,可以包括代码530。代码可以包括在一个或多个存储器中,包括存储器528、532、534、经由网络与系统500耦合的存储器,等等。此外,音频I/O524可以与第二总线520耦合。
[0041]术语“处理器”可以指代处理来自寄存器和/或存储器的电子数据以将该电子数据变换成可存储在寄存器和/或存储器中的其他电子数据的任何设备或设备的部分。处理器或者控制器可以包括意在表示本领域公知的各种控制逻辑中的任一种的控制逻辑,因此,可以很好地实现为微处理器、微控制器、现场可编程门阵列(FPGA)、专用集成电路(ASIC)、可编程逻辑设备(PLD)等。
[0042]实施例可以以代码实现并且可以存储在非暂态存储介质上,非暂态存储介质上存储有指令,指令能够用于对系统编程以执行指令。存储介质可以包括但不限于:任意类型的磁盘,包括软盘、光盘、固态驱动器(SSD)、只读式光盘存储器(CD-ROM)、可重写光盘(CD-RW)以及磁光盘;半导体器件,诸如只读存储器(ROM);随机存取存储器(RAM),诸如动态随机存取存储器(DRAM)、静态随机存取存储器(SRAM)、可擦除可编程只读存储器(EPROM)、闪存、电可擦除可编程只读存储器(EEPROM)、磁卡或光卡;或任何适合存储电子指令的其他类型的介质。
[0043]在本文中参考诸如指令、函数、程序、数据结构、应用程序、配置设定、代码等数据对本发明的实施例进行了描述。当通过机器存取数据时,机器可通过执行任务、定义抽象数据类型、建立低级硬件上下文和/或执行其他操作来做出响应,如本文更详细说明的。数据可以存储在易失性和/或非易失性数据存储设备中。术语“代码”或“程序”涵盖了宽范围的部件和构造,包括应用、驱动器、过程、例程、方法、模块和子程序,并且可指代任何指令集合,当通过处理系统执行时,其执行期望的一种或多种操作。另外,可选的实施例可以包括使用比所披露的全部操作少的过程、使用附加操作的过程、使用不同序列中的相同操作的过程以及本文所披露的各项操作结合、细分或以其他方式改动的过程。部件或模块可以根据需要结合或分离,并且可以定位在设备的一个或多个部分中。