Computer system, communication device, and storage control method with DMA transfer of data

This computer system is configured by connecting a plurality of computers via a communication network. At least one computer among the computers has a storage device and a communication device. The communication device has: a controller that controls data transmission/reception via the communication network; and an intermediate memory that stores data transmitted/received between the storage device and other calculators on the communication network.

TECHNICAL FIELD

The present invention generally relates to storage control.

BACKGROUND ART

Hyper-Converged infrastructure, which attracts attention recently, constructs a computer system through connection of a plurality of computer nodes (Hereinafter, merely termed “nodes”) each accommodating a server apparatus, storage apparatus, communication apparatus, and management apparatus in one enclosure. Patent Literature 1 discloses that, in the case where a firmware in a local side node starts a Direct Memory Access (DMA) controller, the DMA controller transmits a predetermined message to a remote side node, and executes an interrupt (completion notification) toward the firmware without waiting for a completion response from the remote side node.

CITATION LIST

Patent Literature

SUMMARY OF INVENTION

Technical Problem

In accordance with the technology of Patent Literature 1, the latency to a storage apparatus of a node can be reduced.

However, in a computer system based on Hyper-Converged infrastructure, one node accesses not only a storage apparatus in the one node but also storage apparatuses in the other nodes frequently. In the case of the accesses toward storage apparatuses in the other nodes, the latencies are increased by occurrence of, for example, a protocol conversion and an access request toward a CPU, as compared to the case of an access toward a storage apparatus in the one node. This reduces the total input/output (I/O) performance of a computer system based on Hyper-Converged infrastructure. Therefore, the object of the present invention is improvement of the total I/O performance of a computer system based on Hyper-Converged infrastructure.

Solution to Problem

A computer system based on an embodiment comprises a plurality of computers configured to be coupled to one another through a communication network. At least one computer of a plurality of computers comprises a storage device and a communication device. The communication device comprises a controller configured to control data transmission/reception via the communication network, and an intermediate memory configured to store data transmitted and received between a storage device and the other computers on the communication network.

Advantageous Effects of Invention

In accordance with the present invention, the total I/O performance of a computer system based on Hyper-Converged infrastructure can be improved.

DESCRIPTION OF EMBODIMENTS

In the following description, information is sometimes illustrated in such representation as “aaa table”, “aaa queue”, or “aaa list”, however, information may be represented in any data structure. In other words, “aaa table”, “aaa queue”, or “aaa list” may be referred to as “aaa information” to illustrate that information is independent from data structure.

Furthermore, representations “identifying information”, “identifier”, “name”, “appellation”, and “ID” may be used when illustrating contents of each information, and these can be replaced one another.

Further, in the following description, sometimes a process is illustrated with a “program” as a subject, however, as a processor (e.g., a Central Processing Unit (CPU) executes a program, the program performs a predetermined process using at least one of a storage resource (e.g., memory) and a communication interface device, therefore the subject of the process may be the processor or an apparatus comprising the processor. Some or all of the processes executed by the processor may be performed by a hardware circuit.

The computer program may be installed from a program source. The program source may be a program distribution server or a storage media (e.g., portable storage media).

Further, in the following description, when illustrating similar elements distinctly, such reference characters as “Node10A” and “Node10B” may be used, when illustrating similar elements without distinction, such a common numeral in reference characters as “Node10” may be only used.

FIG. 1illustrates a configuration example of a computer system according to an embodiment.

A computer system1based on Hyper-Converged comprises a plurality of nodes10. A plurality of nodes10are bi-directionally communicatively coupled to a switch fabric3based on PCIe. However, the switch fabric3based on PCIe is only an example of a network communication, such other communication networks as SAN (Storage Area Network), LAN (Local Area Network), and SAS (Serial Attached SCSI) and the like may be in this position. Hereinafter, a switch fabric based on PCIe is referred to as merely “fabric”.

The node10comprises a CPU12, a memory14, a flash drive18, and a PCIe card22.

The memory14stores program and/or data. Examples of the memory14include Dynamic Random Access Memory (DRAM), Magnetoresistive Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), and phase-change memory. When using a nonvolatile memory as the memory14, even in the event of power source disorder, data loss does not occur.

The CPU implements a function of node10by reading a program and data from memory14and processing them.

The flash drive18includes a flash memory which is an example of nonvolatile storage media and an Non-Volatile Memory Express (NVMe) controller16which controls data read and write, etc. on the flash memory18. The NVMe controller16controls I/O (write and read) and deleting of data on the flash memory18.

The PCIe card22is an example of communication device, and controls data transmission/reception conforming to PCIe protocol. The PCIe card22may comprise an intermediate memory (e.g., SCM30) storing (cache) data transmission/receive between nodes10. Note that details of the PCIe card22will be described below (seeFIG. 2).

The PCIe card22with built-in SCM, CPU12, and NVMe controller16may be connected with a PCIe bus24. The CPU12and the memory14may be connected with a memory bus26.

In the description of the present embodiment, one node of a plurality of nodes10is referred to as a first node10A, and another is referred to as a second node10B. The elements which the first node10comprises are referred to as a first CPU12A, a first memory14A, a first NVMe controller16A, a first flash memory18A, and a first PCIe card22A. The elements which the second node10comprises are referred to as a second CPU12B, a second memory14B, a second NVMe controller16B, a second flash memory18B, and a second PCIe card22B.

In the present embodiment, the case will be described, where commands of read and write are issued from the second CPU12B of the second node10B to the first node10A. In this case, the first PCIe card22A comprises an intermediate memory (e.g., SCM30). The second PCIe card22B may comprise an intermediate memory, and in this case, acts as a generic PCIe card. In the second memory14B, a submission queue (referred to as “second SQ”)30B and a completion queue (referred to as “second CQ”)32B for the exchange of commands based on NVMe between the second CPU12B and a processor42in the first PCIe card22A may be provided. In the first memory14A, a submission queue (referred to as “first SQ”)30A and a completion queue (referred to as “first CQ”)32A for the exchange of commands based on NVMe between the processor42in the first PCIe card22and the first NVMe controller16A may be provided.

FIG. 2illustrates a configuration example of a PCIe card22according to an embodiment.

The PCIe card22comprises a large-scale integration (LSI) circuit40, a SCM30, a PCIe terminal32, and a terminal for PCIe slots34. The PCIe terminal32is an I/F (InterFace) for connecting the node10to the fabric3through the PCIe card22. The terminal for PCIe slots34is an I/F for connecting the PCIe card22to the PCIe bus24in a node10.

The SCM30is a storage device comprising a nonvolatile storage media. An I/O rate of the SCM30may be slower than that of the memory14of the node10, and faster than that of the flash memory18. Examples of the SCM30are MRAM, FeRAM, and phase-change memory and the like. Note that the PCIe card22may comprise a volatile storage media such as a DRAM in place of the SCM30. In this case, the PCIe card22may comprise a plurality of volatile storage media to secure redundancy.

The LSI40may comprise an arbiter (ARB)50, a memory controller44, a DMA controller46, a cache determination circuit48, and the processor42. These components may be coupled to a predetermined switch52, and may be bi-directionally communicative. The LSI40may be configured in a form of an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA), etc.

The arbiter50is a logic circuit for coupling the PCIe terminal32and/or the terminal for PCIe slots34to the LSI40. The memory controller44is a logic circuit for coupling the SCM30to the LSI40. The DMA controller46is a logic circuit for controlling a DMA-transfer.

The cache determination circuit48is a logic circuit for determining if target data of a read request received by the PCIe card22is stored (cached) in the SCM30. Note that in the case where this cache determination is performed with the processor42, the LSI40does not need to comprise the cache determination circuit48. The processor42is a logic circuit executing processes for implementing functions which the PCIe card22has. In the processor42, an SRAM43may be provided.

FIGS. 3 and 4are processing examples in the case that the second CPU12B of the second node10B issues a read request to the first NVMe controller16A of the first node10A.

FIG. 3shows a ladder chart illustrating a processing example when there is no target data of a read request in the SCM30in the first PCIe card22A (cache miss).

The second CPU12B enqueues a read request command to a second SQ30B of the second memory14B (S102).

The second CPU12B increments the tail pointer of the second SQ30B (S104).

The processor42in the first PCIe card22A detects an update of the tail pointer of the second SQ30B in S104, and fetches (dequeues) the read request command from the second SQ30B (S106). Then the head pointer of the second SQ30B is incremented.

The processor42in the first PCIe card22A determines if a target data of a read request command fetched in S106(referred to as “target read data”) hits a cache data in the SCM30(S108). This determination process is referred to as a cache determination process. Detail of the cache determination process will be described below (seeFIG. 5). Hereinafter, the case in which the result of the cache determination process is “cache miss” is described.

The processor42in the first PCIe card22enqueues the read request command fetched in S106to a first SQ30A of the first memory14A (S110).

The processor42in the first PCIe card22increments the tail pointer of the first SQ30A (S112).

The first NVMe controller16A detects an update of the tail pointer in S110, and fetches the read request command from the first SQ30A (S114). Then the head pointer of the first SQ30A is incremented.

The first NVMe controller16A, in accordance with the read request command in S114, performs DMA-transfer transferring the target read data in the first flash memory18A to the SCM30in the first PCIe card22A (S116).

The first NVMe controller16A, after a completion of a DMA-transfer in S116, enqueues a completion command corresponding to the read request command (referred to as “read completion command”) to a first CQ32A of the first memory14A (S118).

The first NVMe controller16A notifies (executes an MSI-X interrupt) a read completion response to the processor42in the first PCIe card22(S120). Then the tail pointer of the first CQ32A is incremented.

The processor42in the first PCIe card22A receives the read completion response in S120, and fetches the read completion command from the first CQ32A (S122).

The processor42in the first PCIe card22A processes the read completion command in S122(S124).

The processor42in the first PCIe card22A increments the head pointer of the first CQ32A (S126).

The processor42in the first PCIe card22A performs DMA-transfer transferring a DMA-transferred target read data in S116in the SCM30to the second memory14B (S130). This DMA-transfer may be performed by a DMA controller in the first PCIe card22A.

The processor42in the first PCIe card22A, after a completion of a DMA-transfer in S130, enqueues a read completion command fetched in S122to a second CQ32B of the second memory14B (S132).

The processor42in the first PCIe card22A notifies (executes an MSI-X interrupt) a read completion response to the second CPU12B (S134). Then the tail pointer of a second CQ32B is incremented.

The second CPU12B receives the read completion response in S134, and fetches the read completion command from the second CQ32B (S136).

The second CPU12B processes the read completion command in S136(S138).

The second CPU12B increments the head pointer of the second CQ32B (S140).

FIG. 4shows a ladder chart illustrating a processing example when there is target data (cache hit) of a read request in the SCM30in the first PCIe card.

The process from S202to S206is the same as the process from S102to S106inFIG. 3. Accordingly, the description is omitted.

The processor42in the first PCIe card22A, as in S108, performs the cache determination processing (S208). Detail of the cache determination process will be described below (seeFIG. 5). Hereinafter, the case in which the result of the cache determination process is a “cache hit” is described.

The processor42in the first PCIe card22A performs DMA-transfer transferring a cache hit target read data in the SCM30to the second memory14B (S210).

The process from S212to S220is the same as the process from S132to S140inFIG. 3. Accordingly, the description is omitted.

As the I/O rate of the SCM30is faster than that of the flash memory18, in accordance with the process inFIGS. 3 and 4, when the cache of the SCM30of the first PCIe card22is hit, a response time (latency) corresponding to the second node10B can be reduced. Further, when the cache of the SCM30is fit, as the DMA-transfer in the first node10A (S116) does not need to be performed, a bandwidth load of an internal PCIe bus24A in the first node10A can be reduced.

FIG. 5shows a ladder chart illustrating a cache determination processing example. This process is equivalent to S108inFIG. 3and S208inFIG. 4.

The processor42starts the cache determination circuit48(S502).

The cache determination circuit48reads, from the SRAM43, metadata (e.g., index) relating to data stored in the SCM30(S504).

The cache determination circuit48, based on the metadata, determine if there is a target read data of S108or S208in the SCM30(S506).

The cache determination circuit48notifies a determination result in S506to the processor42(S508).

Note that the process described above is an example, a cache determination may be performed in any process. For example, without using the cache determination circuit48, a cache determination may be performed only with the processor42. Also, without using metadata, a retrieval may be performed directly in the SCM30.

FIGS. 6 and 7are processing examples in the case that the second CPU12B of the second node10B issues a write request to the first NVMe controller16A of the first node10A.

FIG. 6shows a ladder chart illustrating a processing example of storing a target data of a write request into a SCM30in the first PCIe card22A.

The second CPU12B stores a target data of the write request (referred to as “target write data”) to the second memory14B. Then the second CPU12B enqueues a write request command to a second SQ30B of the second memory14B (S302).

The second CPU12B increments the tail pointer of the second SQ30B (S304).

The processor42in the first PCIe card22A detects an update of the tail pointer of the second SQ30B in S304, and fetches (dequeues) the write request command from the second SQ30B (S306). Then the head pointer of the second SQ30B is incremented.

The processor42in the first PCIe card22performs DMA-transfer transferring the target write data in the second memory14B designated by the write request command fetched in S306to the SCM30(S308). Then, considering the case of occurring a disorder of one SCM30, dual writing may be performed by copying a target write data to the other SCM30.

The processor42of the first PCIe card22A, after a completion of a DMA-transfer in S308, enqueues a completion command corresponding to the write request command (referred to as “write completion command”) to the second CQ32B (S310).

The processor42in the first PCIe card22A notifies (executes an MSI-X interrupt) a write completion response to the second CPU12B (S312). Then the tail pointer of a second CQ32B is incremented.

The second CPU12B receives the write completion response in S312, and fetches the write completion command from the second CQ32B (S314).

The second CPU12B processes the write completion command in S314(S316).

The second CPU12B increments the head pointer of the second CQ32B (S318).

Due to the process above, a write data issued from the second node10B is stored in the SCM30in the first PCIe card22A.

FIG. 7shows a ladder chart illustrating an example of transferring process (“destage process”) which transfers target write data stored in the SCM30in the first PCIe card22A to the first flash memory18A.

The processor42in the first PCIe card22A, at a predetermined timing, enqueues the write request command to the first SQ30A of the first memory14A (S402). Examples of the predetermined timing are a case in which a process load of a processor is light, a case in which an I/O load of the SCM30is light, a case in which a process load of the first CPU12A or the first NVMe controller16A is light, a case in which an I/O load of the first flash memory18A is light, or a case in which a bandwidth load of the internal PCIe bus24A of the first node10A is light.

The processor42in the first PCIe card22A increments the tail pointer of the first SQ30A (S404).

The first NVMe controller16A detects an update of the tail pointer of the first SQ30A in S404, and fetches (dequeues) the write request command from the first SQ30A (406). Then the head pointer of the first SQ30A is incremented.

The first NVMe controller16A performs DMA-transfer transferring a target write data stored in the SCM30in S308inFIG. 6to the first flash memory18A (S408). This process is referred to as a destage process. This DMA-transfer (destage process) may be performed by a DMA controller46in the first PCIe card22A.

The first NVMe controller16A, after a completion of a DMA-transfer in S408, enqueues a completion command corresponding to the write request command in S406to the first CQ32A of the first memory14A (S410).

The first NVMe controller16A notifies (executes an MSI-X interrupt) a write completion response to the processor42in the first PCIe card22A (S412). Then the tail pointer of the first CQ32A is incremented.

The processor42in the first PCIe card22A receives the write completion response in S412, and fetches the write completion command from the first CQ32A (S414).

The processor42in the first PCIe card22A processes the write completion command in S414(S416).

The processor42in the first PCIe card22A increments the head pointer of the first CQ32A (S418).

Due to the process above, a target write data stored in the SCM30in the first PCIe card22A is transferred (destaged) to the first flash memory18A.

In accordance withFIGS. 6 and 7, as a write completion response is notified to the second node10B at a timing at which a target write data is stored in the SCM30of the first PCIe card22A, a response time (latency) corresponding to the second node10B can be reduced comparing with the case of storing the target write data to the first flash memory18A in the first node10A. In the present embodiment, a destage destination of a write data is a flash memory, however, the destage destination may be HDD (Hard Disk Drive). In this case, latency corresponding to the second node10B becomes significantly low.

Also, as the processor42of the first PCIe card22A shows performing virtually an operation of the first NVMe controller16A to the second node10B, the second CPU12B may be a generic NVMe driver. Similarly, as the processor42of the first PCIe card22A shows performing virtually an operation of the second CPU12B to the first NVMe controller16A, the first NVMe controller16A may be a generic NVMe controller.

Furthermore, when write data whose amount is equal to or larger than the certain reference is stored to the SCM30, or when a process load of the first node10A is light, by destaging the write data stored in the SCM30to the first flash memory18A, the process load of the first node10A and a bandwidth load of an internal PCI bus24A can be equalized.

FIG. 8illustrates an example of a tier management table100.

Depending on an I/O rate of a storage device, the storage device is classified hierarchically. For example, the highest rate class is “tier 1”, the next high rate class is “tier 2”, and the lowest rate class is “tier 3”. Typical order of the I/O rate is, in the order that first is the highest, SCM, SSD (Solid State Drive), and HDD. In this case, conventionally, SCM is in tier 1, SSD is in tier 2, and HDD is in tier 3.

However, in the present embodiment, a certain node10sometimes has smaller latency even including a delay of a fabric3when an I/O request (e.g., write request/read request) is issued to an SSD or an HDD of another node10comprising the PCIe card22with the built-in SCM30, comparing with when an I/O request is issued to an internal SSD or an internal HDD. It is because the PCIe card22returns a completion response at a time point at which data is stored in the SCM30.

Therefore in the present embodiment, tiers to which storage devices of the own node and other nodes belong are determined based on actual latencies.

FIG. 8is an example of a tier management table100in which information regarding the tier determined by the second node10B and information regarding the tier determined by third node10C in the case that the first node10A comprises an SSD and an HDD, and the PCIe card22with the built-in SCM30. In other words, the tier management table100is an example of a table managing a performance of a storage.

As illustrated byFIG. 8, in a first computer system, the second node10B may set the SSD of the first node10A in tier 1 and the SSD of the second node10B in tier 2 when the latency to the SSD comprised by the first node10A which is the other node is smaller than the latency to the SSD comprised by the second node10B itself.

In a second computer system, the second node10B may set the SSD of the second node10B in tier 1 and the SSD of the first node10A in tier 2 when the latency to the SSD comprised by the second node10B itself is smaller than the latency to the SSD comprised by the first node10A which is the other node. For example, when a traffic of a communication network is a bottleneck, or when destage processes inFIG. 7occur extensively, even if the first node10A as the other node comprises the PCIe card22with the built-in SCM30, it is still probable that the latency to the SSD comprised by the second node10B itself is smaller.

The tier management table100may be held in each node10, or held in a predetermined node which can be accessed in common by each node10.

Thereby, each node10can store data required of relatively high speed I/O in a storage device which has an actual small latency. In other words, the total I/O performance of the computer system1may be improved.

FIG. 9shows a table for illustrating a SCM cache mode.

The PCIe card22with the built-in SCM30may comprise a setting for switching do/do not for storing (caching) the SCM30with a write data for a HDD connected to the PCIe card22via an internal bus in a node10. This setting is referred to as a SCM cache mode.

As shown in table120inFIG. 9, when a SCM cache mode is ON126, the PCIe card22stores (caches) the SCM30with write data for both the HDD and the SSD connected via the internal bus. In this case, a latency of a write request to the HDD becomes significantly small, however, as the number of destage times from the SCM30to the HDD becomes large, a cache hit rate of the SCM30concerning a read request for the SSD becomes low. Accordingly, an average latency of a read request for the SSD becomes larger than that in the case in which a SCM cache mode described below is OFF124.

When the SCM cache mode is OFF124, the PCIe card22stores (caches) the SCM30with write data for the SSD connected via the internal bus, however, do not store (cache) the SCM30with write data for the HDD. In this case, an average latency of a read request for the SSD becomes smaller than that in the case in which a SCM cache mode described above is ON126, however, a latency of a write request for the HDD becomes significantly large.

Note that when the HDD is connected via SAS, the PCIe card22has to support SAS as well as NVMe.

Which is proper ON or OFF of a SCM cache mode is different depending on characteristics of data read and write of applications.

The embodiments described above are exemplifications for description of the present invention, thereby no limitation of the scope of the invention only to the embodiments is intended. Those skilled in the art can implement the invention in other various aspects without departing from the spirit of the invention.

REFERENCE SIGNS LIST