Patent ID: 12248709

DETAILED DESCRIPTION

Below, example embodiments of the present disclosure will be described in detail and clearly to such an extent that one skilled in the art easily carries out the present disclosure.

Below, for convenience of description, specific example embodiments are separately described, but the scope of the present disclosure is not limited thereto; it will be understood that various embodiments may be combined with each other or a part of one embodiment may be combined with a part of another embodiment.

FIG.1is a block diagram illustrating a server system according to an embodiment of the present disclosure. Referring toFIG.1, a server system (or referred to as a “computer system” or a “storage system”)1000may include a client server1001and a storage server1002. The server system1000may refer to a data center or a data storage center that performs the maintenance of various data and provides various services. The server system1000may be a search engine or a system for the operation of a database and may be a computing system available in various institutions. The server system1000may refer to a storage system that provides a cloud service or an on-premise service.

The client server1001may refer to a user, a user's terminal, or a user's computing system that uses various data-based services. The client server1001may store data in the storage server1002or may read data stored in the storage server1002.

Based on a request of the client server1001, the storage server1002may store data or may send data to the client server1001. In an embodiment, the client server1001and the storage server1002may communicate with each other over a network (not illustrated).

The storage server1002may include a storage node1100and a plurality of storage devices1200_1to1200_n. The storage node1100may be configured to manage the storage devices1200_1to1200_nincluded in the storage server1002. Under control of the storage node1100, each of the plurality of storage devices1200_1to1200_nmay store data or may output the stored data. Each of the plurality of storage devices1200_1to1200_nmay be a high-capacity storage medium such as a solid state drive (SSD), but the present disclosure is not limited thereto.

The storage node1100may store data in the plurality of storage devices1200_1to1200_nor may read data stored in the plurality of storage devices1200_1to1200_n. For example, to store data in the plurality of storage devices1200_1to1200_n, the storage node1100may send a write command and write data to each of the plurality of storage devices1200_1to1200_n. Alternatively, to read data stored in the plurality of storage devices1200_1to1200_n, the storage node1100may send a read command to each of the plurality of storage devices1200_1to1200_nand may receive data from each of the plurality of storage devices1200_1to1200_n.

In an embodiment, the storage node1100and the plurality of storage devices1200_1to1200_nmay communicate with each other based on a given interface. In an embodiment, the given interface may support at least one of various interfaces such as a universal serial bus (USB) interface, a small computer system interface (SCSI), a peripheral component interconnection (PCI) express (PCIe) interface, an advanced technology attachment (ATA) interface, a parallel ATA (PATA) interface, a serial ATA (SATA) interface, a serial attached SCSI (SAS) interface, a universal flash storage (UFS) interface, a nonvolatile memory express (NVMe) interface, and a compute express link (CXL) interface, but the present disclosure is not limited thereto.

The storage node1100may include a storage node controller1110, a storage node memory1120, and a recovery manager1130. The storage node memory1120may function as a buffer memory for temporarily storing data to be transferred to the plurality of storage devices1200_1to1200_nor data transferred from the plurality of storage devices1200_1to1200_n. In an embodiment, the storage node memory1120may store data or information that is used by the recovery manager1130. For example, the storage node memory1120may store a virtual machine group table VMGT and a recovery sequence table RST.

According to an embodiment, the storage node controller1110and the storage node memory1120may be implemented with separate semiconductor chips. Alternatively, in some embodiments, the storage node controller1110and the storage node memory1120may be implemented in the same semiconductor chip. As an example, the storage node controller1110may be one of a plurality of modules included in an application processor; in this case, the application processor may be implemented with a system on chip (SoC). Also, the storage node memory1120may be an embedded memory included in the application processor or may be a nonvolatile memory or a memory module disposed outside the application processor.

The storage node controller1110may manage an operation of storing data (e.g., write data) of a buffer area of the storage node memory1120to the storage devices1200_1to1200_nor storing data (e.g., read data) of the storage devices1200_1to1200_nto the buffer area.

In an embodiment, the recovery manager1130may provide an optimal recovery sequence to the plurality of storage devices1200_1to1200_n. The recovery manager1130may generate and manage the virtual machine group table VMGT based on a workload characteristic of each virtual machine. The recovery manager1130may collect recovery information from the plurality of storage devices1200_1to1200_n. The recovery manager1130may generate and manage the recovery sequence table RST based on the virtual machine group table VMGT, the recovery information, and storage attributes. The recovery manager1130may provide the optimal recovery sequence to a storage device allocated to a new virtual machine based on the recovery sequence table RST. An optimal recovery sequence providing method of the server system1000according to embodiments of the present disclosure will be described in detail with reference to the following drawings.

FIG.2Ais a diagram illustrating software layers of a server system ofFIG.1. Referring toFIG.2A, an operating system OS, a hypervisor HV, a first virtual machine VM1, and a second virtual machine VM2may be driven on the client server1001. The operating system OS may refer to system software configured to control various hardware and resources included in the client server1001, to drive various programs, and to support various services. The hypervisor HV may be a logical platform configured to drive the first and second virtual machines VM1and VM2that are executed in the client server1001.

Each of the first and second virtual machines VM1and VM2may be driven in the client server1001. In an embodiment, data associated with the first virtual machine VM1may be stored in a first storage area SA1of the storage server1002, and data associated with the second virtual machine VM2may be stored in a second storage area SA2of the storage server1002. In an embodiment, the first storage area SA1may correspond to the first storage device1200_1, and the second storage area SA2may correspond to the second storage device1200_2. Alternatively, the first storage area SA1may correspond to a first namespace, and the second storage area SA2may correspond to a second namespace. The namespace may refer to a storage area of the storage device, which are logically or physically classified. That is, the data that are managed by the first virtual machine VM1may be logically or physically distinguished from the data that are managed by the second virtual machine VM2. Alternatively, the first storage area SA1may correspond to a first zone namespace, and the second storage area SA2may correspond to a second zone namespace. The zone namespace may refer to a namespace of the storage device, which is divided into a plurality of zones.

In an embodiment, the operating system OS and first and second containers (not illustrate) may be driven on the client server1001. For example, data associated with the first container may be stored in the first storage area SA1of the storage server1002, and data associated with the second container may be stored in the second storage area SA2of the storage server1002. A relationship between the first and second containers and the first and second storage areas SA1and SA2may be identical or similar to the relationship between the first and second virtual machines VM1and VM2and the first and second storage areas SA1and SA2, and thus, additional description will be omitted to avoid redundancy.

FIG.2Bis a block diagram illustrating a first storage device1200_1ofFIG.1in detail. Each of the remaining storage devices1200_2to1200_nmay be identical or similar to the first storage device1200_1.

Referring toFIG.2B, the storage device1200_1may include storage mediums for storing data based on a request from the storage node1100. As an example, the storage device1200_1may include at least one of a solid state drive (SSD), an embedded memory, or a removable external memory. In the case where the storage device1200_1is the SSD, the storage device1200_1may be a device that complies with the non-volatile memory express (NVMe) standard. In the case where the storage device1200_1is the embedded memory or the external memory, the storage device1200_1may be a device that complies with the universal flash storage (UFS) or embedded multi-media card (eMMC) standard. Each of the storage node1100and the storage device1200_1may generate a packet that complies with a standard protocol applied thereto and may send the generated packet.

When a nonvolatile memory1220of the storage device1200_1includes a flash memory, the flash memory may include a two-dimensional (2D) NAND flash memory array or a three-dimensional (3D) (or vertical) NAND (VNAND) memory array. As another example, the storage device1200_1may be implemented with various kinds of different nonvolatile memories. For example, the storage device1200_1may include a magnetic RAM (MRAM), a spin-transfer torque MRAM (STT-MRAM), a conductive bridging RAM (CBRAM), a ferroelectric RAM (FeRAM), a phase change RAM (PRAM), a resistive RAM (RRAM), or at least one of various kinds of different memories.

A storage controller1210may include a host device interface1211, a memory interface1212, and a central processing unit (CPU)1213. Also, the storage controller1210may further include a flash translation layer (FTL)1214, a packet manager1215, a buffer memory1216, an error correction code (ECC) engine1217, an advanced encryption standard (AES) engine1218, and a recovery engine1219. The storage controller1210may further include a working memory (not illustrated) to which the flash translation layer1214is loaded, and data write and read operations of nonvolatile memory1220may be controlled as the CPU1213executes the flash translation layer1214.

The host device interface1211may exchange packets with the storage node1100. The packet that is transferred from the storage node1100to the host device interface1211may include a command, data to be written in the nonvolatile memory1220, and the like, and the packet that is transferred from the host device interface1211to the storage node1100may include a response to the command, data read from the nonvolatile memory1220, and the like. The memory interface1212may provide the nonvolatile memory1220with data to be written in the nonvolatile memory1220, and may receive data read from the nonvolatile memory1220. The memory interface1212may be implemented to comply with the standard such as Toggle or ONFI (Open NAND Flash Interface).

The flash translation layer1214may perform various functions (or operations) such as address mapping, wear-leveling, and garbage collection. The address mapping operation refers to an operation of translating a logical address received from the storage node1100into a physical address to be used to actually store data in the nonvolatile memory1220. The wear-leveling is a technology for allowing blocks in the nonvolatile memory1220to be used uniformly such that excessive degradation of a specific block is prevented. The wear-leveling may be implemented, for example, through a firmware technology for balancing erase counts of physical blocks. The garbage collection refers to a technology for securing an available capacity of the nonvolatile memory1220by erasing an existing block after copying valid data of the existing block to a new block.

The packet manager1215may generate a packet that complies with a protocol of an interface agreed with the storage node1100or may parse various kinds of information from the packet received from the storage node1100. Also, the buffer memory1216may temporarily store data to be written in the nonvolatile memory1220or data read from the nonvolatile memory1220. The buffer memory1216may be a component provided within the storage controller1210; however, it may be possible to dispose the buffer memory1216outside the storage controller1210.

The ECC engine1217may perform an error detection and correction function on data read from the nonvolatile memory1220. In detail, the ECC engine1217may generate parity bits for the write data to be written in the nonvolatile memory1220, and the parity bits thus generated may be stored in the nonvolatile memory1220together with the write data. When data are read from the nonvolatile memory1220, the parity bits are read from the nonvolatile memory1220together with the read data and the ECC engine1217may correct an error of the read data by using the parity bits and may output the error-corrected read data.

The AES engine1218may perform at least one of an encryption operation and a decryption operation on data input to the storage controller1210by using a symmetric-key algorithm.

The recovery engine1219may perform recovery operations when a read error occurs. The recovery engine1219may perform the recovery operations based on a default recovery sequence. Before the optimal recovery sequence is received from the storage node1100, the recovery engine1219may correct the read error by performing the recovery operation based on the default recovery sequence.

The recovery engine1219may store a result of performing the recovery operations. The recovery engine1219may store history information about the recovery operations thus performed in the buffer memory1216. The recovery engine1219may store and manage recovery information. The recovery engine1219may send the recovery information to the storage node1100based on the request of the storage node1100.

The recovery engine1219may receive the optimal recovery sequence from the storage node1100. After receiving the optimal recovery sequence, the recovery engine1219may perform the recovery operations based on the optimal recovery sequence when the read error occurs. Accordingly, the reduction of performance of the storage device may be minimized.

FIG.3is a block diagram illustrating a recovery manager ofFIG.1in detail. Referring toFIGS.1and3, the recovery manager1130may include a workload analyzer1131, a workload grouping manager1132, a storage analyzer1133, a recovery collector1134, a recovery sequence table generator1135, and a recovery sequence allocator1136.

The workload analyzer1131may detect and analyze a user input/output. For example, the workload analyzer1131may monitor the user input/output between the client server1001and the plurality of storage devices1200_1to1200_n. It is assumed that the data associated with the first virtual machine VM1are stored in the first storage device1200_1. To extract the workload characteristic of the first virtual machine VM1, the workload analyzer1131may monitor or analyze the read/write request and data that are exchanged between the first virtual machine VM1and the first storage device1200_1.

The workload analyzer1131may extract the workload characteristic of a virtual machine based on a monitoring result (or analysis result). For example, the workload analyzer1131may determine whether the first virtual machine VM1has a first workload characteristic C1.

The workload grouping manager1132may group virtual machines based on workload characteristics of the virtual machines. The workload grouping manager1132may classify virtual machines having similar workload characteristics as a group. The workload grouping manager1132may generate the virtual machine group table VMGT based on the workload characteristics. The virtual machine group table VMGT may include a workload characteristic and information about identifiers of virtual machines corresponding to the workload characteristic.

The storage analyzer1133may analyze attributes of the plurality of storage devices1200_1to1200_n. For example, the storage analyzer1133may send an attribute information request to each of the plurality of storage devices1200_1to1200_n. The storage analyzer1133may receive attribute information from each of the plurality of storage devices1200_1to1200_n. The storage analyzer1133may determine the attribute of each of the storage devices1200_1to1200_nbased on the attribute information. The storage analyzer1133may transfer the determined storage attribute to the recovery sequence table generator1135.

Because the storage attribute changes over time, the storage analyzer1133may monitor the storage attribute of each of the storage devices1200_1to1200_n. To change the optimal recovery sequence based on the changed storage attribute, the storage analyzer1133may periodically request the storage attribute from each of the plurality of storage devices1200_1to1200_n. When the change condition of the optimal recovery sequence is satisfied, the storage analyzer1133may notify the recovery sequence allocator1136that a change condition is satisfied.

The recovery collector1134may collect the recovery information (e.g., recovery result) from the plurality of storage devices1200_1to1200_n. For example, to generate the recovery sequence table RST, the recovery collector1134may collect the recovery information during a given time period. The recovery collector1134may periodically request the recovery information from the plurality of storage devices1200_1to1200_n. The recovery collector1134may receive the recovery information from the plurality of storage devices1200_1to1200_n. The recovery collector1134may transfer the recovery information to the recovery sequence table generator1135.

The recovery sequence table generator1135may generate the recovery sequence table RST. The recovery sequence table generator1135may generate the recovery sequence table RST based on the virtual machine group table VMGT, the storage attribute, and the recovery information. The recovery sequence table generator1135may determine the optimal recovery sequence based on the workload characteristic and the storage attribute. The recovery sequence table generator1135may determine the optimal recovery sequence by changing the order of the default recovery sequence. The default recovery sequence may refer to a given recovery sequence. The recovery sequence table generator1135may store, in the recovery sequence table RST, the optimal recovery sequence determined based on the workload characteristic and the storage attribute.

The recovery sequence (or a set of recovery operations) may refer to the order of recovery operations that the storage device performs when the read error occurs. The recovery sequence (or recovery operations) may be aligned in order from a highest priority to a lowest priority. The optimal recovery sequence may refer to a sequence of recovery operations capable of efficiently correcting a read error when the read error occurs. For example, the optimal recovery sequence may refer to a recovery sequence in which a time required to perform error correction is the shortest.

The recovery operation may refer to an operation of performing read error detection and read error correction when the read error occurs. The recovery operation may include a read retry operation, an operation of changing a read voltage level and performing a read operation, an operation of changing a read voltage level based on the number of program and/or erase cycles and performing a read operation, an operation of changing a read voltage level based on machine learning and performing a read operation, a soft decision or soft decoding operation, a hard decision or hard decoding operation, or the like.

In an embodiment, the recovery sequence table generator1135may align the recovery operations based on the recovery information. For example, the recovery sequence table generator1135may compare parameter values of the recovery operations based on the recovery information and may assign a higher priority to a recovery operation having a greater parameter value. That is, the recovery sequence table generator1135may determine orders of the recovery operations based on the parameter values. The parameter may include at least one of error correction success ratio, a latency, an error correction ratio, and power consumption. However, the present disclosure is not limited thereto.

For example, the error correction success ratio may refer to a ratio of the number of times that the recovery operation is performed and the number of times that an error is corrected. The recovery sequence table generator1135may determine the recovery operation that is performed the most to correct an error, based on the error correction success ratio. The latency may refer to a time taken to correct an error through the recovery operation. The recovery sequence table generator1135may determine the recovery operation requiring the shortest time, based on the latency. The error correction ratio may refer to a ratio of the total number of bits and the number of corrected bits. Alternatively, the error correction ratio may refer to a ratio of the number of error bits and the number of corrected bits. The recovery sequence table generator1135may determine the recovery operation correcting the most data, based on the error correction ratio.

For example, the recovery sequence table generator1135may assign a higher priority to the recovery operation in which the error correction success ratio is higher or the latency is shorter. The recovery sequence table generator1135may assign a lower priority to the recovery operation in which the error correction success ratio is lower or the latency is longer. That is, the recovery sequence table generator1135may place the recovery operation, in which the error correction success ratio is higher or the latency is shorter, at the front of the recovery sequence.

The recovery sequence allocator1136may provide the optimal recovery sequence to each of the plurality of storage devices1200_1to1200_n. For example, the recovery sequence allocator1136may select the optimal recovery sequence for a storage device allocated to a new virtual machine, based on the recovery sequence table RST. The recovery sequence allocator1136may select the optimal recovery sequence from the recovery sequence table RST, based on the workload characteristic of the new virtual machine and the storage attribute of the storage device. The recovery sequence allocator1136may allocate the selected optimal recovery sequence to the storage device allocated to the new virtual machine. The storage device may minimize the reduction of performance by performing the recovery operation based on the optimal recovery sequence. Accordingly, the server system1000with improved performance is provided.

FIG.4is a flowchart illustrating an example of an operation of a storage server ofFIG.1. Referring toFIGS.1and4, in operation S100, the storage server1002may generate the recovery sequence table RST based on the workload characteristic and the attribute of the storage device. For example, the storage server1002may analyze the workload of the virtual machine (or a user, a client, or a container). The storage server1002may extract the workload characteristic of the virtual machine. For example, the workload characteristic may include at least one of read intensive, write intensive, a read ratio, a workload size, a work set size, cache status information (e.g., a hit rate), and a work flow. However, the present disclosure is not limited thereto.

The storage server1002may receive an attribute (or a characteristic, an indicator, or a metric) of a storage device allocated to a virtual machine from each of the plurality of storage devices1200_1to1200_n. The attribute of the storage device (hereinafter referred to as a “storage attribute”) may include at least one of a state of a nonvolatile memory, a type of the nonvolatile memory, a program manner (or type) (or the number of bits stored per memory cell) (e.g., a Single-Layer Cell (SLC), a Multi-Layer Cell (MLC), a Triple-Layer Cell (TLC), or a Quad-Layer Cell (QLC)), the number of programs-erase (PE) cycles, endurance/reliability, durability, an access frequency, or a life of the nonvolatile memory. However, the present disclosure is not limited thereto. The attribute information may refer to information about the storage attribute.

In operation S200, the storage server1002may allocate the optimal recovery sequence to a storage device allocated to a new virtual machine, based on the recovery sequence table RST. The optimal recovery sequence allocation method will be described in detail with reference toFIG.9.

FIG.5is a flowchart illustrating operation S100ofFIG.4in more detail.FIG.6is a diagram illustrating an example of a virtual machine group table. For convenience of description, it is assumed that first to twelfth virtual machines VM1to VM12are driven on the client server1001and data associated with the first to twelfth virtual machines VM1to VM12are stored in a corresponding storage device. In detail, it is assumed that the data associated with the first virtual machine VM1are stored in the first storage device1200_1, the data associated with the second virtual machine VM2are stored in the second storage device1200_2, the data associated with the third virtual machine VM3are stored in the third storage device1200_3, and the data associated with the fourth virtual machine VM4are stored in the fourth storage device1200_4. Data associated with the remaining virtual machines VM5to VM12are stored in a manner similar to those described above, and thus, additional description will be omitted to avoid redundancy.

Referring toFIGS.1,5, and6, operation S100may include operation S110to operation S190. In operation S110, the storage server1002may extract characteristics of workloads of the virtual machines VM1to VM12. For example, the storage server1002may monitor the user input/output I/O. The storage server1002may detect the user input/output I/O between the plurality of virtual machines VM1to VM12and corresponding storage devices. For example, the storage server1002may detect the read/write request and data that are exchanged between the first virtual machine VM1and the first storage device1200_1.

The storage server1002may analyze the user input/output I/O to extract a workload characteristic of a virtual machine. For example, the storage server1002may monitor the read/write request and data that are exchanged between the first virtual machine VM1and the first storage device1200_1. The storage server1002may analyze the read/write request and data that are exchanged between the first virtual machine VM1and the first storage device1200_1and may determine whether the workload of the first virtual machine VM1has the first workload characteristic C1(e.g., the read intensive).

In operation S130, the storage server1002may group virtual machines based on the workload characteristics and may generate the virtual machine group table VMGT. In an embodiment, the storage server1002may classify virtual machines having similar workload characteristics as a group, based on the extracted workload characteristics.

Below, it is assumed that the workload characteristics of the virtual machines include first to fourth workload characteristics C1to C4. That is, it is assumed that plurality of virtual machines are classified into four virtual machine groups according to the first to fourth workload characteristics C1to C4. However, the present disclosure is not limited thereto. For example, the number of workload characteristics used for grouping or the number of virtual machine groups may increase or decrease depending on a way to implement.

Referring toFIG.6, it is assumed that the first, third, and seventh virtual machines VM1, VM3, and VM7have the first workload characteristic C1, the second, fourth, and twelfth virtual machines VM2, VM4, and VM12have the second workload characteristic C2, the fifth, eighth, and eleventh virtual machines VM5, VM8, and VM11have the third workload characteristic C3, and the sixth, ninth, and tenth virtual machines VM6, VM9, and VM10have the fourth workload characteristic C4.

Because the first, third, and seventh virtual machines VM1, VM3, and VM7have the first workload characteristic C1, the storage server1002may classify the first, third, and seventh virtual machines VM1, VM3, and VM7as a first virtual machine group VMGroup1. Because the second, fourth, and twelfth virtual machines VM2, VM4, and VM12have the second workload characteristic C2, the storage server1002may classify the second, fourth, and twelfth virtual machines VM2, VM4, and VM12as a second virtual machine group VMGroup2. Because the fifth, eighth, and eleventh virtual machines VM5, VM8, and VM11have the third workload characteristic C3, the storage server1002may classify the fifth, eighth, and eleventh virtual machines VM5, VM8, and VM11as a third virtual machine group VMGroup3. Because the sixth, ninth, and tenth virtual machines VM6, VM9, and VM10have the fourth workload characteristic C4, the storage server1002may classify the sixth, ninth, and tenth virtual machines VM6, VM9, and VM10as a fourth virtual machine group VMGroup4.

The storage server1002may generate the virtual machine group table VMGT based on the grouping result. The storage server1002may store the virtual machine group table VMGT in the storage node memory1120.

The virtual machine group table VMGT may include the workload characteristic and information about identifiers of virtual machines having the workload characteristic. For example, the virtual machine group table VMGT may include the first workload characteristic C1and information on identifiers (i.e., first, third, and seventh virtual machine identifiers VM1_ID, VM3_ID, and VM7_ID) of virtual machines having the first workload characteristic C1, the second workload characteristic C2and information on identifiers (i.e., second, fourth, and twelfth virtual machine identifiers VM2_ID, VM4_ID, VM12_ID) of virtual machines having the second workload characteristic C2, the third workload characteristic C3and information on identifiers (i.e., fifth, eighth, and eleventh virtual machine identifiers VM5_ID, VM8_ID, and VM11_ID) of virtual machines having the third workload characteristic C3, and the fourth workload characteristic C4and information on identifiers (i.e., sixth, ninth, and tenth virtual machine identifiers VM6_ID, VM9_ID, and VM10_ID) of virtual machines having the fourth workload characteristic C4.

In operation S150, the storage server1002may receive information about attributes of storage devices allocated to virtual machines. For example, the storage node1100may send the attribute information request to each of the storage devices1200_1to1200_12corresponding to the virtual machines VM1to VM12. The storage node1100may receive attribute information from each of the storage devices1200_1to1200_12. The storage node1100may determine the storage attribute of each of the storage devices1200_1to1200_12, based on the received attribute information.

In operation S170, the storage server1002may collect the recovery information (or recovery result) from storage devices. For example, the storage node1100may periodically (or repeatedly) request the recovery information from the storage devices1200_1to1200_12. The storage node1100may receive the recovery information from the storage devices1200_1to1200_12. The recovery information collection method will be described in detail with reference toFIG.7.

In operation S190, the storage server1002may generate the recovery sequence table RST. For example, the storage node1100may generate the recovery sequence table RST based on the virtual machine group table VMGT, attributes of storage devices, and the recovery information. The storage node1100may store the recovery sequence table RST in the storage node memory1120. The recovery sequence table RST will be described in detail with reference toFIGS.8A and8B.

FIG.7is a flowchart illustrating operation S170ofFIG.5in detail. For convenience of description, a method of receiving the recovery information of the first storage device1200_1where the data associated with the first virtual machine VM1are stored will be described. Operation S170may include operation S171to operation S177. Referring toFIGS.1,5, and7, in operation S171, the storage node1100may send the read request to the first storage device1200_1. For example, the read request may include a logical block address LBA corresponding to data that the first virtual machine VM1requires.

In operation S172, the first storage device1200_1may determine whether the read error occurs. For example, the first storage device1200_1may read data corresponding to the read request from the nonvolatile memory1220in response to the read request. The first storage device1200_1may determine whether the read error occurs at the data read from the nonvolatile memory1220. When it is determined that the read error occurs, the first storage device1200_1performs operation S173. When it is determined that the read error does not occur, the first storage device1200_1performs operation S175.

In operation S173, the first storage device1200_1may perform the recovery operation. For example, the first storage device1200_1may perform the recovery operation based on the default recovery sequence.

In operation S174, the first storage device1200_1may update the recovery information. Alternatively, the first storage device1200_1may update a log page associated with the recovery information. For example, the recovery information may include information about execution content (or execution result or history) of the recovery operations of the recovery sequence. The recovery information may include a read error occurrence frequency, an identifier of a recovery operation performed from among recovery operations of the recovery sequence, an identifier of a recovery operation, in which a read error is corrected, from among the recovery operations of the recovery sequence, an error correction success ratio of the recovery operations, latencies of the recovery operations, an error correction ratio of the recovery operations, power consumption of the recovery operations, and the like. For example, the first storage device1200_1may update the read error occurrence frequency and may store an identifier of a recovery operation where the read error is corrected, as the recovery information.

In operation S175, the first storage device1200_1may send the read data and a completion entry corresponding to the read request to the storage node1100. The first storage device1200_1may send the error-corrected read data to the storage node1100.

In operation S176, the storage node1100may send a recovery information request to the first storage device1200_1. For example, the storage node1100may send a “Get Log Page” command including a log identifier corresponding to the recovery information to the first storage device1200_1.

In operation S177, the first storage device1200_1may send the recovery information to the storage node1100in response to the recovery information request. For example, the first storage device1200_1may send a “Get Log Page” completion and log data including the recovery information to the storage node1100in response to the “Get Log Page” command.

In an embodiment, the storage node1100may send the recovery information request to the storage node1100and receive the recovery information from the storage node1100periodically or repeatedly. For example, the storage server1002may repeatedly perform operation S170. The storage server1002may repeatedly perform the recovery information collection operation during a given time period.

In an embodiment, the storage server1002may perform operation S171to operation S175repeatedly. After collecting recovery information plural times, the first storage device1200_1may send the collected recovery information to the storage node1100in response to the recovery information request. In other words, after repeatedly performing the read operation (i.e., operation S171to operation S175), the storage node1100may request the accumulated recovery information from the first storage device1200_1.

FIGS.8A and8Bare diagrams illustrating examples of a recovery sequence table according to an example embodiment of the present disclosure. Referring toFIG.8A, the recovery sequence table RST may include a plurality of tables RT1to RT4. The first recovery table RT1is a table associated with the first virtual machine group VMGroup1having the first workload characteristic C1, the second recovery table RT2is a table associated with the second virtual machine group VMGroup2having the second workload characteristic C2, the third recovery table RT3is a table associated with the third virtual machine group VMGroup3having the third workload characteristic C3, and the fourth recovery table RT4is a table associated with the fourth virtual machine group VMGroup4having the fourth workload characteristic C4.

For convenience of description, below, it is assumed that storage attributes of the storage devices1200_1to1200_12include a first storage attribute A1and a second storage attribute A2. However, the present disclosure is not limited thereto. For example, the number of storage attributes of the storage devices1200_1to1200_12may increase or decrease depending on an embodiment. For example, each of the storage devices1200_1to1200_12may have one of the first storage attribute A1and the second storage attribute A2. It is assumed that the first to sixth storage devices1200_1to1200_6have the first storage attribute A1and the seventh to twelfth storage devices1200_7to1200_12have the second storage attribute A2.

The first recovery table RT1may include information (e.g., a first recovery sequence) about the optimal recovery sequence for the first storage attribute A1with respect to virtual machines having the first workload characteristic C1and may include information (e.g., the second recovery sequence) about the optimal recovery sequence for the second storage attribute A2with respect to virtual machines having the first workload characteristic C1. For example, there are four recovery operations and the first recovery sequence may refer to a sequence that is set (or aligned) to perform a fourth recovery operation, then performs a third recovery operation, then performs a first recovery operation, and then performs a second recovery operation. The second recovery sequence may include a recovery sequence that is set to perform the second recovery operation, then performs the third recovery operation, then performs the first recovery operation, and then performs the fourth recovery operation.

The second recovery table RT2may include information (e.g., a third recovery sequence) about the optimal recovery sequence for the first storage attribute A1with respect to virtual machines having the second workload characteristic C2and may include information (e.g., a fourth recovery sequence) about the optimal recovery sequence for the second storage attribute A2with respect to virtual machines having the second workload characteristic C2. The third recovery sequence may include a sequence that is set to perform the first recovery operation, then performs the second recovery operation, then performs the third recovery operation, and then performs the fourth recovery operation. The fourth recovery sequence may include a sequence that is set to perform the fourth recovery operation, then performs the third recovery operation, then performs the second recovery operation, and then performs the first recovery operation.

The third recovery table RT3may include information (e.g., a fifth recovery sequence) about the optimal recovery sequence for the first storage attribute A1with respect to virtual machines having the third workload characteristic C3and may include information (i.e., a sixth recovery sequence) about the optimal recovery sequence for the second storage attribute A2with respect to virtual machines having the third workload characteristic C3. The fifth recovery sequence may refer to a sequence that is set to perform the third recovery operation, then performs the fourth recovery operation, then performs the first recovery operation, and then performs the second recovery operation. The sixth recovery sequence includes a sequence that is set to perform the first recovery operation, then performs the fourth recovery operation, then performs the third recovery operation, and then performs the second recovery operation.

The fourth recovery table RT4may include information (e.g., a seventh recovery sequence) about the optimal recovery sequence for the first storage attribute A1with respect to virtual machines having the fourth workload characteristic C4and may include information (e.g., an eighth recovery sequence) about the optimal recovery sequence for the second storage attribute A2with respect to virtual machines having the fourth workload characteristic C4. The seventh recovery sequence may include a sequence that is set to perform the second recovery operation, then performs the first recovery operation, then performs the third recovery operation, and then performs the fourth recovery operation. The eighth recovery sequence may include a sequence that is set to perform the third recovery operation, then performs the first recovery operation, then performs the second recovery operation, and then performs the fourth recovery operation.

For example, referring toFIG.6, the first virtual machine group VMGroup1may include the first, third, and seventh virtual machines VM1, VM3, and VM7. The data associated with the first virtual machine VM1may be stored in the first storage device1200_1, the data associated with the third virtual machine VM3may be stored in the third storage device1200_3, and the data associated with the seventh virtual machine VM7may be stored in the seventh storage device1200_7. The first and third storage devices1200_1and1200_3may have the first storage attribute A1, and the seventh storage device1200_7may have the second storage attribute A2.

The storage node1100may select the optimal recovery sequence (i.e., the first recovery sequence) based on the recovery information received from the first and third storage devices1200_1and1200_3. The storage node1100may select the optimal recovery sequence (i.e., the second recovery sequence) based on the recovery information received from the seventh storage device1200_7. The remaining recovery sequences (i.e., the third to eighth recovery sequences) are selected in a manner similar to those described above, and thus, additional description will be omitted to avoid redundancy.

FIG.8Ashows the recovery sequence table RST that stores one optimal recovery sequence with regard to the workload characteristic and the storage attribute. In contrast,FIG.8Bshows that the recovery sequence table RST may store a plurality of optimal recovery sequences with regard to the workload characteristic and the storage attribute. For brevity of drawing,FIG.8Bshows only the first recovery table RT1.

The first recovery table RT1includes recovery sequences associated with the first virtual machine group VMGroup1having the first workload characteristic C1. The first recovery table RT1includes first to third entries with regard to the first storage attribute A1. For example, the first entry includes a recovery sequence that is set (or aligned) to perform the fourth recovery operation, then performs the third recovery operation, then performs the first recovery operation, and then performs the second recovery operation. The second entry includes a recovery sequence that is set to perform the fourth recovery operation, then performs the third recovery operation, then performs the second recovery operation, and then performs the first recovery operation. The third entry includes a recovery sequence that is set to perform the fourth recovery operation, then performs the first recovery operation, then performs the third recovery operation, and then performs the second recovery operation.

The first recovery table RT1further includes fourth to sixth entries with regard to the second storage attribute A2. For example, the fourth entry includes a recovery sequence that is set to perform the second recovery operation, then performs the third recovery operation, then performs the first recovery operation, and then performs the fourth recovery operation. The fifth entry includes a recovery sequence that is set to perform the second recovery operation, then performs the third recovery operation, then performs the fourth recovery operation, and then performs the first recovery operation. The sixth entry includes a recovery sequence that is set to perform the second recovery operation, then performs the first recovery operation, then performs the third recovery operation, and then performs the fourth recovery operation.

The storage node1100may provide the optimal recovery sequence corresponding to one of the first to third entries to a storage device having the first storage attribute A1from among storage devices corresponding to the first virtual machine group VMGroup1. The storage node1100may select one of a plurality of optimal recovery sequences based on other storage attributes or other factors and may provide the selected optimal recovery sequence to the storage device. Alternatively, when the error occurrence frequency is high in the previously provided optimal recovery sequence (e.g., the recovery sequence corresponding to the first entry), the storage node1100may provide the optimal recovery sequence corresponding to the second entry to the storage device.

FIG.9is a flowchart illustrating operation S200ofFIG.4in more detail. Referring toFIGS.1,4, and9, operation S200may include operation S210to operation S270. In operation S210, the storage server1002may detect a new virtual machine. For example, the storage server1002may detect the user input/output from the new virtual machine.

In operation S220, the storage server1002may extract a workload characteristic of the new virtual machine. For example, the storage node1100masy monitor the read/write request and the data that are exchanged between the new virtual machine and a storage device allocated to the new virtual machine. The storage node1100may analyze the read/write request and the data exchanged between the new virtual machine and the corresponding storage device and may determine the workload characteristic of the new virtual machine.

In operation S230, the storage server1002may determine whether there is a group of virtual machines similar in characteristic to the new virtual machine. For example, whether the workload of the new virtual machine has one of the first to fourth workload characteristics C1to C4may be determined. When it is determined that there is a group of virtual machines similar in characteristic to the new virtual machine, the storage server1002performs operation S240to operation S270. When it is determined that there is no group of virtual machines similar in characteristic to the new virtual machine, the storage server1002performs operation S130ofFIG.5.

For example, when there is no group of virtual machines similar in characteristic to the new virtual machine (that is, the method proceeds to operation S130), the storage server1002may update the virtual machine group table VMGT. The storage server1002may add a new workload characteristic (e.g., a fifth workload characteristic) and an identifier of the new virtual machine to the virtual machine group table VMGT. The storage server1002may receive the attribute of the storage device corresponding to the new virtual machine. The storage server1002may receive the recovery information of the storage device corresponding to the new virtual machine. The storage server1002may update the recovery sequence table RST based on the recovery information. The storage server1002may select the optimal recovery sequence of the storage device corresponding to the new virtual machine, based on the recovery information. The storage server1002may generate a fifth recovery table with regard to the fifth workload characteristic. The storage server1002may add the fifth recovery table to the recovery sequence table RST. The fifth recovery table may include the optimal recovery sequence that is selected in consideration of the storage attribute. Alternatively, in an embodiment, the storage server1002may include the new virtual machine in a group of virtual machines having the most similar characteristic by using the clustering technique.

In operation S240, the storage server1002may update the virtual machine group table VMGT. For example, the storage server1002may add the identifier of the new virtual machine to an identifier of a virtual machine corresponding to the workload characteristic, which is the same or similar to the workload characteristic of the new virtual machine, in the virtual machine group table VMGT.

In operation S250, the storage server1002may receive attribute information of the storage device from the storage device allocated to new virtual machine. In operation S260, the storage server1002may select the optimal recovery sequence based on the recovery sequence table RST. The storage server1002may select the optimal recovery sequence with reference to the recovery sequence table RST, that is, based on the workload characteristic of the new virtual machine and the storage attribute of the storage device allocated to the new virtual machine.

In operation S270, the storage server1002may provide the optimal recovery sequence to the storage device. For example, the storage server1002may allocate the selected optimal recovery sequence to the storage device allocated to the new virtual machine.

FIGS.10A to10Care diagrams illustrating examples of an operation of a server system ofFIG.1. A method of allocating an optimal recovery sequence based on a recovery sequence table will be described with reference toFIGS.10A to10C. Below, for brevity of drawing and for convenience of description, additional description associated with components the same as or similar to the above components will be omitted to avoid redundancy. It is assumed that a new virtual machine (i.e., a thirteenth virtual machine VM13) is driven on the client server1001.

In operation S301, the thirteenth virtual machine VM13may send a new request to the storage node1100. In operation S302, the storage node1100may allocate a thirteenth storage device1200_13to the thirteenth virtual machine VM13. For example, the storage node1100may store data associated with the thirteenth virtual machine VM13in the thirteenth storage device1200_13. The storage node1100may allocate the default recovery sequence to the thirteenth storage device1200_13. When the read error occurs, the thirteenth storage device1200_13may perform the recovery operations based on the default recovery sequence.

The storage node1100may detect the user input/output I/O associated with the thirteenth virtual machine VM13. The storage node1100may monitor the read/write request and the data that are exchanged between the thirteenth virtual machine VM13and the thirteenth storage device1200_13. The storage node1100may analyze the read/write request and the data exchanged the thirteenth virtual machine VM13and the thirteenth storage device1200_13and may determine the workload characteristic of the thirteenth storage device1200_13. For example, the storage node1100may determine whether the thirteenth virtual machine VM13has the first workload characteristic C1. The storage node1100may determine that the thirteenth virtual machine VM13is similar to the first virtual machine group VMGroup1with regards to the workload characteristic. The storage node1100may determine that the thirteenth virtual machine VM13belongs to the first virtual machine group VMGroup1.

In operation S303, the recovery manager1130may update the virtual machine group table VMGT. Because the thirteenth virtual machine VM13has the first workload characteristic C1, the recovery manager1130may update the virtual machine group table VMGT such that information about identifiers (i.e., the first, third, seventh, and thirteenth virtual machine identifiers VM1_ID, VM3_ID, VM7_ID, and VM13_ID) of virtual machines having the first workload characteristic C1is included therein. That is, the recovery manager1130may add the thirteenth virtual machine identifier VM13_ID to the information about the identifiers of the virtual machines having the first workload characteristic C1. As such, the virtual machine group table VMGT may include the first workload characteristic C1and the information about the identifiers (i.e., the first, third, seventh, and thirteenth virtual machine identifiers VM1_ID, VM3_ID, VM7_ID, and VM13_ID) of the virtual machines having the first workload characteristic C1.

Referring toFIG.10B, in operation S304, the storage node1100may send the attribute information request to the thirteenth storage device1200_13. For example, the storage node1100may send the “Get Log Page” command including a log identifier. For example, the log identifier may correspond to SMART and/or Health Information. Alternatively, the storage node1100may send a “Get Feature” command. Alternatively, the storage node1100may send a telemetry information request.

In operation S305, the storage node1100may receive the attribute information from the thirteenth storage device1200_13. The thirteenth storage device1200_13may send the storage attribute information to the storage node1100in response to the attribute information request. For example, the thirteenth storage device1200_13may send the attribute information including the first storage attribute A1to the storage node1100. The storage node1100may determine that the thirteenth storage device1200_13has the first storage attribute A1, based on the received attribute information.

Alternatively, in an embodiment, the storage node1100may receive telemetry information from the thirteenth storage device1200_13. The thirteenth storage device1200_13may send the telemetry information to the storage node1100in response to the telemetry information request. The storage node1100may determine the attribute of the thirteenth storage device1200_13based on the received telemetry information.

Referring toFIG.10C, the storage node1100may select the optimal recovery sequence of the thirteenth storage device1200_13. The storage node1100may select the optimal recovery sequence based on the recovery sequence table RST. Because the thirteenth virtual machine VM13has the first workload characteristic C1, the storage node1100may refer to the first recovery table RT1. Because the thirteenth storage device1200_13has the first storage attribute A1, the storage node1100may select the recovery sequence corresponding to the first storage attribute A1from the first recovery table RT1. That is, the storage node1100may select information about the optimal recovery sequence for the first storage attribute A1(i.e., information about the first recovery sequence being the sequence having the order of the fourth recovery operation ROP4, the third recovery operation ROP3, the first recovery operation ROP1, and the second recovery operation ROP2).

In operation S306, the storage node1100may provide the thirteenth storage device1200_13with the optimal recovery sequence including the first recovery sequence (i.e., the sequence having the order of the fourth recovery operation ROP4, the third recovery operation ROP3, the first recovery operation ROP1, and the second recovery operation ROP2). For example, the storage node1100may send the “Set Feature” command including the first recovery sequence. Alternatively, the storage node1100may allocate the first recovery sequence being the optimal recovery sequence through a “Vendor” command including the first recovery sequence.

The thirteenth storage device1200_13may receive the optimal recovery sequence including the first recovery sequence. When the read error occurs, the thirteenth storage device1200_13may perform the recovery operations based on the optimal recovery sequence.

For example, when the read error occurs, the thirteenth storage device1200_13may preferentially perform the fourth recovery operation ROP4. When the read error is corrected through the fourth recovery operation ROP4, the thirteenth storage device1200_13may not perform the remaining recovery operations (i.e., the third recovery operation ROP3, the first recovery operation ROP1, and the second recovery operation ROP2). When the read error is not corrected through the fourth recovery operation ROP4, the thirteenth storage device1200_13may perform the third recovery operation ROP3. When the read error is corrected through the third recovery operation ROP3, the thirteenth storage device1200_13may not perform the remaining recovery operations (i.e., the first recovery operation ROP1and the second recovery operation ROP2). When the read error is not corrected through the third recovery operation ROP3, the thirteenth storage device1200_13may perform the first recovery operation ROP1. When the read error is corrected through the first recovery operation ROP1, the thirteenth storage device1200_13may not perform the remaining recovery operation (i.e., the second recovery operation ROP2). When the read error is not corrected through the first recovery operation ROP1, the thirteenth storage device1200_13may perform the second recovery operation ROP2.

FIG.11is a flowchart illustrating an example of an operation of a storage server ofFIG.1. Referring toFIGS.1,4, and11, in operation S400, the storage server1002may generate the recovery sequence table RST based on the workload characteristic and the attribute of the storage device. In operation S500, the storage server1002may allocate the optimal recovery sequence to a storage device to be allocated to a new virtual machine, based on the recovery sequence table RST. Operation S400and operation S500are similar to operation S100and operation S200ofFIG.4, and thus, additional description will be omitted to avoid redundancy.

In operation S600, the storage server1002may change the optimal recovery sequence. For example, the storage attribute of the first storage device1200_1may change over time. As such, the storage server1002may change the optimal recovery sequence of the first storage device1200_1.

FIG.12is a flowchart illustrating operation S600ofFIG.11in detail. InFIG.12, it is assumed that a storage device is the thirteenth storage device1200_13among the plurality of storage devices1200_1to1200_n. However, the present disclosure is not limited thereto. For example, the remaining storage devices may operate to be similar thereto.

Referring toFIGS.1,11, and12, in operation S610, the storage node1100may repeatedly perform the monitoring operation for the thirteenth storage device1200_13. For example, the storage node1100may monitor the storage attribute of the thirteenth storage device1200_13.

In detail, operation S610may include operation S611to operation S613. In operation S611, the storage node1100may send the attribute information request to the thirteenth storage device1200_13. For example, the attribute information request may include the “Get Log Page” command including the log identifier. For example, the log identifier may correspond to the SMART and/or Health Information. Alternatively, the storage node1100may send the “Get Feature” command. Alternatively, the storage node1100may send the telemetry information request.

In operation S612, the thirteenth storage device1200_13may send the attribute information to the storage node1100. For example, the thirteenth storage device1200_13may send the changed attribute information to the storage node1100in response to the attribute information request. Alternatively, the thirteenth storage device1200_13may send the changed telemetry information to the storage node1100in response to the telemetry information request.

In operation S613, the storage node1100may determine whether the change condition is satisfied. For example, when the storage attribute (or characteristic) of the thirteenth storage device1200_13changes, when the monitored numerical value reaches a threshold value, or when the read error frequency increases, the change condition may be satisfied. When the change condition is satisfied, the storage node1100performs operation S620. When the change condition is not satisfied, the storage node1100performs operation S611.

For example, based on the received attribute information (or telemetry information), the storage node1100may determine that a storage attribute of the thirteenth storage device1200_13changes from the first storage attribute A1to the second storage attribute A2. As such, the storage node1100may determine that the change condition for the recovery sequence of the thirteenth storage device1200_13is satisfied.

In operation S620, the storage node1100may select a new optimal recovery sequence. For example, referring toFIG.10C, because the thirteenth storage device1200_13has the second storage attribute A2, the storage node1100may select the recovery sequence corresponding to the second storage attribute A2from the first recovery table RT1. That is, the storage node1100may select information about the optimal recovery sequence for the second storage attribute A2(i.e., information about the second recovery sequence being the sequence having the order of the second recovery operation ROP2, the third recovery operation ROP3, the first recovery operation ROP1, and the fourth recovery operation ROP4).

In operation S630, the storage node1100may send the new optimal recovery sequence to the thirteenth storage device1200_13. For example, the storage node1100may provide the thirteenth storage device1200_13with the optimal recovery sequence including the second recovery sequence (i.e., the sequence having the order of the second recovery operation ROP2, the third recovery operation ROP3, the first recovery operation ROP1, and the fourth recovery operation ROP4).

FIGS.13A to13Care diagrams illustrating examples of an operation of a server system ofFIG.1. The method in which the storage node1100transfers one optimal recovery sequence to each of the plurality of storage devices1200_1to1200_nhas been described with reference toFIGS.1to12. The method in which the storage node1100transfers a plurality of optimal recovery sequences to each of the plurality of storage devices1200_1to1200_nis described with reference toFIGS.13A to13C. It is assumed that new virtual machines (i.e., fourteenth and fifteenth virtual machines VM14and VM15) are driven on the client server1001.

In operation S701, the fourteenth virtual machine VM14may send a first new request to the storage node1100. In operation S702, the storage node1100may allocate a first namespace NS1of the fourteenth storage device1200_14to the fourteenth virtual machine VM14. For example, the storage node1100may store data associated with the fourteenth virtual machine VM14in the first namespace NS1of the fourteenth storage device1200_14.

In operation S703, the fifteenth virtual machine VM15may send a second new request to the storage node1100. In operation S704, the storage node1100may allocate a second namespace NS2of the fourteenth storage device1200_14to the fifteenth virtual machine VM15. For example, the storage node1100may store data associated with the fifteenth virtual machine VM15in the second namespace NS2of the fourteenth storage device1200_14.

The storage node1100may detect the user input/output I/O associated with the fourteenth virtual machine VM14. The storage node1100may monitor the read/write request and the data that are exchanged between the fourteenth virtual machine VM14and the first namespace NS1of the fourteenth storage device1200_14. The storage node1100may analyze the read/write request and the data exchanged the fourteenth virtual machine VM14and the first namespace NS1of the fourteenth storage device1200_14and may determine the workload characteristic of the fourteenth virtual machine VM14. For example, the storage node1100may determine whether the fourteenth virtual machine VM14has the second workload characteristic C2.

The storage node1100may detect the user input/output I/O associated with the fifteenth virtual machine VM15. The storage node1100may monitor the read/write request and the data that are exchanged between the fifteenth virtual machine VM15and the second namespace NS2of the fourteenth storage device1200_14. The storage node1100may analyze the read/write request and the data exchanged the fifteenth virtual machine VM15and the second namespace NS2of the fourteenth storage device1200_14and may determine the workload characteristic of the fifteenth virtual machine VM15. For example, the storage node1100may determine whether the fifteenth virtual machine VM15has the first workload characteristic C1.

In operation S705, the recovery manager1130may update the virtual machine group table VMGT. For example, because the fourteenth virtual machine VM14has the second workload characteristic C2, the recovery manager1130may update the virtual machine group table VMGT such that information about identifiers (i.e., the second, fourth, twelfth, and fourteenth virtual machine identifiers VM2_ID, VM4_ID, VM12_ID, and VM14ID) of virtual machines having the second workload characteristic C2is included therein. That is, the recovery manager1130may add the fourteenth virtual machine identifier VM14ID to the information about the identifiers of the virtual machines having the second workload characteristic C2. As such, the virtual machine group table VMGT may include the second workload characteristic C2and the information about the identifiers (i.e., the second, fourth, twelfth, and fourteenth virtual machine identifiers VM2_ID, VM4_ID, VM12_ID, and VM14ID) of the virtual machines having the second workload characteristic C2.

For example, because the fifteenth virtual machine VM15has the first workload characteristic C1, the recovery manager1130may update the virtual machine group table VMGT such that information about identifiers (i.e., the first, third, seventh, thirteenth, and fifteenth virtual machine identifiers VM1_ID, VM3_ID, VM7_ID, VM13_ID, and VM15ID) of virtual machines having the first workload characteristic C1is included therein. That is, the recovery manager1130may add the fifteenth virtual machine identifier VM15ID to the information about the identifiers of the virtual machines having the first workload characteristic C1. As such, the virtual machine group table VMGT may include the first workload characteristic C1and the information about the identifiers (i.e., the first, third, seventh, thirteenth, and fifteenth virtual machine identifiers VM1_ID, VM3_ID, VM7_ID, VM13_ID, and VM15ID) of the virtual machines having the first workload characteristic C1.

Referring toFIG.13B, in operation S706, the storage node1100may send the attribute information request to the first namespace NS1of the fourteenth storage device1200_14. In operation S707, the storage node1100may receive the attribute information from the fourteenth storage device1200_14. The fourteenth storage device1200_14may send the storage attribute information of the first namespace NS1to the storage node1100in response to the attribute information request. For example, the fourteenth storage device1200_14may send the attribute information including the first storage attribute A1to the storage node1100. The storage node1100may determine that the first namespace NS1of the fourteenth storage device1200_14has the first storage attribute A1, based on the received attribute information.

In operation S708, the storage node1100may send the attribute information request to the second namespace NS2of the fourteenth storage device1200_14. In operation S709, the storage node1100may receive the attribute information from the fourteenth storage device1200_14. The fourteenth storage device1200_14may send the storage attribute information of the second namespace NS2to the storage node1100in response to the attribute information request. For example, the fourteenth storage device1200_14may send the attribute information including the second storage attribute A2to the storage node1100. The storage node1100may determine that the second namespace NS2of the fourteenth storage device1200_14has the second storage attribute A2, based on the received attribute information.

Referring toFIG.13C, the storage node1100may select the optimal recovery sequence for the first namespace NS1of the fourteenth storage device1200_14. The storage node1100may select the optimal recovery sequence based on the recovery sequence table RST. Because the fourteenth virtual machine VM14has the second workload characteristic C2, the storage node1100may refer to the second recovery table RT2. Because the first namespace NS1of the fourteenth storage device1200_14has the first storage attribute A1, the storage node1100may select the recovery sequence corresponding to the first storage attribute A1from the second recovery table RT2. That is, the storage node1100may select information about the optimal recovery sequence for the first storage attribute A1(i.e., information about the third recovery sequence being the sequence having the order of the first recovery operation ROP1, the second recovery operation ROP2, the third recovery operation ROP3, and the fourth recovery operation ROP4).

In operation S710, the storage node1100may provide the first namespace NS1of the fourteenth storage device1200_14with the optimal recovery sequence including the third recovery sequence (i.e., the sequence having the order of the first recovery operation ROP1, the second recovery operation ROP2, the third recovery operation ROP3, and the fourth recovery operation ROP4).

The storage node1100may select the optimal recovery sequence for the second namespace NS2of the fourteenth storage device1200_14. The storage node1100may select the optimal recovery sequence based on the recovery sequence table RST. Because the fifteenth virtual machine VM15has the first workload characteristic C1, the storage node1100may refer to the first recovery table RT1. Because the second namespace NS2of the fourteenth storage device1200_14has the second storage attribute A2, the storage node1100may select the recovery sequence corresponding to the second storage attribute A2from the first recovery table RT1. That is, the storage node1100may select information about the optimal recovery sequence for the second storage attribute A2(i.e., information about the second recovery sequence being the sequence having the order of the second recovery operation ROP2, the third recovery operation ROP3, the first recovery operation ROP1, and the fourth recovery operation ROP4).

In operation S711, the storage node1100may provide the second namespace NS2of the fourteenth storage device1200_14with the optimal recovery sequence including the second recovery sequence (i.e., the sequence having the order of the second recovery operation ROP2, the third recovery operation ROP3, the first recovery operation ROP1, and the fourth recovery operation ROP4).

As described above, the storage node1100may provide a plurality of optimal recovery sequences to a storage device. That is, the storage node1100may provide the optimal recovery sequence for each namespace of the storage device. The storage device may apply different recovery sequences to namespaces of different attributes. The storage device may prevent the reduction of performance by performing the optimal recovery sequence for each namespace.

FIG.14is a diagram of a data center2000to which a memory device according to an embodiment is applied.

Referring toFIG.14, the data center2000may be a facility that collects various types of pieces of data and provides services and be referred to as a data storage center. The data center2000may be a system for operating a search engine and a database, and may be a computing system used by companies, such as banks, or government agencies. The data center2000may include application servers2100to2100_nand storage servers2200to2200_m. The number of application servers2100to2100_nand the number of storage servers2200to2200_mmay be variously determined according to embodiments. The number of application servers2100to2100_nmay be different from the number of storage servers2200to2200_m.

The application server2100or the storage server2200may include at least one of processors2110and2210and memories2120and2220. The storage server2200will now be described as an example. The processor2210may control all operations of the storage server2200, access the memory2220, and execute instructions and/or data loaded in the memory2220. The memory2220may be a double-data-rate synchronous DRAM (DDR SDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), a dual in-line memory module (DIMM), Optane DIMM, and/or a non-volatile DIMM (NVMDIMM). In some embodiments, the numbers of processors2210and memories2220included in the storage server2200may be variously determined. In an embodiment, the processor2210and the memory2220may provide a processor-memory pair. In an embodiment, the number of processors2210may be different from the number of memories2220. The processor2210may include a single-core processor or a multi-core processor. The above description of the storage server2200may be similarly applied to the application server2100. In some embodiments, the application server2100may not include a storage device2150. The storage server2200may include at least one storage device2250. The number of storage devices2250included in the storage server2200may be variously determined according to embodiments.

The application servers2100to2100_nmay communicate with the storage servers2200to2200_mthrough a network2300. The network2300may be implemented by using a fiber channel (FC) or Ethernet. In this case, the FC may be a medium used for relatively high-speed data transmission and use an optical switch with high performance and high availability. The storage servers2200to2200_mmay be provided as file storages, block storages, or object storages according to an access method of the network2300.

In an embodiment, the network2300may be a storage-dedicated network, such as a storage area network (SAN). For example, the SAN may be an FC-SAN, which uses an FC network and is implemented according to an FC protocol (FCP). As another example, the SAN may be an Internet protocol (IP)-SAN, which uses a transmission control protocol (TCP)/IP network and is implemented according to a SCSI over TCP/IP or Internet SCSI (iSCSI) protocol. In another embodiment, the network2300may be a general network, such as a TCP/IP network. For example, the network2300may be implemented according to a protocol, such as FC over Ethernet (FCoE), network attached storage (NAS), and NVMe over Fabrics (NVMe-oF).

Hereinafter, the application server2100and the storage server2200will mainly be described. A description of the application server2100may be applied to another application server2100_n, and a description of the storage server2200may be applied to another storage server2200_m.

The application server2100may store data, which is requested by a user or a client to be stored, in one of the storage servers2200to2200_mthrough the network2300. Also, the application server2100may obtain data, which is requested by the user or the client to be read, from one of the storage servers2200to2200_mthrough the network2300. For example, the application server2100may be implemented as a web server or a database management system (DBMS).

The application server2100may access a memory2120_nor a storage device2150_n, which is included in another application server2100_n, through the network2300. Alternatively, the application server2100may access memories2220to2220_mor storage devices2250to2250_m, which are included in the storage servers2200to2200_m, through the network2300. Thus, the application server2100may perform various operations on data stored in application servers2100to2100_nand/or the storage servers2200to2200_m. For example, the application server2100may execute an instruction for moving or copying data between the application servers2100to2100_nand/or the storage servers2200to2200_m. In this case, the data may be moved from the storage devices2250to2250_mof the storage servers2200to2200_mto the memories2120to2120_nof the application servers2100to2100_ndirectly or through the memories2220to2220_mof the storage servers2200to2200_m. The data moved through the network2300may be data encrypted for security or privacy.

The storage server2200will now be described as an example. An interface2254may provide physical connection between a processor2210and a controller2251and a physical connection between a network interface card (NIC)2240and the controller2251. For example, the interface2254may be implemented using a direct attached storage (DAS) scheme in which the storage device2250is directly connected with a dedicated cable. For example, the interface2254may be implemented by using various interface schemes, such as ATA, SATA, e-SATA, an SCSI, SAS, PCI, PCIe, NVMe, IEEE 1394, a USB interface, an SD card interface, an MMC interface, an eMMC interface, a UFS interface, an eUFS interface, and/or a CF card interface.

The storage server2200may further include a switch2230and the NIC (Network InterConnect)2240. The switch2230may selectively connect the processor2210to the storage device2250or selectively connect the NIC2240to the storage device2250via the control of the processor2210.

In an embodiment, the NIC2240may include a network interface card and a network adaptor. The NIC2240may be connected to the network2300by a wired interface, a wireless interface, a Bluetooth interface, or an optical interface. The NIC2240may include an internal memory, a digital signal processor (DSP), and a host bus interface and be connected to the processor2210and/or the switch2230through the host bus interface. The host bus interface may be implemented as one of the above-described examples of the interface2254. In an embodiment, the NIC2240may be integrated with at least one of the processor2210, the switch2230, and the storage device2250.

In the storage servers2200to2200_mor the application servers2100to2100_n, a processor (E.g., the processor2110or2210) may transmit a command to storage devices2150to2150_nand2250to2250_mor the memories2120to2120_nand2220to2220_mand program or read data. In this case, the data may be data of which an error is corrected by an ECC engine. The data may be data on which a data bus inversion (DBI) operation or a data masking (DM) operation is performed, and may include cyclic redundancy code (CRC) information. The data may be data encrypted for security or privacy.

Storage devices2150to2150_nand2250to2250_mmay transmit a control signal and a command/address signal to NAND flash memory devices2252to2252_min response to a read command received from the processor. Thus, when data is read from the NAND flash memory devices2252to2252_m, a read enable (RE) signal may be input as a data output control signal, and thus, the data may be output to a DQ bus. A data strobe signal DQS may be generated using the RE signal. The command and the address signal may be latched in a page buffer depending on a rising edge or falling edge of a write enable (WE) signal.

The controller2251may control all operations of the storage device2250. In an embodiment, the controller2251may include an SRAM. The controller2251may write data to the NAND flash memory device2252in response to a write command or read data from the NAND flash memory device2252in response to a read command. For example, the write command and/or the read command may be provided from any one of the processors2210to2210_mof the storage servers2200to2200_m, and the processors2110to2110_nof the application servers2100to2100_n. DRAM3253may temporarily store (or buffer) data to be written to the NAND flash memory device2252or data read from the NAND flash memory device2252. Also, the DRAM3253may store metadata. Here, the metadata may be user data or data generated by the controller2251to manage the NAND flash memory device2252. The storage device2250may include a secure element (SE) for security or privacy.

In an embodiment, the storage servers2200to2200_mmay include the recovery manager described with reference toFIGS.1to13Cor may operate based on the method of providing the optimal recovery sequence, which is described with reference toFIGS.1to13C. For example, based on the method described with reference toFIGS.1to13C, the storage server2200may generate and manage the virtual machine group table VMGT, may collect recovery information, and may generate and manage the recovery sequence table RST.

In an embodiment, the storage server2200may provide the optimal recovery sequence to any other storage server (e.g., the storage server2200_m) over the network NT. For example, the storage server2200may receive the optimal recovery sequence request including the workload characteristic and the storage attribute from the storage server2200_m. The storage server2200may select the optimal recovery sequence with reference to the recovery sequence table RST, based on the workload characteristic and the storage attribute thus received. The storage server2200may provide the optimal recovery sequence to the storage server2200_mover the network NT. The storage server2200_mmay transfer the received optimal recovery sequence to the storage device2250_m. That is, the storage server2200may provide the optimal recovery sequence to the remote storage device2250_mconnected over the network NT.

In an embodiment, the storage server2200may provide or share the recovery sequence table RST to or with the storage server2200_mover the network NT. The storage server2200may send the virtual machine group table VMGT, the collected recovery information, and/or the recovery sequence table RST to the storage server2200_mover the network NT.

The storage server2200_mmay receive the virtual machine group table VMGT, the recovery information, and/or the recovery sequence table RST from the storage server2200_m. The storage server2200_mmay monitor the recovery information and may not internally generate the recovery sequence table RST. The storage server2200_mmay provide the optimal recovery sequence to the storage device based on the recovery sequence table RST provided to the storage server2200.

In the above example embodiments, components according to the present disclosure are described by using the terms “first”, “second”, “third”, etc. However, the terms “first”, “second”, “third”, etc. may be used to distinguish components from each other and do not limit the present disclosure. For example, the terms “first”, “second”, “third”, etc. do not involve an order or a numerical meaning of any form.

In the above example embodiments, components according to embodiments of the present disclosure are described by using a unit, a module, a layer, or a block. The unit, module, layer, or block may be implemented with various hardware devices, such as an integrated circuit (IC), an application specific IC (ASIC), a field programmable gate array (FPGA), and a complex programmable logic device (CPLD); firmware driven in hardware devices; software such as an application; or a combination of a hardware device and software. Also, the block may include circuits implemented with semiconductor elements in an integrated circuit, or circuits enrolled as an intellectual property (IP).

According to the present disclosure, a storage server with improved performance and an operation method of the storage server may be provided.

While the present disclosure has been described with reference to example embodiments thereof, it will be apparent to those of ordinary skill in the art that various changes and modifications may be made thereto without departing from the spirit and scope of the present disclosure as set forth in the following claims and their equivalents.