Patent ID: 12197325

DESCRIPTION OF EMBODIMENTS

Hereinafter, serval examples of the present invention will be described with reference to the drawings. In all the examples, it is assumed that constituents to which the same reference numerals are given are substantially the same constituents. Since processes executed by processors are executed while appropriately using storage resources (for example, memories) and communication interface devices (for example, communication ports), subjects of the processes may be considered to be processors. The processors may have dedicated hardware other than central processing units (CPUs).

Example 1

A storage apparatus according to Example 1 will be described with reference toFIGS.1to9.

FIG.1is a diagram illustrating a storage apparatus according to Example 1.

A storage apparatus100includes controllers110and120and a drive box130that has a plurality of hard disk drives or a plurality of solid state drives.

The controller110includes a processor111, a memory112, a frontend interface113, a backend interface114, a PCI express (PCIe) switch115, and a management processor116. Similarly, the controller120includes a processor121, a memory122, a frontend interface123, a backend interface124, a PCIe switch125, and a management processor126. The processors111and121include a plurality of processor cores (not illustrated) therein.

A host apparatus (not illustrated) that accesses the storage apparatus100is connected to the storage apparatus100via the frontend interfaces113and123. The host apparatus and the frontend interfaces113and123are connected with transfer lines such as Fibre Channel cables or Ethernet cables.

Alternatively, the host apparatus and the frontend interfaces113and123may be configured to be connected via a storage area network that includes a plurality of transfer lines and a plurality of switches. The frontend interfaces113and123convert a data transfer protocol between the host apparatus and the storage apparatus100and a data transfer protocol inside the controllers110and120. The frontend interfaces113and123include protocol chips.

The drive box130is connected to the controllers110and120via the backend interfaces114and124. The backend interfaces114and124convert the data transfer protocol inside the controllers110and120and a data transfer protocol between the controllers110and120and the drive box130. When drives in the drive box are NVMeSSDs of PCIe connection, the backend interfaces114and124are PCIe switches that do not execute protocol conversion.

The processors111and121control data transfer between the host apparatus connected via the frontend interfaces113and123and the drive box130connected via the backend interfaces114and124. Further, the processors111and121control data transfer between the controllers.

The memories112and122are main memories of the processors111and121, respectively, and store programs (storage control programs or the like) which are executed by the processors111and121, and management tables or the like which are referred to by the processors111and121. The memories112and122are also used as cache memories of the controllers110and120, respectively.

FIG.2is a diagram illustrating a configuration of the controllers110and120according to Example 1.

InFIG.2, the backend interfaces114and124inFIG.1are not illustrated to facilitate description.

The processors111and121include NTBs211and231, respectively. The NTBs211and231are connected with a link140. The processors111and121can communicate with each other via the link140. In this way, dual controllers are configured by two controllers110and120in the storage apparatus100. The processors111and121transfer user data duplicated between the two controllers or control data of the controllers over the link140.

The frontend interface113includes interrupt setting registers217and218. The frontend interface113can send an interrupt such as a message signaled interrupt (MSI) to the address set in the interrupt setting registers217and218. For example, in the interrupt setting register217, an address to an interrupt controller (not illustrated) contained by the processor111is set. In the interrupt setting register218, an address to an interrupt controller (not illustrated) contained by the processor121is set. In this way, the frontend interface113can send an interrupt to the processor111or121. The interrupt is used in association with an error notification from the frontend interface to the processor or an operation of a control queue for data transfer to be described later.

Similarly, the frontend interface123includes interrupt setting registers237and238. The frontend interface123can send an interrupt such as an MSI to the address set in the interrupt setting registers237and238. For example, in the interrupt setting register237, an address to an interrupt controller contained by the processor121is set. In the interrupt setting register238, an address to an interrupt controller contained by the processor111is set. In this way, the frontend interface123can send an interrupt to the processor111or121.

The interrupts sent to the processors111and121by the frontend interfaces113and123use only an MSI or an MSI-X which can pass through the NTB as a PCIe write request and do not use a legacy interrupt (INTx).

The PCIe switch115includes NTBs214and215. The NTB214is connected to a port212of the processor111via a link117. The NTB215is connected to a port233of the processor121via a link128.

Similarly, the PCIe switch125includes NTBs234and235. The NTB234is connected to a port232of the processor121via a link127. The NTB235is connected to a port213of the processor111via a link118.

The port213of the processor111and the port233of the processor121set a downstream port containment (DPC) function of the PCIe to be enabled. Accordingly, for example, when the controller110is detached from the storage apparatus100, it is not necessary for the controller120remaining in the storage apparatus100to treat a link-down as an error even if the links118and128are turned down. Therefore, an operation can be continued.

Similarly, for example, when the controller120is detached from the storage apparatus100, it is not necessary for the controller110remaining in the storage apparatus100to treat a link-down as an error even if the links118and128are turned down. Therefore, an operation can be continued.

The management processor116connects a Root Port216to the PCIe switch115and functions as a Root Complex for the frontend interface113and the PCIe switch115.

Similarly, the management processor126connects a Root Port236to the PCIe switch125and functions as a Root Complex for the frontend interface123and the PCIe switch125.

The management processor116is not involved in data transfer control between the host apparatus and the storage apparatus100and executes an initial setting or the like of the frontend interface113and the PCIe switch115including the NTBs214and215. An alternative function of the management processor116may be contained in the PCIe switch115.

Similarly, the management processor126is not involved in data transfer control between the host apparatus and the storage apparatus100and executes an initial setting or the like of the frontend interface123and the PCIe switch125including the NTBs234and235. An alternative function of the management processor126may be contained in the PCIe switch125.

The memories112and122have an outbound queue (OQ) and an inbound queue (IQ) for controlling data transfer between the processors111and121and the frontend interfaces113and123. The OQ is a queue for controlling data transfer from the frontend interface to the processor and the IQ is a queue for controlling data transfer from the processor to the frontend interface.

The memory112includes an OQ201and an IQ202controlling data transfer between the processor111and the frontend interface113. The memory112includes an OQ203and an controlling data transfer between the processor111and the frontend interface123.

Similarly, the memory122includes an OQ221and an IQ222controlling data transfer between the processor121and the frontend interface123. The memory122includes an OQ223and an IQ224controlling data transfer between the processor121and the frontend interface113. In this way, in the storage apparatus according to Example 1, the control queues of the frontend interfaces are disposed over the two controllers.

InFIG.2, the OQ201is displayed as “0000,” the IQ202is displayed as “IQ00,” the OQ203is displayed as “OQ01,” the IQ204is displayed as “IQ01,” the OQ221is displayed as “OQ11,” the IQ222is displayed as “IQ11,” the OQ223is displayed as “OQ10,” and the IQ224is displayed as “IQ10.”

The frontend interface113can switch whether to use the OQ201and the IQ202or use the OQ223and the IQ224in accordance with an instruction from the processor111. Similarly, the frontend interface123can switch whether to use the OQ221and the IQ222or use the OQ203and the IQ204in accordance with an instruction from the processor121.

The instruction from the processor is executed, for example, when the processor writes a predetermined value in the PCIe register of the frontend interface by a PCIe write request.

The foregoing switch setting situation is stored in the registers of the frontend interfaces113and123accessible with the PCIe, and thus can be read by the processors111and121.

Each of the frontend interfaces113and123is connected to any of the processors111and121via the NTB. Accordingly, even when the processor111or121stops and the links117and118are linked down or the links127and128are linked down, links between the frontend interfaces113and123and the PCIe switches115and125are not linked down. That is, link states of the frontend interfaces113and123and the PCIe switches115and125are not affected due to stop of the processors111and121.

FIG.3is a diagram illustrating a configuration of an OQ and an IQ according to Example 1.

The OQ201and the IQ202store entries in a total of N elements from number 0 and number N−1. In the OQ201, content of the entries are, for example, host I/O commands received from the host apparatus. In the IQ202, content of the entries are, for example, responses corresponding to the completed host I/O commands or data transfer lists that the processor directs to the frontend interfaces. Each entry also includes identification information of an exchange (that is, an exchange ID) indicating to which exchange of the host I/O the entry is related. An exchange indicates a series of operations related to a read operation or a write operation between a host and a storage.

In the OQ201ofFIG.3, entries are stored in elements from i−1th to i+5th elements, for example. The other entries of the OQ201are empty. OQ_Producer Index (PI)301indicates a location of an element in which the frontend interface subsequently stores an entry. OQ Consumer Index (CI)302indicates a location of an element in which an entry subsequently read by the processor is stored. When OQ_PI301and OQ_CI302indicate the same element, an empty state where an unprocessed entry is not stored in the OQ201is indicated. In the i−1th element, a processed latest entry is stored. The processor can determine whether a process of an entry related to a certain exchange is completed by investigating an exchange ID of the entry.

In the IQ202ofFIG.3, entries are stored in elements from j−1th to i+4th elements, for example. The other entries of the IQ202are empty. IQ_PI311indicates a location of an element in which the processor subsequently stores an entry. IQ_CI312indicates a location of an element in which an entry subsequently read by the frontend interface is stored. When IQ_PI311and IQ_CI312indicate the same element, an empty state where an unprocessed entry is not stored in the IQ202is indicated. In the j−1th element, a processed latest entry is stored. The processor can determine whether a process of an entry related to a certain exchange is completed by investigating an exchange ID of the entry.

FIG.4is a diagram illustrating a data transfer path at a normal time between frontend interfaces and memories in the storage apparatus100according to Example 1.

Here, it is assumed that the frontend interface113and the processor111use the OQ201and the IQ202ofFIG.2and the frontend interface123and the processor121uses the OQ221and the IQ222ofFIG.2.

The frontend interface113accesses the memory112via a data transfer path400passing through the PCIe switch115, the NTB214, the link117, and the processor111. The frontend interface123accesses the memory122via a data transfer path401passing through the PCIe switch125, the NTB234, the link127, and the processor121. Between the memories112and122, the processors111and121transfer data via a data transfer path402passing through the link140.

The frontend interface113sends an interrupt to the processor111via the PCIe switch115, the NTB214, and the link117. The frontend interface123sends an interrupt to the processor121via the PCIe switch125, the NTB234, and the link127.

FIG.5is a diagram illustrating a data transfer path after a switch in an enqueueing destination queue of the frontend interface in the storage apparatus100according to Example 1.

Here, it is assumed that the frontend interface113and the processor121use the OQ223and the IQ224ofFIG.2and the frontend interface123and the processor121use the OQ221and the IQ222ofFIG.2.

The frontend interface113accesses the memory122via a data transfer path500passing through the PCIe switch115, the NTB215, the link128, and the processor121. The frontend interface123accesses the memory122via a data transfer path401passing through the PCIe switch125, the NTB234, the link127, and the processor121. Between the memories112and122, the processors111and121transfer data via the data transfer path402passing through the link140.

The frontend interface113sends an interrupt to the processor121via the PCIe switch115, the NTB215, and the link128. The frontend interface123sends an interrupt to the processor121via the PCIe switch125, the NTB234, and the link127.

FIG.6is a diagram illustrating a data transfer path after a switch in an enqueueing destination queue of the frontend interface in the storage apparatus100according to Example 1.

Here, it is assumed that the frontend interface113and the processor111use the OQ201and the IQ202ofFIG.2and the frontend interface123and the processor111use the OQ203and the IQ204ofFIG.2.

The frontend interface113accesses the memory112via a data transfer path400passing through the PCIe switch115, the NTB214, the link117, and the processor111. The frontend interface123accesses the memory112via a data transfer path600passing through the PCIe switch125, the NTB235, the link118, and the processor111. Between the memories112and122, the processors111and121transfer data via the data transfer path402passing through the link140.

The frontend interface113sends an interrupt to the processor111via the PCIe switch115, the NTB214, and the link117. The frontend interface123sends an interrupt to the processor111via the PCIe switch125, the NTB235, and the link118.

FIG.7is a diagram illustrating a data transfer sequence at a normal time in the storage apparatus100according to Example 1.

Here, it is assumed that, for example, the frontend interface113and the processor111use the OQ201and the IQ202.

First, a host apparatus700sends a host I/O command701to the frontend interface113.

The frontend interface113receiving the host I/O command701enqueues an entry702including the command content in the OQ201(step703).

Subsequently, the frontend interface113sends an interrupt to an address set in the interrupt setting register217to notify the processor111that the entry is enqueued in the OQ (step704). Further, the frontend interface113updates OQ_PI of the OQ201which is in the memory112(step705).

The processor111receiving the interrupt reads an entry in which content of the host I/O command is stored from the OQ201(step706). Further, the processor111updates OQ_CI of the OQ201which is in the frontend interface113(step707).

Subsequently, the processor111enqueues an entry including a data transfer list corresponding to the host I/O command701in the IQ202(step708). Further, the processor111updates IQ_PI of the IQ202which is in the frontend interface113(step709).

The frontend interface113in which IQ_PI is updated reads the entry including the data transfer list from the IQ202(step710).

Subsequently, between the host apparatus700and the memory the frontend interface113executes data transfer in accordance with the data transfer list included in the entry read from the IQ202(step711).

When the data transfer is completed, the frontend interface113updates IQ_CI of the IQ202which is in the memory112(step712).

In this way, the processor111can process the host I/O command701received by the frontend interface113.

FIG.8is a diagram illustrating a data transfer sequence after a switch in an enqueueing destination queue of the frontend interface in the storage apparatus100according to Example 1.

Here, it is assumed that, for example, the frontend interface113and the processor121use OQ223and IQ224.

First, the host apparatus700sends a host I/O command801to the frontend interface113.

The frontend interface113receiving the host I/O command801enqueues an entry802including the command content in the OQ223(step803).

Subsequently, the frontend interface113sends an interrupt to an address set in the interrupt setting register218to notify the processor121that the entry is enqueued in the OQ (step804). Further, the frontend interface113updates OQ_PI of the OQ223which is in the memory112(step805).

The processor121receiving the interrupt reads an entry in which content of the host I/O command is stored from the OQ223(step806). Further, the processor121updates OQ_CI of the OQ223which is in the frontend interface113(step807).

Subsequently, the processor121enqueues an entry including a data transfer list corresponding to the host I/O command801in the IQ224(step808). Further, the processor121updates IQ_PI of the IQ224which is in the frontend interface113(step809).

The frontend interface113in which IQ_PI is updated reads the entry including the data transfer list from the IQ224(step810).

Subsequently, between the host apparatus700and the memory122, the frontend interface113executes data transfer in accordance with the data transfer list included in the entry read from the IQ224(step811).

When the data transfer is completed, the frontend interface113updates IQ_CI of the IQ224which is in the memory122(step812).

In this way, the processor121can process the host I/O command801received by the frontend interface113.

FIG.9is a flowchart illustrating a case where one-side controller restarts in the storage apparatus100according to Example 1.

Here, for example, a processing flow of a case where the processor121of the controller120takes over a process of a host I/O received by the frontend interface113when the controller110restarts will be described.

First, the processor111confirms whether the controller120is restarting (step901).

Subsequently, in step902, the process returns to step901when the controller120is restarting. When the controller120is not restarting, the process proceeds to step903.

Subsequently, the processor111notifies the processor121that a restart process of the controller110is started (step903).

Subsequently, the processor111sends a queue switch command to switch an operation target queue from the OQ201and the IQ202to the OQ223and the IQ224to the frontend interface113(step904).

The frontend interface113receiving the queue switch command enqueues a subsequently received host I/O command in the OQ223(step905). In other words, the frontend interface113switches the OQ of an enqueueing destination from a newly received exchange. It can be determined whether the exchange is newly started by investigating an exchange ID included in the host I/O command.

That is, the frontend interface113enqueues a host I/O command in which the exchange ID is first switched (that is, a first host I/O command to which a new exchange ID is given) and a subsequent host I/O command among the host I/O command received from the host apparatus700after the reception of the queue switch command from the processor111in the OQ223.

Subsequently, the processor111waits until the OQ201and the IQ202are empty (step906). When OQ_PI and OQ_CI of the OQ201are the same, the processor111can determine whether the OQ201is empty. Similarly, when IQ_PI and IQ_CI of the IQ202are the same, the processor111can determine whether the IQ202is empty. When the OQ201and the IQ202are empty, it can be understood that there is no host I/O which is a host I/O received by the frontend interface113and is an uncompleted host I/O to be processed by the processor111in association with the OQ201.

Further, to guarantee that there is no uncompleted host I/O associated with the OQ201, the processor111may investigate an entry corresponding to the host I/O command processed recently with the OQ201and an entry corresponding to a response processed recently with the IQ202to determine whether the exchange IDs included in the entries match each other. When the OQ201and the IQ202are empty and the exchange IDs of the OQ201and the IQ202match each other, it can be understood that there is no host I/O which is a host I/O received by the frontend interface113and is an uncompleted host I/O to be processed by the processor111.

Subsequently, the storage apparatus100blocks the controller110and executes a process necessary for update or the like of the OS. Then, the storage apparatus100continues to operate with only the one-side controller120. Thereafter, the storage apparatus100restarts the controller110. Thus, the processor111stops and restarts (step907).

After the controller110restarts, the processor111confirms a queue switch setting situation of the frontend interface113(step908).

Subsequently, when a host I/O command enqueueing destination is the OQ223in step909, the process proceeds to step910. Otherwise, the process ends.

Subsequently, the processor111notifies the processor121of the restart of the controller110in a state in which the operation target queue of the frontend interface113is switched (step910).

Subsequently, the processor111sends a queue switch command to switch the operation target queues from the OQ223and IQ224to the OQ201and the IQ202to the frontend interface113(step911).

The frontend interface113receiving the queue switch command enqueues the host I/O command received with a subsequently new exchange ID in the OQ201(step912).

Subsequently, the processor111waits until the OQ223and the IQ224are empty (step913). When OQ_PI and OQ_CI of the OQ223are the same, the processor111can determine that the OQ223is empty. Similarly, when IQ_PI and IQ_CI of the IQ224are the same, the processor111can determine that the IQ224is empty. The fact that the OQ223and the IQ224are empty means that there is no uncompleted host I/O which is received by the frontend interface113and is to be processed by the processor121.

Further, to guarantee that there is no uncompleted host I/O associated with the OQ223, the processor111may investigate an entry corresponding to the host I/O command processed recently with the OQ223and an entry corresponding to a response processed recently with the IQ224to determine whether the exchange IDs included in the entries match each other.

Finally, the processor111notifies the processor121of the restart process completion of the controller110(step914).

As described above, the storage apparatus100according to Example 1 restarts the controller after completion of the switch process of the enqueueing destination OQ by the frontend interface and completion of the host I/O during the process before the switch. Thus, even when the one-side controller restarts, the host I/O is not paused.

In the storage apparatus100according to Example 1, the example in which a pair of OQ and IQ are disposed in each frontend interface for each controller has been described. However, in the storage apparatus100, two or more pairs of OQs and IQs may be disposed. For example, when two pairs of OQs and IQs are disposed in each frontend interface for each controller, the two pairs are set as units of switch processes.

For example, it may be assumed that a failure occurs in the processor111, the memory112, or the like of the controller110. Even in this case, an influence of the failure is not sent to the frontend interface113and the controller120by the NTBs211,214, and215. Here, even when a failure occurs in the processor111, the processor111cannot send a queue switch command to the frontend interface113. In this case, with a heartbeat or the like via the link140, the processor121detecting that a failure occurs in the processor111sends an instruction to switch an enqueueing destination of the host I/O command to the side of the normal controller120to the frontend interface113via the link128, the NTB215, and the PCIe switch115. Accordingly, the storage apparatus100according to Example 1 can continue operating. Here, in this case, it is necessary for the host apparatus to resend the uncompleted I/O which is being processed in the controller110in which the failure occurs.

Example 2

Next, a storage apparatus according to Example 2 will be described with reference toFIGS.10and11. A configuration of the storage apparatus according to Example 2 is similar to that of the storage apparatus according to Example 1, as illustrated inFIGS.1to9, except for differences to be described below. Therefore, the explanation is omitted.

The storage apparatus100according to Example 2 controls data transfer between the frontend interface113and the processors111and121by using both a pair of the OQ201and the IQ202and a pair of OQ223and IQ224at a normal time. In this case, any of the processors111and121receives a host I/O command received by the frontend interface113via the OQ201or the OQ223. In response to an instruction from the processor111, it can be switched whether the pair of OQ201and IQ202are used or the pair of OQ223and IQ224are used for data transfer.

Similarly, the storage apparatus100according to Example 2 controls data transfer between the frontend interface123and the processors111and121by using both a pair of the OQ221and the IQ222and a pair of OQ203and IQ204at a normal time. In this case, any of the processors111and121receives a host I/O command received by the frontend interface123via the OQ221or the OQ203. In response to an instruction from the processor121, it can be switched whether the pair of OQ221and IQ222are used or the pair of OQ203and IQ204are used for data transfer.

Such a switch process in the storage apparatus according to Example 2 is referred to as a “switching queue” process.

FIG.10is a diagram illustrating a data transfer path at a normal time between a frontend interface and a memory in the storage apparatus according to Example 2.

Here, it is assumed that the frontend interface113and the processor111use the OQ201and the IQ202ofFIG.2and the frontend interface113and the processor121use the OQ223and the IQ224ofFIG.2.

It is also assumed that the frontend interface123and the processor121use the OQ221and the IQ222ofFIG.2and the frontend interface123and the processor111use the OQ203and the IQ204ofFIG.2.

The frontend interface113accesses the memory112via the data transfer path400passing through the PCIe switch115, the NTB214, the link117, and the processor111. The frontend interface113accesses the memory122via a data transfer path1000passing through the PCIe switch115, the NTB215, the link128, and the processor121.

The frontend interface123accesses the memory122via the data transfer path401passing through the PCIe switch125, the NTB234, the link127, and the processor121. The frontend interface123accesses the memory112via a data transfer path1001passing through the PCIe switch125, the NTB235, the link118, and the processor111.

Between the memories112and122, the processors111and121transfer data via the data transfer path402passing through the link140.

The frontend interface113sends an interrupt to the processor111via the PCIe switch115, the NTB214, and the link117. The frontend interface113sends an interrupt to the processor121via the PCIe switch115, the NTB215, and the link128. The frontend interface123sends an interrupt to the processor111via the PCIe switch125, the NTB235, and the link118. The frontend interface123sends an interrupt to the processor121via the PCIe switch125, the NTB234, and the link127.

FIG.11is a flowchart illustrating a case where one-side controller restarts in the storage apparatus according to Example 2.

Here, for example, a processing flow of a case where the processor121of the controller120takes over a process of a host I/O received by the frontend interface113when the controller110restarts will be described.

First, the processor111confirms whether the controller120is restarting (step1101).

Subsequently, in step1102, the process returns to step1101when the controller120is restarting. When the controller120is not restarting, the process proceeds to step1103.

Subsequently, the processor111notifies the processor121that a restart process of the controller110is started (step1103).

Subsequently, the processor111sends a switching queue command to switch an operation target queue to only the OQ223and the IQ224to the frontend interface113. The processor121sends a switching queue command to switch an operation target queue only to the OQ221and the IQ223to the frontend interface123(step1104).

The frontend interface113receiving the switching queue command enqueues a host I/O command received with a new exchange ID subsequently in only the OQ223. The frontend interface123receiving the switching queue command enqueues a host I/O command received with a new exchange ID subsequently in only the OQ221(step1105).

Subsequently, the processor111waits until the OQ201, the IQ202, the OQ203, and the IQ204are empty (step1106). When OQ_PI and OQ_CI of the OQ201are the same, the processor111can determine that the OQ201is empty. Similarly, when IQ_PI and IQ_CI of the IQ202are the same, the processor111can determine that the IQ202is empty. When the OQ201and the IQ202are empty, it can be understood that there is no host I/O which is a host I/O received by the frontend interface113and is an uncompleted host I/O to be processed by the processor111in association with the OQ201.

Further, to guarantee that there is no uncompleted host I/O associated with the OQ201, the processor111may investigate an entry corresponding to the host I/O command processed recently with the OQ201and an entry corresponding to a response processed recently with the IQ202to determine whether the exchange IDs included in the entries match each other. When the OQ201and the IQ202are empty and the exchange IDs of the OQ201and the IQ202match each other, it can be understood that there is no host I/O which is a host I/O received by the frontend interface113and is an uncompleted host I/O to be processed by the processor111in association with the OQ201and the IQ202.

When OQ_PI and OQ_CI of the OQ203are equal, the processor111can determine that the OQ203is empty. Similarly, when OQ_PI and OQ_CI of the IQ204are equal, the processor111can determine that the IQ204is empty. When the OQ203and the IQ204are empty, it can be understood that there is no host I/O which is a host I/O received by the frontend interface123and is an uncompleted host I/O to be processed by the processor111in association with the OQ203.

Further, to guarantee that there is no uncompleted host I/O associated with the OQ203, the processor111may investigate an entry corresponding to the host I/O command processed recently with the OQ203and an entry corresponding to a response processed recently with the IQ204to determine whether the exchange IDs included in the entries match each other. When the OQ203and the IQ204are empty and the exchange IDs of the OQ203and the IQ204match each other, it can be understood that there is no host I/O which is a host I/O received by the frontend interface123and is an uncompleted host I/O to be processed by the processor111in association with the OQ203and IQ204.

Subsequently, the storage apparatus100according to Example 2 blocks the controller110and executes a process necessary for update or the like of the OS. Then, in the storage apparatus100according to Example 2, only the one-side controller120continues to operate. With the blockage process, the processor111stops. Thereafter, the storage apparatus100restarts the controller110. Thus, the processor111stops and restarts (step1107).

After the controller110restarts, the processor111confirms a switching queue setting situation of the frontend interface113(step1108).

Subsequently, when a host I/O command enqueueing destination is only the OQ223in step1109, the process proceeds to step1110. Otherwise, the process ends.

Subsequently, the processor111notifies the processor121of the restart of the controller110in a state in which the operation target queue of the frontend interface113is switched (step1110).

Subsequently, processor111sends a command to release the switching queue state to the frontend interface113. The processor121sends a command to release the switching queue state to the frontend interface123(step1111).

The frontend interface113receiving the switching queue releasing command enqueues the host I/O command received with a new exchange ID subsequently in the OQ201or the OQ223. The frontend interface123receiving the switching queue releasing command enqueues the host I/O command received with a new exchange ID subsequently in the OQ221or the OQ203(step1112).

Finally, the processor111notifies the processor121of the restart process completion of the controller110(step1113).

As described above, the storage apparatus100according to Example 2 restarts the controller after completion of the switching queue process of the enqueueing destination OQ by the frontend interface and completion of the host I/O during the process before the switch in the enqueueing destination OQ by the switching queue process. Thus, even when the controller restarts, the host I/O is not paused.

In the storage apparatus100according to Example 2, the example in which a pair of OQ and IQ are disposed in each frontend interface for each controller has been described. However, in the storage apparatus100, two or more pairs of OQs and IQs may be disposed. For example, when two pairs of OQs and IQs are disposed in each frontend interface for each controller, the two pairs are set as units of switching queue processes.

Example 3

Next, a storage apparatus according to Example 3 will be described with reference toFIGS.12to15. A configuration of the storage apparatus according to Example 3 is similar to that of the storage apparatus according to Example 1, as illustrated inFIGS.1to9, except for differences to be described below.

FIG.12is a diagram illustrating a configuration of controllers in the storage apparatus according to Example 3.

InFIG.12, to facilitate description, the backend interfaces are omitted as inFIG.2. The storage apparatus according to Example 3 does not have a link (the link140inFIG.2) linking the processors and corresponding to the link140inFIG.2. Instead, the processors111and121execute data transfer between the processors along a path passing through links118and128and the PCIe switches115and125.

FIG.13is a diagram illustrating a data transfer path at a normal time between frontend interfaces and memories in the storage apparatus according to Example 3.

At the normal time, the frontend interface113and the processor111use the OQ201and the IQ202ofFIG.12and the frontend interface113and the processor121use the OQ223and the IQ224ofFIG.12to control data transfer.

The frontend interface113accesses the memory112via a data transfer path1300passing through the PCIe switch115, the NTB214, the link117, and the processor111. The frontend interface123accesses the memory122via a data transfer path1301passing through the PCIe switch125, the NTB234, the link127, and the processor121. Between the memories112and122, the processors111and121transfer data via a data transfer path1302passing through the processor111, the link117, the NTB214, the PCIe switch115, the NTB215, the link128, and the processor121. Between the memories112and122, the processors111and121transfer data via a data transfer path1303passing through the processor121, the link127, the NTB234, the PCIe switch125, the NTB235, the link118, and the processor111. As a use method for two data transfer paths1302and1303between the processors, it is conceivable that one of the data transfer paths is used for user data transfer in which a transfer data amount is large and the other of the data transfer paths is used for control data transfer in which a transfer data amount of the storage apparatus is small. Accordingly, it is possible to expect the effect of shortening a control data transfer time.

The frontend interface113sends an interrupt to the processor111via the PCIe switch115, the NTB214, and the link117. The frontend interface123sends an interrupt to the processor121via the PCIe switch125, the NTB234, and the link127.

FIG.14is a diagram illustrating a data transfer path after a switch in an enqueueing destination queue of the frontend interface in the storage apparatus according to Example 3.

Here, it is assumed that the frontend interface113and the processor121use the OQ223and the IQ224ofFIG.12and the frontend interface123and the processor121use the OQ221and the IQ222ofFIG.12.

The frontend interface113accesses the memory122via a data transfer path1400passing through the PCIe switch115, the NTB215, the link128, and the processor121. The frontend interface123accesses the memory122via a data transfer path1301passing through the PCIe switch125, the NTB234, the link127, and the processor121. Between the memories112and122, the processors111and121transfer data via the data transfer path1303passing through the processor121, the link127, the NTB234, the PCIe switch125, the NTB235, the link118, and the processor111. Between the memories112and122, the processors111and121transfer data via the data transfer path1302(FIG.13) passing through the processor111, the link117, the NTB214, the PCIe switch115, the NTB215, the link128, and the processor121.

The frontend interface113sends an interrupt to the processor121via the PCIe switch115, the NTB215, and the link128. The frontend interface123sends an interrupt to the processor121via the PCIe switch125, the NTB234, and the link127.

FIG.15is a diagram illustrating a data transfer path after a switch in the enqueueing destination queue of the frontend interface in the storage apparatus according to Example 3.

Here, it is assumed that the frontend interface113and the processor111use the OQ201and the IQ202ofFIG.12and the frontend interface123and the processor111use the OQ203and the IQ204ofFIG.12.

The frontend interface113accesses the memory112via the data transfer path1300passing through the PCIe switch115, the NTB214, the link117, and the processor111. The frontend interface123accesses the memory112via a data transfer path1500passing through the PCIe switch125, the NTB235, the link118, and the processor111. Between the memories112and122, the processors111and121transfer data via the data transfer path1302passing through the processor111, the link117, the NTB214, the PCIe switch115, the NTB215, the link128, and the processor121. Between the memories112and122, the processors111and121transfer data via the data transfer path1303(FIG.13) passing through the processor121, the link127, the NTB234, the PCIe switch125, the NTB235, the link118, and the processor111.

The frontend interface113sends an interrupt to the processor111via the PCIe switch115, the NTB214, and the link111. The frontend interface123sends an interrupt to the processor111via the PCIe switch125, the NTB235, and the link118.

As in the storage apparatus100according to Example 1, the storage apparatus100according to Example 3 restarts the controller after completion of the switch process of the enqueueing destination OQ by the frontend interface and completion of the host I/O during the process before the switch. Thus, even when the one-side controller restarts, the host I/O is not paused. In the storage apparatus100according to Example 3, the number of links connected between the controllers is smaller than in the storage apparatus100according to Example 1. Therefore, mounting is easy.

The system according to an embodiment of the present invention may be configured as follows.

(1) A storage apparatus (for example, the storage apparatus100) that processes a request from a host apparatus includes a plurality of storage controllers. A first storage controller (for example, the controller110) in the plurality of storage controllers includes a first frontend interface (for example, the frontend interface113) that controls a protocol of communication with the host apparatus, and a first processor (for example, the that controls the storage apparatus. A processor111) second storage controller (for example, the controller120) in the plurality of storage controllers includes a second processor (for example, the processor121) that controls the storage apparatus. The first storage controller further includes a first address translation unit (for example, the NTB214) that executes translation between a first address used by the first processor and a second address used by the first frontend interface, a second address translation unit (for example, the NTB215) that executes translation between a third address used by the second processor and the second address used by the first frontend interface, a first outbound queue (for example, the OQ201) that controls data transfer from the first frontend interface to the first processor through the first address translation unit, and a first inbound queue (for example, the IQ202) that controls data transfer from the first processor to the first frontend interface through the first address translation unit. The second storage controller further includes a second outbound queue (for example, the OQ223) that controls data transfer from the first frontend interface to the second processor through the second address translation unit, and a second inbound queue (for example, the IQ224) that controls data transfer from the second processor to the first frontend interface through the second address translation unit. The first frontend interface receives a first enqueueing destination switch instruction to designate the second outbound queue as an enqueueing destination of a request from the host apparatus and then switches an enqueueing destination to the second outbound queue from a request which comes from the host apparatus and to which an identifier (for example, an exchange ID) of a series of operations related to a subsequently received new host request is given (for example, steps904and905or steps1104and1105).

Accordingly, even when the processor of one-side controller of dual controllers stops the process, the processor of the other controller can take over an I/O request from the host apparatus and continuously respond to the I/O request.

(2) In the above (1), the first processor sends the first enqueueing destination switch instruction to the first frontend interface before a process is stopped (for example, step904or step1104) and, stops the process after the first outbound queue and the first inbound queue are empty (for example, steps906and907or steps1106and1107).

Accordingly, for example, in order to update an OS, when the processor of one-side controller is planned to stop, the processor of the other controller can take over the I/O request from the host apparatus and continuously respond to the I/O request.

(3) In the above (2), the first processor stops the process when an identifier of a series of operations related to a request which comes from the host apparatus and which is recently processed with the first outbound queue and an identifier of a series of operations related to a host request to a response recently processed with the first inbound queue among responses to requests from the host apparatus indicate the same host request (for example, steps906and907and steps1106and1107).

Accordingly, it is possible to reliably determine that there is no uncompleted host I/O and stop the processor at an appropriate timing.

(4) In the above (1), the first frontend interface individually sets a transfer destination address of a first interrupt related to the first outbound queue and a transfer destination address of a second interrupt related to the second outbound queue, sends the first interrupt to the first processor through the first address translation unit (for example, via the data transfer path400), and sends the second interrupt to the second processor through the second address translation unit (for example, via the data transfer path500).

Accordingly, when the processor of one-side controller stops the process, the processor of the other-side controller can take over the I/O request from the host apparatus.

(5) In the above (1), the first processor is able to read an enqueueing destination setting of the first frontend interface.

Accordingly, it is possible to confirm a current enqueueing destination.

(6) In the above (1), the first processor switches an enqueueing destination to the first outbound queue when the enqueueing destination is the second outbound queue as a result obtained by reading the enqueueing destination setting of a request from the host apparatus from the first frontend interface after restart (for example, Yes in step909to step912or Yes in step1109to step1112).

Accordingly, it is possible to resume the I/O by the restarted processor.

(7) In the above (1), the first storage controller further includes a third address translation unit (for example, the NTB235) that executes translation between the first address used by the first processor and a fourth address used by a second frontend interface, and a fourth address translation unit (for example, the NTB234) that executes translation between the third address used by the second processor and the fourth address used by the second frontend interface. The first and second processors communicate with each other through the third and fourth address translation units (for example, via the data transfer path1303).

Accordingly, it is possible to reduce the number of links connected between the controllers, and thus mounting is easy.

(8) In the above (1), a switch (for example, the switch115) including the first and second address translation units is included. The first and second processors communicate with each other through the switch (for example, via the data transfer path1302).

Accordingly, it is possible to reduce the number of links connected between the controllers, and thus mounting is easy.

(9) In the above (1), the first frontend interface enqueues a request from the host apparatus to the first or second outbound queue before the first enqueueing destination switch instruction is received (for example, sends via the data transfer path400or1000). The first processor sends the first enqueueing destination switch instruction to the first frontend interface before the process is stopped (for example, step1104). When the first enqueueing destination switch instruction is received from the first processor, the first frontend interface enqueues requests in only the second outbound queue from a request from the host apparatus to which an identifier of a series of operations related to a subsequently received new host request is given (for example, step1105). The first processor stops the process after the first outbound queue and the first inbound queue are empty (for example, steps1106and1107).

Accordingly, when one frontend interface enqueues at a normal time in regard to queues related to the plurality of processors, switching queues process with respect to queues related to the other-side processor is processed before one-side processor stops. Therefore, even when the processor of the one-side controller stops the process, the processor of the other-side controller can take over the I/Q request from the host apparatus and continuously responds to the I/O request.

(10) In the above (9), the second storage controller further includes a second frontend interface (for example, the frontend interface123) that controls a protocol of communication with the host apparatus, a third address translation unit (for example, the NTB235) that executes translation between the first address used by the first processor and a fourth address used by the second frontend interface, a fourth address translation unit (for example, the NTB234) that executes translation between the third address used by the second processor and a fourth address used by the second frontend interface, a third outbound queue (for example, the OQ221) that controls data transfer from the second frontend interface to the second processor through the fourth address translation unit, and a third inbound queue (for example, the IQ222) that executes data transfer from the second processor to the second frontend interface through the fourth address translation unit. The first storage controller further includes a fourth outbound queue (for example, the OQ203) that controls data transfer from the second frontend interface to the first processor through the third address translation unit, and a fourth inbound queue (for example, the IQ204) that controls data transfer from the first processor to the second frontend interface through the third address translation unit. The second frontend interface enqueues a request from the host apparatus in the third or fourth outbound queue. The second processor sends a second enqueueing destination switch instruction to designate the third outbound queue as an enqueueing destination of a request from the host apparatus to the second frontend interface before the first processor stops the process (for example, step1104). When the second enqueueing destination switch instruction is received, the second frontend interface enqueues requests in only the third outbound queue from a request from the host apparatus to which an identifier of a series of operations related to a subsequently received new host request is given (for example, step1105). The first processor stops the process after the first outbound queue, the first inbound queue, the fourth outbound queue, and the fourth inbound queue are empty (for example, steps1106and1107).

Accordingly, when one frontend interface enqueues at a normal time in regard to queues related to the plurality of processors, switching queues process with respect to queues related to the other-side processor is processed before one-side processor stops. Therefore, even when the processor of the one-side controller stops the process, the processor of the other-side controller can take over the I/Q request from the host apparatus and continuously respond to the I/O request.

(11) In the above (10), an identifier of a series of operations related to a request which comes from the host apparatus and which is recently processed with the first outbound queue and an identifier of a series of operations related to a host request to a response recently processed with the first inbound queue among responses to requests from the host apparatus indicate the same host request. The first processor stops the process when an identifier of a series of operations related to a request which comes from the host apparatus and which is recently processed with the fourth outbound queue and an identifier of a series of operations related to a host request to a response recently processed with the fourth inbound queue among responses to requests from the host apparatus indicate the same host request (for example, steps1106and1107).

Accordingly, it is possible to reliably determine that there is no uncompleted host I/O and stop the processor at an appropriate timing.

(12) In the above (1), when a failure occurs in the first storage controller, the second processor detecting the failure sends the first enqueueing destination switch instruction to the first frontend interface.

Accordingly, even when a failure occurs in the processor of the one-side controller, the processor of the other-side controller can take over the I/Q request from the host apparatus and continuously respond to the I/O request.

The present invention is not limited to the foregoing examples and includes various modifications. For example, the foregoing examples have been described in detail to further understand the present invention and all the described configurations are not necessarily included. Some of the configurations according to a certain example can be replaced with the configurations according to another example, and the configurations according to another example can be added to the configurations according to a certain example. Other configurations can be added to, deleted from, or replaced with some of the configurations according to each example.

Some or all of the foregoing configurations, functions, processing units, processing mechanism, and the like may be implemented with hardware by designing, for example, integrated circuits. The foregoing configurations, functions, and the like may be implemented with software by causing processors to interpret and execute programs implementing the functions. Information regarding a program, a table, a file, or the like implementing each function can be stored in a storage apparatus such as a nonvolatile semiconductor memory, a hard disk drive, a solid state drive (SSD) or a computer-readable non-transitory data storage medium such as an IC card, an SD card, or a DVD.

Control lines and information lines indicate lines considered to be required for description, and cannot be said to be all control lines and information lines on products. Actually, it may be considered that almost all the configurations are connected to each other.