Storage device and method for managing size of storage device

The invention relates to a storage device in which MR-IOV is applied to the internal network of a storage controller, whereby the size of the storage device can be easily expanded. The storage device is expanded on the basis of a network having processor-connected RPs, FE I/F, BE I/F, and CM I/F that are connected with a switch. In the switch, a plurality of ports other than those connected to the RPs, FE I/F, BE I/F, and CM I/F are connected with a cross-link. Each processor is allowed to control the FE I/F, BE I/F, or CM I/F either via a path that passes through the cross-link or via a path that does not pass through the cross-link within the unit device. When unit devices are connected to expand the size of a storage device, the cross-link is removed first and then the unit devices are connected with a new cross-link (see FIG. 4).

TECHNICAL FIELD

The present invention relates to a storage device and a method for managing the size (scalability) of the storage device.

BACKGROUND ART

Storage devices typically have a storage controller and a nonvolatile storage medium that can be accessed randomly. Such a storage medium is, for example, a disk array with a number of hard disk drives (HDDs) or nonvolatile semiconductor memory drives (SSDs). The storage controller has a front-end interface (hereinafter abbreviated as FE I/F) for connection to a host system, a back-end interface (hereinafter abbreviated as BE I/F) for connection to the disk array, cache memory (hereinafter abbreviated as CM) for temporarily storing data that is read from or written to the disk array by the host system, and its interface (hereinafter, CM I/F). The storage controller also has a processor for controlling data transfer between the host system and CM I/F and between the disk array and CM I/F.

As a communication network standard specification for connecting the processor to the EF I/F, BE I/F, and the like, there is known “PCI Express.” Meanwhile, as an extended standard of the PCI Express, there is also known “Multi-Root I/O Virtualization and Sharing Specification” (hereinafter, “MR-IOV”) that is the standard to enable sharing of an I/O device among a plurality of processors. For example, Patent Literature 1 discloses a technique related to a communication network that uses the MR-IOV. As the MR-IOV is the standard specification, it is considered that components such as switches that comply with the MR-IOV can be commoditized. That is, it is expected that constructing the internal network of a storage controller using the MR-IOV that is the standard specification allow cost reduction of the storage controller.

In the MR-IOV, a communication network includes, for example, a plurality of root complexes (hereinafter, “RCs”), to each of which is connected a processor, a plurality of root ports (hereinafter, “RPs”) provided in the RCs, a plurality of endpoints (hereinafter, “EPs”) that serve as the base points for data input/output, and a plurality of switches for connecting the RPs and EPs. Each EP is configured to be capable of, when accessed from a processor via an RP, providing its function (a data transfer function with which input data is transferred to another device, for example) to the processor (so that the processor can control data transfer on the each EP). With such a configuration, a plurality of processors can share each EP and can independently access each EP via an RP (each processor can independently control data transfer on each EP). Accordingly, the plurality of processors can independently perform data transfer operations without the need of increasing the number of EPs, whereby the performance of the data transfer processing can be improved.

When focus is placed on a single RP in the MR-IOV, a tree-like topology that has the RP, and EPs and switch logically connected to the RP, is referred to as a “virtual hierarchy” (hereinafter, “VH”). In a communication network that complies with the MR-IOV (hereinafter, “MR-IOV network”), VHs exist in the same number as a plurality of RPs that reside in the MR-IOV network. A single VH represents an address space used for data transfer controlled by a processor for each RP. Assume, for example, that there exist the first VH that has an RP1, EP1, and EP2and the second VH that has an RP2, EP1, and EP2in the MR-IOV network. It is also assumed that the RP1is provided in an RC1connected to a processor1, and the RP2is provided in a RC2connected to a processor2. In such a case, the processor1and processor2can independently control data transfer from the EP1to the EP2(or in the reverse direction) via the RP1on the first VH and via the RP2on the second VH, respectively.

CITATION LIST

Patent Literature

SUMMARY OF INVENTION

Technical Problem

As can be understood from the aforementioned example, an EP (the EP1and EP2in the aforementioned example) is, when there exist a plurality of VHs to which the EP belongs (VHs that the EP supports), shared among the plurality of VHs (the first VH and second VH in the aforementioned example). That is, in the MR-IOV, each EP needs to provide data transfer functions for more than one VH that the EP supports. In the PCI Express, such data transfer functions provided by each EP are referred to as PCI functions.

In the MR-IOV network, when the number of processors is increased to enhance the performance, each EP needs to support VHs corresponding to the number of RPs to which the processors are connected. That is, in order to realize a data transfer function for each VH, each EP needs to have PCI functions corresponding to the number of RPs to which the processors are connected. Typically, an EP is implemented with a solid-state integrated circuit, and the upper limit of the number of PCI functions provided in each EP is determined in the design phase of the solid-state integrated circuits. Thus, each EP should be provided in advance with PCI functions in a number corresponding to a case in which the maximum allowable number of processors are connected.

However, when the number of the connected processors is less than the maximum allowable number, some of the PCI functions of the EP will not be used. In order to enhance the performance of the storage device, however, it is desirable that all of the PCI functions of each EP can be used even when a few processors are connected.

The present invention has been made in view of the foregoing circumstances, and provides a technique for efficiently using PCI functions of each EP (Endpoint) in a storage device that uses the MR-IOV.

Solution to Problem

In order to solve the aforementioned problem, the present invention relates to a storage device (or system) with an internal network of a storage controller in which components (RPs, FE I/F, BE I/F, and CM I/F) are connected with a switch. The storage device is expanded on the basis of a network having processor-connected RPs, FE I/F, BE I/F, and CM I/F that are connected with a switch. A plurality of switch ports other than those connected to the RPs, FE I/F, BE I/F, and CM I/F are connected with a cross-link. Each processor is allowed to control the FE I/F, BE I/F, or CM I/F either via a path that passes through the cross-link or via a path that does not pass through the cross-link. In such a case, the connection relationship between the downstream bridges of virtual switches in the switch and each interface device (FE I/F, BE I/F, and CM I/F) will not be changed by the change in the number of unit devices to be connected or by the attachment/detachment of the cross-link.

When networks of unit devices are to be connected in order to expand the size of a storage device, the cross-link is removed first, and then the unit devices are connected with a new cross-link.

Further features of the present invention will become apparent from the following embodiments and accompanying drawings for carrying out the present invention.

Advantageous Effects of Invention

According to the present invention, PCI functions of each EP (Endpoint) can be efficiently used without changing the internal connection configuration (mapping) of the MR-IOV switching device. In addition, the size of the storage device can be easily expanded. As a result of the size expansion of the storage device, processing efficiency of the entire device can be improved, that is, the performance of the storage device can be enhanced.

DESCRIPTION OF EMBODIMENTS

The present invention relates to a storage device and a method for managing the size (scalability) of the storage device. In particular, the invention relates to a technique of expanding or reducing the size of a storage device by changing switch connections within the internal network of a storage controller.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that the following embodiments are only illustrative for carrying out the invention. Thus, it is obvious that various modifications and variations can be made without departing from the scope and the spirit of the invention. In addition, structures that are common throughout the drawings are assigned the same reference numbers.

First, a brief summary of the MR-IOV network will be described (FIGS. 1 to 3) for an understanding of the basic knowledge that forms the basis of this embodiment, and then this embodiment will be described on the basis of such knowledge.

<Basic Summary of the MR-IOV Network>

FIG. 1is a diagram for describing the topology of the MR-IOV network. In the MR-IOV network, a plurality of root ports (3301to3304) is connected to an endpoint3306via a switch3305. In the MR-IOV network, a tree structure that corresponds to the topology of the PCI Express Base Specification, starting at each RP is referred to as a virtual hierarchy (VH). In this specification, a virtual hierarchy that has the RP0(3301) as the root port, for example, is represented by VH(RP0).

The switch3305is an MRA (Multi-Root Aware) switch that complies with the MR-IOV specification. The switch3305includes a plurality of virtual switches (3330to3333) corresponding to the virtual hierarchies that the switch supports. An upstream bridge3341of the virtual switch3330is connected to a switch port3320. Likewise, an upstream bridge3341of the virtual switch3331is connected to a switch port3321; an upstream bridge3341of the virtual switch3332is connected to a switch port3322; and an upstream bridge3341of the virtual switch3333is connected to a switch port3323. One of a pair of downstream bridges (3342) of each virtual switch is connected to a switch port3324.

A link3308is an MR-link shared as a data transfer path among the plurality of virtual hierarchies. On the MR-link, each virtual hierarchy is identified using a VH number. In the VH(RP0), a VH number “VH0” is set in a data packet header, and the data is transferred over the link3308. Likewise, in the VH(RP1), VH(RP2), and VH(RP3), “VH1,” “VH2,” and “VH3” are respectively set in data packet headers, and the data is transferred over the link3308.

The endpoint (EP)3306has a plurality of PCI functions (3310to3313). The endpoint that complies with the MR-IOV should have PCI functions corresponding to the virtual hierarchies that the endpoint supports. That is, the EP3306has a PCI function F0corresponding to the VH(RP0), a PCI function F1corresponding to the VH(RP1), a PCI function F2corresponding to the VH(RP2), and a PCI function F3corresponding to the VH(RP3). The PCI function F0is accessed with the VH number “VH0,” the PCI function F1is accessed with the VH number “VH1,” the PCI function F2is accessed with the VH number “VH2,” and the PCI function F3is accessed with the VH number “VH3.”

An MR-PCIM (Multi-Root PCI Manager)3307is a computer program having functions of setting and managing the MR-IOV network. Functions of the MR-PCIM include setting and management of the number and configuration of the virtual switches in the switch3305and the connection relationship between the switch ports (3320to3324) and bridges (3341and3342) of each virtual switch. It should be noted that the functions of the MR-PCIM can either be executed by a CPU in accordance with a program or be implemented as a module.

The EP in the aforementioned internal network of the storage controller should have four PCI functions corresponding to the maximum number (e.g., four) of processors connected to the internal network of the storage controller. As each of the four PCI functions is basically assigned for each individual VH, it would be necessary, in order to use all of the four PCI functions, to connect four processors to the internal network of the storage controller. However, when only two processors are connected to the internal network of the storage controller, only two of the PCI functions will be used, whereas the other two PCI functions will remain unused. In order to enhance the performance of the storage device, it is desired to efficiently utilize all of the PCI functions of the EP even when a few processors are connected to the internal network of the storage controller.

Meanwhile, in the MR-IOV network, even when only two processors are connected to the internal network of the storage controller, it is possible to configure the switch such that all of the four PCI functions in each EP that are assigned to the four VHs can be used.

In such a case, however, it would be necessary, when a total of four processors are connected later to the internal network of the storage controller by adding two more processors, to reconfigure the switch so that each processor can use its associated PCI function in the EP. Further, in the configuration ofFIG. 1, it would also be necessary for the MR-PCIM, in order to change the correspondence relationship between each RP and PCI function accessed by the RP, to change the mapping between the switch ports and downstream bridges of each virtual switch. Such change of the mapping will require very complex operations. Thus, eliminating the need for such operations allows a very user-friendly MR-IOV network to be realized.

The present invention relates to a storage device in which the MR-IOV is applied to the internal network of a storage controller, wherein PCI functions, which are provided by an endpoint when the internal network of the storage controller has the maximum configuration, can be used even when the internal network has the minimum configuration (when a few processors are connected thereto), and that allows the size of the internal network of the storage controller to be easily expanded.

Next, VH numbers will be described.FIG. 2is a diagram for describing the assignment (example) of VH numbers. InFIG. 2, three RPs (3400to3402) are connected to a switch3403. The RP3400is connected to an upstream bridge3441of a virtual switch3430via a port3420. The RP3401is connected to an upstream bridge3441of a virtual switch3431via a port3421. The RP3402is connected to an upstream bridge3441of a virtual switch3432via a port3422. One of a pair of downstream bridges3442of the virtual switch3430is connected to a port3423, and the other downstream bridge3442is connected to a port3424. One of a pair of downstream bridges3442of the virtual switch3431is connected to the port3423, and the other downstream bridge3442is connected to the port3424. Both of downstream bridges3442of the virtual switch3432are connected to a port3425.

In the MR-IOV specification, a VH number is defined as information to identify each VH. However, as the VH number is assigned on each link, there may be cases in which the VH number assigned to the same VH differ on different links (that is, the same VH numbers do not necessarily indicate the same VH). For example, in the VH(RP0) inFIG. 2, a VH number “VH0” is used on a link3404, but a VH number “VH1” is used on a link3405. Meanwhile, in the VH(RP1) inFIG. 2, a VH number “VH1” is used on a link3404, but a VH number “VH0” is used on a link3405. As an alternative method of using VH numbers, for example, in the VH(RP2), two VH numbers “VH0” and “VH1” are used on a link3406.

It should be noted that in the MR-IOV specification, a “global key” is defined as information to uniquely identify the VH across the entire MR-IOV network. The global key is set for each of the MRA switch, MR-IOV-compliant endpoint, and the like.

Next, a cross-link will be described.FIG. 3is a diagram for describing a cross-link. Two RPs (3500and3501) are connected to switches3502and3503, respectively. The RP3500is connected to an upstream bridge of a virtual switch3530via a port3520of the switch3502. In the switch3502, one of a pair of downstream bridges of the virtual switch3530is connected to a port3522, and the other downstream bridge is connected to a port3521. An upstream bridge of a virtual switch3531is connected to the port3521, and one of a pair of downstream bridges of the virtual switch3531is connected to the port3522. The RP3501is connected to an upstream bridge of a virtual switch3533via a port3523of the switch3503. In the switch3503, one of a pair of downstream bridges of the virtual switch3533is connected to a port3525, and the other downstream bridge is connected to a port3524. An upstream bridge of a virtual switch3532is connected to the port3524, and one of a pair of downstream bridges of the switch3532is connected to the port3525. The port3521of the switch3502and the port3524of the switch3503are connected with a link3542. As a result of the aforementioned setting, the RP3500(RP0) and RP3501(RP1) can access an end-point EP0via a link3540and an EP1via a link3541.

The link3542is a cross-link having different upstream and downstream directions depending on the virtual hierarchy. In the switch3502or3503, a port that is closer to the RP is an upstream port, and a port that is closer to the EP is a downstream port. Thus, in the VH(RP0) on the link3542, the ports3521and3524are a downstream port and upstream port, respectively. In such a case, a VH number “VH0” is used to transfer data. Meanwhile, in the VH(RP1) on the link3542, the ports3524and3521are a downstream port and upstream port, respectively. In such a case, a VH number “VH1” is used to transfer data. As described above, in the MR-IOV network, a cross-link can be used in which a plurality of VH numbers are assigned to the single link, and upstream and downstream directions differ depending on the VH.

<Configuration and Operation of the Storage Device of the First Embodiment>

FIG. 4is a diagram showing the schematic configuration of a storage device10in accordance with the first embodiment of the present invention. The storage device10has, for example, a controller20and a DA (Disk Array)30. The DA30has a group of storage devices such as a plurality of HDDs (Hard Disk Drives) or SSDs (Solid-State Drives). An external host system is connected to the controller20is via a channel C00, and an SVP40, which is an administrative terminal, is also connected to the controller20.

The controller20has, for example, two processors205(CPU0and CPU1), two pieces of memory208(MEM0and MEM1), two RCs (Root Complexes)206(RC0and RC1), three EPs (Endpoints) (EP0, EP1, and EP2), cache memory CM203, and a switch100. The three EPs correspond to a front-end interface (FE I/F)200, cache memory interface (CM I/F)202, and back-end interface (BE I/F)201. In the internal network of the controller20, two root ports (RP0and RP1)207, switch100, and three EPs are mutually connected. It should be noted that the number of the components that constitute the controller20is not limited to that exemplarily shown inFIG. 4.

The processor205performs various processing by executing various computer programs stored in the memory208(MEM0for CPU0and MEM1for CPU1). For example, the processor205controls data transfer between the host system (not shown) and CM203and between the CM203and DA30. The memory208has stored therein various programs executed by the processor205, various table information referred to by the processor205, and the like.

The administrator can set or change the values of the table information. For example, the administrator can perform operations such as the maintenance or management of the storage device10which include setting of the table information and the like by operating the SVP40. The administrator, when setting the table information by operating the SVP40, enters information to be set as the table information (setting information) into the SVP40. The SVP40, upon receipt of the entry, sends the entered setting information to the processor205via a management network (not shown) in the controller20. The processor205, upon receipt of the setting information, sets or changes the target table information based on the received setting information. The SVP40has an input device such as a keyboard for the administrator to perform the management operation of the storage device, and a display device such as a display.

The RC0_206is connected to the CPU0, and has a single RP (Root Port)207. The RP207is a root port that complies with the PCI Express specification. Meanwhile, the RC1is connected to the CPU1, and has an RP1.

The RP0_207is connected to the EP0(FE I/F200), EP1(BE I/F201), and EP2(CM I/F202) via the switch100. Likewise, the RP1is connected to the EP0(FE I/F200), EP1(BE I/F201), and EP2(CM I/F202) via the switch100.

The switch100is a switch (an MRA (Multi-Root Aware) switch) that complies with the MR-IOV (Multi-Root I/O Virtualization and Sharing) specification. In addition, each EP (the FE I/F200, CM I/F202, and BE I/F201) is a device (an MRA device) that complies with the MR-IOV specification.

The switch100has, for example, ports (P00to P06) for connection to other components and a management port (not shown) for connection to the management network in the controller20. The SVP40, for example, is connected to the management port of the switch100.

The RP0and RP1are connected to the ports P00and P03of the switch100, respectively. The port P04of the switch100is connected to the EP0via a link300. The port P05of the switch100is connected to the EP2via a link301. The port P06of the switch100is connected to the EP1via a link302. The ports P01and P02of the switch100are connected with a link303. The links300to303are MR-links that are shared as data transfer paths among a plurality of virtual hierarchies.

The switch100includes a plurality of virtual switches (VS0to VS3). Each virtual switch is a switch that physically exists, but its switching configuration is virtual. An upstream bridge101of the virtual switch VS0is connected to the port P00of the switch100. Likewise, an upstream bridge101of the VS1is connected to the port P03; an upstream bridge101of the VS2is connected to the port P01; and an upstream bridge101of the VS3is connected to the port P02. Three of downstream bridges102of each virtual switch are connected to the ports P04to P06, respectively, of the switch100. One of the downstream bridges102of the VS0is connected to the port P01. One of the downstream bridges102of the VS1is connected to the port P02. The P01is a downstream port when seen from the VS0and is an upstream port when seen from the VS2. The P02is a downstream port when seen from the VS1and is an upstream port when seen from the VS3. Thus, the link303that connects the P01and P02is a cross-link.

The FE I/F200corresponding to the EP0is connected to a host system via a channel C00. The host system is, for example, a computer that issues I/O requests (write requests or read requests) to the storage device10. The FE I/F200mutually converts the data transfer protocol used on the channel C00and the data transfer protocol used in the internal network of the controller20.

The EP0has PCI functions204(FF0, FF1, FF2, and FF3) that are accessible from the RP0or RP1. Such PCI functions are hardware modules to implement the function (a data transfer function) of the EP having the PCI functions. The phrase “to access a PCI function” refers to an event in which the PCI function204is accessed so that data transfer is controlled with the use of a function realized by the PCI function204. Although the “RP” is described as “accessing the PCI function” in this embodiment, what actually accesses the PCI function204(controls data transfer) is the processor205connected to the RC206with the RP207, via the RP207.

The BE I/F201corresponding to the EP1is connected to the DA30via a channel D00. The BE I/F201mutually converts the data transfer protocol used on the channel D00and the data transfer protocol used in the internal network of the controller20. The EP1has PCI functions204(FB0, FB1, FB2, and FB3) that are accessible from the RP0or RP1.

The CM203is connected to the CM I/F202corresponding to the EP2. In the CM203, data received from the EP0, EP1, or the like is temporarily stored. In addition, control information or the like that is referred to within the controller20is also stored in the CM203. The EP2has PCI functions204(FC0, FC1, FC2, and FC3) that are accessible from the RP0or RP1.

Each EP is connected to the SVP40via the management network in the controller20. The SVP40can transmit setting information entered by the administrator to each EP.

It should be noted that the FE I/F200and BE I/F201can be implemented as a single EP. In that case, the single EP implements both the functions of the FE I/F200and BE I/F201.

The MR-PCIM (Multi-Root PCI Manager) is a computer program having functions of setting and managing the internal network of the controller20(e.g., network components such as the switch100and EPs). In this embodiment and the following embodiments, the MR-PCIM is stored in the memory208provided in the controller20, and is executed by the processor205connected to the memory208. Using the MR-PCIM, the administrator can set or manage the mapping between the RP207and PCI functions204of each EP (i.e., correspondence relationship between the RP207and PCI functions204that are accessible from the RP207(or the EP with the PCI functions204)). For example, the administrator can set the mapping by entering setting information into the MR-PCIM operating on the processor205in the controller20via the SVP40so that the MR-PCIM, upon receipt of the entry, sends the setting information to each EP or the switch100. It should be noted that the MR-PCIM can be incorporated in the SVP40. If the MR-PCIM is incorporated in the SVP40, the MR-PCIM operating on the SVP40can set the mapping by receiving entry from the administrator and sending setting information directly from the SVP40to each EP or the switch100.

FIG. 5is a diagram showing only the components related to the virtual hierarchy VH(RP0) having the RP0as the root port, in the storage device10inFIG. 4. Described below is a path in which the processor205(CPU0) accesses the PCI function in each endpoint (EP0to EP2) from the RP0. The RP0is connected to the upstream bridge101of the virtual switch VS0via the port P00of the switch100. Three of the downstream bridges102of the VS0are connected to the ports P04to P06, respectively, of the switch100. The ports P04to P06are connected to the EP0to the EP2via the links300to302. In the VH(RP0), when an EP is accessed only via the VS0, data is transferred using the VH number “VH0” on the links300to302(400inFIG. 5).

One of the downstream bridges102of the VS0is connected to the port P01of the switch100. In the VH(RP0), the port P01functions as a downstream port and the port P02functions as an upstream port, and they are connected with the link303. In the VH(RP0), data is transferred using the VH number “VH0” on the link303(402inFIG. 5).

The upstream bridge101of the virtual switch VS3is connected to the port P02, and data is transferred using the VH number “VH0” on the link303(403inFIG. 5). The three downstream bridges102of the VS3are connected to the ports P04to P06, respectively, of the switch100. The ports P04to P06are connected to the EP0to the EP2via the links300to302. In the VH(RP0), when an EP is accessed via the VS0, cross-link303, and VS3, data is transferred using the VH number “VH2” on the links300to302(401inFIG. 5).

As described above, the RP0accesses the PCI functions204(FF0, FB0, and FC0) in the EP0to the EP2using the VH number “VH0” on the links300to302. In addition, the RP0accesses the PCI functions204(FF2, FB2, and FC2) in the EP0to the EP2using the VH number “VH2” on the links300to302.

FIG. 6is a diagram showing only the components related to the virtual hierarchy VH(RP1) having the RP1as the root port, in the storage device10inFIG. 4. Described below is a path in which the processor205(CPU1) accesses the PCI function in each endpoint (EP0to EP2). The RP1is connected to the upstream bridge101of the virtual switch VS1via the port P03of the switch100. Three of the downstream bridges102of the VS1are connected to the ports P04to P06, respectively, of the switch100. The ports P04to P06are connected to the EP0to the EP2via the links300to302. In the VH(RP1), when an EP is accessed only via the VS1, data is transferred using the VH number “VHF” on the links300to302(500inFIG. 6).

One of the downstream bridges102of the VS1is connected to the port P02of the switch100. In the VH(RP1), the port P02functions as a downstream port and the port P01functions as an upstream port, and they are connected with the link303. In the VH(RP1), data is transferred using the VH number “VH1” on the link303(502inFIG. 6).

The upstream bridge101of the virtual switch VS2is connected to the port P01, and data is transferred using the VH number “VH1” on the link303(503inFIG. 6). The three downstream bridges102of the VS2are connected to the ports P04to P06, respectively, of the switch100. The ports P04to P06are connected to the EP0to the EP2via the links300to302. In the VH(RP1), when an EP is accessed via the VS1, cross-link303, and VS2, data is transferred using the VH number “VH3” on the links300to302(501inFIG. 6).

As described above, the RP1accesses the PCI functions204(FF1, FB1, and FC1) in the EP0to the EP2using the VH number “VH1” on the links300to302. In addition, the RP1accesses the PCI functions204(FF3, FB3, and FC3) in the EP0to the EP2using the VH number “VH3” on the links300to302.

As described above, in this embodiment, the switch100has a configuration in which the ports P01and P02of the plurality of ports (P00to P03) are connected with a cross-link. Each VS of the switch100has a single upstream bridge and a plurality of downstream bridges. The internal connection of each downstream bridge and each of the plurality of downstream ports (P04to P06) to be linked to each EP is fixed. Accordingly, when the cross-link is connected, the resident RPs (RP0and RP1) can use all of the PCI functions in each EP. Meanwhile, even when the cross-link is removed and the RP (CPU) is connected to each port (P00to P03), it is also possible for the RP to use all of the PCI functions. In such a case, there is no need to change the mapping between each VS and the downstream port to be linked to each EP. That is, RPs can be easily added or removed only by attaching or detaching a cross-link without changing the internal configuration of the switch100, whereby the PCI functions in each EP can be efficiently used (there will be no unusable PCI functions).

<Size Expansion of the Storage Device>

FIGS. 7 to 10are diagrams for describing a connection configuration when the size of the storage device is expanded by connecting two storage devices10in accordance with this embodiment. AlthoughFIG. 7has the same configuration asFIG. 4,FIG. 7only shows the principal parts of the storage device10for the sake of simplicity of the description. In this embodiment, the size of the storage device is expanded by adding to the storage device10another storage device10with the same configuration, as an expansion unit. The storage device10is configured as a stand-alone storage device.

In the storage device10, the processor CPU0accesses the PCI functions FF0, FB0, and FC0in the EP0to the EP2via the RP0, using the VH0on the links300to302. The processor CPU0also accesses the PCI functions FF2, FB2, and FC2in the EP0to the EP2via the RP0, using the VH2on the links300to302. Meanwhile, the processor CPU1accesses the PCI functions FF1, FB1, and FC1in the EP0to the EP2via the RP1, using the VH1on the links300to302. The processor CPU1also accesses the PCI functions FF3, FB3, and FC3in the EP0to the EP2via the RP1, using the VH3on the links300to302.

FIG. 8is a diagram showing an exemplary configuration of a storage device11that is connected to the storage device10. Although the storage device11has the same configuration as the storage device10inFIG. 4,FIG. 8shows only the principal parts of the storage device11for the sake of simplicity of the description.

The storage device11has an RP506and RP507(RP2and RP3) to which two processors (CPU2and CPU3) are connected, respectively, three EPs (Endpoints) (EP3, EP4, and EP5), and a switch500. The three EPs correspond to a front-end interface (FE I/F), cache memory interface (CM I/F), and back-end interface (BE I/F).

The switch500is an MRA switch that complies with the MR-IOV specification. In addition, each EP is a device (an MRA device) that complies with the MR-IOV specification.

The switch500has ports (P10to P16) for connection to other components and a management port (not shown) for connection to the management network in the controller20. The RP2and RP3are connected to the ports P10and P13, respectively, of the switch500. The port P14of the switch500is connected to the EP3via a link501. The port P15of the switch500is connected to the EP5via a link502. The port P16of the switch500is connected to the EP4via a link503. The ports P11and P12of the switch500are not connected. The links501to503are MR-links.

The switch500includes a plurality of virtual switches (VS4to VS7). An upstream bridge101of the virtual switch VS4is connected to the port P10. Likewise, an upstream bridge101of the VS5is connected to the port P13; an upstream bridge101of the VS6is connected to the port P11; and an upstream bridge101of the VS7is connected to the port P12. Three of downstream bridges102of each virtual switch are connected to the ports P14to P16, respectively. One of the downstream bridges102of the VS4is connected to the port P11. One of the downstream bridges102of the VS5is connected to the port P12.

The EP3to the EP5have PCI functions505(the EP3has FF4, FF5, FF6, and FF7; the EP4has FB4, FB5, FB6, and FB7; and the EP5has FC4, FC5, FC6, and FC7) that are accessible from the RP2or RP3when the P11and P12are connected with a cross-link.

In the storage device11when the ports P11and P12are connected with a cross-link as with the storage device10, the processor CPU2accesses the PCI functions FF4, FC4, and FB4in the EP3to the EP5via the RP2, using the VH0on the links501to503. The processor CPU2also accesses the PCI functions FF6, FC6, FB6in the EP3to the EP5via the RP2, using the VH2on the links501to503. Meanwhile, the processor CPU3accesses the PCI functions FF5, FC5, and FB5in the EP3to the EP5via the RP3, using the VH1on the links501to503. The processor CPU3also accesses the PCI functions FF7, FC7, and FB7in the EP3to the EP5via the RP3, using the VH3on the links501to503. On the cross-link that connects the ports P11and P12, data is transferred using the VH0for the VH(PR2) and using the VH1for the VH(RP3).

Before the storage device11is connected to the storage device10, the MR-IOV network is already initialized and set. However, the storage device11is configured such that it will not operate as a stand-alone device. For example, the processor CPU2and processor CPU3can be prevented from accessing the switch500by initially setting the ports P10and P13, to which the RP2and RP3are connected, respectively, to function as downstream ports.

FIG. 9is a diagram showing a configuration with a total of two storage devices which is obtained by adding the storage device11to the storage device10. First, the cross-link303between the ports P01and P02of the storage device10that has been operating before the storage device11is added is removed. Next, the port P01of the storage device10and the port P12of the storage device11are connected with a cross-link304. Further, the port P02of the storage device10and the port P11of the storage device11are connected with a cross-link305. Accordingly, a total of two storage devices can be mutually connected. It should be noted that the connection conditions are described below with reference toFIG. 13.

FIG. 10is a diagram showing only the components related to the virtual hierarchy VH(RP0) having the RP0as the root port, in the (expanded) storage device inFIG. 9. Described below is a path in which the processor205(CPU0) accesses the PCI function in each endpoint (EP0to EP2) from the RP0. The RP0is connected to the upstream bridge101of the virtual switch VS0via the port P00of the switch100. Three of the downstream bridges102of the VS0are connected to the ports P04to P06, respectively, of the switch100. The ports P04to P06are connected to the EP0to the EP2via the links300to302. In the VH(RP0), when an EP is accessed only via the VS0, data is transferred using the VH number “VH0” on the links300to302(700inFIG. 10). Such operation is the same as that when the storage device10operates alone.

One of the downstream bridges102of the VS0is connected to the port P01of the switch100. In the VH(RP0), the port P01of the switch100functions as a downstream port, and the port P12of the switch500functions as an upstream port, and they are connected with the link304. In the VH(RP0), data is transferred using the VH0on the link304.

In the storage device11, the upstream bridge101of the virtual switch VS7of the switch500is connected to the port P12, and data is transferred using the VH0on the link304. In addition, the three downstream bridges102of the VS7are connected to the ports P14to P16, respectively, of the switch500. The ports P14to P16are connected to the EP3to the EP5via the links501to503. In the VH(RP0), when an EP is accessed via the VS0, cross-link304, and VS7, data is transferred using the VH2on the links501to503(701inFIG. 10).

As described above, the RP0accesses the PCI functions204(FF0, FC0, and FB0) in the EP0to the EP2using the VH0on the links300to302. Further, the RP0accesses the PCI functions505(FF6, FC6, and FB6) in the EP3to the EP5using the VH0on the cross-link304and using the VH2on the links501to503.

The other RPs can access each EP in a similar way. For example, the RP1accesses the PCI functions204(FF1, FC1, and FB1) in the EP0to the EP2using the VH1on the links300to302. Further, the RP1accesses the PCI functions505(FF7, FC7, and FB7) in the EP3to the EP5using the VH1on the cross-link305and using the VH3on the links501to503.

The RP2accesses the PCI functions204(FF2, FC2, and FB2) in the EP0to the EP2using the VH0on the cross-link305and using the VH2on the links300to302. Further, the RP2accesses the PCI functions505(FF4, FC4, and FB4) in the EP3to the EP5using the VH0on the links501to503.

The RP3accesses the PCI functions204(FF3, FC3, and FB3) in the EP0to the EP2using the VH1on the cross-link304and using the VH3on the links300to302. Further, the RP3accesses the PCI functions505(FF5, FC5, and FB5) in the EP3to the EP5using the VH1on the links501to503.

The storage device10, when operating alone before the storage device11is added thereto, was configured such that each processor can use two PCI functions in each EP. The storage device shown inFIG. 9that has been expanded by connecting the storage device10and storage device11is also configured such that each processor can use two PCI functions in each EP. Thus, according to this embodiment, the size of the storage device can be expanded with the number of PCI functions in each EP that can be used by each processor kept constant.

As described above, when the storage device10and storage device11are connected with the two cross-links304and305, it becomes possible to construct the internal network of the storage device in which all of the RPs in the storage device can access any EP in the storage device.

According to this embodiment, processors in the storage controller can use asymmetrical (an equal number of) PCI functions regardless of from which root port each endpoint is seen. In other words, when the numbers of RPs and EPs are increased in the same proportion with the expansion of the storage device, all of the processors can equally use the PCI functions in each interface of the storage device. What is more, the number of the PCI functions that can be used by each processor can be kept constant relative to the size of the storage device (the number of the processors), and I/O processing associated with each PCI function can equally be performed regardless of the size of the storage device. As a result of the size expansion of the storage device, processing efficiency of the entire device can be improved, that is, the performance of the storage device can be enhanced.

Further, as there is no need to change the mapping between the switch ports and virtual switch bridges in connecting an additional storage device, the processing of expanding the size of the storage device can be simple.

<Configuration of the Cross-Link>

FIG. 11is a diagram showing an exemplary configuration of a cross-link in the storage device. A storage device80has a similar configuration to the storage device10inFIG. 4. Ports P01and P02of the switch800are arranged adjacent to each other. Further, a single plug801is connected to the ports P01and P02. The plug801includes a wire for cross-link (303) connecting the ports P01and P02. In this manner, arranging the switch ports adjacent to each other and cross-link connecting them with a single plug (a connector-type plug) allows the internal space of the storage device to be reduced and the cabling operation to be simplified.

FIG. 12shows another exemplary configuration of a cross-link between storage devices. A storage device81has a similar configuration to the storage device11inFIG. 8. Ports P11and P12of a switch802are arranged adjacent to each other. The port P01of the switch800and the port P12of the switch802are connected with a cross-link304. Likewise, the port P02of the switch800and the port P11of the switch802are connected with a cross-link305. The cross-links304and305are packaged in a single cable. In this manner, arranging the switch ports adjacent to each other and cross-link connecting them with a single cable900allows the internal space of the storage device to be reduced and the cabling operation to be simplified.

<Arrangement of the MR-PCIM and Fail-Over Processing>

FIG. 13is a diagram illustrating the arrangement of the MR-PCIM in accordance with this embodiment. An MRA switch typically includes a plurality of virtual switches (VSs). Some of the VSs are authorized VSs. An authorized VS refers to a VS having functions of setting and managing the MRA switch when the VS is accessed from the MR-PCIM. Further, one of the plurality of authorized VSs is a management VS. The MR-PCIM operates on a processor connected to an RP that is associated with the virtual hierarchy to which the management VS belongs. That is, the MR-PCIM sets and manages the MRA switch by accessing one of the plurality of authorized VSs that is specified as a management VS. It should be noted that the setting of the management VS can be changed by accessing any of the authorized VSs from the MR-PCIM.

The MRA device has a base function (BF) that is a PCI function for setting and managing the MR-IOV capability. The BF belongs to the virtual hierarchy in which each device is accessed with the VH0. The MR-PCIM, which operates on a processor connected to an RP associated with the virtual hierarchy in which the MRA device is accessed with the VH0, sets and manages the device by accessing the BF.

In this embodiment, an MR-PCIM is arranged in each storage device that is an expansion unit. Management of EPs in each storage device is executed by the MR-PCIM in each storage device, and management of mutual connection between the storage devices (e.g., the MRA switch and links) is executed by an MR-PCIM (1000) in the storage device10. The MR-PCIM (1000) and MR-PCIM (1001) can communicate with each other via a shared memory area provided in the cache memory of the storage device or via the management network.

When the storage device10is operating alone before the storage device11is added thereto (before the size expansion), the processor CPU0executes the MR-PCIM (1000). In the switch100, the VS0and VS3are set as the authorized VSs and the VS0is set as the management VS. The MR-PCIM (1000) sets and manages the switch100by accessing the VS0. In addition, for the VS0, VS Suppress Reset Propagation is set active so that the switch100, the EP0to the EP2, and the like will be prevented from, even when the MR-PCIM (1000) is reset for rebooting purposes, being reset concurrently.

The MR-PCIM (1000) initializes and manages the MR-IOV network using resources in the MR-IOV network that are assigned to the VH0. For example, as the processor CPU0in the storage device10can access the BF in each EP using the VH0on the links300to302, the MR-PCIM (1000) can set and manage each EP in the storage device10.

The processor CPU2of the storage device11executes the MR-PCIM (1001). In the switch500, the VS4and VS7are set as the authorized VSs, and the VS7is set as the management VS. The MR-PCIM (1000) in the storage device10sets and manages the switch500by accessing the VS7via the link304. In addition, for the VS4, VS Suppress Reset Propagation is set active so that the switch500and the EP3to the EP5will be prevented from, even when the MR-PCIM (1001) is reset for rebooting purposes, being reset concurrently.

In the storage device11, the processor CPU2can access the BF in each EP using the VH0on the links501to503. Thus, the MR-PCIM (1001) can manage the setting of each EP in the storage device11.

The MR-PCIM (1001), upon occurrence of an error in the MR-PCIM (1000), takes over the management of the MR-IOV network (e.g., the MRA switch and links). Hereinafter, a fail-over operation in which the MR-PCIM (1000) is switched over to the MR-PCIM (1001) when a need arises to reset and reboot the MR-PCIM (1000) upon occurrence of a software error will be described.

The MR-PCIM (1001), upon detection of an error in the MR-PCIM (1000), first accesses the VS4that is the authorized VS of the switch500, and changes the management VS of the switch500from the VS7to the VS4. Next, the MR-PCIM (1001) accesses the VS3that is the authorized VS of the switch100via the link305, and changes the management VS of the switch100from the VS0to the VS3. Thereafter, the MR-PCIM (1001) instructs the processor (CPU0) to reset and reboot the MR-PCIM (1000). After the reboot, the MR-PCIM (1000) sets and manages the EP0to the EP2in the storage device10. The MR-PCIM (1001) sets and manages the EP3to the EP5in the storage device11, switch100, switch500, links, and the like. That is, the functions of the MR-PCIM (1000) and MR-PCIM (1001) are switched after the execution of the MR-PCIM fail-over.

The MR-PCIM initializes the MR-IOV network using resources assigned to the VH0. In this embodiment, two storage devices are connected using two links. The two links are assigned VH0one by one in opposite directions. That is, the MR-PCIM (1000) that operates on the processor CPU0can use the VH0on the link304, and the MR-PCIM (1001) that operates on the processor CPU2can use the VH0on the link305. As described above, as the VH0can be used in both directions using the two links, either the MR-PCIM in the storage device10or in the storage device11can execute initialization and management of the MR-IOV network (e.g., the MRA switch and links) using the VH0.

The aforementioned description is concluded as follows. Each of the two MR-PCIMs1000and1001has its own defined function. Examples of tasks of the MR-PCIM include setting and management of the switch and setting and management of the EPs. In this embodiment, each of the storage devices10and11has an MR-PCIM, and such an MR-PCIM manages the EPs included in each storage device. That is, the MR-PCIM1000manages the EP0to the EP2and the MR-PCIM1001manages the EP3to the EP5. Meanwhile, the switches100and500are managed by the MR-PCIM1000. When an error occurs in the MR-PCIM1000under such circumstances, the function of the MR-PCIM1000that manages the switches100and500is failed over to the MR-PCIM1001that is operating normally. It should be noted that in this embodiment, the 0th VH (VH0) is very important as the network is initialized using the VH0. VHs that follow the 0th VH are assigned only after the VH0has operated satisfactorily. In addition, the VH0is assigned to both the links304and305as described above. In this manner, when the storage device is configured such that the VH0can be used in the two directions, the network can be managed regardless of which storage device has an MR-PCIM.

<Processing of Adding another Storage Device>

FIG. 14is a flow chart for describing the details of the processing of adding another storage device. Procedures for constructing the storage device inFIG. 9by connecting the storage device10and storage device11will be described. It should be noted that the connection state of each port of the storage device when another storage device is added thereto is displayed as needed on the screen of the display device of the SVP40so that the administrator can check the connection state on the display.

First, it is assumed that the storage device10operates alone. The processor CPU0of the storage device10, upon receipt of an instruction to add another storage device, hot-removes the FF2, FC2, and FB2in the EP0to the EP2. In addition, the processor CPU1, upon receipt of the instruction to add another storage device, also hot-removes the FF3, FC3, and FB3in the EP0to the EP2(S1101). At this time, as each processor in the storage device10can use the PCI functions F0(FF0, FC0, and FB0) and PCI functions F1(FF1, FC1, and FB1) in the EP0to the EP2, it can continue the data processing as the storage device.

Next, an operation of removing the cross-link303is carried out. The administrator of the storage device10, in accordance with an instruction displayed on the SVP screen, disconnects the ports01and P02(S1102). At this time, information that can identify switch ports connected to the target link to be removed, e.g., positional information on the ports is displayed on the SVP screen.

Next, the MR-PCIM1000(seeFIG. 13) updates the global key of each of the VS2, VS3, PCI functions F2(FF2, FC2, and FB2), and PCI functions F3(FF3, FC3, and FB3) (S1103). The global key is the information defined to uniquely identify the virtual hierarchy in the MR-IOV specification. As RPs connected to the upstream ports of the VS2and VS3are changed with the change of the cross-link connection along with the adding processing, the virtual hierarchy to which each of the VS2, VS3, PCI functions F2(FF2, FC2, and FB2), and PCI functions F3(FF3, FC3, and FB3) belongs is also changed. As a global key is assigned for each virtual hierarchy, it is necessary to, when the virtual hierarchy to which each VS belongs is changed with the change of the cross-link connection, update the global key in the VS. That is, the global key is updated in order to maintain the integrity after the change of the cross-link connection.

Next, the administrator, in accordance with the screen display (not shown) of the SVP40, connects the ports P01and P12and the ports P02and P11(S1104). On the SVP screen, information that can identify switch ports connected to the target link to be attached, e.g., positional information the ports is displayed as in step S1102.

Thereafter, the MR-PCIM1000recognizes the switch500, EP3, EP4, and EP5that are added to the MR-IOV network (S1105).

Next, the processor CPU0hot-adds the FF6, FB6, and FC6in the EP3to the EP5of the storage device11. In addition, the processor CPU1hot-adds the FF7, FB7, and FC7in the EP3to the EP5of the storage device11(S1106).

Then, the MR-PCIM1001sets the port P10of the switch500to be communicable. For example, it changes the port P10, which has been set as a downstream port, to an upstream port (S1107). As a result, the processor CPU2hot-adds the PCI functions F2(FF2, FC2, and FB2) in the EP0to the EP2and the PCI functions F4(FF4, FC4, and FB4) in the EP3to the EP5(S1108).

Next, the MR-PCIM1001sets the port P13of the switch500to be communicable. For example, it changes the port P13, which has been set as a downstream port, to an upstream port (S1109). As a result, the processor CPU3hot-adds the PCI functions F3(FF3, FC3, and FB3) in the EP0to the EP2and the PCI functions F5(FF5, FC5, and FB5) in the EP3to the EP5(S1110).

Through the aforementioned procedures for adding another storage device, it becomes possible for all of the processors (CPU0to CPU3) in the whole storage device with the expanded size to access each EP (EP0to EP5) in the storage device. In this embodiment, during the adding processing, PCI functions in a number corresponding to the number of the added RPs becomes unusable as a result of the hot-remove processing. However, as the other usable PCI functions remain in each EP, the operation of the storage device can be continued without difficulty.

It should be noted that the aforementioned adding processing can also be applied to cases in which a generalized configuration of a storage device (FIGS. 26 and 27) or a redundant network configuration (FIG. 33) is used.

<Processing of Removing Storage Device>

FIG. 15is a flow chart for describing the processing procedures for removing (detaching) a storage device from two storage devices that are connected together. Specifically, procedures for removing the storage device11from the connected storage device10and storage device11shown inFIG. 9will be described. It should be noted that the connection state of each port of the storage device in removal of the storage device is displayed as needed on the screen of the display device of the SVP40as in the adding processing so that the administrator can check the connection state on the display.

First, the processor CPU0hot-removes the PCI functions F6(FF6, FC6, and FB6) in the EP3to the EP5of the storage device11. The processor CPU1also hot-removes the PCI functions F7(FF7, FC7, and FB7) in the EP3to the EP5of the storage device11. Further, the processor CPU2hot-removes the PCI functions F2(FF2, FC2, and FB2) in the EP0to the EP2of the storage device10, and the PCI functions F4(FF4, FC4, and FB4) in the EP3to the EP5of the storage device11. In addition, the processor CPU3hot-removes the PCI functions F3(FF3, FC3, and FB3) in the EP0to the EP2of the storage device10, and the PCI functions F5(FF5, FC5, and FB5) in the EP3to the EP5of the storage device11(S1201).

Thereafter, the administrator, in accordance with the SVP screen display, removes the cross-link connections between the port P01of the switch100and the port P12of the switch500and between the port P02of the switch100and the port P11of the switch500(S1202). At this time, information that can identify switch ports connected to the target link to be removed, e.g., positional information on the ports is displayed on the SVP screen.

Next, the MR-PCIM1000updates the global key of each of the VS2, VS3, and PCI functions F2(FF2, FC2, and FB2) and PCI functions F3(FF3, FC3, and FB3) in the EP0to the EP2(S1203).

Then, the administrator connects the ports P01and P02of the switch100with a cross-link in accordance with the SVP screen display (S1204). At this time, information that can identify switch ports connected to the target link to be attached, e.g., positional information on the ports is displayed on the SVP screen.

Thereafter, the processor CPU0detects the cross-link connection and hot-adds the PCI functions F2(FF2, FC2, and FB2) in the EP0to the EP2. Likewise, the processor CPU1detects the cross-link connection and hot-adds the PCI functions F3(FF3, FC3, and FB3) in the EP0to the EP2(S1205).

Through the aforementioned procedures of the removal processing, it becomes possible for the processors (CPU0and CPU1) in the storage device10, after the removal processing, to access the EP0to the EP2in the same manner as when the storage device10operates alone. In this embodiment, during the removal processing, PCI functions in a number corresponding to the number of the removed RPs becomes unusable as a result of the hot-remove processing. However, as the other usable PCI functions remain in each EP, the operation of the storage device can be continued without difficulty.

It should be noted that the aforementioned removal processing can also be applied to cases in which a generalized configuration of a storage device (FIGS. 26 and 27) or a redundant network configuration (FIG. 33) is used.

<Internal Configuration of the Front-End Interface>

FIG. 16is a diagram showing the internal configuration of the front-end interface (FE I/F)200. The FE I/F200has a PCI Express (PCIe) I/F unit3100corresponding to the EP (EP0) that complies with the MR-IOV specification, an embedded processor3103, and memory3104accessed by the processor3103. Further, the FE I/F200includes a protocol IC3102that controls data transfer over the channel C00, and an internal switch3101.

The PCIe I/F unit3100has, in addition to the PCI functions (FF0to FF3) for controlling data transfer between the EPs in the storage controller, a PCI function3105for setting and managing the FE I/F200. The PCI function3105can be accessed from both the processor (e.g., CPU0) on the MR-IOV network side and the embedded processor3103in the FE I/F200. The processor (e.g., CPU0) in the storage controller accesses the PCI function3105using the VH0on the link300, and sets and manages sections other than the PCIe I/F unit of the FE I/F200.

When the FE I/F200controls a management program (e.g., MR-PCIM) by communicating with the processor in the storage controller, the embedded processor3103communicates with the processor (e.g., CPU0) in the storage controller via the PCI function3105. The internal switch3101switches a data transfer path of data transmitted/received by the protocol IC3102, specifically, data transferred between the protocol IC3102and the PCI functions (FF0to FF3) for transferring data in the PCIe I/F unit3100. The change of the data transfer path with the internal switch3101is controlled by the embedded processor3103based on a management table stored in the memory3104.

<Management Table for Switching Data Transfer Path>

FIG. 17shows a management table3200for controlling the switching of a data transfer path with the internal switch3101. This management table3200is stored in the memory3104, and is referred to by the embedded processor3103. Accessing the management table3200from the processor (e.g., CPU0) in the internal network of the storage controller or from the administrative terminal SVP40allows the setting of the table to be changed.

The management table3200has stored therein information3201on the data write target or data read target included in a packet header transmitted/received over the channel C00, and setting of the PCI function (3202to3204) in the PCIe I/F unit3100that is the transmission source/destination of the data. As the information3201, a logical unit number is stored, for example. The PCI function setting3202is the setting when the storage device operates normally. The PCI functions FF0to FF3(herein, only the PCI functions of the FE I/F are shown) are assigned to the logical unit numbers0to3, respectively. The PCI function setting3203is the setting when only the two PCI functions FF2and FF3are used. The PCI function setting3204is the setting when only the two PCI functions FF0and FF1are used. Which of the PCI function settings3202to3204is to be used is determined by the embedded processor3103based on an instruction from the processor (e.g., CPU0) in the storage controller or from the SVP40.

In the storage device of the present invention, part of the PCI functions is, during the processing of adding or removing a storage device, temporarily hot-removed and thus becomes unusable in that period. In such a case, if the PCI functions FF2and FF3are hot-removed, for example, information on the change of the data transfer path with the internal switch3101is changed from3202to3204of the management table. As a result, data transmitted/received by the protocol IC3102can be processed by the PCI functions FF0and FF1, and thus, the data processing of the storage device can be continued even during the adding processing. It should be noted that each of the other BE I/F201and CM I/F202also has a function capable of switching the PCI functions for handling the data processing according to circumstances.

(2) Second Embodiment

A storage device in accordance with the second embodiment of the present invention will be described with reference toFIGS. 18 to 22. The second embodiment relates to scalable storage device to which a maximum of two additional storage devices can be connected, and each storage device that is an expansion unit has a single RP.
<Configuration of each Storage Device>
FIG. 18is a diagram showing the schematic configuration of a single storage device in accordance with the second embodiment. InFIG. 18, a storage device1320has a single RP, single switch1300, and single EP1301. The EP1301has one or more of the functions of the FE I/F, CM I/F, and BE I/F. AsFIG. 18shows only a single EP, PCI functions provided by the EP1301are represented as F0, F1, and F2.

P01and P02of the switch1300are connected with a cross-link3701. The RP0accesses a PCI function1302in the EP1301via a VS0and P03, using the VH0(on a link that connects the P03and EP1301). In addition, the RP0accesses a PCI function1303in the EP1301via the VS0, P02, cross-link3701, P01, VS1, and P03, using the VH1on the link that connects the P03and EP1301. In such a case, the VH1is used on the cross-link3701. Further, the RP0accesses a PCI function1304in the EP1301via the VS0, P01, cross-link3701, P02, VS2, and P03, using the VH2on the link that connects the P03and EP1301. In such a case, the VH0is used on the cross-link3701. The RP0accesses the EP1301through the three kinds of access paths.

<Adding Another Storage Device: Configuration with Two Storage Devices>

FIG. 19is a diagram showing a configuration with a total of two storage devices which is obtained by adding a storage device1321to the storage device1320. The storage device1321only differs from the storage device1320in that ports P11and P12are not connected with a cross-link.

In order to add the storage device1321, the cross-link3701between the P01and P02of the storage device1320(FIG. 18) is removed first. Next, the P01of the storage device1320and the P12of the storage device1321are connected with a cross-link1311. Further, the P02of the storage device1320and the P11of the storage device1321are connected with a cross-link1310. Accordingly, a total of two storage devices can be mutually connected.

FIG. 20is a diagram showing only the components related to the virtual hierarchy (VH(RP0)) having the RP0as the root port, in the expanded storage device inFIG. 19. Described below is a path in which the RP0accesses the PCI function in each end-point (1301and1306). The RP0is connected to an upstream bridge of the virtual switch VS0via a port P00of the switch1300. One of downstream bridges of the VS0is connected to the port P03of the switch1300. The port P03is connected to the EP1301. In the VH(RP0), when the EP1301is to be accessed only via the VS0, the PCI function1302in the EP1301is accessed using the VH0on the link that connects the P03and EP1301. This is the same as when the storage device1320operates alone.

One of the downstream bridges of the VS0is connected to the port P01of the switch1300. In the VH(RP0), the port P01of the switch1300functions as a downstream port and the port P12of the switch1305functions as an upstream port, and they are connected with the link1311. In the VH(RP0), data is transferred using the VH0on the link1311.

In the storage device1321, an upstream bridge of a virtual switch VS5of the switch1305is connected to the P12so that data can be transferred using the VH0on the link1311. In addition, a single downstream bridge of the VS5is connected to a P13of the switch1305. The port P13is connected to the EP1306. In the VH(RP0), when the EP1306is to be accessed via the VS0, cross-link1311, and VS5, a PCI function1309in the EP1306is accessed using the VH2on the link that connects the P13and EP1306.

One of the downstream bridges of the VS0is connected to the port P02of the switch1300. In the VH(RP0), the P02of the switch1300functions as a downstream port and the P11of the switch1305functions as an upstream port, and they are connected with the link1310. In the VH(RP0), data is transferred using the VH1on the link1310.

In the storage device1321, an upstream bridge of a virtual switch VS4of the switch1305is connected to the P11, and data is transferred using the VH1on the link1310. In addition, a single downstream bridge of the VS4is connected to the P13of the switch1305. In the VH(RP0), when the EP1036is to be accessed via the VS0, cross-link1310, and VS4, a PCI function1308in the EP1306is accessed using the VH1on the link that connects the P13and EP1306.

Accordingly, the RP0accesses the PCI function1302in the EP1301and the PCI functions1308and1309in the EP1306. Likewise, the RP1can, in the VH(RP1), access the PCI functions1303and1304in the EP1301and a PCI function1307in the EP1306.

<Adding Another Storage Device: Configuration with Three Storage Devices>

FIG. 21is a diagram showing a configuration with a total of three storage devices which is obtained by adding a storage device1322to the storage device inFIG. 19.

First, inFIG. 19, the cross-link1311that connects the P01of the storage device1320and the P12of the storage device1321is removed. Next, the port P01of the storage device1320and a P22of the storage device1322are connected with a cross-link1505. Further, the P12of the storage device1321and a P21of the storage device1322are connected with a cross-link1506. Accordingly, a total of three storage devices can be mutually connected.

FIG. 22is a diagram showing only the components related to the virtual hierarchy VH(RP0) having the RP0as the root port, in the storage device inFIG. 21. Described below is a path in which the RP0accesses the PCI function in each endpoint (EPs1301,1306, and1501). The path in which the RP0accesses the PCI function1302in the EP1301and the path in which the RP0accesses the PCI function1308in the EP1306are the same as those inFIG. 20. The RP0accesses a PCI function1504in the EP1501instead of accessing the PCI function1309in the EP1306.

One of the downstream bridges of the VS0in the switch1300is connected to the P01of the switch1300. In the VH(RP0), the P01of the switch1300functions as a downstream port, and the P22of a switch1500functions as an upstream port, and they are connected with the link1505. In the VH(RP0), data is transferred using the VH0on the link1505.

In the storage device1322, an upstream bridge of a virtual switch VS8in the switch1500is connected to the P22so that data can be transferred using the VH0on the link1505. In addition, a single downstream bridge of the VS8is connected to a P23of the switch1500. The port P23is connected to the EP1501. In the VH(RP0), when the EP1501is to be accessed via the VS0, cross-link1505, and VS8, the PCI function1504in the EP1501is accessed using the VH2on the link that connects the P23and EP1501.

Accordingly, the RP0accesses the PCI function F0_1302in the EP1301, the PCI function F4_1308in the EP1306, and the PCI function F2_1504in the EP1501. Likewise, the RP1, in the VH(RP1), accesses the PCI function F2_1304in the EP1301, the PCI function F3_1307in the EP1306, and the PCI function F1_1503in the EP1501. The RP2can, in the VH(RP2), access the PCI function F1_1303in the EP1301, the PCI function F5_1309in the EP1306, and the PCI function F0_1502in the EP1501.

Hereinafter, a storage device in accordance with the third embodiment will be described with reference toFIG. 23.FIG. 23is a diagram showing a configuration with a total of two storage devices which is obtained by mutually connecting the storage device10and a storage device12with different configurations.

In comparison with the storage device10, the storage device12includes a less number of connectable EPs. Although the storage device12has an EP3corresponding to the FE I/F and an EP5corresponding to the CM I/F, it does not have an EP corresponding to the BE I/F, and such BE I/F cannot be added later. Therefore, the number of the downstream ports of a switch500and the number of the downstream bridges of each virtual switch (VS4to VS7) are less than those of the switch100in the storage device10by one. The other configurations of the storage device12are the same as those of the aforementioned storage devices10and11.

As in the first embodiment, in this embodiment, the two storage devices are connected by connecting the port P01of the storage device10and a port P12of the storage device12with a cross-link1700and connecting the port P02of the storage device10and a port P11of the storage device12with a cross-link1701, whereby the size of the storage device can be expanded. After the size expansion, each processor in the storage device can access each EP.

As described above, it is obvious that the configuration of the storage device to be added need not have the totally same configuration as the storage device as a connection target, and thus, even a storage device with a different configuration can be added.

A storage device in accordance with the fourth embodiment will be described with reference toFIG. 24.FIG. 24is a diagram showing a configuration with a total of two storage devices which is obtained by mutually connecting the storage device10and a storage device13with different configurations.

In comparison with the storage device10, the storage device13includes a less number of connected EPs. Although the storage device13has an EP3corresponding to the FE I/F and an EP4corresponding to the BE I/F, it does not have an EP corresponding to the CM I/F. However, unlike with the storage device12, the storage device13is configured such that the CM I/F can be added later. Therefore, the number of the downstream ports of a switch500and the number of the downstream bridges of each virtual switch (VS4to VS7) are the same as those of the switch100in the storage device10. The configuration of the storage device13only differs from those of the aforementioned storage devices10and11in that the CM I/F is not connected.

As in the first embodiment, in this embodiment, the two storage devices are connected by connecting the port P01of the storage device10and a port P12of the storage device13with a cross-link1800and connecting the port P02of the storage device10and a port P11of the storage device13with a cross-link1801, whereby the size of the storage device can be expanded. After the size expansion, each processor in the storage device can access each EP.

A storage device in accordance with the fifth embodiment will be described with reference toFIG. 25.FIG. 25is a diagram showing a configuration with a total of two storage devices which is obtained by mutually connecting the storage device10and a storage device14with different configurations. In comparison with the storage device10, the storage device14includes a less number of connected RPs. That is, although the storage device14has a single RP506(RP2), it does not have an RP corresponding to the RP3. However, the storage device14is configured such that the RP3can be added later to a port P13. Therefore, the configuration of the storage device14only differs from those of the storage devices10and11in that the RP3is not connected.

As in the first embodiment, in this embodiment, the two storage devices are connected by connecting the port P01of the storage device10and a port P12of the storage device14with a cross-link1900and connecting the port P02of the storage device10and a port P11of the storage device14with a cross-link1901, whereby the size of the storage device can be expanded. After the size expansion, each processor in the storage device can access each EP. However, when such two storage devices are connected, one of the PCI functions in each EP cannot be used until the RP3is added to the storage device14.

A storage device in accordance with the sixth embodiment will be described with reference toFIGS. 26 and 27. The sixth embodiment relates to a generalized configuration of a storage device that is a connection unit.

FIG. 26is a diagram for describing the configuration of a switch used in the internal network of a storage controller that is necessary to connect a maximum of N storage devices.FIG. 27is a table that collectively indicates the number of switch components.

It is assumed that a storage device that is an expansion unit includes Nrp root ports (Nrp is an integer not less than 1) and Nep endpoints (Nep is an integer not less than 1). As each EP is accessed from N×Nrp RPs included in the N storage devices (N is an integer not less than 2) connected together, each EP has PCI functions corresponding to N×Nrp VHs.

In a switch2000of each storage device that is an expansion unit, a total of Nrp root ports are connected to RP-connection switch ports (2001inFIG. 26) in each storage device that is an expansion unit. In that case, the number of RP-connection ports that the switch2000should have is Nrp (2101inFIG. 27).

In addition, in the switch2000of each storage device that is an expansion unit, a total of Nep EPs are connected to EP-connection switch ports (2003inFIG. 26). In that case, the number of EP-connection ports that the switch2000should have is Nep (2102inFIG. 27).

In addition, it is assumed that the number of cross-link ports (2002inFIG. 26) that are necessary when a maximum of N storage devices are connected is Ncr (Ncr is an even integer not less than 2). Ncr is given by the following formula (2103inFIG. 27).
Ncr=(N−1)×Nrp(Formula 1)

It should be noted that as a storage device that is an expansion unit can operate alone, the number of cross-link-connection ports Ncr is set to an even number. That is, when Npr is an even number, N storage devices (N is two or more) can be mutually connected. Meanwhile, when Npr is an odd number, N should also be set to an odd number.

The switch2000has virtual switches (2010inFIG. 26) whose upstream bridges are connected to the RP-connection switch ports, and virtual switches (2011inFIG. 26) whose upstream bridges are connected to the cross-link-connection switch ports. The number of the virtual switches for connection to the RP is Nrp (2104inFIG. 27) which is equal to the number of RP-connection switch ports. The number of virtual switches for connection to the cross-link is Ncr (2105inFIG. 27) which is equal to the number of the cross-link-connection switch ports.

The number of the downstream bridges (2020inFIG. 26) of each virtual switch for connection to the RP is given by the sum of Nep and Ncr/2 (2106inFIG. 27).

The number of the downstream bridges (2021inFIG. 26) of each virtual switch for connection to the cross-link is Nep (2107inFIG. 27).

With the switch configuration ofFIGS. 26 and 27according to this embodiment, it is possible to mutually connect a maximum of N storage devices whose quantity (Nrp and Nep) of the switch components can be set freely.

It should be noted that such a generalized expression can also be applied to the configuration with a redundant network (seeFIG. 33) described below.

A storage device in accordance with the seventh embodiment will be described with reference toFIGS. 28 to 32. This embodiment relates to a connection configuration of a scalable storage device to which a maximum of three additional storage devices can be connected by adding a storage device one by one.

<Configuration of Each Storage Device>

FIG. 28shows the configuration of a single storage device. A storage device2200that is an expansion unit includes two root ports as with the storage device inFIG. 4. The storage device2200has six ports (2201to2206) for connecting three additional storage devices. Such add-on ports of the storage device are divided into two types depending on the way in which the VH0is assigned. Specifically, an add-on port to which the VH0is assigned in a downstream direction is referred to as an A-type port, whereas an add-on port to which the VH0is assigned in an upstream direction is referred to as a B-type port. The ports2201,2203, and2205are A-type ports and the ports2202,2204, and2206are B-type ports. An each pair of the A-type port and B-type port is connected with a cross-link.

<Expanded Storage Device: Configuration with Two Storage Devices>

FIG. 29is a diagram showing a configuration with a total of two storage devices which is obtained by adding a storage device2300to the storage device2200.

First, the cross-link of the ports2201and2202of the storage device2200are removed. Next, the port2201(A-type) of the storage device2200and a port2302(B-type) of the storage device2300are connected with a cross-link2303. Further, the port2202(B-type) of the storage device2200and a port2301(A-type) of the storage device2300are connected with a cross-link2304. The other ports (2303to2306) of the storage device2300are cross-link connected within the storage device2300. Accordingly, a total of two storage devices can be mutually connected.

<Expanded Storage Device: Configuration with Three Storage Devices>

FIG. 30is a diagram showing a configuration with a total of three storage devices which is obtained by adding a storage device2400to the storage device inFIG. 29.

First, the cross-link of the ports2203and2204of the storage device2200is removed. Next, the port2203(A-type) of the storage device2200and a port2402(B-type) of the storage device2400are connected with a cross-link2407. Further, the port2204(B-type) of the storage device2200and a port2401(A-type) of the storage device2400are connected with a cross-link2408.

In addition, the cross-link of the ports2303and2304of the storage device2300is removed. Next, the port2303(A-type) of the storage device2300and a port2404(B-type) of the storage device2400are connected with a cross-link2409. Further, the port2304(B-type) of the storage device2300and a port2403(A-type) of the storage device2400are connected with a cross-link2410. The other ports (2405and2406) of the storage device2400are cross-link connected within the storage device2400. Accordingly, a total of three storage devices can be mutually connected.

<Expanded Storage Device: Configuration with Four Storage Devices>

FIG. 31is a diagram showing a configuration with a total of four storage devices which is obtained by adding a storage device2500to the storage device inFIG. 30.

First, the cross-link of the ports2205and2206of the storage device2200is removed. Next, the port2205(A-type) of the storage device2200and a port2502(B-type) of the storage device2500are connected with a cross-link2507. Further, the port2206(B-type) of the storage device2200and a port2501(A-type) of the storage device2500are connected with a cross-link2508.

In addition, the cross-link of the ports2305and2306of the storage device2300is removed. Next, the port2305(A-type) of the storage device2300and a port2504(B-type) of the storage device2500are connected with a cross-link2510. Further, the port2306(B-type) of the storage device2300and a port2503(A-type) of the storage device2500are connected with a cross-link2509.

Further, the cross-link of the ports2405and2406of the storage device2400is removed. Next, the port2405(A-type) of the storage device2400and a port2506(B-type) of the storage device2500are connected with a cross-link2512. Furthermore, the port2406(B-type) of the storage device2400and a port2505(A-type) of the storage device2500are connected with a cross-link2511. Accordingly, a total of four storage devices can be mutually connected.

<Expanded Storage Device: Another Configuration with Three Storage Devices>

FIG. 32shows another connection configuration of storage devices in which a total of three storage devices are mutually connected. As withFIG. 30,FIG. 32is a diagram showing a configuration with a total of three storage devices which is obtained by connecting the storage device2400to the storage device inFIG. 31.

First, inFIG. 29, the cross-link2303that connects the port2201of the storage device2200and the port2302of the storage device2300is removed. Next, the port2201(A-type) of the storage device2200and the port2402(B-type) of the storage device2400are connected with a cross-link2601. Further, the port2302(B-type) of the storage device2300and the port2401(A-type) of the storage device2400are connected with a cross-link2602. Accordingly, a total of three storage devices can be mutually connected.

As described above, according to this embodiment, the size of a storage device can be expanded by connecting thereto a maximum of three additional storage devices. It should be noted that the maximum number of the connectable storage devices can be arbitrarily set by changing the number of the switch internal components in accordance with Embodiment 6.

A storage device in accordance with the eighth embodiment will be described with reference toFIGS. 33 to 37.

<Configuration of a Storage Device that is an Expansion Unit>

FIG. 33is a diagram showing a storage device2700that is an expansion unit in accordance with the eighth embodiment. The storage device2700is characterized in that its storage controller has redundant components. The storage device2700has a storage controller2701and disk array2705, and an administrative terminal SVP2706for an administrator is connected thereto.

The storage controller2701has a processor (CPU0)2710; processor (CPU1)2720; memory2712and2722; and RCs2711and2721. The RC2711has two RPs (RP0and RP1), and the RC2721has two RPs (RP2and RP3). Further, the storage controller2701has switches2750and2770, various interface devices2713to2715, and various interface devices2723to2725. In this embodiment, a plurality of RPs is provided in each RC.

The switches2750and2770are MRA switches. The switch2750has RP-connection ports2751and2752, EP-connection ports2753to2758, and cross-link-connection ports2759and2760. Likewise, the switch2770has RP-connection ports2771and2772, EP-connection ports2773to2778, and cross-link-connection ports2779and2780. In this embodiment, the RP1of the RC0_2711is connected to the RP-connection port2772of the switch2770, and the RP2of the RC1_2721is connected to the RP-connection port2752of the switch2750, so that a cross-coupled configuration between the networks is provided. That is, the RP0and RP1are connected to different networks, and the RP2and RP3are also connected to different networks.

In this embodiment, as with the storage device2200in accordance with the seventh embodiment, an add-on port to which the VH0is assigned in a downstream direction is referred to as an A-type port, whereas an add-on port to which the VH0is assigned in an upstream direction is referred to as a B-type port. The ports2759and2779are A-type ports and the ports2760and2780are B-type ports. An each pair of the A-type port and B-type port is connected with a cross-link2761or2781.

The FE I/F2713has two EPs (EP01and EP02), the CM I/F2714has two EPs (EP0S and EP06), and the BE I/F2715has two EPs (EP03and EP04). In addition, the FE I/F2723has two EPs (EP11and EP12), the CM I/F2724has two EPs (EP15and EP16), and the BE I/F2725has two EPs (EP13and EP14).

The FE I/F2713is connected to a host system (not shown) via a channel2702. The FE I/F2723is also connected to a host system (not shown) via a channel2704. The BE I/Fs2715and2725are connected to the disk array2705via a channel2703. The CM I/F2714is connected to CM2716. The CM I/F2724is connected to CM2726.

The internal network of the storage controller2701includes two MR-IOV networks. One of the networks is a network that mutually connects the RP0, RP2, switch2750, EP01, EP03, EP0S, EP11, EP13, and EP15. The other network is a network that mutually connects the RP1, RP3, switch2770, EP02, EP04, EP06, EP12, EP14, and EP16. As the two MR-IOV networks are independent networks, they are managed by different MR-PCIMs. That is, the former network is managed by an MR-PCIM executed by the processor2710, and the latter network is managed by an MR-PCIM executed by the processor2720.

FIG. 34is a diagram showing the internal configuration of the switch2750. The switch2750differs from the switch100inFIG. 4in the number of connectable EPs. The switch2750has virtual switches2800and2803whose upstream bridges are connected to the RP-connection ports2751and2752, respectively, and virtual switches2801and2802whose upstream bridges are connected to the cross-link-connection ports2759and2760, respectively. Each of the virtual switches2801and2802has the same number of downstream bridges (6) as the number of EPs connected thereto. Each of the virtual switches2800and2803has downstream bridges (6+1) for connection to the EPs and to the cross-link-connection port. The switch2770has a similar configuration to the switch2750.

<Configuration of the Expanded Storage Device: Configuration with Two Storage Devices>

FIG. 35is a diagram showing a configuration with a total of two storage devices which is obtained by adding to the storage device2700shown inFIG. 33another storage device2900with the same configuration.

The storage device2900has ports2959,2960,2979, and2980for connection to the storage device2700. When the storage device2900operates alone, the ports2959and2960are cross-link connected and also the ports2979and2980are cross-link connected. The ports2959and2979of the storage device2900are A-type ports, and the ports2960and2980thereof are B-type ports.

First, the cross-link2761between the ports2759and2760of the storage device2700and the cross-link2781between the ports2779and2780thereof are removed. Next, the port2759(A-type) of the storage device2700and the port2960(B-type) of the storage device2900are connected with a cross-link2902. Further, the port2760(B-type) of the storage device2700and the port2959(A-type) of the storage device2900are connected with a cross-link2901. Next, the port2779(A-type) of the storage device2700and the port2980(B-type) of the storage device2900are connected with a cross-link2903. Further, the port2780(B-type) of the storage device2700and the port2979(A-type) of the storage device2900are connected with a cross-link2904. Accordingly, a total of two storage devices can be mutually connected.

FIG. 36is a diagram showing another connection configuration of the storage device2700and storage device2900. The port2759(A-type) of the storage device2700and the port2980(B-type) of the storage device2900are connected with a cross-link2905. Further, the port2760(B-type) of the storage device2700and the port2979(A-type) of the storage device2900are connected with a cross-link2906. Next, the port2779(A-type) of the storage device2700and the port2960(B-type) of the storage device2900are connected with a cross-link2907. Further, the port2780(B-type) of the storage device2700and the port2959(A-type) of the storage device2900are connected with a cross-link2908. With the aforementioned connection configuration, a total of two storage devices can be mutually connected.

<Configuration of the Expanded Storage Device: Configuration with Three Storage Devices>

FIG. 37is a diagram showing a configuration with a total of three storage devices which is obtained by further adding a storage device3000to the storage device inFIG. 35.

The storage device3000has ports3059,3060,3079, and3080for connection to the storage device2700or storage device2900. When the storage device3000operates alone, the ports3059and3060are cross-link connected and also the ports3079and3080are cross-link connected. The ports3059and3079of the storage device3000are A-type ports, and the ports3060and3080thereof are B-type ports.

First, inFIG. 35, the cross-link2901that connects the port2760of the storage device2700and the port2959of the storage device2900and the cross-link2904that connects the port2780of the storage device2700and the port2979of the storage device2900are removed. Next, the port2959(A-type) of the storage device2900and the port3060(B-type) of the storage device3000are connected with a cross-link3002. Further, the port2760(B-type) of the storage device2700and the port3059(A-type) of the storage device3000are connected with a cross-link3001. In addition, the port2979(A-type) of the storage device2900and the port3080(B-type) of the storage device3000are connected with a cross-link3003. Further, the port2780(B-type) of the storage device2700and the port3079(A-type) of the storage device3000are connected with a cross-link3004. Accordingly, a total of three storage devices can be mutually connected.

As described above, according to this embodiment, the size of a storage device can be expanded by connecting thereto a maximum of two additional storage devices each with a redundant internal configuration. It should be noted that the maximum number of the connectable storage devices can be arbitrarily set by changing the number of the switch internal components in accordance with Embodiment 6.

A storage device in accordance with the ninth embodiment will be described with reference toFIGS. 38 to 40.

<Configuration of the Expanded Storage Device: Configuration with Two Storage Devices>

FIG. 38is a diagram showing a configuration with a total of two storage devices which is obtained by adding the storage device2900to the storage device2700as inFIG. 35. InFIG. 38, FE interface devices are shown representatively among the FE, CM, BE interfaces, and the other CM and BE interface devices are omitted. However, such omitted interface devices have a similar configuration to the FE interface device.

An FE (Front-End) interface device2713has an EP01(3824) that is accessible from an RP0, RP2, RP4, and RP6, and an EP02(3825) that is accessible from an RP1, RP3, RP5, and RP7. The EP01has PCI functions FF0to FF3. The EP02has PCI functions FF4to FF7. An internal switch3826, with a protocol IC (not shown), switches a data transfer path of data transmitted/received over a channel2702, between the protocol IC and PCI functions (FF0to FF7).

The FE interface device3813has an EP31(3827) that is accessible from the RP0, RP2, RP4, and RP6, and an EP32(3828) that is accessible from the RP1, RP3, RP5, and RP7. The EP31has PCI functions FF0to FF3. The EP32has PCI functions FF4to FF7. An internal switch3829, with a protocol IC (not shown), switches a data transfer path of data transmitted/received over a channel3830, between the protocol IC and PCI functions (FF0to FF7). Although only the FE (Front-End) interface devices are described herein, it is needless to mention that each of the storage devices2700and2900includes a CM (Cache Memory) interface device and BE (Back-End) interface device.

As described in Embodiment 8, the internal network of the storage controller includes two MR-IOV networks. One of the networks is a network that mutually connects the RP0, RP2, RP4, RP6, switches2750and3850, EP01, and EP31(interfaces other than the FE interface are omitted herein). The other network is a network that mutually connects the RP1, RP3, RP5, RP7, switches2770and3870, EP02, and EP32(interfaces other than the FE interface are omitted). As the two MR-IOV networks are independent networks, they are managed by different MR-PCIMs. That is, the former network is managed by MR-PCIMs3820and3822, and the latter network is managed by MR-PCIMs3821and3823. The MR-PCIM3820handles the management of the switches2750and3850and the EP (e.g., EP01) in each interface that is accessible from the RP0in the storage device2700. The MR-PCIM3822handles the management of the EP (e.g., EP31) in each interface that is accessible from the RP4in the storage device2900. Meanwhile, the MR-PCIM3821handles the management of the switches2770and3870and the EP (e.g., EP02) in each interface that is accessible from the RP3in the storage device2700. Further, the MR-PCIM3823handles the management of the EP (e.g., EP32) in each interface that is accessible from the RP7in the storage device2900.

The switch2750has four VSs (2800to2803). Initially, the VS2800and VS2802are set as the authorized VSs, and the VS2800is set as the management VS. The switch3850has four VSs (3800to3803). Initially, the VS3800and VS3802are set as the authorized VSs, and the VS3802is set as the management VS.

<Management Table for Switching Data Transfer Path>

FIG. 39is a diagram showing a management table3900for controlling the switching of a data transfer path with the internal switches3826and3829. This management table3900is, as with the management table3200(seeFIG. 17) of the first embodiment, stored in memory in the front-end interface and is referred to by an embedded processor in the front-end interface. Accessing the management table3900from the processor (e.g., CPU0) in the internal network of the storage controller or from the administrative terminal SVP allows the setting of the table to be changed.

The management table3900has stored therein information3901on the data write target or data read target included in a packet header transmitted/received over the channel2702or3830, and setting of the PCI function (3902to3906) that is the transmission source/destination of the data. Specifically, the management table3900has stored therein setting of the PCI functions FF0to FF3of the EP01(EP31) and the PCI functions FF4to FF7of the EP02(EP3). As the information3901on the data write target or data read target, a logical unit number is stored, for example. The PCI function setting3902is the setting when the storage device operates normally. The PCI functions FF0to FF7are assigned to the logical unit numbers0to7, respectively.

The PCI function setting3903is the setting when only the two PCI functions FF2and FF3or FF6and FF7are used in each EP. The PCI function setting3904is the setting when only the two PCI functions FF0and FF1or FF4and FF5are used in each EP. Case1_3903and case2_3904are related to the assignment of PCI functions used when a storage device is added or removed.

The PCI function setting3905is the setting when only the four PCI functions FF0to FF3of the EP01(EP31) are used. The PCI function setting3906is the setting when only the four PCI functions FF4to FF7of the EP02(EP32) are used. Case3_3905and case4_3906are related to the assignment of PCI functions when all tasks are distributed to only one of the networks.

FIG. 40is a flow chart for describing the fail-over processing of the MR-PCIM. Herein, processing when a software error has occurred in the MR-PCIM3820in the storage device shown inFIG. 38will be described. During the normal operation, the MR-PCIM3820handles the management of the switches2750and3850and the management of the EP (e.g., EP01) in each interface that is accessible from the RP0in the storage device2700. Meanwhile, the MR-PCIM3822handles the management of the EP (e.g., EP31) in each interface that is accessible from the RP4in the storage device2900.

First, the MR-PCIM3822, upon detecting an error in the MR-PCIM3820, changes the setting of the internal switch path for each interface device in the storage device (S4001). For example, the MR-PCIM3822changes the data transfer path of the internal switch3826(3829) from3902to3906of the management table3900. As a result, only the PCI functions F4to F7in the EP of each interface device will be used, and thus the storage devices2700and2900will not transfer data via the switches2750and3850.

Next, the MR-PCIM3822changes the setting of the management VS in the switch2750from the VS2800to the VS2802. In addition, the MR-PCIM3822also changes the setting of the management VS in the switch3850from the VS3802to the VS3800(S4002). It should be noted that such procedures for changing the management VS can be omitted if an MR-PCIM, which does not manage the switches, is reset.

Then, the MR-PCIM3822instructs the processor CPU0to reset and reboot the MR-PCIM3820(S4003).

Next, the MR-PCIM3822changes the setting of the internal switch path back to the original setting for each interface device in the storage device (S4004). For example, the MR-PCIM3822changes the data transfer path of the internal switch3826(3829) from3906to3902of the management table3900.

After the execution of the aforementioned MR-PCIM fail-over, the MR-PCIM3820handles the management of the EP (e.g., EP01) in each interface that is accessible from the RP0in the storage device2700. Meanwhile, the MR-PCIM3822handles the management of the switches2750and3850and the EP (e.g., EP31) in each interface that is accessible from the RP4in the storage device2900.

The present invention can be widely applied, not only to storage devices with internal networks, but also to computers such as a blade server, its internal network technique, and the like.

In the switch of the storage device in accordance with the present invention, a plurality of ports other than those connected to the RPs, FE I/F, BE I/F, and CM I/F are connected with a cross-link. Each processor is allowed to control the FE I/F, BE I/F, or CM I/F either via a path that passes through the cross-link or via a path that does not pass through the cross-link. In such a case, the connection relationship between the downstream bridges of each virtual switch in the switch and each interface device (FE I/F, BE I/F, and CM I/F) will not be changed by the change in the number of added storage devices or by the attachment/detachment of the cross-link. Accordingly, PCI functions provided by each EP can be effectively used in a single MR-IOV network. In addition, as a plurality of storage devices can be easily connected by removing the cross-link, the size of the storage device can be easily expand or reduced.

In addition, in the expanded storage device (a configuration with a plurality of unit devices connected together), if an error has occurred in an MR-PCIM which manages a switch in the expanded storage device in one of the unit devices, an MR-PCIM in another unit device resets and reboots the error occurred MR-PCIM. After the execution of such fail-over processing, the functions for management of the switch of the MR-PCIMs are switched before and after the occurrence of the error. Accordingly, in the expanded storage device, an MR-PCIM in any unit device can initialize and manage the MR-IOV network.

In addition, when the cross-link is removed in changing the size of the storage device (in adding or removing a storage device), assignment of PCI functions in each EP that is accessible from each processor is temporarily changed to limit the usable PCI functions (seeFIG. 17). Accordingly, even when some of the PCI functions are hotremoved, the storage device will continue to operate adequately, and thus, the operation of the storage device need not be completely stopped while another storage device is being added or removed. Accordingly, adverse effect on the operation of the storage device can be minimized.

REFERENCE SIGNS LIST