Information processing apparatus, computer-readable recording medium having stored program for controlling information processing apparatus, and method for controlling information processing apparatus

A control device controls the switch device and the plurality of processing apparatuses such that, when one of replicated data pieces stored in two or more different storage devices among the plurality of storage devices is lost, the replicated data piece stored in a second storage device other than a first storage device storing lost replicated data piece among the two or more different storage devices is copied through the switch device to reconstruct the lost replicated data piece. Thus, a redundancy of replicated data is restored without affecting a bandwidth of a network.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of the prior Japanese Application No. 2013-128606 filed on Jun. 19, 2013 in Japan, the entire contents of which are hereby incorporated by reference.

FIELD

The present invention relates to an information processing apparatus, a computer-readable recording medium having a stored program for controlling an information processing apparatus, and a method for controlling an information processing apparatus.

BACKGROUND

In an object storage system including a plurality of servers (a server cluster), actual data (object) is transmitted between a user or a web application and each server via a network.

For example, in an object storage system illustrated inFIG. 37, the user transmits or receives actual data (object) to or from a server group including a plurality of servers300A to300D (four servers inFIG. 37) through a network100such as the Internet or a local area network (LAN) and a gateway200. In the following description, when one of the servers is specified, one of reference numerals300A to300D is used, but when an arbitrary server is designated, a reference numeral300is used. As a storage place of actual data, a disk drive (hard disk drive (HDD)) (see reference numerals303ato303cinFIGS. 38 and 39) equipped in each server300is used.

At this time, as a counter-measure against a drive failure, a replica that is a replicated data piece of the same object is stored in HDDs of a plurality of different servers300in order to prevent an object from being lost when a drive failure occurs. A certain replica number of the same object stored in the object storage system is referred to as a “multiplicity” or “redundancy” of a replica.FIG. 37illustrates an example in which a replica multiplicity is 3.

A gateway (front end server)200interposed between the network100and the server group generates a certain number (3inFIG. 37) of replicas on an object when a write access of an object or the like is received from the user. Further, the gateway200distributedly arranges and stores the certain number of generated replicas in the different servers300A to300C.

Next, processing when a drive failure occurs in the object storage system illustrated inFIG. 37will be described with reference toFIGS. 38 and 39. Each server300is connected to the LAN100, and includes a central processing unit (CPU)301, a memory302, and a plurality of HDDs303ato303c(three HDDs inFIG. 38) as illustrated inFIGS. 38 and 39. The actual data (object) accessed by the user is stored in the HDDs303ato303cof each server300. The memory302of each server300stores various kinds of data, programs, and the like for processing of the CPU301, and temporarily stores an object to be stored in the HDDs303ato303c. InFIGS. 38 and 39, the gateway200illustrated inFIG. 37is not illustrated.

First, three replicas (refer to a star inFIG. 38) of the same object are stored in the HDD303aof the server300A, the HDD303bof the server300B, and the HDD303cof the server300C, respectively, as illustrated inFIG. 38. At this time, the HDD303aof the server300A is assumed to be faulty as illustrated inFIG. 39. In this case, the object stored in the HDD303aof the server300A is lost, the replica number (multiplicity or redundancy) of the object in the system is reduced from 3 to 2. For this reason, the replica number of the object is restored to a certain number (here, 3) before the replica number of the object becomes zero (0).

In this regard, at least one of the server300B and the server300C having the replica of the object transmits the replica in the HDDs303bor303cto the server300A via the LAN100as indicated by an arrow A1or A2inFIG. 39, and copies the replica of the object to the server300A. As a result, the replica number of the object in the system is restored to the certain number, that is, 3. A specific means for selecting, for example, a server or a HDD in which a replica is to be transmitted at the time of the replica number restoration process depends on an implementation.

Here, the restoration process speed of the replica number is not allowed to exceed the band of the LAN100. When the user writes or reads an object during the replica number restoration process, it competes with the restoration process. At this time, when the user process is given a priority, the speed of the restoration process decreases, and when the restoration process is given a priority, the speed of the user process decreases.

As a related art, a technique of performing a backup process from a disk to a tape via a storage area network (SAN) without using a LAN has been known (see JP 2002-007304A). However, this technique aims to simply increase the speed of the backup process from the disk to the tape, and does not related to a replica number restoration process in an environment in which a certain number of replicas (replicated data pieces) of the same data are distributedly arranged in a certain number of servers (HDDs).

In a storage system including a server cluster, a bandwidth of a LAN connecting servers with one another serves a bottle neck of processing performance. For this reason, when a drive failure occurs and so the replica number restoration process is performed using a LAN between servers as described above, the restoration process speed is limited. Thus, when a processing request of a system user becomes a high load during the restoration process, the restoration process hinders the user process, or vice versa.

SUMMARY

An information processing apparatus of the present disclosure includes a plurality of storage devices, a plurality of processing apparatuses, a switch device that switches a connection state between the plurality of storage devices and the plurality of processing apparatuses, and a control device that controls the switch device and the plurality of processing apparatuses. The control device controls the switch device and the plurality of processing apparatuses such that, when one of replicated data pieces stored in two or more different storage devices among the plurality of storage devices is lost, the replicated data piece stored in a second storage device other than a first storage device storing lost replicated data piece among the two or more different storage devices is copied through the switch device to reconstruct the lost replicated data piece.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, exemplary embodiments will be described with reference to the appended drawings.

[1] Configuration of Object Storage System and Outline of Replica Redundancy Restoration Process According to Present Embodiment

First, a configuration of an object storage system (information processing apparatus)1and an outline of a replica redundancy restoration process according to the present embodiment will be described with reference toFIGS. 1 to 6.

The object storage system (information processing apparatus)1according to the present embodiment includes a LAN2, a server group3, a drive enclosure4, a drive connection selector switch5, and a control device6as illustrated inFIGS. 1 to 6.

The LAN2is a network such as the Ethernet (a registered trademark), and connects servers3A to3C configuring the server group3with one another and connects the server group3with the control device6so that communication can be performed therebetween.

The server group (a server cluster or a storage node group)3includes a plurality of servers (nodes or processing apparatuses)3A to3C. The server group3illustrated inFIGS. 1 to 6includes the three servers3A to3C but may include four or more servers. Each of the servers3A to3C of the present embodiment includes a CPU (processing unit)31and a memory32. Storage devices (drives)4ato4j,41, and42used by the servers3A to3C are aggregated into a drive pool (drive enclosure)4. The memory32stores various kinds of data, various kinds of programs, and the like which are necessary for processing of the CPU31, and temporarily stores objects to be stored in the storage devices4ato4j,41, and42. Further, various kinds of storage devices such as a random access memory (RAM), a HDD, or a solid state drive (SSD) may be used as the memory32.

Various kinds of drives (for example, a HDD and an SSD) may be mounted in the drive pool4together. Further, the drive pool4is configured so that many drives can be stored, and an extra drive (third storage device) for a backup can be mounted as well. In the present embodiment, the ten HDDs4ato4jand the two SSDs41and42are equipped as the drives, but the present invention is not limited to the number of drives. Further, in the present embodiment, the HDD4jis equipped as the extra drive, but two or more HDDs may be equipped as an extra drive. Further, in the present embodiment, the SSDs41and42are used as a storage device for data replication which will be described later, but three or more SSDs may be equipped as a storage device for data replication.

A drive connection selector switch (switch device)5switches a connection state between the plurality of storage devices4ato4j,41, and42and a plurality of CPU3A to3C, and is configured with, for example, an SAS expander which will be described later with reference toFIGS. 8 and 9. The switch5is controlled by a function of a manager server of the control device6which will be described later. As the switch5switches the connection state, each of the servers3A to3C can exclusively access any one of the storage devices4ato4j,41, and42.

In the example illustrated inFIG. 1, one node including the server3A and the drive group4A is configured such that the drive group4A including the three HDDs4ato4care connected to the server3A by the switch5. Further, one node including the server3B and the drive group4B is configured such that the drive group4B including the three HDDs4dto4fare connected to the server3B by the switch5. Furthermore, one node including the server3C and the drive group4C is configured such that the drive group4C including the three HDDs4gto4iare connected to the server3C by the switch5.

As a result, the server3A and the HDDs4ato4care managed as one node, and objects accessed through the server3A are stored in the HDDs4ato4c. Further, the server3B and the HDDs4dto4fare managed as one node, and objects accessed through the server3B are stored in the HDDs4dto4f. Further, the server3C and the HDDs4gto4iare managed as one node, and objects accessed through the server3C are stored in the HDDs4gto4i. At this time, the HDD4jused as an extra drive for a backup and the SSDs41and42used for data copy are not connected to any of the servers3A to3C, that is, in the disconnected state.

The control device6is connected to perform communication with the switch5via a universal serial bus (USB), connected to perform communication with the servers3A to3C (the CPUs31) via the LAN2, and controls the switch5and the servers3A to3C (the CPUs31). The control device6controls the switch5and the servers3A to3C (the CPUs31) and functions as a manager server that performs a replica redundancy restoration process which will be described later. Further, the control device6has a function of a front end server accessing a corresponding object in the drive groups4A to4C through the LAN2and the servers3A to3C according to access from the user.

At this time, in the present object storage system1, as a counter-measure against a drive failure, a replica that is a replicated data piece of the same object is stored in HDDs of a plurality of different servers in order to prevent an object from being lost when a drive failure occurs.FIGS. 1 to 6illustrate an example in which a replica redundancy (multiplicity) is 2. In other words, the front end server function of the control device6generate two replicas on a corresponding object and distributedly arranges the generated replicas in two HDDs belonging to different drive groups. Through this operation, in the example illustrated inFIG. 1, two replicas (replicated data pieces) D1and D1of a certain object are distributedly arranged in the HDDs4aand4e, and two replicas (replicated data pieces) D2and D2of another object are distributedly arranged in the HDDs4aand4i.

Further, the manager server function of the control device6performs a process of restoring the replica redundancy when one of the replicas is lost and the replica redundancy is reduced from 2 to 1. In other words, the control device6controls the switch5and the CPUs31of the servers3A to3C such that a corresponding replica from the HDDs (second storage devices)4eand4istoring the corresponding replica other than the HDD (first storage device)4athat has stored the lost replica is copied through the switch5, and the lost replica is reconstructed in the drive group4A. At this time, for example, the control device6copies the replicas D1and D2stored in the HDDs4eand4ito the HDD (third storage device)4jother than the HDD4athrough the switch5.

Next, an outline of the restoration process performed by the control device6when the HDD4astoring the replicas D1and D2is faulty as illustrated inFIG. 2in a state in which the replicas D1and D2are stored with the redundancy of 2 as illustrated inFIG. 1will be described with reference toFIGS. 2 to 6. Here, a node including the faulty HDD4ais referred to as a “first node”, and two nodes including other HDDs4eand4istoring the replicas D1and D2that has been stored in the faulty HDD4aare referred to as “second nodes”. Further, the server (processing apparatus)3A included in the first node is referred to as a “first processing apparatus”, and the servers (processing apparatuses)3B and3C included in the two second nodes are referred to as “second processing apparatuses”.

Here, when the HDD4astoring the replicas D1and D2is faulty as illustrated inFIG. 2, the control device6controls the switch5such that the SSDs41and42that are unused data replication storage devices are connected to the servers (second processing apparatuses)3B and3C configuring the second node. As a result, the SSDs41and42used for data copy are included in the drive groups4B and4C, respectively. Thus, the servers3B and3C storing the replicas D1and D2of the object stored in the faulty HDD4anewly exclusively use the SSDs41and42, respectively.

Further, the control device6controls the servers3B and3C (the CPUs31) such that the replicas D1and D2stored in the HDDs4eand4iare copied to the SSDs41and42connected to the servers3B and3C through the switch5. As a result, in the server3B and the second nodes belonging to the drive group4B, the replica D1to be restored is locally copied from the HDD4eto the SSD41as indicated by an arrow A3inFIG. 3. Similarly, in the server3C and the second nodes belonging to the drive group4C, the replica D2to be restored is locally copied from the HDD4ito the SSD42as indicated by an arrow A4inFIG. 3. At this time, the replica number (the replica redundancy) is 2 and equal to or larger than before a drive failure occurs, and thus a risk of object loss is avoided.

Thereafter, the control device6controls the switch5such that the SSDs41and42to which the replicas D1and D2are copied are disconnected from the servers3B and3C and connected to the server (first processing apparatus)3A configuring the first node. As a result, the SSDs41and42used for data copy are included in the drive group4A as illustrated inFIG. 4. Thus, the SSDs41and42are dedicated to the server3A in which a drive failure occurred. At this time, the server3A and the nodes belonging to the drive group4A are logically reconstructed to the state before the drive failure occurs.

Further, the control device6controls the switch5such that the HDD4jserving as the unused third storage device is connected to the server3A instead of the HDD (first storage device)4a. As a result, as illustrated inFIG. 4, the HDD4jthat is an extra drive for a backup replaced with a faulty drive is included in the drive group4A, and is dedicated to the server3A instead of the faulty HDD4a.

Further, the control device6controls the server3A (the CPU31) such that the replicas D1and D2stored in the SSDs41and42used for data copy are stored in the HDD4jconnected to the server3A through the switch5. As a result, in the server3A and the first nodes belonging to the drive group4A, the replica D1to be restored is copied from the SSD41to the HDD4jas indicated by an arrow A5inFIG. 5. Similarly, in the first node, the replica D2to be restored is copied from the SSD42to the HDD4jas indicated by an arrow A6inFIG. 5.

Lastly, the control device6stores the replicas D1and D2in the HDD4j, and then controls the switch5such that the SSDs41and42used for data copy is disconnected from the server3A to return to an unused state. As a result, as illustrated inFIG. 6, the SSDs41and42return to a free state, that is, a disconnected state in which the SSDs41and42are not connected to any of the servers3A to3C. At this time, one extra drive for a backup enters the used state from the unused state, and the object storage system1is also restored to the same state as before a drive failure occurs.

Since the replica redundancy restoration process is performed using the switch5between the server group3and the drive enclosure4as described above with reference toFIGS. 1 to 6, in the present embodiment, the replica redundancy is restored without affecting the bandwidth of the LAN2. Thus, even when a load in the LAN2is large, the replica redundancy restoration process is not affected by the LAN2. For this reason, even in the configuration in which the bandwidth of the LAN2is limited, it is possible to rapidly restore a reduction in the replica number, and even when a HDD being used is faulty, it is possible to immediately return a distribution arrangement situation of a replica to an initial ideal allocation.

Further, as the storage device for data replication, a HDD may be used instead of the SSDs41and42. Since HDDs are cheaper in capacity unit price than SSDs, when a HDD is used as the data replication storage device, the system1of the present embodiment can be provided at the lower cost when an SSD is used.

On the other hand, SSDs are faster in transfer speed at the time of a reading process and a transfer speed at the time of a writing process than HDDs. For this reason, as an SSD is used as the storage device for data replication as in the present embodiment, the replica redundancy restoration process can be performed at a higher speed than when a HDD is used as the storage device for data replication. In the present embodiment, it is possible not only to effectively use a small number of expensive devices but also to perform the replica redundancy restoration process at a high speed at a low cost as possible by equipping two SSDs, for example, as the storage device for data replication.

[2] Concrete Hardware Configuration and Function of Object Storage System of Present Embodiment

Next, a hardware configuration and a function of an object storage system (information processing apparatus)1will be more concretely described with reference toFIGS. 7 to 17. First, a concrete hardware configuration and a functional configuration of the object storage system1according to the present embodiment will be described with reference toFIG. 7.FIG. 7is a block diagram illustrating a concrete hardware configuration and a functional configuration of the object storage system1. In the drawings, the same reference numerals as the reference numerals described above denote the same parts or almost the same parts, and thus a detailed description thereof is omitted.

The object storage system1illustrated inFIG. 7includes a LAN (network switch)2, a server group3, a drive enclosure4, a drive connection selector switch5, and a control device6, similarly to the system1illustrated inFIGS. 1 to 6.

In the system1illustrated inFIG. 7, the control device6includes a CPU (processing unit)61and a memory62. The memory62stores various kinds of data, various kinds of programs, and the like which are necessary for processing of the CPU61. Among various kinds of programs, there is a control program performed by the CPU61when the control device6functions as the manager server and the front end server. Further, among various kinds of data, there are tables T1to T4which are referred to, collated, searched for, and updated by the CPU61when the control device6functions as the manager server and the front end server. The tables T1to T4will be described later with reference toFIGS. 13 to 16. As the memory62, various kinds of storage devices such as a RAM, a HDD, or an SSD may be used.

In the system1illustrated inFIG. 7, the switch5is configured with a serial attached small computer system interface (SCSI) (SAS) expander illustrated inFIG. 8. The SAS expander5is connected to the server group3and the drive enclosure4through an SAS interface. Further, the SAS expander5is connected to the control device6via a USB interface. When the control device6functions as the manager server, the control device6transmits a serial management protocol (SMP) command to the SAS expander5through the USB interface, and operates zoning information of the SAS expander5. As a result, it is possible to arbitrarily set a combination of servers belonging to each of a plurality of nodes (three nodes inFIGS. 7 to 9) and one or more HDDs and/or SSDs. Further, the SAS expander5may be connected to the control device6through an interface such as the Ethernet.

Next, the SAS expander5will be described with reference toFIGS. 8 and 9.FIG. 8is a diagram for describing the SAS expander (switch device)5in the object storage system1according to the present embodiment.FIG. 9is a diagram for describing a zoning setting in the SAS expander5illustrated inFIG. 8.

The SAS expander5includes PHY (physical layer ports)51ato51cconnected to host bus adapters (HBAs)33of the servers3A to3C, and PHYs52ato52lconnected to the HDD4ato4jand the SSDs41and42. The HBAs33of the respective servers are associated with the drives4ato4j,41, and42by a zoning setting of the SAS expander5. The zoning setting may be changed by an SMP command from the control device6.

In the SAS expander5illustrated inFIG. 8, the zoning setting is performed as illustrated inFIG. 9. In other words, the PHYs51aand52ato52care set to a zone ID (IDentification) “Z1” corresponding to the server3A. As a result, one node in which the PHY51ais connected the PHYs52ato52c, and the server3A is associated with the HDDs4ato4cis configured in the zone Z1. Further, the PHYs51band52dto52fare set to a zone ID “Z2” corresponding to the server3B. As a result, one node in which the PHY51bis connected the PHYs52dto52f, and the server3B is associated with the HDDs4dto4fis configured in the zone Z2. Furthermore, the PHYs51cand52gto52iare set to a zone ID “Z3” corresponding to the server3C. As a result, one node in which the PHY51cis connected the PHY52gto52i, and the server3C is associated with the HDDs4gto4iis configured in the zone Z3.

Further, in the SAS expander5illustrated inFIG. 8, a zone ID “Z4” that does not correspond to any server and is used for disconnection is set in addition to the zones Z1to Z3respectively corresponding to the servers3A to3C. In other words, since the PHYs52j,52k, and52lconnected to the HDD4jused as an extra drive for a backup and the SSDs41and42used for data copy are set to the zone ID “Z4” as illustrated inFIG. 9, the HDD4jand the SSDs41and42are not connected to any of the servers3A to3C, that is, are in the disconnected state.

[2-2] Exemplary Setting of Drive ID

Next, an exemplary setting of a drive ID used in the object storage system1according to the present embodiment, that is, an ID of each drive (each of the HDD4ato4jand the SSDs41and42) aggregated in the drive pool4will be described with reference toFIGS. 10 to 12.

In the present embodiment, for example, designation of a drive to be connected to or disconnected from the server group3through the switch5and management of a connection state of each drive are performed using a drive ID. Any value can be set as the drive ID as long as the value is a unique value identifying each drive. In the present embodiment, for example, the drive ID is set based on a drive storage position in the drive enclosure4as illustrated inFIGS. 10 to 12.

When four units #0 to #3 are mounted in one drive enclosure, 16 slots #0 to #15 are equipped in each unit, and a HDD or an SSD is mounted in each lost as illustrated inFIG. 10, the drive ID is set as follows. In other words, for example, for a drive mounted in the slot #3 of the unit #2, “2-3” is set and assigned as the drive ID. Either a HDD or an SSD can be mounted in each slot through the SAS interface or an SATA (serial advanced technology attachment (SATA) interface.

Further, when the size of the drive pool4is increased by connecting the drive enclosures illustrated inFIG. 10in a cascade form as illustrated inFIG. 11, the drive ID is set as follows.FIG. 11illustrates an example (an example of a two-step cascade) in which three drive enclosures illustrated inFIG. 10are connected in a cascade form. At this time, the drive enclosures may be assigned numbers of #0, #1, #2, and the like in order from the enclosure in the head of the cascade, and in this case, for example, for a drive mounted in the slot #3 of the unit #2 in the enclosure #1, “1-2-3” may be set and assigned as the drive ID.

Further, when the size of the drive pool4is increased by connecting each server to a plurality of cascade sets (the enclosure sets illustrated inFIG. 11) as illustrated inFIG. 12, the drive ID is set as follows.FIG. 12illustrates an example in which the two enclosure sets illustrated inFIG. 11are connected to each server. Each enclosure set is referred to as a “domain”. At this time, two sets of domains are assigned #0 and #1, and in this case, for example, for a drive mounted in the slot #3 of the unit #2 in the enclosure #1 of the domain #0, “0-1-2-3” may be set and assigned as the drive ID.

Next, the tables T1to T4used in the object storage system1according to the present embodiment will be described with reference toFIGS. 13 to 16. Here, the device management table T1and the replica management table T2are used when the front end server function of the control device6performs a distribution arrangement of objects as will be described later with reference toFIG. 17. Further, the server/drive connection table T3is updated each time the manager server function of the control device6performs a connection or disconnection of a drive. Further, the drive attribute table T4is used to decide an SSD for data copy when the manager server function of the control device6performs the replica redundancy restoration process.

[2-3-1] Device Management Table T1

First, the device management table T1in the object storage system1according to the present embodiment will be described with reference toFIG. 13. In the device management table T1, at least a device ID logically identifying a device such as a HDD or an SSD is associated with a drive ID physically specifying an actual drive such as a HDD or an SSD corresponding to the device ID.

In the device management table T1illustrated inFIG. 13, a device ID (devID), a drive ID (driveID), a mount point, and a server IP (Internet Protocol) address are associated with one another. The drive ID is an ID described above with reference toFIGS. 10 to 12. The mount point represents a path of a directory in which a drive associated with a device ID is mounted in a file system of a server. A directory name serving as the mount point may be arbitrary, and in the present embodiment, a character string as the drive ID is used as the mount point. Further, the server IP address is used to perform communication with a drive associated with a device ID via the LAN2. In implementing the function of the present embodiment, a correspondence relation between the device ID and the drive ID may be held in the device management table T1, and the mount point and the server IP address may be omitted.

[2-3-2] Replica Management Table T2

The replica management table T2in the object storage system1according to the present embodiment will be described with reference toFIG. 14. The replica management table T2is a table having a hash correspondence of each replica used when replicas of an object are distributedly arranged. In the following description, the replica redundancy is assumed to be 3, andFIG. 14illustrates an example in which the replica redundancy is 3.

In the replica management table T2illustrated inFIG. 14, device IDs specifying three devices (drives) in which three replicas #1 to #3 are to be stored are associated with a upper n-bit value of a hash value of an object. The hash value of the object is a bit string of a certain width obtained by applying a one-way hash function such as a message digest algorithm 5 (MD5) to a path name of an object including the user's account name, an object name, or the like. As the replica management table T2is collated and search using the upper n-bit value of the hash value as a key, the three devices in which the three replicas #1 to #3 of the object are to be stored are specified. The replica management table T2is generated before the system1is activated and managed by the control device (front end server)6.

Further, a correspondence between the upper n-bit value of the hash value of the object and the three device IDs is randomly decided basically. Here, three devices associated with the same upper n-bit value are assumed to belong to different zones (nodes). In other words, three replicas of the same hash value are assumed to be assigned devices belonging to different zones (nodes).

In the replica management table T2illustrated inFIG. 14, for example, three device IDs “dev0”, “dev1”, and “dev2” are assigned to a value “00001” of the upper n bits of the hash value. Thus, three replicas of an object in which the upper n-bit value of the hash value is “00001” are arranged in three devices specified by the device IDs “dev0”, “dev1”, and “dev2”. The device IDs “dev0”, “dev1”, and “dev2” are recognized as HDDs corresponding to the drive IDs “0-0-0-0”, “0-0-0-1”, and “0-0-0-2” based on the device management table T1illustrated inFIG. 13.

Next, a storage example of an object (replica) in each HDD will be described with reference toFIG. 14. In each HDD, a directory is set to each upper n-bit value of a hash value of an object. All objects having the hash values starting from the upper n-bit value set to the directory are stored in each directory. For example, when a replica distribution arrangement is performed using the replica management table T2illustrated inFIG. 14as will be described with reference toFIG. 17, all objects having hash values starting from “00001”, “00002”, and “00010” are stored in the HDD of the device ID “dev2”.

The server/drive connection table T3in the object storage system1according to the present embodiment will be described with reference toFIG. 15. In the server/drive connection table T3illustrated inFIG. 15, a server ID (serverID) specifying a server is associated with a drive ID (driveID) specifying a drive to be connected to a server specified by the server ID. The server/drive connection table T3is updated by the control device6(manager server) each time a connection or disconnection of a drive is performed by the switch5.

As the server ID, for example, a value decided by a hardware component such as a number of a port/PHY to be connected to a corresponding server may be used, and a value decided using a table in which a server is associated with an IP address separately or like may be used. Further, as the drive ID, for example, the ID described above with reference toFIGS. 10 to 12is used.

Further, in the server/drive connection table T3, “free” is set as a server ID, and an ID of a drive in the disconnected state that is not connected to any of the servers3A to3C is associated with the server ID “free”. In the server/drive connection table T3illustrated inFIG. 15, for example, drive IDs “0-0-1-0”, “0-0-1-1”, and “0-0-1-2” of the HDD4jused as an extra drive for a backup and the SSDs41and42used for data copy are associated with the server ID “free”.

[2-3-4] Drive Attribute Table T4

The drive attribute table T4in the object storage system1according to the present embodiment will be described with reference toFIG. 16. In the drive attribute table T4, a drive ID (driveID) is associated with an attribute of a drive specified by the drive ID. Here, an attribute is information representing whether a corresponding drive is a HDD or an SSD as illustrated inFIG. 16, and used when the control device (manager server)6decides an SSD for data copy. As the drive ID, for example, the ID described above with reference toFIGS. 10 to 12is used.

The drive attribute table T4is managed by the control device (manager server)6. The drive attribute table T4may be generated in advance and then stored in the memory62, or may be generated such that the control device (manager server)6investigates drives belonging to the system1when the system1is activated and collects drive IDs and attributes (HDD or SSD) of respective drives.

[2-4] Replica Distribution Arrangement During Normal Operation

Next, a replica distribution arrangement during a normal operation in the object storage system1according to the present embodiment will be described with reference toFIG. 17.

The control device (front end server)6of the present embodiment manages, for example, a PUT command and a GET command given from the user. At this time, the control device6obtains a hash value of an object (a bit string of a certain width) by applying one-way hash function such as MD5 to a path name of an object including the user's account name, an object name, or the like. Then, the control device6collates and searches the replica management table T2using the obtained upper n-bit value of the hash value as a key, and acquires three device IDs (for example, dev0, dev1, and dev2) in which three replicas #1 to #3 of a corresponding object are to be stored.

Thereafter, the control device6collates and searches the device management table T1using the acquired three device IDs as a key, and acquires three drive IDs (for example, 0-0-0-0, 0-0-0-3, and 0-0-0-6) corresponding to the three device IDs (for example, dev0, dev1, and dev2). Further, the control device6collates and searches the server/drive connection table T3using the acquired three device IDs (for example, 0-0-0-0, 0-0-0-3, and 0-0-0-6), and acquires IDs (for example, 0, 1, and 2) of three servers to be connected to the three devices.

Then, the control device6transfers the three replicas #1 to #3 of the object to be written in the HDDs specified by the acquired three device IDs (for example, 0-0-0-0, 0-0-0-3, and 0-0-0-6) through the servers specified by the acquired three server IDs (for example, 0, 1, and 2) as illustrated inFIG. 17(see arrows A7to A9).

Through this operation, even in the present object storage system1, as a counter-measure against a drive failure, in order to prevent an object from being lost when a drive failure occurs, replicas that are replicated data pieces of the same object are stored in three HDDs that are under control of three different servers.

[3] Operation of Object Storage System of Present Embodiment

Next, the replica redundancy restoration process performed by the object storage system1according to the present embodiment having the above-described configuration will be described with reference toFIGS. 18 to 36.

In the following description, the replica redundancy restoration process is performed using the tables T1to T4described above with reference toFIGS. 13 to 16, and the replica redundancy is assumed to be 3.

Further, 0, 1, 2, and the like are assumed to be the server IDs of the servers3A,3B,3C, and the like, and 0-0-0-0, 0-0-0-1, 0-0-0-3, 0-0-0-4, 0-0-0-6, and 0-0-0-7 are assumed to be the drive IDs of the HDDs4a,4b,4d,4e,4g, and4h, as illustrated inFIGS. 19, 22, 25, 28, 31, and 34. Further, 0-0-1-0 is assumed to be the drive ID of the HDD4jused as an extra drive for a backup, and 0-0-1-1 and 0-0-1-2 are assumed to be the drive IDs of the SSDs41and42used for data copy. Furthermore, inFIGS. 19, 22, 25, 28, 31, and 34, a hyphen (-) inserted in each ID is omitted.

[3-1] Faulty Drive Disconnection Process

First, a faulty drive disconnection process performed by the object storage system1according to the present embodiment will be described according to a flowchart (steps S11to S14) illustrated inFIG. 18with reference toFIGS. 19 and 20.FIG. 19illustrates a node configuration of the system1corresponding to the process ofFIG. 18, andFIG. 20is a diagram for describing a process on the table T3corresponding to the process ofFIG. 18.

As illustrated inFIGS. 19 and 20, in the system1directly before a drive failure occurs, three storage nodes (the zone Z1to Z3) are constructed through the switch5. The HDD4ahaving the drive ID “0-0-0-0” and the HDD4bhaving the drive ID “0-0-0-1” and the server3A having the server ID “0” belong to the node in the zone Z1. The HDD4dhaving the drive ID “0-0-0-3” and the HDD4ehaving the drive ID “0-0-0-4” and the server3B having the server ID “1” belong to the node in the zone Z2. The HDD4ghaving the drive ID “0-0-0-6” and the HDD4hhaving the drive ID “0-0-0-7” and the server3C having the server ID “2” belong to the node in the zone Z3. Further, the HDD4jhaving the drive ID “0-0-1-0” and the SSDs41and42having the drive IDs “0-0-1-1” and “0-0-1-2” are associated with the server ID “free”, and in the disconnected state (free state) in which the drives are not connected to any of the servers3A to3C.

When a failure occurs in the HDD4aduring the operation of the system1, an operator or a monitoring device (not illustrated) detects the failure through a certain means, and notifies the control device (manager server which is also referred to simply as a “manager”)6of the occurrence of the failure together with the drive ID “0-0-0-0” of the faulty HDD4a(see step S11ofFIG. 18and an arrow A11ofFIG. 19). At this time, as the failure occurs in the HDD4a, in the system1, a replica stored in the HDD4ais lost, the replica redundancy is reduced from 3 to 2, and the manager6performs a process of restoring the replica redundancy to 3 (steps S12to S33) as follows.

The manager6that is notified of the occurrence of the failure collates and searches the server/drive connection table T3using the drive ID “0-0-0-0” of the faulty HDD4aas a key, and acquires the server ID “0” corresponding to the faulty HDD4a(see step S12ofFIG. 18and an arrow A12ofFIG. 20). Thereafter, the manager6changes the zoning setting in the SAS expander5based on the server ID “0” and the drive ID “0-0-0-0” through the USB interface, and performs the disconnection process of disconnecting the HDD4aof the drive ID “0-0-0-0” from the server3A of the server ID “0” (see step S13ofFIG. 18and an arrow A13ofFIG. 19). At this time, as the zoning setting is changed, in the SAS expander5, the connection state between the PHY51aand the PHY52a(seeFIG. 8) is released, and the PHY52aand the HDD4aconnected to the PHY52abelong to the disconnection zone Z4.

After disconnecting the HDD4afrom the server3A, the manager6updates the server/drive connection table T3according to the disconnection process of the HDD4a(step S14ofFIGS. 18 and 20). In other words, the manager6deletes the drive ID “0-0-0-0” associated with the server ID “0” in the server/drive connection table T3(see the table T3ofFIGS. 20 and 23). Thereafter, the manager6causes the process to proceed to step S15ofFIG. 21.

[3-2] Copy Target Server Selection Process

Next, a copy target server selection process performed by the object storage system1according to the present embodiment will be described according to a flowchart illustrated inFIG. 21(steps S15to S18) with reference toFIGS. 22 and 23.FIG. 22illustrates a node configuration of the system1corresponding to a process ofFIG. 21, andFIG. 23is a diagram for describing a process on a the tables T3and T4corresponding to the process ofFIG. 21.

In step S15ofFIG. 21, after deleting the faulty drive ID “0-0-0-0” in step S14, the manager6collates and searches the device management table T1using the faulty drive ID “0-0-0-0” as a key, and acquires the faulty device ID “dev0” corresponding to the faulty drive ID “0-0-0-0” (see an arrow A15ofFIG. 23).

Further, the manager6collates and searches the replica management table T2using the acquired faulty device ID “dev0” and acquires IDs “dev1” and “dev2” of devices holding the same replica as the replica held in the HDD4athat is the faulty device (see step S16ofFIG. 21and an arrow A16ofFIG. 23).

Further, the manager6collates and searches the device management table T1using the acquired device IDs “dev1” and “dev2” as a key and acquires the drive IDs “0-0-0-3” and “0-0-0-6” associated with the devices holding the same replica as the replica held in the faulty HDD4a(see step S17ofFIG. 21and an arrow A17ofFIG. 23).

Then, the manager6collates and searches the server/drive connection table T3using the acquired drive IDs “0-0-0-3” and “0-0-0-6” and acquires IDs “1” and “2” of two servers in which a replica was lost (see step S18ofFIG. 21and an arrow A18ofFIG. 23). Here, a “server in which a replica was lost” refers to a server that has held a replica whose redundancy was changed from 3 to 2. Thereafter, the manager6causes the process to proceed to step S19ofFIG. 24.

Next, an SSD selection/connection process performed by the object storage system1according to the present embodiment will be described according to a flowchart (steps S19to S22) illustrated inFIG. 24with reference toFIGS. 25 and 26.FIG. 25is a node configuration of the system1corresponding to a process ofFIG. 24, andFIG. 26is a diagram for describing a process on the tables T3and T4corresponding to the process ofFIG. 25.

In step S19ofFIG. 24, the manager6acquires a list of drive IDs in the free state from the server/drive connection table T3(see an arrow A19ofFIG. 26).

The manager6compares the acquired list of the drive IDs with the drive attribute table T4, and decides a necessary number of SSDs for data copy (see step S20ofFIG. 24and an arrow A20ofFIG. 26). At this time, since the number of the server IDs acquired in step S18ofFIG. 21is 2, the two SSDs41and42having the drive IDs “0-0-1-1” and “0-0-1-2” are selected and decided as an SSD for data copy.

Further, the manager6changes the zoning setting in the SAS expander5through the USB interface, and connects the SSDs41and42having the drive IDs “0-0-1-1” and “0-0-1-2” decided in step S20ofFIG. 24to the servers3B and3C of the server IDs “1” and “2” acquired in step S18ofFIG. 21(see step S21ofFIG. 24and an arrow A21ofFIG. 25). At this time, as the zoning setting is changed, in the SAS expander5, the PHY51bconnected to the server3B is connected with the PHY52kconnected to the SSD41, and the PHY51cconnected to the server3C is connected with the PHY52lconnected to the SSD42. As a result, as illustrated inFIG. 25, the servers3B and3C holding the same replica as the replica stored in the faulty HDD4anewly exclusively use the SSDs41and42.

After connecting the SSDs41and42with the servers3B and3C, the manager6updates the server/drive connection table T3according to the new connection (step S22ofFIG. 24). In other words, the manager6deletes the drive IDs “0-0-1-1” and “0-0-1-2” associated with the server ID “free” from the line of the server ID “free” in the server/drive connection table T3. Then, the manager6moves and adds the drive IDs “0-0-1-1” and “0-0-1-2” to the server IDs “1” and “2”, respectively (see an arrow A22ofFIG. 26). Thereafter, the manager6causes the process to proceed to step S23ofFIG. 27.

[3-4] Process of Copying Replica to SSD

Next, a process of copying a replica to an SSD which is performed by the object storage system1according to the present embodiment will be described according to a flowchart (steps S23to S25) illustrated inFIG. 27with reference toFIGS. 28 and 29.FIG. 28is a diagram illustrating a node configuration of the system1corresponding to a process ofFIG. 27, andFIG. 29is a diagram for describing a replica copy process corresponding to the process (step S24) ofFIG. 27.

In step S23ofFIG. 27, the manager6gives a copy start instruction to the storage nodes to which the servers3B and3C belong via the LAN2(see an arrow A23ofFIG. 28). At this time, the manager6notifies the server3B of the drive ID “0-0-0-3” the copy source, the drive ID “0-0-1-1” of the copy destination, and the hash value (the upper n-bit value) of the object to be copied. Similarly, the manager6notifies the server3C of the drive ID “0-0-0-6” the copy source, the drive ID “0-0-1-2” of the copy destination, and the hash value (the upper n-bit value) of the object to be copied. The hash value (the upper n-bit value) of the object to be copied is obtained by collating and searching the replica management table T2using the device ID “dev0” of the faulty HDD4aas a key.

In each of the servers3B and3C that have received the copy instruction, the CPU31performs the process of copying the replica to the SSDs41and42(step S24ofFIG. 27). At this time, in the server3B, the replica having the designated hash value (the upper n-bit value) is copied from the HDD4d(dev1) having the drive ID “0-0-0-3” to the SSD41having the drive ID “0-0-1-1” through the switch5and the CPU31(see an arrow A241ofFIG. 28). Similarly, in the server3C, the replica having the designated hash value (the upper n-bit value) is copied from the HDD4g(dev2) having the drive ID “0-0-0-6” to the SSD42having the drive ID “0-0-1-2” through the switch5and the CPU31(see an arrow A242ofFIG. 28).

As illustrated inFIG. 29, the copy process from the HDD4g(dev2) to the SSD42is performed by setting only the replica present in the faulty HDD4aas a target. In other words, in the example illustrated inFIG. 29, replicas having hash values (upper n-bit values) of 00001 and 00010 are stored in the faulty HDD4aand thus set as a copy target, a replica having a hash value (upper n-bit value) of 00002 is not stored in the faulty HDD4aand thus not set as a copy target.

After ending the copy process, each CPU31gives an end notification to the manager6via the LAN2(see step S24ofFIG. 27and an arrow A243ofFIG. 28).

The manager6is on standby until the end notifications are received from all the storage nodes to which the servers3B and3C belong (step S25ofFIG. 27), and when the end notifications are received from all the storage nodes, the process proceeds to step S26ofFIG. 30. At this time, the replica number (the replica redundancy) is 3 and the same as before the drive failure occurs, and thus a risk of object loss is avoided.

[3-5] SSD Movement Process and Backup HDD Addition Process

Next, an SSD movement process and a backup HDD addition process performed by the object storage system1according to the present embodiment will be described according to a flowchart (steps S26to S30) illustrated inFIG. 30with reference toFIGS. 31 and 32.FIG. 31is a node configuration of the system1corresponding to a process ofFIG. 30, andFIG. 32is a diagram for describing a process on the tables T1, T3, and T4corresponding to the process ofFIG. 30.

In step S26ofFIG. 30, the manager6changes the zoning setting in the SAS expander5through the USB interface, and disconnects the SSDs41and42connected to the servers3B and3C in step S21ofFIG. 24from the servers3B and3C. At this time, as the zoning setting is changed, in the SAS expander5, the PHY51bconnected to the server3B is disconnected from the PHY52kconnected to the SSD41, and the PHY51cconnected to the server3C is disconnected from the PHY52lconnected to the SSD42. After disconnecting the SSDs41and42from the servers3B and3C, the manager6updates the server/drive connection table T3according to the disconnection process of the SSDs41and42. In other words, the manager6deletes the drive IDs “0-0-1-1” and “0-0-1-2” associated with the server IDs “1” and “2” in the server/drive connection table T3.

Then, in step S27ofFIG. 30, the manager6changes the zoning setting in the SAS expander5through the USB interface, and connects the SSDs41and42disconnected from the servers3B and3C to the server3A having the faulty HDD4a. At this time, as the zoning setting is changed, in the SAS expander5, the PHY51aconnected to the server3A is connected to the PHYs52kand52lconnected to the SSDs41and42. As a result, as illustrated inFIG. 31, the server3A newly exclusively uses the SSDs41and42. After connecting the SSDs41and42with the server3A, the manager6updates the server/drive connection table T3according to the connection process of the SSDs41and42. In other words, the manager6moves and adds the drive IDs “0-0-1-1” and “0-0-1-2” of the SSDs41and42to the line corresponding to the server ID “0” of the server3A in the server/drive connection table T3(an arrow A27ofFIG. 32). At this time, the node to which the server3A belongs is logically restored to the state before the drive failure occurs.

Thereafter, step S28ofFIG. 30, the manager6searches a drive ID that is associated with the server ID “free” and has an attribute of HDD based on the server/drive connection table T3and the drive attribute table T4. In the example illustrated inFIG. 32, the manager6decides “0-0-1-0” as a drive ID that is associated with the server ID “free” and has an attribute of HDD. The decided HDD of the drive ID “0-0-1-0” is the unused HDD4j.

Then, in step S29ofFIG. 30, the manager6changes the zoning setting in the SAS expander5through the USB interface, and connects the unused HDD4jwith the server3A having the faulty HDD4a. At this time, as the zoning setting is changed, in the SAS expander5, the PHY51aconnected to the server3A is connected with the PHY52jconnected to the HDD4j. As a result, as illustrated inFIG. 31, the server3A newly exclusively use the HDD4j. After connecting the HDD4jwith the server3A, the manager6updates the server/drive connection table T3according to the connection process of the HDD4j. In other words, the manager6moves the drive ID “0-0-1-0” of the HDD4jfrom the line corresponding to the server ID “free” and adds the drive ID “0-0-1-0” of the HDD4jto the line corresponding to the server ID “0” of the server3A in the server/drive connection table T3(see an arrow A29ofFIG. 32).

Further, the manager6changes the drive ID “0-0-0-0” of the faulty HDD4aassociated with the device ID “dev0” to the drive ID “0-0-1-0” of the HDD4jnewly added in step S29in the device management table T1(see step S30ofFIG. 30and an arrow A30ofFIG. 32). Thereafter, the manager6causes the process to proceed to step S31ofFIG. 33.

[3-6] Process of Copying Replica to Backup HDD

Next, a process of copying a replica to a backup HDD which is performed by the object storage system1according to the present embodiment will be described according to a flowchart (steps S31to S33) illustrated inFIG. 33with reference toFIGS. 34 to 36.FIG. 34is a node configuration of the system1corresponding to a process ofFIG. 33,FIG. 35is a diagram for describing a replica copy process corresponding to a process (step S32) ofFIG. 34, andFIG. 36is a diagram for describing a process on the table T3corresponding to a process (step S33) ofFIG. 34.

In step S31ofFIG. 33, the manager6gives a copy start instruction to a storage node to which the server3A having the faulty HDD belong via the LAN2(see an arrow A31ofFIG. 34). At this time, the manager6notifies the server3A of the drive IDs “0-0-1-1” and “0-0-1-2” of the copy source, the drive ID “0-0-1-0” of the copy destination, and the hash value (the upper n-bit value) of the object to be copied. The hash value (the upper n-bit value) of the object to be copied is the same as the hash value notified in step S23ofFIG. 27and is obtained by collating and searching the replica management table T2using the device ID “dev0” of the faulty HDD4aas a key.

In the server3A that has received the copy instruction, the CPU31performs the process of copying the replica from the SSDs41and42to the HDD4j(step S32ofFIG. 33). At this time, the replica having the designated hash value (the upper n-bit value) is copied from the SSD41having the drive ID “0-0-1-1” to the HDD4j(dev0) having the drive ID “0-0-1-0” through the switch5and the CPU31(see an arrow A321ofFIGS. 34 and 35). Similarly, the replica having the designated hash value (the upper n-bit value) is copied from the SSD42having the drive ID “0-0-1-2” to the HDD4j(dev0) having the drive ID “0-0-1-0” through the switch5and the CPU31(see the arrow A322ofFIGS. 34 and 35).

After ending the copy process, the CPU31of the server3A gives an end notification to the manager6via the LAN2(see step S32ofFIG. 33and an arrow A323ofFIG. 34).

Then, in step S33ofFIG. 33, upon receiving the end notification from the storage node to which the server3A belongs, the manager6changes the zoning setting in the SAS expander5through the USB interface, and disconnects the SSDs41and42from the server3A. At this time, as the zoning setting is changed, in the SAS expander5, the PHY51aconnected to the server3A is disconnected from the PHYs52kand52lconnected to the SSDs41and42. After disconnecting the SSDs41and42from the servers3B and3C, the manager6updates the server/drive connection table T3according to the disconnection process of the SSDs41and42. In other words, as illustrated inFIG. 36, the manager6returns the drive IDs “0-0-1-1” and “0-0-1-2” associated with the server ID “0” to the section of the server ID “free” in the server/drive connection table T3. As a result, the SSDs41and42returns to the free state, that is, the disconnected state in which the SSDs41and42are not connected to any of the servers3A to3C. At this time, one extra drive for a backup changes from the unused state to the used state, and the object storage system1is restored to the same state before the drive failure occurs.

According to the object storage system1according to the present embodiment, when a drive failure occurs in the server3A of the server cluster 3 and so the replica number is reduced, the servers3B and3C individually copy replicas to the SSDs41and42for restoration. Then, after copying, the SSDs41and42are connected to the server3A of the restoration target through the switch (drive connection selector device)5, and so a reduction in the replica number is restored without using the LAN2between the servers.

Since the replica redundancy restoration process is performed using the switch5as described above, the replica redundancy is restored without affecting the bandwidth of the LAN2, and it is possible to avoid a situation in which the restoration process and the user process compete with each other and hinder each other. Thus, even when a load in the LAN2is large, the replica redundancy restoration process is not affected by the LAN2. For this reason, even in the configuration in which the bandwidth of the LAN2is limited, it is possible to rapidly restore a reduction in the replica number, and even when the HDD4abeing used is faulty, it is possible to immediately return a distribution arrangement situation of a replica to an initial ideal allocation.

Further, as an SSD is used as the storage device for data replication, the replica redundancy restoration process can be performed at a high speed. In the present embodiment, it is possible not only to effectively use a small number of expensive devices but also to perform the replica redundancy restoration process at a high speed at a low cost as possible by equipping the two SSDs41and42, for example, as the storage device for data replication.

The exemplary embodiment of the present invention have been described above, but the present invention is not limited to a related specific embodiment, and various modification or changes can be made within the scope not departing from the gist of the present invention.

All or some of the functions of the control device6serving as the manager server or the front end server are implemented by executing a certain application program (control program) through a computer (for example, a CPU, an information processing apparatus, or various kinds of terminals).

The application program is provided in the form in which the application program is recorded in a computer-readable recording medium such as a flexible disk, a CD (CD-ROM, CD-R, CD-RW, or the like), a DVD (DVD-ROM, DVD-RAM, DVD-R, DVD-RW, DVD+R, DVD+RW, or the like), or a Blu-ray disc. In this case, the computer reads the program from the recording medium, transfers the program to be stored in an internal storage device or an external storage device or, and then uses the program.

According to an embodiment, a redundancy of replicated data pieces can be restored without affecting a bandwidth of a network.