Method to use multipath to reduce IO error handle duration

A system for handling errors during interfacing with storage devices comprises a communications interface and an error handler. The error handler is for: indicating to provide a first instruction to the first storage device using the first path; determining whether an error or time out message is received; in the event the error handler determines that either message is received: indicating to provide a second instruction to the first storage device using a second path; determining whether a second error or second time out message is received; in the event the error handler determines that either message is received, indicating to provide a likely first storage device error message; and in the event the error handler determines that either message is not received, indicating to provide a likely first path error message and indicating to communicate using the second path.

BACKGROUND OF THE INVENTION

Backup servers including a plurality of data storage drives frequently encounter data communications errors when trying to access a drive. When a data communications error occurs, the drive access is retried multiple times, consuming time and adding time delays to client applications, resulting in system-wide down time. Despite the amount of time required for error retries, the error translation is not able to determine the cause of the error (e.g., whether the error is caused by a faulty drive or a faulty connection to the drive) quickly.

DETAILED DESCRIPTION

A system for handling errors during interfacing with storage devices is disclosed. The system comprises a communications interface for communicating with a first storage device using a first path or a second path. The system comprises an error handler for indicating to provide a first instruction to the first storage device using the first path. The error handler is for determining whether 1) an error message or 2) a short time out message is received in response to the first instruction. In the event the error handler determines that 1) the first error message or 2) the first short time out message is received, indicating to provide a second instruction to the first storage device using the second path and determining whether 1) the second error message or 2) the second short time out message is received in response to the second instruction. In the event that the error handler determines that the 1) second error message or 2) the second short time out message is received, indicating to provide a likely first storage device error message. In the event that the error handler determines that 1) the second error message or 2) the second short time out message is not received, indicating to provide a likely first path error message and an indicating to communicate using the second path.

In some embodiments, a system for handling errors during interfacing with storage devices comprises a storage controller connected to each of a plurality of storage devices using a first path and a second path. In some embodiments, the plurality of storage devices comprises a plurality of storage devices in a storage device rack. In some embodiments, the first path and the second path comprise alternate paths through the storage device rack. In some embodiments, each storage device includes a multiplexer to determine which of the first path and/or the second path to send and/or receive data on. In some embodiments, in the event a communications error occurs (e.g., the system for handling errors receives either a communications error message or a timeout message), communication using the alternate path is attempted. In the event that a communications error does not occur using the second path, it is determined that a first path error has occurred, and future communications are routed using the second path. In the event that a communications error occurs also using the second path, it is determined that a storage device error has occurred.

FIG. 1is a block diagram illustrating an example of an embodiment of a network system. In the example shown, administrator system102, user system104, storage server106, and application server108communicate with one another via network100. In various embodiments, network100comprises one or more of the following: a local area network, a wide area network, a wired network, a wireless network, the Internet, an intranet, a storage area network, or any other appropriate communication network. In various embodiments, administrator system102, user system104, storage server106, and application server108and subsets thereof are in physical proximity to one another (e.g., in the same building, in the same room, etc.) or are physically remote from one another (e.g., in separate buildings of a campus, in separate locations of a business, at separate offices of a collaboration between organizations, in separate countries, etc.). In various embodiments, the processors comprising administrator system102, user system104, storage server106, and application server108comprise any one of a variety of proprietary or commercially available single or multi-processor systems (e.g., an Intel-based processor) or other type of commercially available processor able to support communications in accordance with each particular embodiment and application. In various embodiments, the network system comprises any appropriate number of administrator systems (e.g., as in administrator system102), user systems (e.g., as in user system104), storage servers (e.g., as in storage server106), and application server (e.g., as in application server108).

In the example shown, administrator system102comprises a system accessed by an administrator for administrating the network system ofFIG. 1or subsets of the network system ofFIG. 1. In various embodiments, administrating the network system ofFIG. 1comprises configuring user system104(e.g., configuring application settings, network settings, security settings, etc.), configuring application server108(e.g., configuring applications, user settings, security settings, network settings, etc.), configuring storage server106(e.g., configuring storage devices, storage device mappings, storage device ports, backup schedules, security settings, etc.), configuring hardware comprising network100, or administering any other appropriate attributes of the network system ofFIG. 1. User system104comprises a system accessed by a user for accessing the network system ofFIG. 1. In various embodiments, a user accesses the network system ofFIG. 1to store data, access data, backup data, execute applications, communicate with other users, or for any other appropriate purpose. Application server108comprises an application server for providing applications for use via the network system ofFIG. 1. Storage server106comprises a storage server for storing data accessible via the network system ofFIG. 1. In some embodiments, storage server106comprises a storage server for storing backup data. In some embodiments, storage server106comprises a deduplicating storage server. In some embodiments, storage server106comprises a storage server storing files as collections of file segments. In some embodiments, storage server106comprises a storage server storing files as collections of file segments and storing a file including a collection of segments and a segment reference in place of a storing a segment that is a duplicate of a segment already stored.

FIG. 2is a block diagram illustrating an embodiment of a storage server. In some embodiments, storage server200comprises storage server106ofFIG. 1. In the example shown, storage server200comprises storage controller202communicating with a plurality of storage devices (e.g., storage device210). In various embodiments, storage server200comprises 1, 2, 4, 5, 7, 8, 22, or any other appropriate number of storage devices. In the example shown, storage controller202comprises data processor204. In various embodiments, data processor204comprises a data processor for determining data to provide to a storage device, determining commands to provide to a storage device, determining data to read from a storage device, determining a storage device to communicate with, determining a path or port to use in communicating with a storage device, or determining any other appropriate data processing function. In some embodiments, data processor204receives communications from a user system, an administrator system, an application server, a storage server, or any other appropriate system. In some embodiments, data processor204is implemented using a processor. Storage controller202comprises error handler206. In some embodiments, error handler206comprises an error handler for determining whether an error message is received. In various embodiments, an error message comprises a communications error message (e.g., an error message from a controller or interface associated with a path), a short time out message (e.g., a message indicating a communication has been provided to a device but no response has been received within an appropriate short time period, where the short time period is shorter than a long time period), a long time out message (e.g., a message indicating a communication has been provided to a device but no response has been received within an appropriate long time period, where the long time period is longer than the short time period), or any other appropriate error message. In some embodiments, error handler206is implemented using a processor. Communications interface208comprises a communications interface for communicating with the one or more storage devices. In various embodiments, communications interface208provides data to a storage device, provides commands to a storage device, receives data read from a storage device, receives error messages from a storage device, determines time out messages, or provides any other appropriate communications interface function. In some embodiments, communications interface208is implemented using a processor. In various embodiments, the elements of storage controller202are implemented each on their own processor, all on a single processor, or combined onto multiple processors in any other appropriate way.

In some embodiments, communications interface208communicates with the plurality of storage devices via port212and via port214. In some embodiments, port212and port214comprise output ports for providing communications. In some embodiments, port212comprises a controller (e.g., for controlling data output). In some embodiments, port214comprises a controller (e.g., for controlling data output). In some embodiments, communications interface208communicates with each of the plurality of data storage devices (e.g., data storage device210) via both port212and port214. In some embodiments, a communication path from a first port (e.g., port212) to a first data storage device (e.g., data storage device210) comprises a first cable. In some embodiments, a communication path from a second port (e.g., port214) to a first data storage device comprises a second cable. In some embodiments, a communication path from a first port (e.g., port212) to a first data storage device (e.g., data storage device210) comprises a first port on the first data storage device. In some embodiments, a communication path from a second port (e.g., port212) to a first data storage device (e.g., data storage device210) comprises a second port on the first data storage device. In some embodiments, port212communicates with the first port of each of the plurality of data storage devices. In some embodiments, port214communicates with the second port of each of the plurality of data storage devices. In some embodiments, in the event a first communications path between storage controller202and a storage device of the plurality of storage devices is determined to not be functioning correctly, communications can be accomplished using a second path. In some embodiments, in the event a first communications path and a second communications path between storage controller202and a storage device of the plurality of storage devices are both determined to not be functioning, a determination is made that the storage device is not functioning correctly. In some embodiments, a first path between a port of storage controller202and a storage device of the plurality of storage devices and a second path between a port of storage controller202and a storage device of the plurality of storage devices are not the same (e.g., one of the paths is favored over the other path—for example, one path is shorter). In some embodiments, each storage device of the plurality of storage devices is stored in a storage device shelf. In some embodiments, port212communicates with a shelf of storage device via its top side (e.g., accessing storage devices at the top of the shelf first) and port214communicates with a shelf of storage devices via its bottom side (e.g., accessing storage devices at the bottom of the shelf first). In some embodiments, each storage device of the plurality of storage devices comprises a multiplexer for selecting between a first port (e.g., a first port of the storage device) and a second port (e.g., a second port of the storage device).

FIG. 3is a block diagram illustrating an embodiment of a storage server. In some embodiments, storage server300ofFIG. 3comprises storage server106ofFIG. 1. In the example shown, storage server300comprises application server302and application server304. In various embodiments, application server302and application server304comprise application servers for executing data storage applications, for executing user applications, for executing backup server applications, for executing deduplication algorithms, or for executing any other appropriate application server function. In various embodiments, application server304and application server304are implemented within the computing device of storage server300, are implemented within the same hardware rack as storage server300, are implemented within the same room as storage server300, communicate with storage server300via a network, or are part of storage server300in any other appropriate configuration. Storage server300additionally comprises a plurality of storage devices (e.g., storage device306). In the example shown, application server302communicates with each of the plurality of storage devices. In some embodiments, the communication path between application server302and a storage device of the plurality of storage devices comprises a first cable and a first port on the storage device. In the example shown, application server304communicates with each of the plurality of storage devices. In some embodiments, the communication path between application server304and a storage device of the plurality of storage devices comprises a second cable and a second port on the storage device. In some embodiments, in the event communications between application server302and a storage device of the plurality of storage devices is determined to not be functioning correctly, communications can be accomplished via application server304. In some embodiments, in the event communications between both application server302and application server304and a storage device of the plurality of storage devices is determined to not be functioning correctly, a determination is made that the storage device is not functioning correctly. Storage controller308comprises a storage controller for providing storage control commands to application server302and application server304. In various embodiments, storage control commands comprise read commands, write commands, directory listing commands, backup configuration commands, network configuration commands, or any other appropriate commands.

FIG. 4is a flow diagram illustrating an embodiment of a process for handling errors during interfacing with storage devices. In some embodiments, the process ofFIG. 4is executed by storage controller202ofFIG. 2. In the example shown, in400, a first instruction is provided to a first storage device using a first path. In402, it is determined whether a first error message or a first short time out message is received. For example, in response to the first instruction, an error message or a time out message is received. In the event it is determined that a first error message or a first short time out message is not received, the process ends. In some embodiments, it is indicated to communicate using the first path. In the event it is determined that a first error message or a first short time out message is received, control passes to404. In404, a second instruction is provided to the first storage device using a second path. In406, it is determined whether a second error message or a second short time out message is received. For example, in response to the second instruction, an error message or time out message is received. In the event it is determined that a second error message or a second short time out message is received, control passes to408. In408, a likely first storage device error message is provided (e.g., indicating the error messages or short time out messages are due to a storage device error), and the process ends. In the event it is determined in406that a second error message or a second short time out message is not received, control passes to410. In410, a likely first path error message is provided (e.g., indicating the error message or short time out message is due to a first path error). In412, an indication to communicate via the second path is provided. In some embodiments, the indication to communicate via the second path is stored.

In some embodiments, the process ofFIG. 4is used by a storage server, one application server of a storage server, or any other appropriate server. In some embodiments, the process ofFIG. 4is expanded to use more than two attempts to communicate with a storage device using more than two paths. In some embodiments, in the event that each path of an attempt to communicate with a first storage device fail, then the server attempt to communicate with a second storage device.

FIG. 5is a flow diagram illustrating an embodiment of a process for handling errors during interface with storage devices. In some embodiments, the process ofFIG. 4is executed by storage controller308ofFIG. 3. In the example shown, in500, an indication is provided to a first application server to provide a first instruction to a first storage device using a first path (e.g., the path from the first application server to the first storage device). In502, it is determined whether a first error message or a first short time out message is received (e.g., in response to the first instruction). In the event it is determined that a first error message or a first short time out message is not received, the process ends. In some embodiments, it is indicated to communicate with the first storage device using the first path. In the event it is determined that a first error message or a first short time out message is received, control passes to504. In504, an indication is provided to a second application server to provide a second instruction to the first storage device using a second path (e.g., the path from the second application server to the first storage device). In506, it is determined whether a second error message or a second short time out message is received (e.g., in response to the second instruction). In the event it is determined that a second error message or a second short time out message is received, control passes to508. In508, a likely first storage device error message is received (e.g., indicating the error messages or short time out messages are due to a storage device error), and the process ends. In the event it is determined in506that a second error message or a second short time out message is not received, control passes to510. In510, a likely first path error message is provided (e.g., indicating the error message or short time out message is due to a first path error). In512, an indication to communicate via the second application server is provided. In some embodiments, the indication to communicate via the second application server is stored.

In some embodiments, the process ofFIG. 5is used by a storage server, two or more application servers of a storage server, or any other appropriate server. In some embodiments, the process ofFIG. 5is expanded to use more than two attempts to communicate with a storage device using more than two paths (e.g., providing a third instruction to the first storage device using a third path, etc.). In some embodiments, in the event that each path of an attempt to communicate with a first storage device fail, then the server attempts to communicate with a second storage device (e.g., providing a third instruction to a second storage device using a third path, etc.).