Apparatus and methods for CRC error injection in a storage system

Apparatus and methods for Cyclic Redundancy Check (CRC) error injection between storage controllers and storage devices in a storage system. A plurality of bridge devices are configured in a storage system each coupled persistently coupled to a corresponding one of the plurality of storage devices. Each bridge device may couple to one or more Serial Attached SCSI (SAS) initiators for transferring exchanges between one or more SAS initiators and the attached target storage device. Each bridge device receives parameters from a SAS initiator or an administrative client directing the bridge regarding injection of CRC errors. A log memory in each bridge may log information regarding the injected CRC errors.

RELATED PATENTS

This patent is related to commonly owned U.S. patent application Ser. No. 11/644,549 entitled “Serial Advanced Technology Attachment (SATA) and Serial Advanced Small Computer System Interface (SCSI) (SAS) Bridging” which is hereby incorporated by reference. This patent is also related to commonly owned U.S. patent application Ser. Nos. 12/138,315 and 12/138,309 filed herewith which are also hereby incorporated by reference.

BACKGROUND

1. Field of the Invention

The invention relates generally to testing in storage system and more specifically relates to improvements in injection of CRC errors in communications with storage devices in a storage system for testing error recovery in the storage system.

2. Discussion of Related Art

Storage systems typically comprise a storage controller coupled to one or more storage devices. In large scale storage systems, multiple controllers and a large number of storage devices (e.g., disk drives) are typically housed in an enclosure. One or more host systems are coupled to one or more of the storage controller through storage networking protocols and media. The host systems apply I/O requests to the storage system through the storage controllers which, in turn, apply appropriate I/O operations to one or more of the storage devices within the storage system to write data or to retrieve previously written data.

The storage controllers are adapted to detect errors in the communications with the storage devices and to perform various types of error recovery processing to attempt to recover from the various types of error conditions. One common type of error condition is a Cyclic Redundancy Check (CRC) error detected in exchanges between the storage controller(s) and one or more storage devices. A CRC error represents detection of a bit (or multi-bit) error in the communication link between the storage controller(s) and a storage device. A CRC code is computed and transmitted with data in such an exchange. The receiving device computes its own CRC code based on the data as received. The receiving device compares its computed CRC code with the received CRC code to detect an error in transmission/reception of the data. CRC errors may arise in operational storage system due to electromagnetic noise or other environmental aspects of the operating storage system.

Design engineers and/or field support engineers dealing with storage systems often need to test the ability of elements of the storage system to properly recover from CRC errors. Design engineers may wish to test the design of their storage controller or storage device to verify proper detection and recovery from CRC errors. In like manner, field support engineers may wish to test CRC error recovery processing of a storage controller or a storage device to isolate a fault detected in a field installation of a storage system.

A common prior technique for such testing involves inserting a “jammer” device in the communication link between the storage controller(s) and a storage device. The jammer device is physically and electronically inserted between the two components and controllably injects bit errors in the exchanges between the controller(s) and the storage device. These injected bit errors will cause a CRC error to arise in the exchanges between the controller(s) and the storage device and thus enable the engineer to evaluate or debug recovery processing from CRC errors. Jammer devices for injecting CRC errors are well known and widely available for insertion into any of several widely used communication media and protocols.

Ad hoc insertion of a jammer device to test CRC error recovery gives rise to a number of problems. Insertion of a jammer in a storage system may cause physical/mechanical problems in that the jammer may not physically fit in the nominal mounting structure of the storage device to which it is to be attached. For example, in the context of larger storage systems, storage devices (e.g., disk drives) are typically mounted into a tray or carrier so that they may be readily inserted and removed from the storage system enclosures (e.g., for “hot swap” functionality). Further, electronic insertion of the jammer into the nominal connection between the storage controller(s) and the storage device alters the electronic characteristics of the coupling such that signal timings may change and other unintended errors may be introduced thereby. Still further, to test each storage device in a large storage system, a jammer would have to be inserted for each drive to be tested. Thus one jammer would have to be inserted, removed, and re-inserted numerous times to test each of a large number of drives or a large number of jammer devices would have to be provided at significant cost. Numerous other problems arise in use of such jammer devices in that they have little flexibility to alter the style of testing or to adapt to the specific testing needs of a particular application.

Thus it is an ongoing challenge to flexibly and effectively test CRC error recovery in storage systems.

SUMMARY

The present invention solves the above and other problems, thereby advancing the state of the useful arts, by providing methods and apparatus for improved testing of CRC error recovery in storage systems. Features and aspects hereof provide for a bridge device that couples one or more Serial Attached SCSI (SAS) initiators to a storage device. The storage device may be a Serial Advanced Technology Attachment (SATA) storage device or any other non-SAS attached storage device. The bridge device is coupled to the storage device in its nominal mechanical and electrical coupling to the storage controller and therefore does not alter the physical mounting of the storage device in the storage system or the electrical characteristics of the coupling between the storage controller(s) and the storage device. The bridge device is enhanced in accordance with features and aspects hereof to controllably enabling injection of CRC errors in the communication path between the storage controller(s) and the storage device. The enhanced bridge device may provide programmable options regarding the specific CRC error to be generated as well as the number and frequency of such CRC errors to be injected. Still further, the enhanced bridge device may maintain log information regarding the injected errors and the recovery processing of the errors. Still further, the enhanced bridge device may provide such CRC error injection on behalf of any of multiple SAS initiators (e.g., multiple storage controllers) coupled through the bridge to the storage device.

In one aspect hereof, a communication bridge is provided. The bridge includes a Serial Attached SCSI (SAS) port for coupling to a SAS initiator device and a second port, coupled to the SAS port, for coupling to a target device. The SAS initiator device and the target device exchange information through the bridge. The bridge also includes a Cyclic Redundancy Check (CRC) error injector adapted to controllably inject a CRC error in an exchange between the target device and the SAS initiator device. In one aspect, the bridge may be further adapted to receive parameters from the SAS initiator as a SCSI Mode Page containing Vendor Unique fields and/or non-vendor unique fields directing the bridge regarding injection of CRC errors. In another aspect, the bridge includes a maintenance port for coupling to an administrative client and receives parameters from the administrative client directing the bridge regarding injection of CRC errors. In still another aspect, the bridge is adapted to receive parameters from the SAS initiator as an out of band communication directing the bridge regarding injection of CRC errors.

Another aspect hereof provides a storage system including a plurality of SAS initiators and a plurality of target storage devices. Further the system includes a plurality of bridge devices each coupled to a corresponding one of the plurality for target storage devices and each coupled to one or more of the plurality of SAS initiators to enable exchanges between the one or more SAS initiators and the corresponding target storage device. Each bridge device further comprises a Cyclic Redundancy Check error injector for controllably injecting CRC errors in exchanges between its corresponding target storage device and the one or more SAS initiators coupled to the bridge device. Each bridge device may be adapted to receive parameters from the one or more SAS initiators as a SCSI Mode Page containing Vendor Unique fields and/or non-vendor unique fields directing the bridge device regarding injection of CRC errors. In another aspect each bridge device further comprises a maintenance port for coupling to an administrative client. Each bridge device is then adapted to receive parameters from the administrative client directing the bridge device regarding injection of CRC errors. In still another aspect, each bridge device is adapted to receive parameters from the one or more SAS initiators as out of band communications directing the bridge regarding injection of CRC errors. In yet another aspect, each bridge device further comprises a log memory adapted to store log information relating to injected CRC errors.

DETAILED DESCRIPTION OF THE DRAWINGS

FIG. 1is a block diagram of a communication bridge100enhanced in accordance with features and aspects hereof to provide automated CRC error injection into the data exchanges, in either direction, between an initiator device114and a target device118each coupled to the bridge. In one exemplary embodiment, communication bridge100serves as a bridge for communication between one or more SAS initiator devices114coupled to the bridge via path152and a corresponding, non-SAS target device118coupled to the bridge via path158. Communication bridge100may be persistently coupled to target device118so as to permit mechanical and electrical coupling within a larger storage enclosure utilizing standard mounting and electronic coupling features within the enclosure. By contrast to prior jammer technologies, communication bridge100may be a persistent component within a storage system and thus persistently coupled with an associated target device118.

Communication bridge100includes one or more CRC error injectors110positioned intermediate paths152and158to inject CRC errors on various of the communication paths through the bridge between initiator114and target device118. Communication bridge100includes SAS port102for coupling to one or more SAS initiator devices114via paths152and153through CRC error injector110. Those of ordinary skill in the art will readily recognize that path152may be a direct SAS connection or may be a more flexible connection through a SAS fabric of a SAS domain. Such a fabric may include, for example, any number of SAS expanders to provide flexible, selectable connectivity between any of the initiators and any of the targets. SAS port102therefore provides a mechanism for exchanges between bridge100via path152and one or more SAS initiator devices114.

Communication bridge100also includes a second port as target device port106for coupling a target device118to the bridge via paths158and156through CRC error injector110. In one exemplary embodiment, target device port106may be a SATA port for coupling a SATA target device to the bridge. Those of ordinary skill in the art will recognize numerous protocols and communication media other than SAS that may be applied in coupling the bridge communication device100to a target device118. For example, Parallel ATA (PATA), Universal Serial Bus (USB), and numerous other potential attachments for supporting target devices may be utilized.

Such SAS/SATA exchanges in either direction are converted by SAS port102and target device port106to an appropriate protocol and media for eventual transfer through bridge100to target device118or to the initiator device114.

CRC error injectors110of communication bridge100enhance operation of the bridge100by providing controllable injection of a CRC error conditions. Control logic within CRC error injectors110responds to external signals (e.g., signals applied to path150from SAS port102labeled “A”) to enable injection of CRC errors in accordance with particular requested parameters from, for example, SAS port102. Other control signals (or lack of signals) on path150causes all data transmissions to pass through CRC error injectors110directly between target device port106and the target device118without any alteration. In particular, SAS port102may detect information transmitted over the SAS communication media152from SAS initiator devices114requesting that one or more of the CRC error injectors110begin injecting specified CRC errors. Such a transmission may be, for example, a SCSI Mode Page command including information in Vendor Unique fields and/or non-Vendor Unique fields of such a command. Alternatively, SAS initiator devices114may forward out of band communications over SAS link152that may be detected by SAS port102for generation of an appropriate signal on path150to cause one or more CRC error injectors110to operate as desired. Parameters of the injection may include, for example, random error injection, injection of altered CRC codes in a transmission, injection of errors in particular types of transmission, injection of errors at particular positions of data in transmissions, etc.

As shown inFIG. 1bridge100may include one or more CRC error injectors positioned in different portions of the paths in bridge100between path152and path158. For example, a first CRC error injector110may be positioned between paths152and153to inject errors in SAS transmissions in either direction between the SAS port102and the initiator devices114. A second CRC error injector110may be positioned between paths154and155to inject CRC errors in transmissions in either direction between SAS port102and target device port106. Yet another CRC error injector110may be positioned between paths156and158to inject CRC errors in transmissions in either direction between target device118and target device port106. Each such CRC error injector is coupled to path150to receive parameters for errors to be injected and each injector110may be individually controlled by such parameters. Further, those of ordinary skill in the art will recognize that fewer may be configured in bridge100or additional injectors may be added to bridge100.

FIG. 2is a block diagram of another exemplary embodiment of features and aspects hereof comprising a communication bridge200similar in many respects to communication bridge100ofFIG. 1. A plurality of SAS initiator devices114are each coupled to communication bridge200via corresponding communication media152through a corresponding plurality of SAS ports102in communication bridge200. Communication bridge200includes multiple SAS ports102each adapted for coupling to a specific SAS initiator device114or adapted to couple through a fabric152to any of several SAS initiator devices114. Bridge control logic (not shown but discussed in the related patents) controls multiplexor104to select among the various SAS ports102based on criteria discussed in the bridge patents noted above and incorporated herein by reference. Multiplexer104may controllably select one or another of the multiple initiators (114) for coupling with a SATA target device118coupled to communication bridge200through path158. Thus a selected one of the plurality of SAS initiators114may be coupled through communication bridge200to the SATA target device port106for exchanges with the SATA target device118.

As above inFIG. 1, a first CRC error injector110of communication bridge200is electronically coupled between target device port106and SATA target device118coupled thereto (i.e., between paths156and158). Based on control signals and other information applied to CRC error injector110via path150, data communications may be passed unchanged from target device port106to target device118or, in the alternative, CRC errors may be injected into the information to be exchanged between target device port106and target device118. As noted above, CRC error injector110may receive control signals via path150(label “A”) from any of SAS ports102requesting particular parameters for the injection of CRC errors. Exemplary of the particular parameters of the injection may include, for example, random error injection, injection of altered CRC codes in a transmission, injection of errors in particular types of transmission, injection of errors at particular positions of data in transmissions, etc.

In accordance with features and aspects hereof, communication bridge200may include an optional maintenance port108coupled to receive parameters from an administrative client116via path254indicating the particular parameters to be applied for testing the standard recovery error processing. Administrative client116may be any process operable on any computing device coupled to the bridge200via path254and optional maintenance port108. Optional maintenance port108then communicates received parameters via path150to CRC error injector110.

In addition, bridge200may include additional CRC error injectors in other paths within the bridge to permit controlled CRC error injection in other paths. For example, additional CRC error injectors110are positioned between each SAS port102and Mux204to permit injection of errors in the SAS communications applied through those intermediate paths of bridge200. As above, and as discussed with respect toFIG. 1each CRC error injector110may receive signals via path150to provide parameters of the CRC errors to be injected. Also, as shown inFIG. 1though not shown inFIG. 2for simplicity, still more CRC error injectors may be interposed between each SAS port102and the communication medium152coupled thereto to permit injection of CRC errors in the SAS communications with the initiators114. As noted inFIG. 1those of ordinary skill in the art will recognize that any number of CRC error injectors my be configured in the bridge200to provide further flexibility in determining where errors are to be injected in communications through bridge200.

FIG. 3is a block diagram of an exemplary storage system300in accordance with features and aspects hereof. System300integrates multiple SAS initiators302(e.g., storage controllers) with multiple target storage devices306(e.g., disk drives). Each target storage device306is persistently coupled to a bridge device310(e.g., such as bridge circuits100or200ofFIGS. 1 and 2, respectively). Each bridge device310is persistently coupled to (e.g., integral with) a corresponding target storage device306so that the bridge device310may be persistently positioned mechanically and electronically within a storage system enclosure thus alleviating problems of prior techniques with ad hoc insertion of a jammer device. Each bridge device310couples its corresponding target storage device306to a SAS domain fabric360within storage system300. In addition, each of the plurality of SAS initiators302is also coupled to SAS domain fabric360to permit coupling of any of the plurality of SAS initiators302to any of the plurality of targets storage devices306each persistently coupled to a corresponding bridge device310. Host systems308are coupled to storage system300through any suitable communication medium such as Ethernet, parallel SCSI, SAS, Fibre Channel, etc.

As discussed above, each bridge device310includes appropriate control logic to permit transmissions to pass directly through the bridge between a SAS initiator308and a corresponding target storage device306. When appropriately controlled, one or more bridge devices310begin inserting/injecting CRC errors into communications along their corresponding paths. The CRC errors are injected in accordance with parameters of a request from the corresponding SAS initiator or, as noted above, from an administrative client coupled through a maintenance port of the bridge device.

FIG. 4is a block diagram providing additional exemplary details of the structure of a CRC error injector110such as discussed above with respect toFIGS. 1 and 2. CRC error injector110receives transmissions from a frame source destined for a frame destination. As noted above, CRC error injectors110ofFIGS. 1 and 2operate for transmission in either direction. The exemplary details ofFIG. 4suggest one direction of transmissions for which errors may be injected. The exemplary circuitry ofFIG. 4may thus be replicated and/or controllably switched to provide for CRC error injection in the opposite direction of data transmissions. Such replication or switching would be readily apparent to those of ordinary skill in the art and thus such details are omitted herein for simplicity and brevity of this discussion.

When appropriately controlled, multiplexer406passes such transmissions directly through from the frame source to the frame destination. Control logic402generates an appropriate selection signal for multiplexer406to either pass through the data transmissions between the source and destination unaltered or to inject a CRC error generated by CRC error generator404. CRC error generator404receives the data transmission from the frame source and alters the received information in accordance with control parameters received from control logic402on path454. The altered information is then forwarded to the intended destination through multiplexer406.

Parameters for injection of the CRC errors are received on path150as discussed above from a SAS port of the bridge and/or from a maintenance port within the bridge. These error parameters may be stored in CRC error injector110in a suitable memory408. CRC error injector110may also include a CRC error log memory410coupled to control logic402for logging information regarding generation and transmission of the various CRC errors. CRC error log memory410may be any suitable memory including flash memory and other non-volatile or volatile memory components. A SAS initiator or an administrative client coupled through a maintenance port of the bridge may request return of the logged information through control port150. Logging of information may use well known circular buffer techniques to store more recent information as older information is discarded. In the alternative, a fixed maximum size of logged information may be stored such that when the log memory410is full, logging of further error information will cease until the contents are fetched and cleared. Similar communication techniques to those described above to control the error injection may be employed to fetch and/or clear the log memory410content (e.g., out of band communications, vendor unique and non-vendor unique fields in SAS commands/responses, maintenance port communications, etc.).

Those of ordinary skill in the art will readily recognized numerous additional and equivalent elements present in fully functional circuits as generally depicted inFIGS. 1 through 4. Such additional and equivalent elements are omitted herein for simplicity and brevity of this discussion. Further, those of ordinary skill in the art will recognize that each CRC error injector, positioned in any communication path of the bridge, may receive parameters to inject CRC errors from any signal source coupled to its control signal path. Thus a CRC error injector may receive parameters to initiate CRC error injection from any SAS initiator coupled to the bridge on any SAS port. For example, a CRC error injector coupled in a path of the bridge associated with a first SAS port (and/or a first SAS initiator) may receive parameters to start or stop error injection from that first SAS port/initiator or from any second/other SAS port of the bridge associated with any other SAS initiator. Further, for example, through the maintenance port of the bridge, the administrative client may provide parameters to start or stop error injection on any of the CRC error injectors of the bridge.

FIG. 5is a flowchart describing an exemplary method in accordance with features and aspects hereof for providing CRC error injection capability in a bridge device coupling multiple SAS initiators to a non-SAS target device. Step500represents processing to receive parameters to begin injection of CRC errors in the exchanges between a presently selected one of the plurality of SAS initiators and the target device associated with the bridge device (e.g., persistently coupled to the bridge device). As noted, the received parameters may be provided by an administrative client coupled to the bridge through a maintenance port or from any of the SAS ports (coupled to any of the SAS initiators using the bridge). The parameters may specify, for example, the bit position of desired CRC error to be injected in a buffer or frame as well as the frequency or other timing aspects of the CRC error to be injected. Step502then injects a CRC error in the next transmission between the identified SAS initiator and the target device in accordance with the parameters provided by the request received a step500. Step504then optionally logs information related to the injected error in a log memory associated with the CRC error injection of the bridge device. Step506then determines whether the parameters received at step500require ongoing further errors be injected in the same buffer/frame or in subsequent buffers/frames. If additional errors remain to be injected in the data stream, processing continues looping back to step502. Otherwise, processing continues at step508to discontinue injection of the CRC errors and to continue normal processing of data exchanges between each of the SAS initiators and the target device.

While the invention has been illustrated and described in the drawings and foregoing description, such illustration and description is to be considered as exemplary and not restrictive in character. One embodiment of the invention and minor variants thereof have been shown and described. Protection is desired for all changes and modifications that come within the spirit of the invention. Those skilled in the art will appreciate variations of the above-described embodiments that fall within the scope of the invention. As a result, the invention is not limited to the specific examples and illustrations discussed above, but only by the following claims and their equivalents.