Storage device error simulator tool

The system and method provide establishment of hooks in a send-path at inter-object interfaces of a layered stack of the storage driver and hooks in the completion-path execution sequence of storage driver of a storage system, the completion-path hook inserts replacement storage response messages to simulate the specified storage access error.

BACKGROUND

A conventional storage array provides disk storage and multiple storage processors allowing access by one or more external host computers to the disk storage. The system can produce a large number of hardware and software errors. To test error handling and response in the conventional storage array an injection tool for injecting an error on the send path of an I/O request is used. The injection tool receives from an individual testing the storage array a Logical Block Address (LBA) or LBA range of addresses and an error code to be created upon access to the LBA or LBA range. The individual testing the storage array also runs a test sequence that attempts to access blocks in the LBA on the storage array. The storage array recognizing the LBA in the I/O request catches the I/O request and returns the error code back up the software stack of the I/O request. The operating software in storage array above the point where the I/O request is caught then exercises error correction routines in response to the returned error code.

SUMMARY

Unfortunately, there are deficiencies with the above-described testing of the conventional storage array. The catching of a particular I/O requests to a given LBAs or LBA ranges to produce a given error code requires extensive knowledge of the storage array and its sub-systems. Additionally, the catching of the I/O request on the send path and returning of an error code when no actual I/O is performed on the disk drives of the storage array can leave the storage array in an unknown state. The unknown state is caused by the disk or other component not being in the state that the error code returned by the I/O requests indicates. For example, a write request would update the disk and return an error. However, where the write is never applied to the LBA, the disk is not in the state that the write would place it in. Further, the storage array includes multiple storage processors. The conventional test inserts the error code in the send path of a single storage processor of storage array. Each storage processor may access the LBAs of the storage array, thus an error for a given LBA or LBA range should be reflected in both storage processors. Also, the conventional testing of the storage array does not allow for insertion of delays to simulate saturation in the storage array.

Advantageously, the improved technique provides for establishment of hooks in a send-path at inter-object interfaces of a layered stack of the storage driver and hooks in the completion-path execution sequence of storage driver of a storage system, the completion-path hook inserts replacement storage response messages to simulate the specified storage access error.

The use of hooks in the completion-path execution sequence of storage driver allows the I/O request to execute prior to modifying the result thereby leaving the disk in a consistent state. The hooks are established in a send-path at inter-object interfaces of a layered stack of the storage driver and hooks in the completion-path execution sequence of storage driver in both storage processors allowing both storage processors to be executed. Further, as the tester is not returning an error by short circuiting the send-path, the user does not need to know the relationship between interface objects in order to insert the hook to return the correct data.

One embodiment of the improved technique provides a method of testing storage software executing on one or more storage systems. The storage processor in response to input establishes send-path hooks at inter-object interfaces of a send path of a layered stack of the storage driver, the send-path hooks usable to selectively modify storage command messages. The storage processor in response to a test command input specifying a storage access error to be simulated to test storage software by selection of one or more of the send-path hooks and configuring the selected one or more of send-path hooks to monitor for a specified storage command message and insert at least one of (1) a replacement storage command message to simulate the specified storage access error and (2) a completion-path hook into a completion-path object execution sequence. The storage processor after having performed the I/O operation executes the completion-path hook in the completion-path object execution sequence, inserting a replacement storage response message to simulate the specified storage access error.

Another embodiment of the improved technique is directed to a system of testing storage software executing on one or more storage systems. The storage system includes a storage processor circuit configured and arranged to establish send-path hooks at inter-object interfaces of a send path of a layered stack of a storage driver, the send-path hooks usable to selectively modify storage command messages. The storage processor circuit also responds to a test command input that specifies a storage access error to be simulated to test storage software by selection of one or more of the send-path hooks and configures the selected one or more of send-path hooks to monitor for a specified storage command message and insert at least one of (1) a replacement storage command message to simulate the specified storage access error and (2) a completion-path hook into a completion-path object execution sequence. The storage system also executes the completion-path hook in the completion-path object execution sequence and inserts a replacement storage response message to simulate the specified storage access error.

One embodiment of the improved technique is directed to a non-transitory computer readable storage medium with a computer program stored thereon. The computer program upon execution by a processor performs the method of testing storage software executing on one or more storage systems.

DETAILED DESCRIPTION

FIG. 1is a block diagram of a data storage system100which is configured to insert hooks in a send path and execute hooks in a return path of a software stack executed to perform I/O. The storage system100includes a set of test driver101(1) and101(2), (i.e. test driver hosts101) and a communications conduit103(1) and103(2) (i.e. communication conduits103) for connecting the test drivers101to storage processors configured with test hooks105(A) and105(B) (i.e. storage processors105). The storage processors configured with test hooks105(A) and105(B) are connected to each other via bus109allowing communication there between. The storage processors105are connected to a set of disks107(1) through107(n).

The storage processors105are configured to perform data storage operations (e.g., read operations, write operations, etc.) on behalf of test drivers101. It should be understood that the test drivers101run scripts requesting I/O to a given disk drive107. The set of disk drives107enables the data storage system100to store and retrieve data on behalf of the test drivers101in a fault tolerant, non-volatile manner (e.g., using a RAID scheme).

Each storage processor105is configured using DEST115to catch data storage operations performed by the I/O software113that includes a read or write command to a given LBA on a given disk107. It should be understood that the software to operate the storage processor are stored on a non-transitory computer readable medium loaded into a memory and executed by a processor of the storage processor105. A tester inputs the tests into DEST115via a console111attached to the system100via network113. Each storage processor105has inserted in its send path, send hooks that are placed between the object interfaces in the driver code of the storage processor105. The send path hook searches for the storage command message with a given LBA for a given disk drive107. The send path hook upon catching a matching storage command message alters the storage command message to insert an error into the test command and further inserts a completion path hook into a completion-path object execution sequence. Upon the storage command message completing the completion-path hook is automatically executed and it either alters the result of the storage command message to simulate the error or inserts a delay in returning further up the layered stack of the storage driver, thereby simulating saturation of the disk drive107or other error.

For example, an individual tester enters a command to instruct the storage processors105to intercept read operations to disk drive107(1). The command also set an LBA range of 0x0 to 0x1d7FF. The tester also defines the error type, for example, SCSI, and that the error is an unrecovered read error (0x031100). The tester further defines that the number of errors to insert is 4294967294 and should be inserted every 5thI/O.

The test driver101begins a sequence of four reads of LBA 0x1-0x04 on the disk drive107(1). The storage processor has a send-path hook in the send path of a layered stack of the storage driver examine the test command input. The send-path hook determines that the LBA's of each read are in the correct range, the disk drive107(1) is the drive that is being tested and I/O type is a read. However, only the every fifth read is acted upon. As such, each of the reads of LBA 0x1-0x04 on the disk drive107(1) are allowed to complete without any changes to the storage command message or by specifying a completion-path hook to be execute on the completion-path of the test command input.

However, when the test driver begins a fifth read of LBA 0x5 on disk drive107(1), the send-path hook in the send path of a layered stack of the storage driver examines the values which match the criteria inputted by the tester. The send-path hook further determines this is the fifth read which matches all the criteria required by the tester. The send-path hook inserts a completion-path hook into a completion-path object execution sequence.

The storage processor executing the completion-path object execution sequence executes the completion-path hook. The completion-path hook inserts the error code 0x031100 into the result returned by the completion-path object execution sequence. It should be understood that software objects above the object where the error code is inserted would perform error correction and any other error handling required by the error code, thus testing the error path of the completion-path object execution sequence.

FIG. 2is a graphic illustrating a storage driver200with execution objects and inter-object interfaces between the objects with hooks. The storage driver200includes a logical data object (LDO)201, a physical data object (PDO)203a port object (P.O.)207. The send-path hook205can be placed at any interface. In the instant figure the send-path hook205is inserted between the physical data object203and the port object207. The send-path hook205could also, for example, be inserted between the logical data object201and the physical data object203.

The send-path hook205is defined by an individual tester. The send-path hook205defines under what conditions a storage command message215will be processed to simulate a drive error. The send-path hook205catches and individual storage command message215that match the criteria defined by the individual tester. The storage driver200in response to the storage command message215executes the logical data object201. As there is no send-path hook205at the interface211between the logical data object201and the physical data object203, execution proceeds to the physical data object203. The physical data object203performs its functionality and then, at the interface213between the physical data object203and the port object207, the send-path hook205executes. The send-path hook205compares the criteria defined by the individual tester. Upon the criteria matching, the send-path hook205inserts the completion-path hook into the completion-path object execution sequence or modifies the storage command message215to form a replacement storage command message217. Execution on the send-path of the storage then proceeds to the port object207. The storage driver200after executing the port object207performs the read from the disk drive107(1).

The storage driver200reads from the disk drive107(1) and returns no error. The port object207of the storage driver200returns the result to completion-path hook209. The completion-path hook209alters the result to 0x031100 and returns that value to the physical data object203. The physical data object203may have error handing functionality for dealing with the error code 0x031100 and if so the error handing functionality will be exercised for the inserted error. The physical data object203having completed execution returns the resulting error code 0x031100 to the logical data object201where error functionality therein may act on the error. The logical data object201completes execution and returns the resulting error code to the object above it (not shown) in the storage driver software stack.

FIG. 3is a graphic illustrating the transformation of a read result301of a read operation by the completion-path hook209. As discussed in the example above, a fifth read of disk drive107(1) to LBA 0x5 is intercepted by a send-path hook which inserts the completion-path hook209in completion-path object execution sequence. The read result301for disk107(1) targeting LBA 0x5 has a return code of 0x0. The return code of 0x0 indicates that the read was successful. However, the completion path hook209executed in completion-path object execution sequence modifies the return code. Thus, the modified read result303has the value 0x031100 indicated an unrecovered read error. The modified read result is returned to the objects above the completion-path hook in completion-path object execution sequence.

FIG. 4is a graphic illustrating a completion-path execution sequence with and without a completion-path hook. A first completion-path execution sequence401is the completion-path execution sequence where no completion-path hook is inserted by the send-path hook205. The first completion-path execution sequence401contains a return path executing the return sequence of port object (PO)409, then a physical data object (PDO)407and then a logical data object (LDO)405.

A second completion-path execution sequence403illustrates the send-path hook205inserting the completion-path hook411into the completion-path execution sequence such that after the port object409executes, the completion-path hook411executes prior to the physical data object407when returning from a read transaction.

FIGS. 5A and 5Billustrate a command line interface500of a drive error simulation tool. In particularFIG. 4Aincludes the main command line interface commands. The storage system provides a command line interface500for inserting the criteria required for catching storage command message215when testing the storage system100. The command line interface500provides a “dest-h” command501entered on a command line of the DEST115for displaying all the commands the command line interface500can accept. The individual performing the test may setup test criteria by entering “dest-init” command”503to have DEST115load an extensible markup language file (XML) defining the criteria to be used to test a disk drive107. It should be understood that the XML file contains the required fields and any optional fields required to set up the criteria for a given test that will invoke the send-path hook205.

The command line interface500also provides a “dest-add” command505for adding individual records to DEST115. It should be understood that when entering the “dest-add” command505, the individual will also enter at least the required parameters for any given test record.

The drive error simulation tool (DEST)115also provides for adding a predefined scenario by entering “dest-add_scenario”507for a named disk drive107. For example, default scenarios are SLOW_IO for slowing a disk drive107's response time to reads and writes and RANDOM_MEDIA_ERROR for injecting a random media error to the named disk drive107at random intervals when the hooks are executed. It should be understood that disk drives are identified in the form of bus/enclosure/slot. The command line interface500also provides for defining the point that errors start being injected using “dest-start”509and when errors are stopped being injected using “dest-stop”511.

The command line interface500also provides commands for listing port errors that can be injected using the “dest-list_port_errors” command513; listing opcodes using “dest-list_opcodes”515that are searched for by the send-path hook205, for example, READ or WRITE operations; or scenarios using the “dest-list_scenerios” command517as set by the “dest-add_scenarios” command507.

The command line interface500also provides for displaying active test records using the “dest-display” command519.

The command line interface500also provides for saving any input record in the form of an XML file using the “dest-save” command521.

FIG. 4Bin particular list the parameters/arguments that the “dest-add” command505uses. The “dest-add” command505requires the naming of disk drive107to be tested. Thus, the command further includes “-et<type>”527for the error type to be tested. For example, the error type could be one of a SCSI error529or a port error537. When the tester defines the error type to be SCSI529, they must further define on the command line they type of sense key, “-sk<SK/ASQ/ASCQ>”531). The tester may optionally enter at the command line that the error should be invalid send data by entering the argument “-ilba”533. If the tester wishes to set the deferred bit in sense data, he or she enters a parameter of “-def”535. If the error type inputted on the command line500is “port”535, then a required parameter defining the port error “-perr”539is also entered.

As discussed in our example above, an LBA is normally defined. The command line interface includes an input for an LBA or LBA by inputting “-lr<start><end>”541. It should be understood, that an LBA is not defined at the command line interface500then the default is to test all LBAs on the disk drive107. A parameter for entering a delay in milliseconds is included by inputting “-delay<msec>”543on the command lane. Likewise, the opcode to be tested is entered as the parameter “-oc<value>”545.

The command line interface includes an input for defining the number of errors to be injected in any one test sequence by adding parameter “-num<value>”547on the command line. Optionally, the number of errors inserted can also define the frequency in which the errors are inserted during the test by inputting “-freq<N>”549or a semi random frequency by inputting the parameter “-rfreq<N>”. The insertions may also have a reactivation gap that is added by time between I/Os by inputting “-react_gap<type><value>”553on the command line or pseudo randomly based on a count by inputting “-react_rgap<type><value>”555on the command line. The tester can further define the number of reactivations by entering the parameter “-n_react<N>”557on the command line or the number of pseudo random reactivations (559) by entering the parameter “-n_rreact<N>”559on the command line. It should be understood by entering the commands through the workstation111hooks are placed in the inter object interfaces211and213of the storage driver200and catch and modify test commands exercising the disk107.

FIG. 6is a flowchart illustrating a method of storage software executing on one or more storage systems100. The storage system100establishes send-path hooks205at inter-object interfaces211and213of a send path of a layered stack of a storage driver200, the send-path hooks205usable to selectively modify storage command messages300. The storage system100in response to a test command input specifies a storage access error to be simulated to test storage software, selects one or more of the send-path hooks205and configuring the selected one or more of send-path hooks205to monitor for a specified storage command message215and insert at least one of (1) a replacement storage command message217to simulate the specified storage access error and (2) a completion-path hook209into a completion-path object execution sequence403. The storage system100further executes the completion-path hook209in the completion-path object execution sequence403, inserting a replacement storage response message303to simulate the specified storage access error.

While various embodiments of the present disclosure have been particularly shown and described, it will be understood by those skilled in the art that various changes in the form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims.

For example, the storage processor105include a processor and memory for loading a computer program stored on non-transitory computer readable storage medium such as the disk107. It should be understood that a system including circuits is formed by the execution of the computer program in the processor at any given point in time during the execution of the computer program by the processor.