Abstract:
An apparatus including a storage array, a primary controller, a secondary controller and a solid state device. The storage array may be configured to be accessed by a plurality of controllers. A first of the plurality of the controllers may be configured as the primary controller configured to read and write to and from the storage array during a normal condition. A second of the plurality of the controllers may be configured as the secondary controller configured to read and write to and from the storage array during a fault condition. The solid state device may be configured to (i) store data and (ii) be accessed by the storage array and the secondary controller.

Description:
FIELD OF THE INVENTION 
       [0001]    The present invention relates to storage arrays generally and, more particularly, to a method and/or apparatus for responding to handling interrupted writes using multiple cores. 
       BACKGROUND OF THE INVENTION 
       [0002]    Conventional systems address potential double fault conditions in a variety of ways. A conventional controller enters an NVSRAM interrupted write mode condition and the owning controller is rebooted by the test. Due to the controller reboot (i) the controller firmware will regenerate parity for the data stripe involved in a write, and (ii) a forced transfer to the surviving controller takes place. 
         [0003]    The two conditions above cause the next N writes to be implemented using old/new data to generate a new parity bit. While performing the previous tasks associated with the next N write cycles, a data drive can fail unexpectedly in a volume group before the host retries the write. In such a condition, the controller does not know whether the write completed to the data drive and/or parity drive. If the write completes to the data drive, but does not complete to the parity drive, or vice versa, a potential data corruption will be detected due to the inconsistency between data and parity. 
         [0004]    It would be desirable to implement a method and/or apparatus for handling interrupted writes using multiple cores that avoids data corruption. 
       SUMMARY OF THE INVENTION 
       [0005]    The present invention concerns an apparatus comprising a storage array, a primary controller, a secondary controller and a solid state device. The storage array may be configured to be accessed by a plurality of controllers. A first of the plurality of the controllers may be configured as the primary controller configured to read and write to and from the storage array during a normal condition. A second of the plurality of the controllers may be configured as the secondary controller configured to read and write to and from the storage array during a fault condition. The solid state device may be configured to (i) store data and (ii) be accessed by the storage array and the secondary controller. 
         [0006]    The objects, features and advantages of the present invention include providing a method and/or apparatus that may (i) have multiple cores to handle  10  processing, (ii) have certain cores handle reconstruction and IC write process, (iii) have certain cores handle  10  read process and stripe-set preservation (e.g., SSP and/or previous state preservation), (iv) have a nonvolatile RAM (e.g., a solid state drive) to store the data for SSP, (v) prevent data corruption when double faults occur, and/or (vi) provide performance enhancement with multiple cores handling next N writes by reading all stripes to generate new parity. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0007]    These and other objects, features and advantages of the Present invention will be apparent from the following detailed description and the appended claims and drawings in which: 
           [0008]      FIG. 1  is a flow diagram illustrating a double fault condition; 
           [0009]      FIG. 2  is a block diagram of a double fault condition; 
           [0010]      FIG. 3  is a conceptual diagram of a double fault condition; 
           [0011]      FIG. 4  is a block diagram of an example embodiment of the present invention; 
           [0012]      FIG. 5  is a flow diagram illustrating an example embodiment of the present invention; and 
           [0013]      FIG. 6  is a conceptual diagram of an example embodiment of the present invention. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       [0014]    Referring to  FIG. 1 , a flow diagram of a process  50  is shown illustrating a double fault condition. The process  50  generally comprises a state  52 , a state  54 , a state  56 , a state  58 , a state  60 , and a state  62 . In the state  52 , a controller enters an NVSRAM interrupted write mode. In the state  54 , the controller reboots. In the state  56 , a LUN undergoing an input/output (IO) is forced to transfer to an alternate controller. In the state  58 , subsequent write operations have old/new data in the stripe set to regenerate a new parity. In the state  60 , one of the physical drives under the LUN 0  fails before completing the interrupted write. In the state  62 , a potential data corruption state occurs. 
         [0015]    Referring to  FIG. 2 , a block diagram of a system  70  is shown illustrating a double fault condition. The system  70  generally comprises a block  72 , a block  74 , a block  76 , and a block  78 . The block  72  may be implemented as a controller (e.g., controller A). The block  74  may be implemented as a controller (e.g., controller B). The block  76  may be implemented as a LUN 0 . The block  78  may be implemented as a drive array. The system  70  may operate in a number of states as shown in the following TABLE 1: 
         [0000]    
       
         
               
               
             
               
               
             
           
               
                 TABLE 1 
               
               
                   
               
               
                 STATE 
                 ACTION 
               
               
                   
               
             
             
               
                   
               
             
          
           
               
                 1 
                 Controller A and Controller B 
               
               
                 2 
                 LUN0 created on top of physical drives 
               
               
                 3 
                 Stripe Set A used to generate parity PA 
               
               
                 4 
                 State 3 located on the Stripe Set A location of 
               
               
                   
                 LUN0 
               
               
                 5 
                 Write IO sent to LUN0 
               
               
                 6 
                 DOA′ written to P1, Controller A reboots 
               
               
                 7 
                 Forced transfer of LUN0 to Controller B 
               
               
                 8 
                 Pending write and the next “n” writes involve 
               
               
                   
                 generation of parity PA′ by reading all strip 
               
               
                   
                 segments 
               
               
                 9 
                 Controller B regenerates parity PA′ for stripe set 
               
               
                   
                 A having an interrupted write mode 
               
               
                 10 
                 Controller B reads new data and old data to 
               
               
                   
                 generate parity PA′ 
               
               
                 11 
                 P2 fails during step 10 
               
               
                 12 
                 If state 11 occurs, regeneration of data for P2 
               
               
                   
                 results in wrong data 
               
               
                 13 
                 Data corruption for next “x” writes due to state 
               
               
                   
                 11 
               
               
                   
               
             
          
         
       
     
         [0016]    In the state 1 of TABLE 1, the controller A is implemented as an owning controller for the LUN 0   76 . The controller B is implemented as an alternate controller. The controller B is in a passive state and implemented for redundancy. In the state 2 of TABLE 1, the LUN 0   76  is created on top of the drive array  78 . In one example, the drive array  78  is implemented as four physical drives (or disks) (e.g., P1, P2, P3, P4). However, the particular number of drives may be varied to meet the design criteria of a particular implementation. In the state 3 of TABLE 1, a number of data segments D 0 A, D 1 A, D 2 A are implemented in a stripe set (e.g., A) which is used to generate a parity (e.g., PA). In the state 4 of TABLE 1, the state 3 is located on the stripe set A location of the LUN 0   76  created in state 2. In the state 5 of TABLE 1, the controller A sends a write IO (e.g., the data segment D 0 A′) to the LUN 0   76 . In the state 6 of TABLE 1, the data D 0 A′ is written in a first disk (e.g., P1) of the drive array  78 . Before regenerating and writing a parity (e.g., PA′) to a fourth disk (e.g., P4), the controller A reboots. In the state 7 of TABLE 1, a forced transfer of the LUN 0   76  to the controller B (e.g., the alternate controller) happens. In the state 8 of TABLE 1, the action of the state 7 triggers the pending write and the next “N” writes to involve generation of the parity PA′ by reading all data segments (whether the LUN 0   76  has old/new data). 
         [0017]    In one example, a scenario may be considered where the controller B tries to regenerate the parity PA′ for the stripe set A, which has an interrupted write mode (e.g., the state 9). In the state 10 of TABLE 1, the controller B reads new data (e.g., D 0 A′ from the first disk), old data (e.g., D 1 A from the second disk) and old data (e.g., D 2 A from the third disk) to generate the parity PA′ as new parity in place of the parity PA (from the fourth disk). If the second disk fails during the state 10, one new data and old parity results (the state 11 of TABLE 1). In the state 11 scenario (e.g., the second disk fails), regenerating data for the second disk will result in wrong data since one new data and the parity PA that was not generated using the new data stripes is used. In the State 13 of TABLE 1, data corruption occurs for the next ‘x’ writes due to the second disk failing. 
         [0018]    Referring to  FIG. 3 , a conceptual diagram of a double fault condition of the system  70  is shown. In the STATE 1, the stripe set A (e.g., D 0 A, D 1 A, D 2 A, and PA) is stored in the physical drives (e.g., P1, P2, P3, and P4) of the drive array  78 . In the STATE 2, a write IO is sent for a new data (e.g., D 0 A′) to be written to the first disk (e.g., P1). In the STATE 3, the write DO is completed for the data D 0 A′. After the write IO has completed in the STATE 3, an interrupted write mode occurs (e.g., a first fault) without completing the write to the second disk (e.g., P2). In the STATE 4, while trying to generate the new parity PA′, the second disk (e.g., P2) fails (e.g., a second fault) resulting in the wrong generation of D 1 A′ using the parity PA with the data D 0 A′ and the data D 2 A. The system  70  may experience both the first fault and the second fault (e.g., the double fault condition). 
         [0019]    Referring to  FIG. 4 , a block diagram of the system  100  is shown. The system generally comprises a module  102 , a module  104 , a module  106 , a module  108 , and a module  110 . The module  102  may be implemented as a controller (e.g., controller A). The module  104  may be implemented as a controller (e.g., controller B). The module  106  may be implemented as a LUN 0 . The module  108  may be implemented as a storage array. For example, the module  108  may represent an array of disk drives or other storage devices (e.g., solid state storage, etc.). The modules  102 ,  104  and  106  may be implemented as hardware, software, a combination of hardware and software, or other implementations. The module  110  may be implemented as a storage device. In one example, the storage device  110  may be implemented as a solid state drive (or device). The system  100  may be implemented in a number of states as shown in the following TABLE 2: 
         [0000]    
       
         
               
               
             
               
               
             
           
               
                 TABLE 2 
               
               
                   
               
               
                 STATE 
                 ACTION 
               
               
                   
               
             
             
               
                   
               
             
          
           
               
                 1 
                 Controller A and Controller B 
               
               
                 2 
                 LUN0 created on top of physical drives 
               
               
                 3 
                 Stripe Set A used to generate parity PA 
               
               
                 4 
                 State 3 located on the stripe set A location of 
               
               
                   
                 LUN0 
               
               
                 5 
                 Write IO sent to LUN0 
               
               
                 6 
                 Stripe-set State Preservation of data 
               
               
                 7 
                 DOA′ written to P1, Controller A reboots 
               
               
                 8 
                 Forced transfer of LUN0 to Controller B 
               
               
                 9 
                 Pending write and the next “n” writes involve 
               
               
                   
                 generation of parity PA′ by reading all strip 
               
               
                   
                 segments 
               
               
                 10 
                 Controller B regenerates parity PA′ for stripe set 
               
               
                   
                 A having an interrupted write mode 
               
               
                 11 
                 Controller B reads new data and old data to 
               
               
                   
                 generate parity PA′ 
               
               
                 12 
                 P2 fails during step 11 
               
               
                 13 
                 If state 12 occurs, D1A generated/read from SSP 
               
               
                 14 
                 Data stored in state 6 is erased 
               
               
                   
               
             
          
         
       
     
         [0020]    In the state 1 of TABLE 2, the controller A may be implemented as an owning controller for the LUN 0   106  (e.g., during a normal condition). The controller B may be implemented as an alternate controller. The controller B may be in a passive state and may be implemented for redundancy. In the state 2 of TABLE 2, the LUN 0   106  may be created on top of the drive array  108 . IO requests are normally sent to the LUN 0   106 , which translates such requests to the storage devices in the storage array  108 . While one LUN 0  is shown, a number of LUNs may be implemented (e.g., up to 2048 or more) in a particular design. In the example shown, the storage array  108  may be implemented as four physical drives (or disks) (e.g., P1, P2, P3, P4). However, the particular number of drives may be varied to meet the design criteria of a particular implementation. In the state 3 of TABLE 2, a number of data segments D 0 A, D 1 A, D 2 A may be implemented in a stripe set (e.g., A) which is normally used to generate a parity (e.g., PA). In the state 4 of TABLE 2, the state 3 may be located on the stripe set A location of the LUN 0   106  created in the state 2. In the state 5 of TABLE 2, the controller A may send a write IO (e.g., the data segment D 0 A′) to the LUN 0   106 . 
         [0021]    In the state 6 of TABLE 2, the system  100  may implement a Stripe-set State Preservation (SSP). In an SSP, the previous state of the stripe set A (e.g., D 0 A, D 1 A, D 2 A and PA) may be stored by the storage device  110 . In one example, the previous state of the stripe set A may be stored before the stripe set A is written to the LUN 0   106 . Corresponding mappings may be maintained by the controller B. In one example, the stripe set having the data segment to be written may be read and stored in the storage device  110  for each write IO request. In the state 7 of TABLE 2, the data segment D 0 A′ may be written in a first disk (e.g., P1) of the drive array  108 . Before regenerating and writing a parity (e.g., PA′) to a fourth disk (e.g., P4), the controller A may reboot (e.g., a fault condition). In one example, the controller B may not be sure that the state 7 has completed correctly (e.g., an incomplete write, a recovered error state, etc.). The controller B may retry the write with the new data in cache and make sure it is written to the first disk before generating the parity PA′. In the state 8 of TABLE 2, a forced transfer of the LUN 0   106  to the controller B (e.g., the alternate controller) may happen. In the state 9 of TABLE 2, the state 8 may trigger the pending write and the next “N” writes to generate the parity PA′ by reading all stripe segments (whether the LUN 0   106  has old/new data). 
         [0022]    In one example, a scenario may be considered where the controller B tries to regenerate the parity PA′ for the stripe set A, which may have an interrupted write mode (e.g., the state 10 of TABLE 2). In the state 11 of TABLE 2, the controller B may read new data (e.g., D 0 A′ from the first disk), old data (e.g., D 1 A from the second disk) and old data (e.g., D 2 A from the third disk) to generate the parity PA′ as new parity in place of the parity PA (e.g. from the fourth disk). In the state 12 of TABLE 2, the second disk may fail in the state 11. If the second disk fails, one new data and the old parity may result. Regenerating data for the second disk may not be done with a present stripe-set state. The data D 1 A (stored in the state 6) may be generated or read from the storage device  110  (the state 13). The storage device  110  may generate the data D 1 A. The LUN 0   106  may use the data D 0 A′, D 1 A, D 2 A to regenerate the parity PA′. In the state 14 of TABLE 2, the data stored in the storage device  110  during the state 6 may be erased. 
         [0023]    The system  100  may involve multiple cores (or controllers). Based on design implementation, certain cores (e.g., write cores) may handle data reconstruction and IO write operations while other cores (e.g., read cores) may focus on read operations. In one example, the read cores may handle the Stripe-set State Preservation (SSP). For example, the SSP may involve reading an entire data segment in the stripe set (e.g., for a write operation) before the write operation begins. The data segment to be written may be read by the read cores in response to each write IO request. The data segment may also be stored in the storage device  110  (e.g., a NVRAM, solid state drive, etc.). The corresponding data state with respect to the LUN 0   106  may be mapped and maintained in a separate table by the controller B. The system  100  may prevent possible data corruption. 
         [0024]    Referring to  FIG. 5 , a flow diagram of the process  200  is shown illustrating an example embodiment of the present invention. The process  200  generally comprises a state  202 , a state  204 , a state  206 , a state  208 , a state  210 , a state  212 , a state  214 , a state  216 , a state  218 , a state  220 , a state  222 , and state  224 . In the state  202 , a write IO may be sent to a stripe set A of the LUN 0   106 . In the state  204 , the controller A may trigger the read cores for a Stripe-set State Preservation (SSP) operation. In the state  206 , the storage device  110  may store the SSP data. In the state  208 , the controller A may trigger the write cores for a  10  write. In the state  210 , the controller A may enter a nonvolatile SRAM interrupted write mode. In the state  212 , the controller A may reboot. In the state  214 , the LUND  106  undergoing an IO may be forced to transfer to the controller B (e.g., an alternate controller). In the state  216 , the next “N” writes may have old/new data in the stripe set A to regenerate the parity PA′. In the state  218 , one physical drive under the LUN 0   106  may fail before completion of the interrupted write. In the state  220 , the LUN 0   106  may retrieve the SSP data from the storage device  110 . In the state  222 , the parity PA′ may be generated with correct old/new data stripes and may avoid data corruption. In the step  224 , the storage device  110  may erase the SSP data. 
         [0025]    Referring to  FIG. 6 , a conceptual diagram of Stripe-set State Preservation by the system  100  is shown. In the STATE 1, the stripe set A (e.g., a data block D 0 A, a data block D 1 A, a data block D 2 A, and a parity block PA) may be shown written to the drive array  108 . The data block D 0 A may represent a single data block or a plurality of data blocks. The data blocks D 1 A and D 2 A, as well as the parity block PA, may each represent one or more data blocks. In the STATE 2, a write IO may be sent for a new data block (e.g., D 0 A′). In the STATE 3, the stripe set A may be written to the storage device  110 . The write operation for the data block D 0 A′ may be completed in the STATE 3. After the STATE 3, an interrupted write mode may occur without completing the write to the second disk (e.g., P2). While the LUN 0   106  may try to generate the parity PA′ in the STATE 4, the write operation to the second disk (e.g., P2) may fail. The failure of the write to the second disk may result in the loss of the data block D 1 A. In the STATE 5, the data block D 1 A may be read from the storage device  110 . In the STATE 6, the parity block PA′ may be generated. In the STATE 7, the stripe Set A (stored in the STATE 3) may be erased. The STATES 1-7 generally describe different states used to implement a read modify/write implementation. In general, the symbol “+” in  FIG. 6  represents an exclusive OR. 
         [0026]    The system  100  may handle interrupted write modes using dual cores. For example, one or more cores may handle the IO thread for write and one or more cores may handle the IO thread for reading the old data in the stripe set where the write is intended to be performed. The old data read (e.g., by the read cores) may be stored in the storage device  110  (e.g., a NVRAM) and the controller B may map the old data with respect to the stripe set location. Data corruptions may be prevented during a double fault situation where a NVSRAM interrupted write mode condition happens within the controller (e.g., the owning controller A is rebooted) and one of the hard disk drives fails. 
         [0027]    As used herein, the term “NVSRAM interrupted write mode” is meant to describe the condition where a controller is in the middle of writing IO to a series of drives (or storage devices) and the controller hits an exception (e.g., a reboot, failed state, etc.) thus interrupting the write sequence to a particular stripe set. 
         [0028]    While the invention has been particularly shown and described with reference to the preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the scope of the invention.