Patent Publication Number: US-7899988-B2

Title: Video media data storage system and related methods

Description:
FIELD OF THE INVENTION 
     The present invention relates to the field of data storage, and, more particularly, to video data storage networks and related methods. 
     BACKGROUND OF THE INVENTION 
     In the computer industry the term “crash” is defined as the sudden failure of a software application or operating system, or of a hardware device such as a hard disk. While a software crash usually creates an interruption in service, the dreaded hard disk crash also tends to cause the loss of data. Due to the mechanical nature of a fast-spinning disk drive (e.g., 10,000 RPM), crashes are usually permanent and the need for data protection becomes critical. 
     Along with data errors that occur due to outright drive failure, expressed as a mean time between failure (MTBF), drives also experience bit errors over the amount of data read, expressed as a bit error rate (BER). Other errors that can result in seek, read and write failures are usually masked by successful retries. Since many drives now come with MTBF&#39;s upward of one million hours, BER&#39;s on the order of 1 in 1×10 15  and warranties up to 5 years, it makes economic sense for the vendors to be able to distinguish between a drive that has failed from one that is merely “having occasional trouble.” To reduce warranty replacement costs, drives hedge against excessive “failures” by employing many internal data recovery mechanisms, including error correction, retries, and sector remapping. Only when a drive exceeds its retry count and runs out of sectors for remapping is it considered “failed.” 
     Because of the need for deterministic data performance, video server designs usually cannot afford to allow drives the luxury to attempt all of the internal data correction mechanisms designed to conceal errors. Retries need to be limited and managed at a systemic level, and problematic drives generally are not acceptable. 
     Drive specifications also typically specify an annualized failure rate (AFR), which is equal to the operational duty cycle multiplied by the number of hours in a year and divided by the MTBF. Drives are usually divided into 3 classes, namely desktop, notebook and enterprise. Although more costly, only enterprise or server class drives are specified around a 100% duty cycle. So, a server drive with an MTBF of 1,000,000 hours would have an AFR of 0.876%, while a drive with an MTBF of 1,500,000 hours would have an AFR of 0.584%. 
     As systems increase in performance, size, and complexity, drive failure and error handling become a critical issue to video server design. Assuming a drive has a BER of 1 per 1×10 15  (errors per bits read), as data rates increase, the error frequency approaches 1 every few hours. Since even a single uncorrected bit error in a critical data location may result in an unacceptable video anomaly, it is typically necessary to implement some form of data protection. 
     One disk drive configuration that is used to help guard against data loss is the redundant array of independent drives (RAID) configuration. Generally speaking, in a RAID data is divided and/or replicated among multiple hard disk drives, which can lead to increased data reliability as well as increased input/output (I/O) performance. 
     Various levels of RAID configurations have been developed, and different RAID levels take advantage of different storage/data protection techniques. For example, some RAIDs employ “striping,” meaning that the data stream is divided into segments or blocks that are stored on separate drives in the RAID. Parity data may be used with data striping, which allows data from a faulty drive or sector(s) to be reconstructed from data on the other data drives in the RAID and the parity data. Another RAID technique is mirroring data on multiple drives within a same RAID set. That is, a copy of a data set is stored/maintained on a separate drive so that if one of the drives goes down, the duplicate data set is immediately available from the mirror drive. 
     Various prior art RAID implementations have been used to provide increased read performance and data redundancy. By way of example, U.S. Pat. No. 7,225,315 is directed to a file system including a storage system having a plurality of volumes, a volume allocation table adapted to set the plurality of volumes for each directory, a file allocation table that stores attributes and divided block information of the file, a block reading table in which numbers of blocks read out in one reading operation for each volume are respectively set, and a read control module that controls data read from the volume. A read control module, when a read command is received, determines a volume to be read from the volume allocation table. The read control module further determines the number of blocks read for each volume by referring to the block reading table, determines the blocks read for each volume based on the volume, the number of blocks, and the block information, and reads from each volume in parallel. 
     Despite the advantages that such configurations provide in certain applications, further data reading and recovery features may be desirable in high-bandwidth, high-reliability data storage applications, such as broadcast video applications, for example. 
     SUMMARY OF THE INVENTION 
     In view of the foregoing background, it is therefore an object of the present invention to provide a system and related methods for enhanced video media data storage and recovery. 
     This and other objects, features, and advantages are provided by a video media data storage system which may include first and second pluralities of data storage devices each arranged in a redundant array of independent drives (RAID) configuration for permitting writing and reading of video media data. The system may further include at least one memory controller coupled to the first and second pluralities of data storage devices for performing mirrored writing of video media data to both the first and second pluralities of data storage devices. The at least one memory controller may also be for generating respective first and second file allocation tables (FATs) including video media data time stamps and validity information for both of the first and second pluralities of data storage devices, and selectively reading video media data from the first and second pluralities of data storage devices based upon the first and second FATs. As such, more flexibility in video media data reading is provided to therefore enhance performance. 
     In addition, the at least one memory controller may also be for performing data recovery between the first and second plurality of data storage devices from the mirrored video media data based upon the first and second FATs. Moreover, the at least one memory controller may write the video media data in each of the first and second pluralities of data storage devices as striped video media data, and generate parity data from the striped video media. The at least one memory controller may therefore also perform data recovery within a given one of the first and second pluralities of data storage devices based upon the parity data. Additionally, the at least one memory controller may also advantageously select between using the mirrored video media data and the parity data for performing data recovery based upon a number of data storage devices having faults within the given one of the first and second pluralities of data storage devices. 
     The validity information may include data storage device fault information, for example. As such, the at least one memory controller may give reading preference to data storage devices without a fault and with a most recent video media data time stamp. The at least one memory controller may also perform load balancing for reading the video media data from the first and second pluralities of data storage devices. 
     By way of example, the first and second pluralities of data storage devices may each be arranged in a RAID 3 or higher configuration. Furthermore, the at least one memory controller may include a first memory controller for the first plurality of data storage devices, and a second memory controller for the second plurality of data storage devices. Additionally, the system may further include first and second fibre channel (FC) switches respectively coupling the first and second memory controllers to the first and second pluralities of data storage devices. Also, the first memory controller may be coupled to the second FC switch, and the second memory controller may be coupled to the first FC switch. 
     The first and second pluralities of data storage devices and the at least one memory controller may be connected together in a storage area network (SAN), for example. Further, the at least one memory controller may be at least one broadcast video server. 
     A video media data storage method aspect may include providing first and second pluralities of data storage devices each arranged in a redundant array of independent drives (RAID) configuration for permitting writing and reading of video media data, and performing mirrored writing of video media data to both the first and second pluralities of data storage devices. The method may further include generating respective first and second file allocation tables (FATs) including video media data time stamps and validity information for both of the first and second pluralities of data storage devices. The method may also include selectively reading video media data from the first and second pluralities of data storage devices based upon the first and second FATs. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a schematic block diagram of a video media data storage system in accordance with the invention. 
         FIG. 2  is a schematic block diagram of an alternative embodiment of the video media data storage system of  FIG. 1 . 
         FIG. 3  is a RAID map of the mirrored RAID sets of  FIG. 1  illustrating mirrored writing thereto. 
         FIG. 4  is a flow diagram illustrating video media data writing operations of the system of  FIG. 1 . 
         FIG. 5  is a schematic diagram of the first and second pluralities of data storage devices from the system of  FIG. 1  with accompanying file allocation tables (FATS) and data stripes. 
         FIG. 6  is a flow diagram illustrating video media data reading operations of the system of  FIG. 1 . 
         FIG. 7  is a RAID map of the first and second pluralities of data storage devices of the system of  FIG. 1  and their associated FAT tables. 
         FIGS. 8A and 8B  are respective FAT tables for the first and second pluralities of data storage devices of the system of  FIG. 1 , and  FIG. 8C  is a corresponding operation table showing read operations by the memory controller(s) of the system of  FIG. 1  based upon the information in the FAT tables of  FIGS. 8A and 8B . 
         FIGS. 9-13  are RAID drive maps illustrating various video media data recovery scenarios of the system of  FIG. 1 . 
         FIG. 14  is a flow diagram generally illustrating reading and writing operations in accordance with the invention. 
         FIG. 15  is flow diagram illustrating data recovery operations in accordance with another advantageous aspect of the invention. 
         FIG. 16  is a graph comparing RAID 3 vs. ECC overhead for parity configurations that may be implemented in the system of  FIG. 1 . 
     
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which preferred embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Like numbers refer to like elements throughout, and prime notation is used to indicate similar elements in alternate embodiments. 
     Referring initially to  FIG. 1 , a video media data storage system  30  and associated video data storage method are first described. The system  30  illustratively includes first and second pluralities or sets of data storage devices D 1  and D 2  each arranged in a redundant array of independent drives (RAID) configuration for permitting writing and reading of video media data. As used herein, “video media data” includes video, graphics, automation and/or animation, and associated audio data, for example. Furthermore, one or more memory controllers  31  are illustratively coupled to the first and second RAID sets D 1 , D 2 . In a typical implementation, the data storage devices may be magnetic or optical disks, although other suitable data storage devices (e.g., FLASH drives, etc.), may also be used. 
     The system  30  may be particularly well suited for applications where a large bandwidth is required, such as in broadcast or streaming video data applications. In the illustrated example, the video data feed is generated by video cameras  32 , and video media data from the system  30  may be communicated to recipients by one or more communications mediums, such as a satellite link (illustrated by a satellite station  33  in  FIG. 1 ), co-axial/fiber connections, wide area networks (WANs) such as the Internet, etc., as will be appreciated by those skilled in the art. 
     Turning now additionally to  FIGS. 3 and 14 , beginning at Block  140 , writing and reading operations of the system  30  will first be generally described, and further detail will be provided below. The memory controller(s)  31  performs mirrored writing of video media data to both the first and second RAID sets D 1 , D 2  (Blocks  141 - 142 ). In one exemplary implementation, the data in a given RAID set D 1 , D 2  is written in data stripes (i.e., divided in blocks or sections each written to a different storage device), and the striped data is mirrored in both of the RAID sets D 1 , D 2 , as will be discussed further below. This mirrored data writing is illustrated in  FIG. 3 . 
     The memory controller(s)  31  also generates respective first and second file allocation tables (FATs) including video media data time stamps and validity information (e.g., drive/sector fault information, etc.) for both of the first and second RAID sets D 1 , D 2 , at Block  143 . Since both of the FATs are available to the memory controller(s)  31 , it can advantageously selectively read video media data from the first and second RAID sets D 1 , D 2  based upon the first and second FATs, at Block  144 , thus concluding the method illustrated in  FIG. 14  (Block  145 ). That is, the memory controller(s)  31  has more flexibility in performing read operations, which may be particularly important where there are multiple drive faults. For example, the memory controller(s)  31  may give reading preference to data storage devices without a fault and with a most recent video media data time stamp. Moreover, this also advantageously allows load balancing to be used to reduce read times and increase data throughput, for example. 
     In addition, this configuration also allows the memory controller(s)  31  to perform data recovery between the first and second RAID sets D 1 , D 2  from the mirrored video media data based upon the first and second FATs. Stated alternatively, this allows inter-RAID data recovery (i.e., from RAID set D 1  to D 2 , or vice-versa). More particularly, the memory controller(s)  31  may write the video media data in each of the first and second RAID sets D 1 , D 2  as striped video media data, and generate parity data from the striped video media data to perform data recovery within a given RAID set based thereon (i.e., intra-RAID data recovery), as will be discussed further below. The memory controller(s)  31  will, generally speaking, select between using the mirrored video media data (inter-RAID recovery) and the parity data (intra-RAID recovery) for performing data recovery based upon a number of data storage devices having faults within the first and second RAID sets D 1 , D 2 , as will also be discussed further below. 
     Turning now additionally to  FIG. 2 , in an alternative embodiment the system  30 ′ is implemented as a storage area network (SAN) and includes respective broadcast video servers as the memory controllers  31   a ′,  31   b ′. By way of example, one suitable broadcast video server that may be used is the Nexio Advanced Media Platform (AMP) from Harris Corporation, the present Assignee. However, other suitable memory controllers/servers may also be used in different embodiments, as will be appreciated by those skilled in the art. 
     In the SAN system  30 ′, the first and second RAID sets D 1 ′, D 2 ′ are arranged in a RAID level  3  (or higher) configuration and implement data striping and parity (e.g., error correction code (ECC) parity), as will be appreciated by those skilled in the art. The parity data is stored in one or more dedicated parity data storage devices  40   a ′,  40   b ′, Furthermore, first and second fibre channel (FC) switches  36   a ′,  36   b ′ are respectively coupled to the first and second memory controllers  31   a ′,  31   b ′ and the first and second RAID sets D 1 ′, D 2 ′. That is, the first and second RAID sets D 1 ′, D 2 ′ are on separate FC domains to advantageously provide high redundancy and availability, as will be appreciated by those skilled in the art. 
     Moreover, the first memory controller  31   a ′ is illustratively coupled to the second FC switch  36   b ′, and the second memory controller  31   b ′ is illustratively coupled to the first FC switch  36   a ′. This advantageously gives the first memory controller  31   a ′ access to the second RAID set D 1 ′, and the second memory controller  31   b ′ access to the second RAID set D 2 ′. In some embodiments, additional FC switches may also be used to provide additional access points to the RAID sets D 1 ′, D 2 ′, for example. However, it should be noted that a SAN and/or FC network configuration is not required in all embodiments. 
     As noted above, since even a single uncorrected bit error in a critical data location may result in an unacceptable video anomaly, it is generally desired to implement some form of data protection. In a typical RAID 3 implementation, the parity information is simply used to reconstruct errant data after the problem is discovered. However, in the system  30  parity information may be continuously read and decoded, improving the capability of error detection as well as data performance during error correction. 
     More particularly, intra-RAID set parity may be implemented in the system  30  or  30 ′ as follows. Data is organized into buffers or stripes and is stored on respective drives  1 -P, and each partial data block size is defined as: 
     Data Buffer: D 
     Partial: d 
     Parity: P 
     Buffer size=S d    
     Partial size: S d    
     Number of drives=n 
     Partial size (s d )=S d /n, where S d  is divisible n, and
 
D=Σ i=n   i=0 di
 
Representing the XOR contribution of each partial as d, the parity equation is defined as:
 
P=xorΣ i=n   i=0 di
 
A single failing data block derr can be reconstructed via the inverse operation:
 
 derr=Pxor ( xorΣ   i&lt;err   i=0   di+xorΣ   i=n   i&lt;err   di )
 
     Unlike many RAID 3 implementations, in read mode the memory controllers  31   a ′,  31   b ′ advantageously read the parity data and use it to verify data integrity, rather than simply using this information for after-the-fact data recovery. Since the parity data is typically based on XOR contributions from each partial, error handling can take the form of either detection or correction. In this application, the parity equation serves three purposes depending on the error cases. A first error case is when a partial data block is not returned, or is reported in error. The defective partial is still reconstructed by the XOR resultant of the good partials and the parity. This case is known as single-error correcting (SEC). 
     A second case is when all partial data and parity is returned and no errors are reported by the drive. If all the data and parity is in fact good, the equation simply rebuilds the buffer. If any partial is detective, the error is detected and the damaged buffer is discarded. For the sake of video (and audio) information, data may be represented as black and silence can be substituted. This case is known as single-error detecting (SED). Still another case is where the partial data set has multiple defective elements. In this eventuality the data cannot be corrected, and again for the sake of video (and audio) information, data may be represented as black and silence can be substituted. This case is known as multiple-error detecting (MED). 
     In the case of a RAID disk storage sub-system, a failing partial usually represents a specific failing drive (resolving the issue of knowing which partial is in error), which reduces the model to the first and third cases noted above. By relying on the drive to know when it is having a problem and not always running data through XOR verification, most RAID systems typically ignore the second case, leaving the system vulnerable or erroneous unreported data. In the application of a video server, the present approach implemented by the memory controllers  31   a ′,  31   b ′ of using parity data for error detection may potentially detect this erroneous data and replace it with black to maintain a continuous uninterrupted video stream. This same approach may also be used for audio to reduce “pops” and “clicks”. 
     Another advantage of always keeping a running, on the fly, parity count is an overall performance improvement in the case of single corrected errors. If a drive fails, there is no need to go back and re-read the previous partials. Rather, the memory controllers  31   a ′,  31   b ′ may skip over the failed device and reconstruct its data contribution using the accrued parity information. This advantageously results in no net decrease in system performance during a “degraded” mode of operation. 
     Since a single parity drive RAID can only protect a data stripe from a single data element failure, the odds of an uncorrectable failure increase with the amount of partials involved in the stripe. As more data elements (drives in this case) are added, the greater the likelihood of an uncorrectable double data failure. The primary tradeoff of using a single parity RAID is that as sets grow larger they become much more susceptible to uncorrectable double errors. The system  30 ′ therefore advantageously utilizes multiple, mirrored RAID sets D 1 ′, D 2 ′ to provide further error recovery flexibility (in addition to the additional data reading flexibility noted above), as will be discussed further below. 
     An improved approach toward overcoming the deficiencies of single parity block per data stripe can be realized using a multi-parity block approach. The first part of this approach is defining a set of equations to represent parity information of a given data set. Hamming codes are used to define these equations. By using Hamming distance-3 codes to define a set of parity equations, desired data protection of the following nature may be achieved: 
     Number of parity equations (parity drives): r 
     Number of partial data blocks (data drives): n=2 r −1−r 
     Examining the example of a typical (15,11,3) code SEC set of Hamming equations, the standard parity matrix for the given distance-3 code is defined as follows: 
                                        Position                                                                                 p1   p2   d3   p4   d5   d6   d7   p8   d9   d10   d11   d12   d13   d14   d15                                                                                         H=   0   0   0   0   0   0   0   1   1   1   1   1   1   1   1           0   0   0   1   1   1   1   0   0   0   0   1   1   1   1           0   1   1   0   0   1   1   0   0   1   1   0   0   1   1           1   0   1   0   1   0   1   0   1   0   1   0   1   0   1                    
Columns having only a single entry determine parity positions. This approach assures the independence of the parity equations, yielding the following set:
 
 p 1 =d 3+ d 5+ d 7+ d 9+ d 11+ d 13+ d 15
 
 p 2 =d 3+ d 6+ d 7+ d 10+ d 11+ d 14 +d 15
 
 p 4 =d 5+ d 6+ d 7+ d 12 +d 13+ d 14 +d 15
 
 p 8 =d 9 +d 10+ d 11+ d 12 +d 13+ d 14 +d 15
 
The decoding of a received data stripe in the vector form r=(r1, r2, . . . r15) uses the following syndrome equations derived from the corresponding data equations above:
 
 s 1 =r 1+ r 3+ r 5+ r 7+ r 9+ r 11+ r 13+ r 15
 
 s 2 =r 2+ r 3+ r 6+ r 7+ r 10+ r 11 +r 14+ r 15
 
 s 4 =r 4 +r 5 +r 6 +r 7 +r 12 +r 13 +r 14+ r 15
 
 s 8= r 5+ r 9+ r 10+ r 11+ r 12+ r 13+ r 14+ r 15
 
     The syndrome vector obtained from the preceding equation set, written as s=(s4+s3+s2+s1), identifies the partial data position of a single error. Once the error is located, the failing data can be reproduced by solving the original parity equations. This is the standard application of a distance-3 Hamming SEC decoder. 
     In a RAID application it is possible (and likely) that the location of a failing partial (or partials) will be known due to the previously mentioned internal drive error reporting mechanisms. Knowing the location of the failing data increases the performance of the distance-3 Hamming code in different ways depending upon the error scenario. For example, if a single partial data block is not returned or reported in error, the data can be reconstructed as described above (without the need of solving the syndrome equations). This case is single-error correcting (SEC). If two partial data blocks are not returned or reported in error, the data can be also be reconstructed. Based upon the above matrix above, for any two known bad data partials there exists at least one equation that only contains one of the failing vectors. Solving the corresponding data equation reconstructs that partial, and reduces the syndrome to the preceding case. This case is double-error correcting (DEC). 
     A third case involves a combination of multiple unknown partials. In this event, the data cannot be corrected. For the case of video (and audio) information, data representing black and/or silence can be inserted. This case is multiple-error detecting (MED). Notwithstanding this last case, advantages of the above-described approach are that double data errors can be corrected, and single, unknown location data errors can also be corrected. 
     Comparing the relative cost of single parity and ECC in terms of additional overhead can be useful in determining which strategy is best suited for the given application. This can be done by comparing the parity overhead of a drive population containing a multiple RAID 3 set to the same population arranged in a single ECC set. The graph of  FIG. 16  examines both approaches for an example in which the maximum RAID 3 stripe length is set to 9 drives (8 data+1 parity). As can be seen, ECC systems begin to become advantageous at around 25 drives (data and parity) with an overhead less than twice that of the multiple RAID 3 set. As systems begin to reach 50 drives, the overhead of ECC actually becomes less than that of RAID 3, while continuing a higher level of protection. It should be noted that either of these approaches (or other suitable approaches) may be used at the RAID set level in the system of  30 ′. 
     Nonetheless, while localized RAID 3 and ECC protect against drive related media and data errors within a given RAID, data mirroring between different RAIDS affords protection against not only multiple drive errors, but also drive enclosure failures. Although this added layer of redundancy increases the storage system costs (i.e., because it uses an additional RAID set), it significantly increases data survivability and on-air availability. With respect to writing operations, mirroring can be done between two identical RAID 3 or ECC drive sets. Writing is accomplished by sending the data partials and parities to two identical sets of drives D 1 ′, D 2 ′, which results in the creation of a fully redundant data set, as seen in  FIG. 3 . It should be noted that by “identical” it is meant that the RAID sets D 1 ′, D 2 ′ have at least some corresponding drives that are the same, although these RAID sets could have other or extra drives which are not part of the mirror configuration in some embodiments. 
     Turning now to  FIGS. 4 and 5 , an exemplary write flow process by the memory controller(s)  31  is described. Beginning at Block  40 , the memory controller(s)  31 , initially attempts to write the video media data stripe to one of the RAID data sets (D 1  in the present example), at Block  41 . If writing to the RAID set D 1  cannot be completed successfully, such as due to a drive fault, etc., at Block  42 , then the memory controller(s)  31  attempts to write the video media data stripe to the second RAID set D 2 , at Block  43 . If this is successful, then the memory controller(s)  31  updates the FAT tables to indicate the RAID data set D 1  storage device(s) that is bad, and that the RAID data set D 2  is ON, at Block  45 , and to write a time stamp TS indicating the time of writing. Otherwise, both of the RAID sets D 1  and D 2  have one or more bad storage devices/sectors, and the first and second FATS are updated accordingly, at Block  46 . 
     If writing to the RAID data set D 1  is initially successful (Block  42 ), then the mirrored writing to the RAID data set D 2  is attempted, at Block  47 , and the memory controller(s)  31  determines whether this writing operation is successful, at Block  45 . The first and second FAT tables are updated accordingly for the case where the write operation to the RAID set D 2  was successful (Block  49 ), and where it was not (Block  50 ), as described above. At this point, the memory controller(s)  31  writes the first and second FAT tables to their respective first and second RAID sets D 1 , D 2 , at Blocks  50 - 51 , thus concluding the illustrated writing operations (Block  52 ). 
     Exemplary FATs for a mirrored RAID set configuration are shown in  FIG. 5 . Since a likely failure scenario involves not only write failures but also the possibility of loss of an entire RAID set, it is desirable to protect against these failure modes. The FATs for each RAID set D 1 , D 2  advantageously include sector indexing information, timestamps, and data OK flags for both of the RAID sets, as seen in  FIG. 5 . If a write fails on either side of the mirror (i.e., in either RAID set), the corresponding data OK flags in both FATS are cleared. This way subsequent reads will know to ignore these data elements, as discussed further below. 
     One particularly advantageous feature of the writing/reading/recovery operations described herein is that they may be implemented using a software application running at the memory controller/sever. Once suitable software application that may be used to implement the approach described herein is the RAIDSoft application from Harris Corporation. The parity operations/calculations are performed by the memory controller(s)  31 , and the data and localized parity are then written in a mirrored fashion to both of the RAID sets D 1 , D 2 , as discussed above. All of the information is mirrored at the block level, with the exception of the FAT tables. The FAT tables differ between the two RAID sets D 1 , D 2  to track bad sectors/drives and offline conditions. The FAT information for both RAID sets D 1 , D 2  may also be stored in a memory such as a random access memory (RAM) in some embodiments as well to provide faster operation by the memory controller(s)  31 . 
     Turning now additionally to  FIGS. 6 and 7 , the reading process will now be described. Generally speaking, data reads are performed once and load balanced between the first and second RAID sets D 1 , D 2  if both sets contain good (i.e., without fault and/or up-to-date) data. Unrecorded or faulty sectors/drives are noted in the FATS and not used in read operations. If the load balanced read returns an error, then the other copy of the data is immediately read with little or no disturbance to air playout in broadcast scenarios, for example. In the case of a RAID set being down, the down set will not be accessed again until the problem is cleared (e.g., drive replaced, etc.). Probe commands may be periodically sent to determine when the down set returns to operations, for example, as will be appreciated by those skilled in the art. 
     More particularly, beginning at Block  60 , when performing a read operation the memory controller(s)  31  first checks the first and second FAT tables to see if the data is good for a given block of data in both of the first and second RAID sets D 1 , D 2  (Blocks  61 ,  62 ), as illustrated in  FIG. 7 . The results of the FAT tables are compared, such as through a logical OR operation (Block  63 ), to determine the four possible fault conditions for the drives in questions, namely: (a) the RAID set D 1  is OK but RAID set D 2  is BAD (Block  64 ); (b) both RAID sets are OK (Block  65 ); (c) RAID set D 1  is BAD but RAID set D 2  is OK (Block  66 ); and (d) both RAID sets are bad for the given block of data (Block  67 ). 
     In the first case (a), the memory controller(s)  31  attempts to read the first RAID set D 1 , at Block  68 , and if the read is completed successfully (Block  69 ) then the data is output accordingly, at Block  70 . If the read operation is not successful for whatever reason, then black and/or silence may be output, at Block  71 . In the second case (b), since both RAID sets are OK (i.e., without fault), then the memory controller(s)  31  checks to see which RAID set D 1 , D 2  has the most up-to-date data (i.e., compares the time stamps), at Block  72 . If the time stamp for the RAID set D 1  time stamp is more recent, then the memory controller  31  reads the first RAID set as described above at Blocks  68 - 71 . 
     If the time stamps are the same, then the memory controller(s)  31  may read from either RAID set D 1 , D 2 , and the choice between the two may be based upon a load balancing algorithm (i.e., checks which one is busiest at the time and uses the other), and/or a preferential (i.e., default) scheme may be used. For example, in the illustrated embodiment an affinity is given to odd or even RAID sets, such that odd-numbered memory controllers (e.g., controller # 1 ) would first access an odd-numbered RAID set (i.e., RAID set D 1 ), and vice-versa. In the present example, the affinity is for the first RAID set D 1 , which the memory controller(s)  31  attempts to read at Block  73 . If the read is successful, at Block  74 , then the data is output (Block  75 ). If not, the memory controller(s)  31  still has the option of reverting back to the second RAID set D 2 . If the attempted read from the second RAID set D 2  is successful, at Blocks  76 - 77 , then the data is output, at Block  78 . Otherwise, black and/or silence data is output, at Block  79 . 
     For the third case (c), the memory controller(s)  31  attempts to read from the second RAID set D 2  in the same manner described above with reference to Blocks  76 - 79 . For the final case (d) where both RAID sets D 1 , D 2  have faults, etc., then black and/or silence data is output, at Block  79 . In the exemplary embodiment shown in  FIG. 7 , all of the drives are without fault and have equal time stamps (meaning both sets have the freshest or most up-to-date data). It should be noted that in some embodiments reading may also be performed from a faulty drive using the parity data described above if the mirrored data is bad or “stale”, as will be appreciated by those skilled in the art, although this is not illustrated in  FIG. 6  for clarity of explanation. Typically this option is available if a single drive within a RAID set is faulty but the rest are not. That is, as the number of faulty drives increases past one, the less likely it will be that intra-RAID set parity can be used to read and/or recover this data, as will be appreciated by those skilled in the art. This is why the mirrored writing/reading/recovery operations are particularly advantageous where high throughput and availability are required of the system  30 . 
     Referring now additionally to  FIGS. 8A-8C , the above-described reading operations will be further understood with reference to an exemplary read scenario. For the first block of data (stored on drive d 1  in both RAID sets D 1 , D 2 ), it can be seen in the operation table of  FIG. 8  that both FATs reflect that drive d 1  is OK in both RAID sets, and that their time stamps are equal. Thus, both data stripes have “good” data and either can be successfully read based upon affinity, load balancing, etc. In the case of the second data block, both FATS indicate that the drive d 2  is OK in the first RAID set D 1 , but the second FAT indicates that the drive d 2  in the second RAID set D 2  is bad, and that the time stamp for drive d 1  in RAID set D 1  is newer than for the drive d 2  of the second RAID set D 2 . As such, only the first data stripe (i.e., RAID set D 1 ) is considered valid for reading. 
     For the fourth data block, the drive d 4  is listed as bad in the second FAT table, and the time stamp for the drive d 4  in the second FAT is also newer. As such, the first data stripe is again considered bad, and the second data strip (i.e., RAID set D 2 ) is read. For the seventh data block, both FAT tables indicate that the RAID sets D 1  and D 2  are OK, but the time stamp for the drive d 7  of the second RAID set is newer, indicating that the data on the drive d 7  of the first RAID set is stale. Accordingly, the memory controller(s)  31  considers the first data stripe to be old or stale, and reads from the drive d 7  of the second RAID set. In the last illustrated case for the ninth data block, both FAT tables indicate that the RAID sets D 1  and D 2  are OK, but the time stamp for the drive d 7  of the first RAID set is newer, indicating that the data on the drive d 7  of the second RAID set is stale. Accordingly, the memory controller(s)  31  considers the second data stripe to be old or stale, and reads from the drive d 7  of the first RAID set. 
     Referring now to  FIGS. 9-13  and  FIG. 15 , the data recovery steps performed by the memory controller(s)  31  are now further described. Generally speaking, in cases where both RAID sets are available, all unused, bad, or out of sync sectors on all disks will be written with recovered data. The recovered data is preferably always the “freshest” copy, taken from whichever RAID set D 1 , D 2  is appropriate. If the freshest data had a read fault from an individual disk, then the inter-RAID set mirror data is used to recover the data. 
     More particularly, beginning at Block  150 , the parity data is generated for the first and second RAID sets D 1 , D 2  as discussed above (Block  150 ). In particular, the parity data may be generated for one of the RAID sets D 1 , D 2 , and then copied over as part of the mirrored data set, although in some embodiments parity data could be generated independently for each of the RAID sets, if desired. The memory controller(s)  31  determines which of the storage devices in the first and second RAID sets require recovery or data freshening, at Block  152 , based upon the OK flags and the time stamps, for example. 
     The way in which the memory controller(s) will perform data recovery will depend upon the number of storage devices/drives that are affected. More particularly, if only a single drive in one of the RAID sets D 1  or D 2  is affected, the data is recovered on the fly from the intra-RAID set parity data, at Block  154 , thus concluding the illustrated example (Block  155 ). In the example shown in  FIG. 9 , the drive d 4  in the RAID set D 1  is bad (or has stale date), as indicated by the dashed markings. Here, the data may be recovered using the parity data stored in the drive P in the first RAID set, and thus it is not necessary to go to the mirrored data in the second RAID set D 2  for recovery (although this could be done, if desired). 
     In the case of a bad drive, the user would replace the bad drive and the recovery operations may advantageously be performed automatically by the memory controller(s)  31  once it detects that the drive is no longer faulty, for example. It should be noted that even if the corresponding partials from both RAID sets D 1 , D 2  are damaged, parity data from either set can be used to reconstruct the data buffer. 
     However, if more than a single drive is affected, depending upon the particular scenario, inter-RAID recovery will be used in addition to, or instead of, the intra-RAID parity-based recovery, at Blocks  156 - 157 . In the example illustrated in  FIG. 10 , two drives in the first RAID set D 1  are bad or have stale data, namely the drives d 4  and d 9 . Here, the memory controller(s)  31  uses the mirrored data from the corresponding drives d 4  and d 9  in the second RAID set D 2  to recover the data to the drives d 4  and d 9  of the first RAID set D 1 , which again may advantageously be done on the fly without disruption to air time. 
     A more extended disk failure scenario is shown in  FIG. 11 , where multiple drives in the first RAID set D 1  have failed (i.e., the drives d 4 , d 9 , d 13 , and P), and the corresponding disk d 9  in the second RAID set also fails. Here, the data in the drive d 9  of the second RAID set D 2  is recovered using the inter-RAID parity data from its drive P, and the faulty drives in the first RAID set D 2  (i.e., the drives d 4 , d 9 , d 13 , and P) may then be recovered using the data from the corresponding drives in the second RAID set. This can still advantageously be performed by the memory controller(s)  31  on the fly and without disruption to air. 
     Still another fault scenario is illustrated in  FIG. 12 , where the entire RAID set D 1  goes down for some period of time. By way of example, this could happen when an FC cable is unplugged from a memory controller/server, drive chassis, or switch. This could also happen where an FC switch is accidentally unplugged or fails, or where a drive chassis loses power or fails, as will be appreciated by those skilled in the art. In such case, the memory controller(s)  31  will lose access to the first RAID set D 1 , but it will still advantageously be able to read and write from the second RAID set D 2  with little or no effect to the output video stream. In this situation, sectors written by the affected servers (e.g., the server  31   a ′ in  FIG. 2 ) are remembered as bad in the FAT of the opposite RAID set, and therefore will not be used in subsequent read operations by any server. Once power/connection to the affected RAID set is restored, then the memory controller(s) performs data recovery from the mirrored data in the RAID set D 2  to the corresponding drives in the RAID set D 1 . 
     Thus, it will be appreciated that with a mirrored RAID 3 (or higher) set as discussed above, loss of all the partial elements (i.e., individual drives) in one of the RAID sets as well as the loss of an additional single drive in the other RAID set can be recovered. Mirrored ECC RAID sets can advantageously recover from loss of an entire RAID set as well as the loss of two additional elements on the other RAID sets in some situations. If an entire RAID set is not available, as in the prior example, when it is returned to service it is likely that it will contain stale data elements. By comparing timestamps during reading, stale data elements can likewise be ignored. This may be done by ORing together the corresponding data values and comparing the timestamps, for example. 
     Many modifications and other embodiments of the invention will come to the mind of one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is understood that the invention is not to be limited to the specific embodiments disclosed, and that modifications and embodiments are intended to be included within the scope of the appended claims.