Patent Publication Number: US-10332613-B1

Title: Nonvolatile memory system with retention monitor

Description:
BACKGROUND 
     NAND-based flash memories are widely used as the solid-state memory storage due to their compactness, low power consumption, low cost, high data throughput and reliability. Solid state drive (SSD) devices commonly employ NAND-based flash memory chips and a flash controller to manage the flash memory and to transfer data between the flash memory and a host computer. 
     An issue for SSDs is the reliability of the storage elements over the life of the SSD. Over time, relatively high gate voltages applied to the storage elements during program and erase (P/E) cycles in the SSD may cause cumulative permanent changes to the storage element characteristics. Charge may become trapped in the gate oxide of the storage elements through stress-induced leakage current (SILC). As the charge accumulates, the effect of programming or erasing a storage element becomes less reliable and the overall endurance of the storage element decreases. Additionally, an increasing number of P/E cycles experienced by a storage element decreases the storage element&#39;s data retention capacity, as high voltage stress causes charge to be lost from the storage element&#39;s floating gate, resulting in increased Bit Error Rate (BER) of the memory storage device. 
     Design capabilities of SSD&#39;s are driven by application use cases. Consumer applications are driven primarily by cost, requiring low cost devices that can have limited endurance and limited retention, as long as a lifespan of a few years is obtained for a single-user usage model in which the SSD is operated for only a few hours a day. In contrast, enterprise applications require high reliability, high endurance and long service life. Some enterprise applications also require high retention. However, the factors dictating retention and endurance are related, allowing for varying specifications to accommodate specific use cases. For example, a SSD may have a write endurance of 10,000 cycles/block. By making the specification for retention less stringent, write endurance can be extended. In transaction-oriented applications, where data retention of a few weeks is acceptable, block write endurance can be extended to more than 10,000 cycles/block. 
     Accordingly it is important to be able to accurately determine both write endurance and retention. Prior art models for determine retention capabilities of NAND-based flash memory chips are typically based on delta read calculations and the assumption that delta read is monotonic. However, with scaled NAND geometries, delta read is not monotonic. Delta read can be both positive and negative for a particular retention time. Accordingly, prior art models based on the assumption that delta read are based on an incorrect assumption. This can lead to incorrect estimation of retention values for a particular NAND Device. 
     Accordingly, what is needed in the art is a method and apparatus that will allow for accurately determining retention capabilities of NAND-Flash devices and SSD&#39;s and assuring that NAND-Flash devices and SSD&#39;s maintain the determined retention capabilities. 
     SUMMARY 
     In various embodiments, a nonvolatile memory system is disclosed that includes a nonvolatile memory storage module for storing encoded data and a nonvolatile memory controller. The nonvolatile memory storage module includes a plurality of memory cells that are controlled by the nonvolatile memory controller. The nonvolatile memory controller includes a retention monitor that is configured for storing test characteristics corresponding to a use case and determining, each time that a read of a codeword is performed, whether the number of errors in the codeword exceed a retention threshold. If the number of errors in the codeword exceed the retention threshold, the block that includes the codeword that exceeds the retention threshold is retired. Retention tests are performed during the operation of the memory controller and the retention threshold is adjusted when the results of the retention tests indicate deviation from the test characteristics corresponding to a use case. 
     A method for assuring retention is disclosed that includes storing test characteristics corresponding to a use case and, each time that a read of a codeword of a nonvolatile memory device is performed by the controller, determining the number of errors in the codeword. Each time that a read of a codeword of a nonvolatile memory device is performed by the controller, the method includes determining whether the number of errors in the codeword exceed a retention threshold and, if the number of errors in the codeword exceed the retention threshold, retiring the block of the nonvolatile memory device that includes the codeword that exceeds the retention threshold. The method further includes performing retention tests during the operation of the memory controller and adjusting the retention threshold when the results of the retention tests indicate deviation from the test characteristics corresponding to a use case. 
     The method and apparatus of the present invention allow for assuring a level of retention can be maintained over the life of the nonvolatile memory devices. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The accompanying drawings are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention, and together with the description, serve to explain the principles of the invention. 
         FIG. 1  is a block diagram illustrating a nonvolatile memory system in accordance with an embodiment of the present invention. 
         FIG. 2  is block diagram illustrating a memory logic organization in accordance with an embodiment of the present invention. 
         FIG. 3  is a diagram illustrating a NAND array in accordance with an embodiment of the present invention. 
         FIG. 4  is a flow diagram illustrating a method for assuring retention of nonvolatile memory devices coupled to a memory controller in accordance with the present invention. 
         FIG. 5  is a flow diagram illustrating an offline retention test and updating the retention threshold when a calculated DeltaWorst retention threshold exceeds the current retention threshold in accordance with an embodiment of the present invention. 
         FIG. 6  is a flow diagram illustrating an online retention test and updating the retention threshold when a calculated ΔR(t,ts) multiplied by an acceleration factor exceeds ΔR(t,ts) CHAR  in accordance with an embodiment of the present invention. 
         FIG. 7  is a diagram illustrating test results for a test NAND device during retention, showing the number of errors on the vertical axis and retention time on the horizontal axis with the worst number of errors in each block plotted for a particular use case in accordance with an embodiment of the invention. 
         FIG. 8  is a diagram having a vertical axis representing the number of errors and a horizontal axis representing retention time and shows DeltaWorst for the use case shown in  FIG. 7  in accordance with an embodiment of the invention. 
         FIG. 9  is a diagram having a vertical axis representing the number of errors and a horizontal axis representing retention time and shows Delta-DeltaWorst for the use case shown in  FIG. 7  in accordance with an embodiment of the invention. 
         FIG. 10  is a diagram illustrating test results for a test NAND device during retention, showing the number of errors on the vertical axis and retention time on the horizontal axis with the worst number of errors in each block plotted for a use case in accordance with an embodiment of the invention. 
         FIG. 11  is a diagram illustrating test results for a test NAND device during retention, showing DeltaR(t) on the vertical axis and retention time on the horizontal axis for the use case shown in  FIG. 7  in accordance with an embodiment of the invention. 
         FIG. 12  is a diagram illustrating test results for a test NAND device during retention, showing errors on the vertical axis and retention time on the horizontal axis for the use case shown in  FIG. 11  and shows maximum DeltaR(t), minimum DeltaR(t) and average DeltaR(t) in accordance with an embodiment of the invention. 
         FIG. 13  is a diagram illustrating test results for a test NAND device during retention, showing maximum absolute DeltaR(t) on the vertical axis and retention time on the horizontal axis for the use case shown in  FIG. 11  in accordance with an embodiment of the invention. 
     
    
    
     DETAILED DESCRIPTION 
     A nonvolatile memory system  100  is shown in  FIG. 1  to include a nonvolatile memory controller  110  in communication with a nonvolatile memory storage module  140 . The nonvolatile memory storage module  140  includes a plurality of nonvolatile memory devices for storing data. In the present embodiment the nonvolatile memory devices are NAND devices  150 . In the present embodiment each NAND device  150  is a packaged semiconductor die that is coupled to nonvolatile memory controller  110  by conductive pathways that couple instructions, data and other information between each NAND device  150  and nonvolatile memory controller  110 . The nonvolatile memory controller  110  includes a retention monitor  120 . Retention monitor  120  includes a characteristics module  121  that is configured to store test characteristics corresponding to one or more use case. In one embodiment the test characteristics are stored in memory storage on nonvolatile memory controller  110 . Alternatively, the test characteristics can be stored in one or more NAND devices  150 . 
     Retention monitor  120  is operable for determining, each time that a read of a codeword is performed, whether the number of errors in the codeword exceed a retention threshold, and if the number of errors in the codeword exceed the retention threshold, retiring the block of the nonvolatile memory device that includes the codeword that exceeds the retention threshold. Retention monitor  120  is also operable for performing retention tests during the operation of the memory controller and adjusting the retention threshold when the results of the retention tests indicate deviation from the test characteristics corresponding to a use case. Retention monitor  120  includes an online test module  123  that is configured to perform online testing and an offline test module  122  that is configured to perform testing on the results of the offline retention. 
     In one exemplary embodiment each NAND device  150  is coupled to nonvolatile memory controller  110  by chip enable line (CE#), a command latch enable (CLE) line, a read enable signal line (RE#), an address latch enable (ALE) signal line, a write enable single line (WE#), a read/busy (RB) signal line and input and output (I/O) signal lines. 
     Referring now to  FIG. 2 , each NAND device  21  includes memory cells that are organized into blocks  22  and pages  23 , with each block  22  composed of NAND strings that share the same group of word lines. A logical page  23  is composed of cells belonging to the same word line. The number of logical pages  23  within logical block  22  is typically a multiple of 16 (e.g.  64 ,  128 ). Each logical page  23  is composed of a main data area and a spare area. The main data area may have the size of 4 kB, 8 kB, 16 kB or larger. The spare area is made up of hundreds of bytes for every 4 kB of main data storage area. 
     In the present embodiment, a logical page  23  is the smallest addressable unit for reading from and writing to the NAND memory and a logical block  22  is the smallest erasable unit. However, it is appreciated that in embodiments of the present invention programming to less than an entire page may be possible, depending on the structure of the NAND array. 
     An exemplary NAND array  30  is shown in  FIG. 3  that is made of memory cells connected in series to form NAND strings. Each NAND string is isolated from the rest of the array by select transistors, such as, for example, select transistor  31  and select transistor  32 . Multiple memory cells share the gate voltage (Vg) through a word line, and the drain of one memory cell is the source of the adjacent one. For example, memory cells  34 - 39  of  FIG. 3  share word line 0 (WL0). Though  FIG. 2  illustrates an embodiment in which memory cells are single level cells, it is appreciated that NAND devices  150  can also be multi-level cell NAND devices and can store, for example, 2 bits per cell, 3 bits per cell or 4 bits per cell. 
       FIG. 4  illustrates a method  400  for assuring retention that includes storing test characteristics corresponding to a use case  401 . In the present embodiment characteristics module  121  is configured to store test characteristics, either in memory storage on nonvolatile memory controller  110  or in one or more NAND device  150 . In one embodiment test characteristics module  121  is also operable to store test codewords to be read in online and offline retention testing. 
     As shown by step  402 - 403 , each time that a read of a codeword is performed, the number of errors in the codeword are determined. A determination is then made  404  as to whether the number of errors in the codeword exceed a retention threshold (RT) and if the number of errors in the codeword exceed the retention threshold, the block of the nonvolatile memory device that includes the codeword that exceeds the retention threshold is retired as shown by steps  404 - 405 . Alternatively, when the number of errors in the codeword exceed a retention threshold (RT) and if the number of errors in the codeword exceed the retention threshold, the page of the nonvolatile memory device that includes the codeword that exceeds the retention threshold is retired. 
     As shown by step  406 - 412  retention tests are performed during the operation of nonvolatile memory controller  110 . As shown by step  408 - 409  and  412  the retention threshold is adjusted when the results of the retention tests indicate deviation from the test characteristics corresponding to a use case. 
     In the present embodiment the retention tests include both offline retention tests  407  and online retention tests  411 . Alternately, only offline retention testing  407  or only online retention testing  411  could be used for adjusting the retention threshold  409 . 
     In the present embodiment, offline retention tests are performed  407  when the memory controller is being shut off and when it is turned back on as shown by steps  406 - 407 . In the present embodiment a first portion of the offline retention test  407  is performed as the memory controller is shut down (e.g., as a part of the shut-down process) and a second part of the offline retention test  407  is performed as the memory controller is restarted (e.g., as a part of the power-on-start-up process) in the next start-up of the nonvolatile memory controller  110 . 
     As shown by steps  408 - 409  the retention threshold is adjusted  409  when the result of the offline retention test indicates deviation from an online test characteristic corresponding to a model use case  408 . 
     Online retention tests  411  are performed during normal operation nonvolatile memory controller  110  at certain times  410 , which may be regular intervals of time such as hourly, daily, weekly, monthly or after a predetermined number of operating hours. In one exemplary embodiment, online retention tests  411  are performed after every 12 operating hours of nonvolatile memory controller  110 . 
     When the results of the online retention test indicate deviation from online test characteristics corresponding to a model use case  412  the retention threshold is adjusted  409 . 
     When a read is not being performed, when the number of errors is not greater than the retention threshold  404 , and after retiring the block  405  normal operation of the nonvolatile memory controller is continued  414 . Also, when online retention test does not indicate deviation from a model use case and offline retention test does not indicate deviation from a model use case and after adjusting the retention threshold  409  normal operation is continued  414 . 
       FIG. 5  illustrates an embodiment of steps  407 - 409  of  FIG. 4 . More particularly a method  500  is illustrated for performing an offline retention test and updating a retention threshold in accordance with an embodiment of the invention. In one embodiment some or all of the steps of method  500  are performed by offline test module  122 . As shown by step  501  when a power-off indication is received test codewords are read and the number of errors in each codeword is determined  502 . In the present embodiment test codewords are one or more pages that are dedicated to storing data for retention testing. In one embodiment each test codeword is a logical page. Alternatively, a page may contain more than test codeword. The number of test codewords read in step  502  may be as few as two to three or as many as two to three codewords in each block of each NAND device  150 . In one exemplary embodiment only three codewords are read so as to keep shutdown time to a minimum. In another embodiment a page of each block of each NAND device  150  is read. 
     As shown by step  503  the highest number of errors in the tested codewords is determined. More particularly, the number of errors in the codeword having the highest number of errors is determined. The highest number of errors can also be referred to as the “worst number of errors”. In one embodiment the highest number of errors is determined by initializing a highest number of errors value at  0  and comparing the number of errors in each codeword read in step  502  to the highest number of errors value. If the errors in a codeword exceed the highest number of errors value, the highest number of errors value is replaced by the number of errors in the codeword having a number of errors exceeding the highest number of errors value. The highest number of errors value is then stored as shown by step  505 . In one embodiment the highest number of errors is stored in nonvolatile memory controller  110  or on a NAND device  21  prior to powering off the nonvolatile memory controller  110 . In the present embodiment the time at which test codewords  502  were read is determined and is stored along with the highest number of errors. The stored time, that can be referred to as initial time (t 1 ) can be the time that the codeword having the highest number of errors was read. Alternatively, the time can be the time that the read operation of step  502  commenced or ended. The initial time may be determined by setting a timing device to an initial time of 0. Alternatively, the initial time may be determined by determining the time of an internal or external clock, e.g., a time t REF1  of an internal or external reference clock that is not reset to a time of 0. 
     The nonvolatile memory controller is then powered off as shown by step  505 . 
     After powering-on the nonvolatile memory controller at a subsequent time  510 , the codewords that were read in step  502  are again read as shown by step  511  and the number of errors in each codeword is determined as each codeword is decoded. In embodiments in which all of the active codewords are read on startup to refresh active memory pages, the reading of step  511  is integrated into the startup-refresh process so as to provide a quick startup of the nonvolatile memory controller  110 . 
     The number of errors in the codeword having the highest number of errors (at the subsequent time) is determined  512 . In one embodiment the highest number of errors is determined by initializing an after-offline-retention-highest number of errors value at  0  and comparing the number of errors in each codeword read in step  511  to the after-offline-retention-highest number of errors value. If the errors in a codeword exceed the after-offline-retention-highest number of errors value, the after-offline-retention-highest number of errors value is replaced by the number of errors in the codeword. The after-offline-retention-highest number of errors value may then be stored along with the time of the reading of the codeword having the highest number of errors in step  511 , that can be referred to as the subsequent read time (t 2 ). 
     The offline retention time t is then determined  513 . In the embodiment in which the timing device is initialized to a time of 0 at step  502 , the time indicated by the timing device at step  512  is the offline retention time (t). It is appreciated that offline retention time t is the relative time period between the initial offline retention time of step  502  and the offline retention time of step  511 , and that memory controllers do not typically include timing devices that are operable during power-off. As previously discussed, nonvolatile memory controller may include offline timer that is operable during power-off. 
     In one embodiment when a power-off signal is received, nonvolatile memory controller  110  does not completely power-off but rather enters a low power mode in which certain modules are operable such as the offline timer. In this mode all NAND memory devices are powered off and nonvolatile memory controller  110  returns to full-power mode when a power-on signal or start-up signal is received or when power is applied to one or more power pin of nonvolatile memory controller  110 . 
     Alternatively time is measured using a system clock or other timing resource external to the nonvolatile memory controller. In these embodiments the time of the read of step  511  is determined (T REF2 ) and the read time (T REF1 ) of step  502  is subtracted from T REF2  of step  511  to obtain the subsequent read time (offline retention time=t). 
     Referring now to step  514 , a delta worst (ΔWorst) value is calculated by subtracting the number of errors in the codeword having the highest number of errors from step  504  Worst(0) from the number of errors in the codeword having the highest number of errors from step  512  Worst(t) and can be represented by the equation:
 
ΔWorst( t )=Worst( t )−Worst(0)
 
where t represents offline time.
 
As shown by step  515  a delta worst retention threshold (DWRT) is determined. In the present embodiment the delta worst retention threshold is determined by subtracting delta worst determined in step  514  from the maximum error correction capacity of the error correction code used to generate the codewords (ECC MAX ) and can be represented by the equation:
 
DWRT=ECC MAX −ΔWorst( t ).
 
     The retention threshold is updated if the delta worst retention threshold exceeds the retention threshold  516 . The retention threshold is initially set at a characteristic retention threshold (RT CHAR ) that was stored in step  401 . Accordingly, the retention threshold is updated if the delta worst retention threshold exceeds the characteristic retention threshold. 
     In the present embodiment a determination is made as to whether the offline retention time determined in step  513  is within specification and when the retention time is above the specification retention time the retention time is not updated in step  518 . In one embodiment a specification retention time Tspec is stored in step  401  for use in step  517 . 
     It is appreciated that step  517  is optional and that, in embodiments that do not include step  517 , the retention threshold is updated in step  518  even when the offline retention time exceeds the specification retention time. 
     In the present embodiment the retention threshold is updated  518  by replacing the retention threshold with the delta worst retention threshold determined in step  515  when the delta worst retention threshold of step  515  exceeds the retention threshold. Accordingly, the retention threshold is initially updated if the delta worst retention threshold exceeds the characteristic retention threshold. After the initial update in which the characteristic retention threshold is used to determine whether the offline retention test indicates deviation from the model use case, subsequent comparisons in step  516  are based on a previous delta worst retention threshold calculated in step  515  or an update based on online retention, which is further discussed in step  607 . 
     Though the test codewords read in steps  502  and  511  may be codewords stored in NAND devices  150  for the exclusive purpose of retention testing, alternatively, any codeword could be chosen as a test codeword for offline retention testing. In this embodiment test codewords read in steps  502  and  511  are selected from the active codewords stored in NAND devices  150 . 
       FIG. 6  illustrates an embodiment of steps  410 - 412  and  414 . More particularly, method  600  illustrates a method for performing an online retention test and updating a retention threshold. In one embodiment some or all of the steps of method  600  are performed by online test module  123 . As shown by step  601  test codewords are read and the number of errors is determined for each codeword that is read. In the present embodiment test codewords are one or more pages that are dedicated to storing data for retention testing. In one embodiment each test codeword is a logical page. Alternatively, a page may contain more than test codeword. The number of test codewords read in step  601  may be as few as two to three or as many as an entire block of each NAND device  150 . In one exemplary embodiment each codeword is a logical page and step  601  reads all of the pages of a dedicated test block of a single NAND device  150 . 
     The read  601  is performed on a regular basis an interval is that may be, for example, 12, 24, 36 or 48 hours. In the present embodiment read  601  is performed every 12 operating hours of nonvolatile memory controller  110 . 
     Marginal error rate is determined  602  for the codewords read in step  601 . In one embodiment marginal error rate, that can also be referred to as Delta Read and ΔR(t,ts) is determined by subtracting the number of errors at time (t) from the number of errors at the following interval (t+ts) as is illustrated by the equation:
 
Δ R ( t,ts )=#Errors( t+ts )−#Errors( t ).
 
     In one embodiment, each time that the test codewords are read in step  601 , the number of errors are stored for use in the following calculation of marginal error rate. 
     Temperature is determined as shown by step  603 . In the present embodiment nonvolatile memory controller  110  includes a temperature sensor that is operable for determining the temperature of the nonvolatile memory controller at the time of the test. 
     There is feedback between online and offline parameters to as to enable a cross-correlation between them. Because temperatures can be different between online and offline periods (and they usually are) an acceleration factor (AF) is used to normalize measured retention errors. More particularly, as shown by step  604 , an acceleration factor is determined that corresponds to the temperature determined in step  603 . In the present embodiment the acceleration factor is determined using conventional methodology such as, for example, the Arrhenius equation. 
     As shown by step  605 - 607  the retention threshold is updated if the determined marginal error rate multiplied by the acceleration factor exceed a characterized marginal error rate. More particularly, the marginal error rate (ΔR(t,ts)) determined in step  602  is multiplied by the acceleration factor (AF) determined in step  603  and the results are compared to a corresponding characterized marginal error rate (ΔR(t,ts) CHAR ). 
     In the present embodiment the test characteristics stored in step  401  include a plurality of characterized marginal error rates. In one embodiment a table is stored that includes characterized marginal error rates for each test time (t) for the particular testing interval (ts). In one embodiment a characterized marginal error rate is stored for each test time (t). Alternatively, to save storage space, a characterized marginal error rate may be used for more than one different test time (t). For example, though the testing of steps  601 - 607  may be done daily, a single characterized marginal error rate may be used for testing during a time interval, such as, for example, using a single characterized marginal error rate during all tests within a given week or month or specific operating hours. 
     In the embodiment shown in  FIG. 6 , the retention threshold is updated  606 - 607  using a correction factor (f). First, the correction factor is determined as shown by step  606 . In one embodiment the correction factor is determined using the equation:
 
 f=|ΔR ( t,ts )* AF−ΔR ( t,ts ) CHAR |*α
 
     where α&gt;1 if ΔR(t,ts)*AF&gt;ΔR(t,ts) CHAR . Otherwise α≤1. 
     In the present embodiment the test characteristics stored in step  401  include one or more correction factor characterization value α. In one embodiment only a single correction factor characterization value α is stored. In another embodiment a plurality of correction factor characterization values α are stored and the correction factor characterization value α to be used to update the retention threshold is chosen based on the use case. 
     Correction factor characterization value α is a fitting parameter that is determined by testing NAND test chips and looking at how errors evolve over time, focusing on ECC codewords that deviate from the typical behavior. For example, when ts is 12 hours the testing can look at how many additional errors develop in a 12 hour period. More particularly, the testing could indicate that, on the average 20% more errors result. Single correction factor characterization value α is not an absolute value but rather it scales with the absolute number of errors coming from the read operation. Accordingly, depending on the use case, a single value may be sufficient for the life of nonvolatile memory system  100 . However, in other use cases multiple values of α are used, with a different α used after a predetermined number of P/E cycles have occurred or after a predetermined number of read cycles have occurred. In these embodiments a table is stored in step  401  that includes the α values and the index to be used for selecting the appropriate α value (e.g., number of P/E cycles or read cycles). 
     The retention threshold is updated  607  by multiplying the current retention threshold by the correction factor determined in step  606  and storing the results as the new correction factor. 
     The testing of steps  601 - 607  proceeds until all test codewords have been read  608 . The test then ends and normal operation continues as shown by step  609 . 
     Accordingly, the amount of correction corresponds to the variance between the online retention tests and the expected results, providing an updated retention threshold that has been corrected in proportion to the deviation between the calculated marginal error rate and the characterized marginal error rate. 
       FIGS. 7-12  illustrate the results of an exemplary test of a sample NAND device and illustrate the calculation of RT CHAR  and ΔR(t,ts) CHAR . In the test, codewords are stored in blocks of a test NAND device and are read at a time period (ts) and the number of errors in each codeword is determined. 
       FIG. 7  shows errors on the vertical axis and retention time on the horizontal axis and shows the highest error in each block as a function of retention time. In this specific example the time period ts is daily and the test is performed at a temperature of 85 degrees (F-85-2). 
     Following is an exemplary illustration as to how a characteristic retention threshold can be determined. In the present embodiment the first time represents the start of the test and is performed at a time (t=0). First, the number of errors in the codeword having the highest number of errors at an initial offline time (t=0) are determined. At the start of the test the codeword having the highest number of errors Worst(0) has 45 errors. 
     In this example, when the target retention time is 5 months, t=5 months and the highest number of errors Worst(t)=66. Using the equation ΔWorst(t)=Worst(t)−Worst(0) gives a ΔWorst(t=5 mo.)=21 errors. 
       FIG. 8  shows an exemplary plot of ΔWorst for the tested NAND devices tested in  FIG. 7 . It can be seen that ΔWorst increases with retention time. 
     When the maximum error correction capacity of the error correction code used to generate the codewords (ECC MAX ) is 100, the DeltaWorst retention threshold can be calculated using the equation: DWRT=ECC MAX −ΔWorst(t O =5 mo.) which gives a DWRT of 79. 
     The DWRT to be used for characterizing the NAND device may be reduced to account for statistical variations (die-to-die or intra-die). In one embodiment an additional margin (MM) is subtracted from DWRT to account for manufacturing variability. In one embodiment the ΔWorst is determined for a retention time that is greater than the target retention time. In one embodiment an additional month is added to the retention time. For example, adding one month gives 6 months retention. Referring now to  FIGS. 6 and 7  it can be seen that this gives a ΔWorst+MM=27 and a corresponding Delta Worst Retention Threshold Characterization value (DWRT CHAR ) of 73 errors. 
     In another embodiment the additional margin to be subtracted from to account for statistical variation is determined by calculating the change between successive calculations of ΔWorst that can be referred to as Delta-DeltaWorst or ΔΔWorst. Delta-DeltaWorst can be determined using the equation ΔΔWorst=ΔWorst(t+ts)−ΔWorst (t)).  FIG. 9 . shows ΔΔWorst for the ΔWorst values shown in  FIG. 7 . In one embodiment the additional margin is the greatest value of ΔΔWorst in a given time period. For example, in one embodiment a time period of between one month prior to the target retention time and one month after the target retention time is used and the highest ΔΔWorst in that time period is used as the additional margin. In the embodiment shown in  FIG. 9 , the highest ΔΔWorst between four months and six months is 9. Subtracting 9 to the calculated DWRT of 79 gives a Delta Worst Retention Threshold Characterization value (DWRT CHAR ) of 70 errors. 
     The retention threshold of the present invention is a function of the use case at given retention time. The methods and apparatus of the present invention are available to track how he flash NAND devices age while the SSD is active. The threshold is a function of ts. Accordingly, a different retention threshold is set depending on how frequently correctability is checked. In fact, the longer the ts, the lower the retention threshold should be. 
     It has been found that there is a feedback between online and offline parameters so as to enable a cross-correlation between the two. Because temperatures could be different between online and offline periods an acceleration factor AF is used to normalize measurement retention errors. Temperature is monitored by the controller and is an input value to the algorithm. 
       FIG. 10  shows an embodiment in which four blocks are tested, with the highest number of errors in each block plotted. Thus, the vertical axis shows the worst number of errors in each block and the horizontal axis shows retention time in months.  FIG. 10  illustrates a DeltaWorst retention threshold of 70 and highlights the issue of worst-error plots having a steep slope. When the worst-error plots have a steep slope, the difference between the worst error in any codeword and the DWRT CHAR  of 70, marked as Region S, is relatively large at an early retention time. Codewords starting in Region S were not a part of the characterization batch and their behavior was not measured. The issue is, to what extent should codewords starting in Region S be allowed? One way to deal with this issue is to further reduce the ΔWorst retention threshold. In one embodiment the ΔWorst retention threshold is reduced by an additional 3 errors, producing a DWRT CHAR  of 67. 
     The high-slope problem can also be addressed by using more than one RT CHAR  to represent the model use case. For example, a first RT CHAR  having a reduced value can be used for the first three months of retention and there will be no need to reduce the DWRT CHAR  after the end of the first three months. Therefore the DWRT CHAR  for months 4, 5 and 6 will remain at 70. In other embodiments more than two values of DWRT CHAR  can be used, depending on the characteristics of the model use case. In the present embodiment, when more than one DWRT CHAR  is used a table is stored in step  401  that includes DWRT CHAR  indexed by retention time. 
     Following is an exemplary calculation of ΔR(t,ts) using the test data of  FIG. 7 . Test codewords are read at a time t and the total number of errors in each codeword (#errors(t)) is determined. The determined total number of errors in each codeword is then stored. At a time t+ts the same test codewords are again read and the total number of errors in each codeword is determined (#errors(t+ts)). The determined total number of errors in each codeword at time t (#errors(t)) is then subtracted from the determined total number of errors in that codeword at time ts (#errors(t+ts)) to obtain a ΔR(t,ts) value for each tested codeword. The test is repeated for each successive time interval ts to obtain a marginal error profile that characterizes the use case modeled by the test criteria. 
       FIG. 11  shows a graph of ΔR(t,ts) for the NAND device tested in  FIG. 7  for a use case F85-2. Use case F85-2 is performed at a temperature of 85 degrees and a ts of one day. It can be seen that ΔR(t,ts) is both positive and negative for each time t. In this test 16 die were tested and 192 blocks per die were tested. 
     In one embodiment the ΔR(t,ts) value chosen to be ΔR(t,ts) CHAR  is based on the maximum and minimum ΔR(t,ts) at each time t.  FIG. 12  shows a graph of ΔR(t,ts) on the vertical axis and retention time on the horizontal axis and illustrates the maximum and minimum ΔR(t,ts) at each retention time for use case F85-2. 
       FIG. 13  shows a graph of ΔR(t,ts) on the vertical axis and retention time on the horizontal axis and illustrates the maximum |ΔR(t,ts)| at each retention time for use case F55-2 and use case F85-2. In the present embodiment the absolute value of each ΔR(t,ts) value is determined (|ΔR(t,ts)|) and the maximum |ΔR(t,ts)| value is plotted for each time t. 
     In the present embodiment a single ΔR(t,ts) CHAR  is chosen to be the highest maximum |ΔR(t,ts)| over the interval between t=0 and the time period t. For example, the ΔR(t,ts) CHAR  for the time of 10 days is 9 errors. For each ΔR(t,ts) CHAR  up to 20 days the ΔR(t,ts) CHAR  will remain at 10 errors. At 30 days ΔR(t,ts) CHAR  rises to 10 errors. For periods between 30 days and the end of specification retention time, the ΔR(t,ts) CHAR  will remain at 10 errors. 
     The ΔR(t,ts) CHAR  and corresponding time t for each ΔR(t,ts) CHAR  are stored (step  401 ). This can be, for example, a table with ΔR(t,ts) CHAR  and corresponding time (t) for every time (t) and incremented by is within the lifetime of the NAND devices  110  or until some earlier cutoff period such as, for example, the specification retention time. 
     Though  FIG. 4  describes using both online and offline testing, in one embodiment only offline testing is performed. More particularly, retention monitor  120  does not include an online test module  123  and steps  410 - 412  of  FIG. 4  are not performed. In an alternate embodiment only online testing is performed and retention monitor  120  does not include an offline test module  122  and steps  406 - 408  of  FIG. 4  are not performed. 
     In various embodiments, the system of the present invention may be implemented in a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC). 
     Though the method and apparatus of the present invention is described above with respect to a single level memory cell, it is within the scope of the present invention to extend the methods and apparatus of the present invention to MLC (multiple-level cell) devices, as would be evident to one of skill in the art. In this embodiment, the memory cells of NAND devices  150  are multi-level cells and the steps of  FIGS. 4 and 7-8  are performed using multi-level cells. 
     Although the invention has been described with reference to particular embodiments thereof, it will be apparent to one of ordinary skill in the art that modifications to the described embodiment may be made without departing from the spirit of the invention. Accordingly, the scope of the invention will be defined by the attached claims not by the above detailed description.