Data retention management for data storage device

Data is received for storage in at least one memory of a Data Storage Device (DSD) and metadata associated with the received data is generated. The received data and the generated metadata are stored in the at least one memory and the retention of the received data is managed based on the generated metadata.

CROSS REFERENCE TO RELATED APPLICATION

This application is related to co-pending application Ser. No. 14/871,054 filed on Sep. 30, 2015, and entitled “MEDIA REGION MANAGEMENT FOR DATA STORAGE DEVICE” by Robert Horn, which is hereby incorporated by reference in its entirety.

BACKGROUND

Data Storage Devices (DSDs) are often used to record data onto or to reproduce data from a storage media such as a rotating magnetic disk or a solid-state memory. In some cases, DSDs may be used to archive or store data received from one or more sensing devices such as, for example, video cameras, accelerometers, microphones, or various other sensors. The data received from such sensing devices may arrive at the DSD as a continuous stream of data and eventually consume large amounts of the available storage capacity in the DSD. Since the storage capacity of the DSD is limited, new data may simply overwrite older data without any consideration of the importance of the older data.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are set forth to provide a full understanding of the present disclosure. It will be apparent, however, to one of ordinary skill in the art that the various embodiments disclosed may be practiced without some of these specific details. In other instances, well-known structures and techniques have not been shown in detail to avoid unnecessarily obscuring the various embodiments.

System Overview

FIG. 1shows an example of Data Storage Device (DSD)106which receives data from devices101and104according to an embodiment. Devices101and104can include sensing devices such as, for example, a video camera, accelerometer, microphone, motion sensor, temperature sensor, humidity sensor, light sensor, or other type of sensing device. In addition, DSD106communicates with remote DSD109via network105, which can include a Local Area Network (LAN), Wide Area Network (WAN), or the Internet.

As shown in the example embodiment ofFIG. 1, DSD106includes Non-Volatile Memory (NVM) in the form of rotating magnetic disk150and Non-Volatile Solid-State Memory (NVSM)128. In other embodiments, DSD106can include other NVM media such as magnetic tape. In this regard, one or both of disk150and NVSM128can be omitted or replaced by a different NVM media. For example, NVSM128may be omitted in some embodiments so that the NVM of DSD106includes only disk storage media. In yet other embodiments, each of disk150or NVSM128can be replaced by multiple Hard Disk Drives (HDDs) or multiple Solid-State Drives (SSDs), respectively, so that DSD106includes pools of HDDs and/or SSDs.

DSD106includes controller120which includes circuitry such as one or more processors for executing instructions and can include a microcontroller, a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), hard-wired logic, analog circuitry and/or a combination thereof. In one implementation, controller120can include a System On a Chip (SoC).

Interface126is configured to interface DSD106with devices101and104, and with network105and may interface using, for example, Ethernet or WiFi, and/or one or more bus standards. As will be appreciated by those of ordinary skill in the art, interface126can be included as part of controller120.

The components ofFIG. 1may or may not be physically co-located. In this regard, devices101or104may be located remotely from DSD106. Those of ordinary skill in the art will also appreciate that other embodiments can include more or less than those elements shown inFIG. 1and that the disclosed processes can be implemented in other environments. For example, other embodiments can include a different number of devices providing data to DSD106.

In the example ofFIG. 1, DSD106includes sensor122, which can also provide data for storage in at least one NVM of DSD106. Sensor122can include, for example, an accelerometer or a temperature sensor to detect an environmental condition. In other embodiments, sensor122can be external to DSD106as with devices101and104.

InFIG. 1, disk150is rotated by a spindle motor (not shown) and head136is positioned to read and write data on the surface of disk150. In more detail, head136is connected to the distal end of actuator130which is rotated by Voice Coil Motor (VCM)132to position head136over disk150to read or write data in tracks152. A servo system (not shown) of controller120controls the rotation of disk150with SM control signal38and controls the position of head136using VCM control signal34.

As will be appreciated by those of ordinary skill in the art, disk150may form part of a disk pack including multiple disks that are radially aligned with disk150. In such implementations, head136may form part of a Head Stack Assembly (HSA) including heads arranged to read data from and write data to a corresponding disk surface in the disk pack.

DSD106may also include NVSM128for storing data. While the description herein refers to solid-state memory generally, it is understood that solid-state memory may comprise one or more of various types of memory devices such as flash integrated circuits, Chalcogenide RAM (C-RAM), Phase Change Memory (PC-RAM or PRAM), Programmable Metallization Cell RAM (PMC-RAM or PMCm), Ovonic Unified Memory (OUM), Resistive RAM (RRAM), NAND memory (e.g., single-level cell (SLC) memory, multi-level cell (MLC) memory, or any combination thereof), NOR memory, EEPROM, Ferroelectric Memory (FeRAM), Magnetoresistive RAM (MRAM), other discrete NVM (non-volatile memory) chips, or any combination thereof.

As shown inFIG. 1, NVSM128stores metadata12which is associated with the data received from devices101and104or from sensor122. As discussed in more detail below, controller120generates metadata12for managing the retention or a size of the received data after it has been stored in NVM of DSD106. In this way, it is ordinarily possible to determine which data should be kept and which data can be deleted, compressed or transferred to another DSD when more space is needed in the NVM of DSD106. As used herein, compression can include a reduction in a sampling rate of the data, such as the removal of frames from video data to result in a lower quality video or the application of one or more lossless or lossy compression algorithms as known in the art.

DSD106also includes memory140, which can include, for example, a Dynamic Random Access Memory (DRAM). In some embodiments, memory140can be a volatile memory used by DSD106to temporarily store data. In other embodiments, memory140can be an NVM that can be quickly accessed. Data stored in memory140can include data read from NVM, data to be stored in NVM, instructions loaded from firmware10for execution by controller120, and/or data used in executing firmware10.

In operation, interface126receives data from devices101and104via interface126for storing the data in NVM of DSD106. Controller120may buffer the received data in memory140before storing the data on disk150or in NVSM128.

For data to be written on disk150, a read/write channel (not shown) of controller120may encode the buffered data into write signal36which is provided to head136for magnetically writing data on disk150. A servo system of controller120can provide VCM control signal34to VCM132to position head136over a particular track for writing the data.

In addition, controller120may need to read data from NVM to manage retention of the stored data or to provide the stored data to another device. To read data from disk150, the servo system positions head136over a particular track on disk150. Controller120controls head136to magnetically read data stored in the track and to send the read data as read signal36. A read/write channel of controller120can then decode and buffer the data into memory140for use by controller120or for transmission to another device via interface126.

For data to be stored in NVSM128, controller120receives data from interface126and may buffer the data in memory140. In one implementation, the data is then encoded into charge values for charging cells (not shown) of NVSM128to store the data.

To access data stored in NVSM128, controller120in one implementation reads current values for cells in NVSM128and decodes the current values into data that can be transferred to a host via interface126.

FIG. 2depicts an example of metadata12that is generated by DSD106for data received from sensor122and devices101and102according to an embodiment. As shown inFIG. 2, metadata12includes an address for the data associated with the metadata, a time the data was received, an indication of a source of the received data, an indication of whether the data is related to other data stored in DSD106, and a priority level of the data. Other implementations of metadata12may include different metadata than that shown inFIG. 2.

As discussed below with reference toFIG. 3, controller120may generate metadata12as part of a preprocessing of the data when the data is received by DSD106. Metadata12can then later be used in managing retention of the received data in accordance with one or more retention policies of DSD106.

InFIG. 2, each row represents a different instance of receiving data. The address of metadata12can correspond either directly or indirectly to a location where the associated data is stored in the NVM of DSD106. The time can indicate when the data was received by DSD106. The source indicates which device (e.g., sensor122or devices101or104) provided the data to DSD106.

The related indicator indicates whether the data is related to other data received by DSD106from a different source. For example, in a case where device101is a video camera and device104is a microphone in the same room as device101, audio data received from device104may be indicated as being related to video data received from device101by using a flag or other indicator in metadata12for data received from device104. In other implementations, data received from two or more devices may be indicated as being related by assigning each set of related devices with a particular value in metadata12so that metadata12can be sorted by one or more groups of related devices.

The priority indicator in metadata12indicates a priority level for the associated data. In one implementation, the priority level may be based on data received from another device indicating a high priority for the data. For example, sensor122can be a motion sensor and device101can be a video camera so that when sensor122detects motion, the data received from device101is indicated as having a higher priority over data from when there is no motion detected by sensor122.

The devices used with DSD106are not limited to the examples provided above. For example, devices101or104in some implementations can include sensors located in machinery such as an engine or sensors that are used as part of a manufacturing process. Generally speaking, some embodiments of the invention may be applied to scenarios where a large amount of data is constantly being generated and for which there may be some analytical or historical reference value in later assesses, but the storage space is limited.

Data Retention Management Examples

FIG. 3is a block diagram illustrating preprocessing and storage of received data according to an embodiment. In the example ofFIG. 3, controller120performs a preprocessing of data streams and sensor data to generate metadata and processed data that is stored in an NVM of DSD106.

The generated metadata can include, for example, one or more of the types of metadata shown inFIG. 2for metadata12such as an indication of when the data was received, whether the received data is related to other data stored in NVM, the source of the received data, or a priority level for the received data.

In addition, the metadata can include a result of an analysis of the data performed as part of the preprocessing. In one example, the preprocessing may include an analysis of video data to determine a brightness of the video data such that metadata12includes an indication of when the brightness exceeds or falls below a brightness threshold. In such an example, controller120may later use this metadata to make decisions on which video data to keep and delete portions of the video data that may be too dark or too bright.

The generated metadata for the received data allows controller120to manage retention of the data or a data size of the data without having to rely on processing from an external host or a selection made by a user of DSD106. By using the generated metadata, controller120can ordinarily be more selective in the data it retains or compresses so that certain data is retained or preserved longer than other data that may not be as important in light of retention policies implemented by DSD106.

FIG. 4is a flowchart for a retention management process that can be performed by controller120executing firmware10according to an embodiment. In block402, controller120receives data from storage in at least one memory of DSD106such as disk150or NVSM128. The data may be received as a continuous stream of data or as a burst of data from a sensing device such as device101or device104, or the data may be more discrete such as an isolated value from sensor122.

In block404, controller120generates metadata12that is associated with the data received in block402. For example, the metadata may include a timestamp indicating when the data is received, an indication of whether the data is related to other data stored in DSD106, an indication of the source of the data, a priority level of the data, or a result of an analysis of the data.

In block406, the received data and the generated metadata is stored in the at least one memory of DSD106(e.g., disk150or NVSM128). In some cases, the generated metadata may be stored with the received data such as within or near the same track152on disk150so as to provide nearly simultaneous access of the received data with its associated metadata. In other implementations, the generated metadata may be stored in a different location from its associated data. Such an implementation may be possible inFIG. 1where metadata12is stored in NVSM128and the associated data can be stored on disk150. Storing metadata12and its associated data in different storage media can in some cases allow for a more efficient use of different types of storage media.

In one example, disk150can include overlapping tracks that are generally sequentially written using Shingled Magnetic Recording (SMR) that is well suited for archiving large amounts of streaming data. In such an example, a data steam from a video camera can be sequentially written in the overlapping tracks with little movement of head136. NVSM128may then be used to store associated metadata that is generally smaller in size and may need to be accessed independently when evaluating the metadata to determine which video data should be kept when a remaining storage capacity of disk150reaches a high capacity threshold.

In block408, controller120manages retention of the received data based on the generated metadata. Controller120may manage the retention of the received data by performing a selective retention action such as, for example, deletion of a portion of the data, compression of a portion of the data, or transferring a portion of the data to another storage device, such as remote DSD109.

In block410, controller120optionally uses the metadata generated in block404to analyze the received data or to generate additional metadata. For example, metadata12may be used to identify data received within a certain time period or data having a higher priority. The received data may also be further processed by controller120in block410to generate additional metadata. For example, controller120may analyze data received from device101within a certain timeframe to identify portions of the data including a facial recognition match and generate additional metadata indicating the portions of the data that include a match. In another example, controller120may use metadata12to identify data older than a particular time and compress such data, and generate additional metadata indicating that the data has been compressed.

FIG. 5is a block diagram illustrating the enforcement of at least one retention policy according to an embodiment. InFIG. 5, controller120detects one or more trigger conditions that cause controller120to evaluate metadata12using at least one retention policy. The one or more retention policies can consider, for example, whether data has been stored for longer than a certain period of time, a priority level for data stored longer than a certain period of time, or a relation of data to other data stored in NVM. Based on the evaluation, controller120performs one or more selective retention actions on the data stored in at least one memory of DSD106.

The trigger conditions inFIG. 5can include, for example, an elapse of a predetermined amount of time, an input received from an external device, the receipt of new data for storage, or at least one memory reaching a high threshold storage capacity. In the case where the trigger condition is an elapsed amount of time, controller120may be configured to periodically evaluate the data stored in NVM using the generated metadata. In the case where the evaluation of the data is triggered by receiving an input from an external device, a host device connected to DSD106either directly through interface126or remotely via network105may command DSD106to evaluate its metadata using a retention policy to determine if space can be freed up in the NVM of DSD106.

In the case where a high threshold storage capacity is reached, controller120may monitor the remaining available space for storing data in NVM of DSD106. When the high threshold storage capacity is reached, controller120may perform one or more selective retention actions until a low threshold storage capacity is reached to generally maintain the amount of data stored in NVM between the low and high thresholds. As discussed above, the selective retention action can include, for example, deleting a portion of the data, compressing a portion of the data, or transferring a portion of the data to another DSD.

FIG. 6is a flowchart for a selective retention process that can be performed by controller120executing firmware10according to an embodiment. In block602, controller120detects a trigger condition such as an elapse of a predetermined amount of time, an input received from an external device, or at least one memory (e.g., disk150or NVSM128) reaching a high threshold storage capacity.

In block604, controller120evaluates metadata12using at least one retention policy. In one example, controller120may identify data that has been stored longer than a threshold amount of time and then identify the data that has been stored longer than the threshold amount of time and not indicated as having a high priority.

In block606, controller120performs a selective retention action on the received data stored in NVM based on the evaluation performed in block604. Using the example given above, controller120may delete or compress any data identified as being stored longer than the threshold amount of time and not indicated as having a high priority.

FIG. 7is a flowchart for a selective retention process triggered by reaching a high threshold storage capacity according to an embodiment. The example process ofFIG. 7can, for example, be performed by controller120executing firmware10as a background activity.

In block702, controller120determines that a high threshold storage capacity has been reached. This can be part of a storage capacity monitoring performed by controller120and can include one or more NVMs (e.g., disk150or NVSM128) of DSD106storing a certain amount of data. The high threshold storage capacity may be for the data received from a particular device (e.g., device101) or may be for the data received from all of the devices providing data for storage in DSD106(e.g., sensor122and devices101and104). In other embodiments, the selective retention process may be triggered by other trigger conditions such as an external input or an elapsed amount of time.

In block704, controller120determines whether any of the data received by DSD106and stored in NVM has been stored for longer than or equal to a first time period. For example, controller120may evaluate timestamps of metadata12to determine if there is any data stored in NVM that was received before two weeks ago.

If it is determined in block714that there is data stored longer or equal to the first time period, controller120in block706compresses such data using a first compression method. The first compression method can include one or more known compression methods. In block714, controller120determines whether a low threshold storage capacity has been reached by compressing the data in block706. If so, the process ends in block716.

On the other hand, if the low threshold storage capacity has not been reached in block714, the process returns to block704. In other embodiments, block714may be omitted such that the process ofFIG. 7ends after the performance of a retention action such as the compression, deletion, or transferring of a portion of the data stored in DSD106. In such embodiments, the process ofFIG. 7could then be initiated again if the high threshold storage capacity has been reached in block702.

If there is not any data received by DSD106that has been stored longer than or equal to the first time period in block704, controller120in block708determines whether any of the received data has been stored longer than a second time period that is less than the first time period. As with block704, controller120can evaluate metadata12to determine whether any data has been stored longer than the second time period.

If it is determined in block708that there is data that has been stored longer than the second time period, controller120in block710compresses such data using a second compression method. The second compression method can, for example, use a lower compression ratio than a compression ratio used in the first compression method of block706so that data that has been stored longer is compressed more in block706than data compressed in block710since this data has not been stored for as long. In other implementations, the second compression method can use a higher compression ratio than used in the first compression method or may include a different algorithm for compressing data. In yet other implementations, the second compression method can include a second instance of compression using the first compression method of block706to further compress previously compressed data. In some embodiments, a compression method may include operations such as selective deletion and/or migration to another storage location.

After compressing the data in block710, controller120determines whether the low capacity storage threshold has been reached in block714. If so, the process ends in block716. If not, the process returns to block704for further data retention management.

If it is determined in block708that there is not any data that has been stored for longer than the second period of time, controller120in block712deletes, compresses, or transfers an oldest portion of the data received by DSD106until the low threshold storage capacity is reached. The process ofFIG. 7then ends in block716.

FIGS. 8A and 8Bprovide a flowchart for a selective retention process that is triggered by the receipt of new data according to an embodiment. The process ofFIGS. 8A and 8Bcan be performed by controller120executing firmware10.

In block802, controller120receives data for storage in at least one memory of DSD106. In block804, controller120determines whether the size of the data stored in the at least one memory together with the new data received in block802is less than a critical threshold storage capacity. In one implementation, the critical threshold storage capacity can be a data capacity at which DSD106can no longer store new data in NVM. In other implementations, the critical threshold storage capacity can be a certain capacity before reaching the point when DSD106can no longer store new data in NVM.

In other embodiments, controller120may only determine the size of the data already stored in NVM rather than the size of the new data together with the data already stored in NVM. In addition, the size of the data considered in block804may be for the data received from a particular device or may be a total data size including data received from all devices.

If it is determined that the data size in block804is less than the critical threshold storage capacity, controller120checks in block806whether data received from another device indicates a priority status for the data received in block802. In some implementations, the receipt of other data that is related to the data received in block802can indicate a higher priority of the received data. For example, an input received from a device such as an input indicating that a doorbell was pressed can cause controller120to determine that video data received from a camera has a higher priority for a time period around the receipt of the input indicating the doorbell was pressed.

If it is determined that the data from the other device indicates a priority status, controller120in block808generates priority status metadata for the data received in block802. In the example of metadata12inFIG. 2, this can include setting a flag indicating a high priority for the received data.

The new data is stored in the at least one memory in block810. Metadata associated with the new data, such as any metadata generated in block808, may also be stored in the at least one memory in block810.

If it is determined in block804that the size of the stored data and the new data is not less than the critical threshold storage capacity, controller120determines in block812whether the oldest data stored in the at least one memory is older than an age deletion threshold. In this regard, controller120can use metadata12to determine the age of the data stored in the at least one memory. If there is data older than the age deletion threshold, controller120deletes the oldest data in block814and the process returns to block804to determine whether the size of the stored data and the new data received in block802is less than the critical threshold storage capacity.

If it is determined in block812that the oldest data stored in the least one memory is not older than the age deletion threshold, controller120in block816determines whether there is any uncompressed data that has been stored longer than an age compression threshold. Metadata12may be used to identify data that has been stored longer than the age compression threshold. Analysis of the data or analysis of metadata12indicating the compression of data can be used to determine whether there is any uncompressed data that has been stored longer than the age compression threshold.

If it is determined in block816that there is uncompressed data that is older than the age compression threshold, controller120in block818compresses the oldest uncompressed stored data and the process returns to block804.

If there is no uncompressed data that is older than the age compression threshold in block816, the process proceeds to block820inFIG. 8B. In block820, controller120determines whether there is any data stored in the at least one memory that is older than an age sampling threshold that has a sampling rate greater than a predetermined sample rate value. In one example, the stored data can include video data and controller120in block820can determine whether any of the data that is older than a certain age has a frame rate greater than a particular frame rate or predetermined sample rate value.

If it is determined that there is data meeting the age and sample rate criteria of block820, controller120in block822deletes the oldest data with a sampling rate greater than the sample rate value. In other embodiments, controller120in block822may instead compress or reduce the sample rate of the oldest data.

If there is no data that meets the age and sample rate criteria in block820, controller120in block824determines whether there is any stored data that does not have a priority status that is older than a second age deletion threshold. If so, controller120in block826deletes the oldest non-priority data stored in the at least one memory.

On the other hand, if there is no data meeting the criteria in block824, controller120in block828deletes the oldest stored data until the size of the new data and the stored data is less than the critical threshold storage capacity. The deleted data may, for example, include the deletion of data starting with the oldest data and continuing toward more recently stored data until falling below the critical threshold storage capacity. In other embodiments, the deleted data may include a predetermined amount of data or the oldest data within a predetermined timeframe, such as the deletion of all data older than two weeks. The process ofFIGS. 8A and 8Bthen returns to block806inFIG. 8Ato determine whether data received from another device indicates a priority status of the new data received in block802.

By using metadata generated by DSD106, it is ordinarily possible for DSD106to consider other factors in addition to or in place of only considering an age of the data in determining which data to keep or compress. In addition, the metadata generated by DSD106can also allow DSD106to manage the retention of data on its own without the need for involvement of a host.

Other Embodiments

Those of ordinary skill in the art will appreciate that the various illustrative logical blocks, modules, and processes described in connection with the examples disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. Furthermore, the foregoing processes can be embodied on a computer readable medium which causes a processor or computer to perform or execute certain functions.

To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, and modules have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those of ordinary skill in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The activities of a method or process described in connection with the examples disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The steps of the method or algorithm may also be performed in an alternate order from those provided in the examples. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable media, an optical media, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an Application Specific Integrated Circuit (ASIC).

The foregoing description of the disclosed example embodiments is provided to enable any person of ordinary skill in the art to make or use the embodiments in the present disclosure. Various modifications to these examples will be readily apparent to those of ordinary skill in the art, and the principles disclosed herein may be applied to other examples without departing from the spirit or scope of the present disclosure. The described embodiments are to be considered in all respects only as illustrative and not restrictive and the scope of the disclosure is, therefore, indicated by the following claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.