Retention management for data streams

The described technology is generally directed towards managing data retention policy for stream data stored in a streaming storage system. When a request to truncate a data stream from a certain position (e.g., from a request-specified stream cut) is received, an evaluation is made to determine whether the requested position is within a data retention period as specified by data retention policy. If any data prior to the stream cut position (corresponding to a stream cut time) is within the data retention period, the truncation request is blocked. Otherwise truncation from the stream cut point is allowed to proceed/is performed. Also described is handling automated (e.g., sized based) stream truncation requests with respect to data retention.

TECHNICAL FIELD

The subject application relates generally to data storage, and, for example, to a technology that manages retention policy in storage systems that implement data streams for storing and serving continuous and unbounded data, and related embodiments.

BACKGROUND

Contemporary data storage systems, such as DELL EMC's PRAVEGA system/data storage service, store data in a storage abstraction referred to as a stream. A stream is identified with a name, and can store continuous and potentially unbounded data; more particularly, a stream comprises a durable, elastic, append-only, sequence of stored events. One stream may be divided into one or more segments, such as based on the routing keys associated with the events. Older stream data can be aggregated into chunks and written to an object storage system (e.g. Dell EMC's ECS data storage system) or to a file storage system (e.g. DELL EMC's ISILON data storage system).

New events are added to a tail (front) of a stream. As can be readily appreciated, PRAVEGA is thus ideal for IoT (Internet of Things) data, where devices/sensors may generate thousands of data points per second. Notwithstanding, PRAVEGA may be highly beneficial for storing data corresponding to more traditional workloads, such as financial trading data that regularly changes.

Although a stream is potentially unbounded, storage resources are finite. PRAVEGA provides ways to cut a stream short, including an automatic data expiration feature and an explicit truncate call. When cut, the events are deleted from a head (back) of a stream. Not all stream data can simply be truncated, however, as data retention policies need to be followed for some types of data, typically for regulatory compliance or business reasons.

DETAILED DESCRIPTION

Various aspects of the technology described herein are generally directed towards implementing data retention management in stream-based data storage systems. In one aspect, a retention policy (that is, corresponding to a retention period) is specified for individual streams, or possibly multiple streams. For multiple streams, a retention policy can be specified at higher levels, e.g. the system level or a scope (namespace) level, with the retention policy is propagated to the streams within the system or the scope.

In general, as described herein retention management logic blocks attempts to delete any data that is under retention from a stream. Deletion of data, which can be considered truncation of the stream's older data (events), is allowed by the retention management logic when the data to be deleted is not under a retention policy and/or no longer within the retention period associated with the stream.

It should be understood that any of the examples herein are non-limiting. For instance, some of the examples are based on PRAVEGA data storage technology; however virtually any stream-based data storage system may benefit from the technology described herein. As a more particular example, instead of tracking time for each event written in PRAVEGA, a “stream cut object” or simply a “stream cut” refers to a specific position in the data stream that is generated by a data writer; older data needs to be deleted from a stream cut boundary (rather than arbitrarily), unless any of data to be deleted within the retention policy period as described herein. A stream cut is associated with a time value. Other data stream storage systems can use a similar concept, or can use timestamped data; notwithstanding, as will be understood, the technology described herein can be applied to any stream-based data storage mechanism that tracks position/time of stored data. Thus, any of the embodiments, aspects, concepts, structures, functionalities or examples described herein are non-limiting, and the technology may be used in various ways that provide benefits and advantages in computing and data storage in general.

Reference throughout this specification to “one embodiment,” “an embodiment,” “one implementation,” “an implementation,” etc. means that a particular feature, structure, or characteristic described in connection with the embodiment/implementation is included in at least one embodiment/implementation. Thus, the appearances of such a phrase “in one embodiment,” “in an implementation,” etc. in various places throughout this specification are not necessarily all referring to the same embodiment/implementation. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments/implementations.

FIG.1shows a streaming data storage system100that includes an event stream102comprising a data stream of events, with event writes being appended from a head towards a tail direction. Note that for simplicity,FIG.1does not illustrate the concept of stream segments, (which are shown herein with reference toFIGS.2and3).

In general, an event writer such as a streaming application program104sends data writes (events)106to the streaming data storage system100for appending to the event stream102. As described herein, the application program104can also send truncate requests108to the streaming storage system100, such as by identifying the stream for which truncation is requested, and specifying a particular stream cut.

More particularly, as set forth above, a position in a stream at an event boundary is specified using a stream cut. The application program104can request association of a stream cut with a stream, as represented by block110. Instead of, or in addition to, application-specified stream cuts, stream cuts112can be created automatically, such as periodically, by automated program(s)114or the like. The automated program(s) can also send truncate requests115, such as when the stream reaches a size capacity limit.

The stream cuts for a stream can be considered a series of stream cuts. In one or more implementations, the stream cuts may be maintained as an auxiliary system stream116associated with the main data stream, e.g., event stream102inFIG.1. A time value (obtained from the system time) can be associated with each stream cut, such as corresponding to when a given stream cut object is created. For example, the correct system time (block118) can be obtained via use of the NTP (Network Time Protocol).

Turning to aspects related to data retention,FIG.1exemplifies retention enforcement logic120that enforces retention policy122, corresponding to a retention time period, associated with the event stream102. The retention enforcement logic120can be incorporated into the streaming storage system100as shown inFIG.1, or otherwise coupled thereto. As set forth herein, a retention policy122can be associated with an individual stream, multiple streams within a specified scope, or possibly the entire set of streams of a streaming storage system.

In general, before truncating any data, the retention enforcement logic120evaluates whether the truncation request is allowable based on the retention policy122. For a retention policy122that specifies infinite data retention, it is straightforward for the retention management logic120to block any request, either a manual or automatic attempt, to truncate the stream.

For a data finite retention period, data retention, which is time-based by its nature, has to have some notion of time at the stream level. When a stream is to be truncated, the position in the stream at an event boundary is specified via a stream cut identified in the request. The events that are ahead of the stream cut (that is, older events), are removed from the stream.

As described herein, each stream cut has an associated time, namely its time of creation in one implementation. When a stream is requested to be truncated, the retention enforcement logic120evaluates whether the time of the specified stream cut is within the retention period. If not, the truncation is allowed to occur; otherwise the retention enforcement logic120blocks the request.

FIGS.2and3show examples of truncation requests being blocked (FIG.2) or allowed to proceed (FIG.3). Note thatFIGS.2and3show a segmented stream of events, in that the segments are represented as separate rectangles that make up the stream202.

FIGS.2and3also illustrate the concept of a stream of periodically created stream cuts. The three stream cuts shown are created at times t1, t2, and t3, wherein the times are used as metadata for the stream cuts. A relatively considerable retention period is specified for the stream; as is shown, for the retention period up to the current retention end time of “now”, the retention period starts somewhere between times t1and t2. The stream cannot be truncated using the stream cut Sc2corresponding to time t2(or SC3corresponding to time t3), because doing so would impact (delete) data that is still under retention.

Thus, inFIG.2when a truncation request208is received, the specified stream cut (SC2) is evaluated to determine whether the specified stream cut (SC2) is within the retention period. Accordingly, as described herein and as shown inFIG.2, the truncate request208is blocked (block230)

One way to determine whether the time of the specified stream cut is within the retention period is to calculate a retention end time. The retention end time can be calculated as retention_end_time=stream_cut.time+stream.retention_period, that is, the retention end time equals the retention period configured for the stream added to the time associated with the stream cut. The retention end time is then compared to the current system time, and if the retention end time is in the future, stream truncation is blocked, otherwise stream truncation is allowed. Note that the main stream (e.g., the event stream102) is truncated, the auxiliary stream116of stream cuts can be truncated as well, removing from the auxiliary stream the specified stream cut and any stream cut or stream cuts prior to the specified stream cut.

As can be readily appreciated, other calculations can provide the same decision result. For example, the current system time minus the retention period equals a retention start time; then, if the time associated with the specified stream cut time is before the retention start time, truncation is allowed to proceed, otherwise truncation is blocked.

As shown inFIG.3, however, a truncate request308is received that specifies an earlier stream cut, SC1, corresponding to the time t1. As is understood, such as based on one of the above calculations, it is determined that the specified stream cut is not within the retention period. This means that the stream can be truncated from the events prior to time t1/stream cut SC1. As is represented inFIG.3by the before and after truncate request processing operations, seen, the data (shown shaded before processing) to the left of time t1/stream cut SC1is truncated (not shown) after processing. The truncated stream202′ is reduced in size relative to the stream202. Note that stream cut SC1including its metadata, and any other earlier stream cuts can be removed from the auxiliary stream unless deemed desirable to keep for another reason, e.g., stream cut SC1can be used to track a new starting time for the remaining stream202′.

Turning to another aspect, data retention can operate with size-based data expiration, such as supported in PRAVEGA. With size-based data expiration, an administrator can specify a maximal size for a stream data in the terms of capacity. In general, size-based data expiration automatically cuts the stream short to make the data stream's capacity footprint stay within the limit.

However, when there is also a retention period defined for the stream, the size-based expiration feature needs to be configured using an assumption about the data ingestion rate, because data within a stream cannot be expired before its retention ends. Note that data expiration is not allowed for data retained in the retention period for compulsory legal/regulatory purposes or the like, and thus retention policy needs to override any size-based data expiration; (in contrast, if data is only retained for a non-compulsory business purpose such as data analytics, a business decision can be made as to whether size-based expiration can override retention policy).

As graphically represented inFIG.4, the data ingestion rate (size to time) is around the expected ‘max” value until time t1. Before this time, whenever it is time to cut the stream short because its size reaches the capacity limit (max), the data to be deleted has reached the end of its retention so it can be deleted. However, after time t1the data ingestion rate grows above the assumed value. After time t1, when it is desired to cut the stream short because it has reached the capacity limit, at least a part the data to be deleted is still under retention, so it cannot be deleted. The system is forced to make delayed a data expiration. As a result, the stream may have a capacity footprint over the limit, possibly for a long period of time.

Thus, when the data ingestion rate considerably exceeds the assumed “max” value, there can be a conflict between data expiration and data retention. Assuming that in this example data retention takes priority over data expiration, the capacity footprint of the stream may considerably exceed the desired “max” limit. This possibility needs to be considered during storage capacity planning whenever data retention policy has priority over data expiration policy.

FIG.5is a flow diagram showing example operations that may be performed by retention enforcement logic, beginning at operation502where a request to truncate a stream (e.g., identified in the request) from a specified stream cut is received. Operation504obtains the stream retention period, if any, specified in any associated retention policy for this stream.

At operation506, if there is no retention period, operation514is performed to truncate the stream from the request. Alternatively, if there is an infinite retention period (the branch represented by the dashed line), operation516is performed to block the request, possibly notifying the requestor as to why the request was blocked.

As a likely more typical alternative for a retention policy, for a finite retention period, operation508is performed to obtain the stream cut time associated with the specified stream cut. Operations510and512represent determining whether the stream cut time is within the retention period, which in this example use the “Retention End Time” calculation (operation510) described herein, along with a comparison at operation512as to whether the retention end time is greater than the current system time, that is, the retention end time is in the future. If the retention end time is not in the future (the stream cut time is not within the retention period), truncation is allowed to proceed, as represented via operation514. Otherwise retention is blocked at operation516.

Note that in addition to blocking at operation516, an option is for the system to look for an earlier stream cut that is not within the retention period. The requesting entity can be notified of such an earlier stream cut, e.g., a response code can indicate “blocked because requested stream cut Y is in retention period, but truncation is available from an older stream cut X;” this gives the requestor some context for a subsequent truncation decision, instead of having the requesting entity keep trying to truncate from earlier and earlier stream cuts. Another alternative is to automatically truncate from the most recent stream cut (if any) that is no longer within the retention period, and only block if no earlier stream cut is available. The requesting entity can be notified of the truncation from the different stream cut from the stream cut that was requested.

FIG.6shows another alternative, which may be particularly useful with sized-based data expiration (e.g., received via operation502) or other automated stream truncation. InFIG.6, most of the operations are similar to those ofFIG.5, and thus are not described again for purposes of brevity. However, operation616is different, in that instead of blocking, the requested truncation operation is scheduled, for example for as soon as the stream cut time of the requested stream cut is no longer within the retention period, e.g., based on the retention end time. It is also feasible to automatically cut the stream from the most recent stream cut that is no longer within the retention period, even if not specified. Indeed, a useful call may be made to “truncate based on size-based expiration” that triggers a truncation operation from the most recent stream cut that is no longer within the retention period, as well as schedules a truncation operation to occur from any specified stream cut as soon as that specified stream cut is no longer within the retention period window.

One or more aspects can be embodied in a system, such as represented inFIG.7, and for example can comprise a memory that stores computer executable components and/or operations, and a processor that executes computer executable components and/or operations stored in the memory. Example operations can comprise operation702, which represents receiving a request to truncate a data stream of a streaming data storage system, the request associated with a stream cut maintained for the data stream. Operation704represents obtaining a retention period associated with the data stream. Operation706represents obtaining a stream cut time associated with the stream cut. Operation708represents determining whether the stream cut time is within the retention period. Operation710represents, in response to the stream cut time being determined not to be within the retention period, truncating the data stream based on the stream cut.

Determining whether the stream cut time is within the retention period can comprise determining a retention end time based on the stream cut time and the retention period, and determining whether the retention end time is later than a current system time.

Further operations can comprise, in response to the stream cut time being determined to be within the retention period, blocking the request. The request to truncate the data stream can be received from a size-based expiration requestor, and blocking the request can correspond to increasing a capacity footprint of the data stream beyond a pre-specified size limit. Further operations can comprise scheduling a delayed truncation operation in response to the increasing the capacity footprint of the data stream. Scheduling the delayed truncation operation can comprise scheduling the delayed truncation operation to occur at a stream cut time that is not within the retention time period.

The stream cut can be a second stream cut, and the stream cut time value cam be a second stream cut time value that is after a first stream cut time value of a first stream cut earlier in time than the second stream cut, and further operations can comprise, in response to determining that the second stream cut time value is within the retention period, selecting the first stream cut, and determining whether the first stream cut time value is within the retention period, and in response to determining that the first stream cut time is not within the retention period, truncating the data stream based on the first stream cut.

Further operations can comprise maintaining an auxiliary stream comprising stream cut data for the stream cut in association with the data stream. Further operations can comprise in response to the stream cut time being determined not to be within the retention period, truncating the auxiliary stream based on the stream cut.

One or more example aspects, such as corresponding to example operations of a method, are represented inFIG.8. Operation802represents determining, by a streaming data storage system comprising a processor, whether a request to truncate data stream events in a data stream that are prior to a specified stream cut is allowable; the determining can comprise obtaining a retention period associated with the data stream (operation804), obtaining a stream cut time associated with the specified stream cut (operation806), and evaluating the stream cut time relative to the retention period to determine whether the stream cut time is within the retention period or whether the stream cut time is not within the retention period (operation808). Operation810represents, in response to determining that the stream cut time is within the retention period, determining that the request to truncate the data stream events is not allowable. Operation812represents, in response to determining that the stream cut time is not within the retention period, determining that the request to truncate the data stream events is allowable, and truncating the data stream based on the stream cut in response to the request.

Aspects can comprise, in response to determining that the request to truncate the data stream events is not allowable, blocking the request.

Aspects can comprise, in response to determining that the request to truncate the data stream events is not allowable, scheduling a future truncation operation based on the specified stream cut.

Evaluating the stream cut time relative to the retention period to determine whether the stream cut time is within the retention period or whether the stream cut time is not within the retention period can comprise determining a retention period end time based on the stream cut time and the retention period, and determining whether the retention end time is later than a current system time.

Evaluating the stream cut time relative to the retention period to determine whether the stream cut time is within the retention period or whether the stream cut time is not within the retention period can comprise determining a retention period start time based on a current system time and the retention period, and determining whether the stream cut time is earlier than the retention period start time.

Aspects can comprise maintaining an auxiliary stream comprising stream cut data for the stream cut in association with the data stream, and, in response to determining that the stream cut time has a stream cut time value that is not within the retention period, truncating the auxiliary stream based on the stream cut.

FIG.9summarizes various example operations, e.g., corresponding to a machine-readable storage medium, comprising executable instructions that, when executed by a processor of a streaming data storage system, facilitate performance of operations. Operation902represents receiving a request to truncate events of a data stream of the streaming data storage system, in which the events are relative to a specified stream cut. Operation904represents determining whether the events are subject to a retention policy based on a retention period associated with the data stream Operation906represents, in response to determining that the events are not subject to the retention policy, truncating the events based on the specified stream cut.

Determining whether the events are subject to the retention policy can comprise accessing a stream cut time value associated with the specified stream cut, and determining whether the stream cut time value is within the retention period.

Further operations can comprise, in response to determining that the events are subject to the retention policy, blocking the request.

Further operations can comprise, in response to determining that the events are subject to the retention policy, scheduling a future truncation operation based on the specified stream cut.

Further operations can comprise, in response to determining that the events are subject to the retention policy, attempting to locate an earlier stream cut that is prior to the specified stream cut.

As can be seen, described herein is a technology that facilitates data retention management in stream-based data storage systems. The technology provides for compliance with data retention policies, including for enterprises that need to comply with state and federal regulations, as well as other business reasons. The technology is practical to implement.

FIG.10is a schematic block diagram of a computing environment1000with which the disclosed subject matter can interact. The system1000comprises one or more remote component(s)1010. The remote component(s)1010can be hardware and/or software (e.g., threads, processes, computing devices). In some embodiments, remote component(s)1010can be a distributed computer system, connected to a local automatic scaling component and/or programs that use the resources of a distributed computer system, via communication framework1040. Communication framework1040can comprise wired network devices, wireless network devices, mobile devices, wearable devices, radio access network devices, gateway devices, femtocell devices, servers, etc.

The system1000also comprises one or more local component(s)1020. The local component(s)1020can be hardware and/or software (e.g., threads, processes, computing devices). In some embodiments, local component(s)1020can comprise an automatic scaling component and/or programs that communicate/use the remote resources1010and1020, etc., connected to a remotely located distributed computing system via communication framework1040.

One possible communication between a remote component(s)1010and a local component(s)1020can be in the form of a data packet adapted to be transmitted between two or more computer processes. Another possible communication between a remote component(s)1010and a local component(s)1020can be in the form of circuit-switched data adapted to be transmitted between two or more computer processes in radio time slots. The system1000comprises a communication framework1040that can be employed to facilitate communications between the remote component(s)1010and the local component(s)1020, and can comprise an air interface, e.g., Uu interface of a UMTS network, via a long-term evolution (LTE) network, etc. Remote component(s)1010can be operably connected to one or more remote data store(s)1050, such as a hard drive, solid state drive, SIM card, device memory, etc., that can be employed to store information on the remote component(s)1010side of communication framework1040. Similarly, local component(s)1020can be operably connected to one or more local data store(s)1030, that can be employed to store information on the local component(s)1020side of communication framework1040.

The system bus1108can be any of several types of bus structure that can further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. The system memory1106includes ROM1110and RAM1112. A basic input/output system (BIOS) can be stored in a non-volatile memory such as ROM, erasable programmable read only memory (EPROM), EEPROM, which BIOS contains the basic routines that help to transfer information between elements within the computer1102, such as during startup. The RAM1112can also include a high-speed RAM such as static RAM for caching data.

The computer1102further includes an internal hard disk drive (HDD)1114(e.g., EIDE, SATA), and can include one or more external storage devices1116(e.g., a magnetic floppy disk drive (FDD)1116, a memory stick or flash drive reader, a memory card reader, etc.). While the internal HDD1114is illustrated as located within the computer1102, the internal HDD1114can also be configured for external use in a suitable chassis (not shown). Additionally, while not shown in environment1100, a solid state drive (SSD) could be used in addition to, or in place of, an HDD1114.

Other internal or external storage can include at least one other storage device1120with storage media1122(e.g., a solid state storage device, a nonvolatile memory device, and/or an optical disk drive that can read or write from removable media such as a CD-ROM disc, a DVD, a BD, etc.). The external storage1116can be facilitated by a network virtual machine. The HDD1114, external storage device(s)1116and storage device (e.g., drive)1120can be connected to the system bus1108by an HDD interface1124, an external storage interface1126and a drive interface1128, respectively.

A number of program modules can be stored in the drives and RAM1112, including an operating system1130, one or more application programs1132, other program modules1134and program data1136. All or portions of the operating system, applications, modules, and/or data can also be cached in the RAM1112. The systems and methods described herein can be implemented utilizing various commercially available operating systems or combinations of operating systems.

Computer1102can optionally comprise emulation technologies. For example, a hypervisor (not shown) or other intermediary can emulate a hardware environment for operating system1130, and the emulated hardware can optionally be different from the hardware illustrated inFIG.11. In such an embodiment, operating system1130can comprise one virtual machine (VM) of multiple VMs hosted at computer1102. Furthermore, operating system1130can provide runtime environments, such as the Java runtime environment or the .NET framework, for applications1132. Runtime environments are consistent execution environments that allow applications1132to run on any operating system that includes the runtime environment. Similarly, operating system1130can support containers, and applications1132can be in the form of containers, which are lightweight, standalone, executable packages of software that include, e.g., code, runtime, system tools, system libraries and settings for an application.

A monitor1146or other type of display device can be also connected to the system bus1108via an interface, such as a video adapter1148. In addition to the monitor1146, a computer typically includes other peripheral output devices (not shown), such as speakers, printers, etc.

The computer1102can operate in a networked environment using logical connections via wired and/or wireless communications to one or more remote computers, such as a remote computer(s)1150. The remote computer(s)1150can be a workstation, a server computer, a router, a personal computer, portable computer, microprocessor-based entertainment appliance, a peer device or other common network node, and typically includes many or all of the elements described relative to the computer1102, although, for purposes of brevity, only a memory/storage device1152is illustrated. The logical connections depicted include wired/wireless connectivity to a local area network (LAN)1154and/or larger networks, e.g., a wide area network (WAN)1156. Such LAN and WAN networking environments are commonplace in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which can connect to a global communications network, e.g., the Internet.

When used in a LAN networking environment, the computer1102can be connected to the local network1154through a wired and/or wireless communication network interface or adapter1158. The adapter1158can facilitate wired or wireless communication to the LAN1154, which can also include a wireless access point (AP) disposed thereon for communicating with the adapter1158in a wireless mode.

When used in a WAN networking environment, the computer1102can include a modem1160or can be connected to a communications server on the WAN1156via other means for establishing communications over the WAN1156, such as by way of the Internet. The modem1160, which can be internal or external and a wired or wireless device, can be connected to the system bus1108via the input device interface1144. In a networked environment, program modules depicted relative to the computer1102or portions thereof, can be stored in the remote memory/storage device1152. It will be appreciated that the network connections shown are example and other means of establishing a communications link between the computers can be used.

When used in either a LAN or WAN networking environment, the computer1102can access cloud storage systems or other network-based storage systems in addition to, or in place of, external storage devices1116as described above. Generally, a connection between the computer1102and a cloud storage system can be established over a LAN1154or WAN1156e.g., by the adapter1158or modem1160, respectively. Upon connecting the computer1102to an associated cloud storage system, the external storage interface1126can, with the aid of the adapter1158and/or modem1160, manage storage provided by the cloud storage system as it would other types of external storage. For instance, the external storage interface1126can be configured to provide access to cloud storage sources as if those sources were physically connected to the computer1102.