Storage optimization for event streaming for multiple consumers

Method and system are provided for storage optimization for event streaming for multiple consumers. The method provides an entire event stream for storage in a first tier storage and dynamically determines advance portions of the event stream for at least some of the consumers based on a consumer's position index in the event stream. The advance portions are portions of the event stream that will be consumed next by the consumer and the method provides the advance portions of the event stream for storage in second tier storage that has a higher performance than the first tier storage.

BACKGROUND

The present invention relates to storage optimization, and more specifically, to storage optimization for event streaming for multiple consumers.

Event streaming technologies provide the ability to store a single copy of events, and allows simultaneous access to multiple consumers. Such event streaming technologies provide a distributed streaming platform that is used for building real-time data pipelines and streaming applications.

Event streaming technologies provide the advantage of being able to scale the number of consumers without significant impact to the storage requirements. This is achieved by assigning a consumer a specific pointer into the stream. This allows two consumers to be at different positions within the stream, while maintaining a single copy of the data.

New consumers can subscribe to the stream and should have access to the complete stream history, not only the events after the subscription. This commonly means a large storage requirement for a stream, as it has to include the entire stream history (or at least a configured retention period or size). Often this means that the stream history is stored on traditional hard disk drives (HDD). These are cheap to purchase; however, they can have undesirable performance characteristics. The alternative is to store the stream on solid-state drives (SSD). This can significantly improve the performance; however, it may also increase the associated cost of the solution.

The traditional SSD cache solution stores commonly accessed data on SSD so future access can benefit from improved performance compared to the HDD.

Tier adjusting storage mechanisms are known, which respond to the presence of SSD in a storage pool that also contains HDDs. The system automatically and non-disruptively moves frequently accessed data from HDD managed disks to flash-based storage SSD managed disks, thus placing such data in a faster tier of storage.

SUMMARY

According to an aspect of the present invention there is provided a computer-implemented method for storage optimization for event streaming for multiple consumers, comprising: providing an entire event stream for storage in a first tier storage; dynamically determining advance portions of the event stream for at least some of the consumers based on a consumer's position index in the event stream, wherein advance portions are portions of the event stream that will be consumed next by the consumer; and providing the dynamically determined advance portions of the event stream for storage in a second tier storage that has a higher performance than the first tier storage.

According to another aspect of the present invention there is provided a system for storage optimization for event streaming for multiple consumers, comprising: a processor and a memory configured to provide computer program instructions to the processor to execute the function of the components: a first tier storage component for providing an entire event stream for storage in a first tier storage; an advance portion determining component for dynamically determining advance portions of the event stream for at least some of the consumers based on a consumer's position index in the event stream, wherein advance portions are portions of the event stream that will be consumed next by the consumer; and a second tier storage component for providing the dynamically determined advance portions of the event stream for storage in second tier storage that has a higher performance than the first tier storage.

According to a further aspect of the present invention there is provided a computer program product for storage optimization for event streaming for multiple consumers, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to: provide an entire event stream for storage in a first tier storage; dynamically determine advance portions of the event stream for at least some of the consumers based on a consumer's position index in the event stream, wherein advance portions are portions of the event stream that will be consumed next by the consumer; and provide the dynamically determined advance portions of the event stream for storage in second tier storage that has a higher performance than the first tier storage.

The computer readable storage medium may be a non-transitory computer readable storage medium and the computer readable program code may be executable by a processing circuit.

DETAILED DESCRIPTION

A method and system are provided for reducing the cost of high-performance event streaming, by using a mixture of a first tier of storage of cost-effective storage and a second tier of storage with a higher performance. The second tier of storage is used for data that is predicted to be accessed in the near future by consumers.

In the described method and system, the first tier of storage is used to store the entire event stream history and the second tier of storage is used to store the data that is going to be accessed in the near future by all or a group of consumers. This provides high performance for existing consumers, whilst providing access to the retained stream history for new consumers. The data that is going to be accessed in the near future is referred to as the advance data and is based on the current location in the event stream of a consumer.

The advance data is dynamically loaded on the second storage tier and may be intelligently selected based on the understanding of how consumers work with event streaming technologies. This significantly improves the access time for consumers, while only requiring a fraction of the event stream history to be stored on the second, more expensive, storage tier.

With event streaming systems, it is less about reading one bit of data lots of times and more about a set of consumers making their own way through the stream of data that is coming in. Due to the operation of event streaming systems, they track where every consumer is up to and therefore it is possible to determine what portion of the event stream will be looked at next.

Event streaming provides a growing, ordered set of distinct event objects that a set of consumers typically work forward through, with each consumer potentially starting in a different place and moving at a different speed. An event may, for example, be something like ‘doorbell rang’, ‘transaction processed,’ ‘sensor triggered’ with each event having additional information that the consumers can act upon. An event stream may be other forms of stream of media with multiple consumers acting at different speeds and reading forwards.

The described method analyzes consumers' behavior in relation to an event stream to determine the current location of consumers and their behavior parameters and patterns. Behavior parameters may include, for example, how fast they are moving through events, or wherein in the stream a consumer is in relation to other consumers. The consumer behavior is used to analyze the size of the portion of the event stream that would be optimal to load in the second tier storage. The consumer behavior may be analyzed for each consumer or a group of consumers to optimize the amount of storage in the second tier storage.

Tier adjusting storage mechanisms are provided in some storage products. Storage systems typically have different tiers of storage where each tier has a different ratio of cost to performance. Tier adjusting storage mechanisms typically put the data that is accessed most often on the fastest storage. This provides something approaching the performance of having only the most expensive storage but with only a fraction actually being this expensive storage and the rest being much cheaper. In a dynamically tiered environment, data movement is seamless to the host application regardless of the storage tier in which the data belongs.

In the described method and system, tier adjusting storage mechanism are augmented based on how customers typically consume data in the event stream. The determined advance portions of the event stream for consumers are loaded in second tier storage. The advance portions required by consumers may be scored, with the highest scoring data loaded to the second tier storage.

In this description, the first tier of storage is provided by hard disk drives (HDD) and the second tier of storage is provided by solid-state drives (SSD). It will be appreciated that other types of storage may be used in the tiers.

Referring toFIG.1, a flow diagram100shows an example embodiment of the described method. An event stream may be provided101to be stored is a first storage tier, such as HDD storage. The entire stream history may be stored in the first storage tier.

The method may dynamically determine102the current consumers or a group of consumers of the event stream. Group analysis may be used for a representative proportion of consumers or a group for which it is determined that optimization is required.

The method may determine103current position indexes of at least some of the consumers in the event stream, for example, all consumers of a group to be optimized, in order to identify advance portions to be stored in the second tier storage. The position indexes are used to determine the start of the advance portions of the event stream that are candidates to be stored in the second tier storage.

The amount of second tier storage allocated to each consumer may be an equal amount per consumer. For example, this may be based on one or more of: a static allocated size of the storage per consumer; a static allocated number of events per consumer; a dynamic allocated size of the storage per consumer depending on the number of consumers, and a dynamic allocated number of events per consumer based on the number of consumers.

Optionally, the behavior of the consumers is analyzed104by considering behavior parameters or patterns so that the volume of future events per consumer to be stored in second tier storage takes the behavior of consumers into account. Such parameters may include, for example, the speed the consumer is processing events or the number of events a consumer is behind the head of the event stream. The number of consumers that need access to that data in the future may also be taken into consideration.

The method may optimize105the amount of second tier storage for at least some of the consumers, based on the number of consumers to be optimized and, optionally, based on the consumer behavior.

Using the position index and the optimized amount of second tier storage for a consumer, the method may dynamically determine106the advance event stream portions to be stored on the second tier storage. Each advance portion for a consumer may be scored and the highest scoring portions may be selected for storage on the second tier storage. This also takes into account overlaps of the advance event stream portions across the consumers as a portion effectively get scored multiple times (once for each consumer) so there is a tendency to select data that multiple consumers want to access.

These determined portions may be provided107for loading into the second tier storage, for example, using tier adjusting storage mechanisms. In this way, the volume of the advance events per consumer may vary, and is calculated based on the consumer position and, optionally, behavior.

Known tier adjusting storage demonstrates that smart placement of data on SSDs can drive very significant performance improvements over HDD without anything like the cost of a full SSD implementation. The described method predicts the data that is to be read next so that it can be placed on the SSDs.

The described method and system provide integration between the event stream and storage subsystem so optimization of the stream storage can occur between different tiers of cost-to-performance storage.

FIGS.2A to2Cillustrate some example embodiments of storage environments210,220,230of the described method. In each figure, an event stream201is shown against a timeline202with most recent events at the top of the event stream201. A first tier storage in the form of HDD storage203is shown with the entire event stream201stored204. A second tier storage in the form of SSD storage205is shown with selected portions211,221-223,231-233of the event stream201stored based on the advance data required by consumers206-208.

A first simple embodiment is shown inFIG.2A, the consumers206-208are all currently reading from positions within the newest entries in the event stream201. When an event is published to the event stream it is stored within the HDD203and the SSD205. The SSD205is populated with the most recent events211because these are the advance events required by the consumers206-208. Access to the stream for the existing consumers206-208will be using the SSD205.

As the number of consumers increases, the probability of the SSD being able to hold all the advance events for all consumers decreases. In this case, there are a number of strategies that may be applied.

FIG.2Bshows a second embodiment in which individual consumer coverage is used as the basis for the SSD allocation. The storage of the SSD205is used to cover a proportion of the advance events for each consumer206-208; however, the entire future stream of events is not stored in SSD.

InFIG.2B, each consumer has an equal share221-223of their future events stored in SSD205. The amount stored in SSD for each consumer may be based on one or many of the following factors.

It may be based on a static size of storage for each consumer or it may be based on a static number of events for each consumer. Alternatively, it may be based on a dynamic size of storage or number of events depending on the number of consumers. If more consumers start to consume the event stream201, the size of the portions221-223stored in SSD205for each consumer may decrease.

FIG.2C, shows a third embodiment in which the concept is extended so that the size of the portions231-233for each consumer varies based on the behavior of each of the consumers. The factors of the behavior may include a speed the consumer is processing events. The factors of the behavior may include the number of events a consumer is behind the head of the stream as this will affect the number of consumers that need access to that data in the future.

In a further embodiment, the method may optimize the SSD storage for a cluster or proportion of consumers instead of all existing consumers. This may allow consumers who are behind the head of the stream, quick storage access so they can catch-up. This may alternatively, maximize the benefit of the SSD for median consumers. In this context the median consumers are those that are statistically clustered together.

One further factor to take into account is that new consumers may start reading from the very beginning of the stored event stream history. Optimizing for consumers being at different points of the event stream in this way should have benefits throughout the lifetime of the event stream.

The described approach relies on knowing where the consumer is going to read next and does not assume the same data is accessed repeatedly. The method does not rely on how often someone is accessing data because there are likely to be consumers reading forwards through a stream to parse events.

The method provides a view on where a consumer is in the stream and considers the actual position of the next pointer to indicate what is likely to be read next. The described method covers this by knowing the index consumers are currently working at. Optionally, read speed and other parameters are used to determine how fast a consumer is moving forward to allocate an appropriate amount of storage for the consumer. The method may also consider how a set of consumers are working to balance what gets onto a hot tier across these consumers.

The method takes an ordered stream of events and optimizes the read time for the events in that stream that a consumer or set of consumers are likely to care about without changing the ordering of the stream itself.

Referring toFIG.3, a block diagram shows a computing system300in which the described event stream storage optimization system310is provided. The computing system300may include at least one processor301, a hardware module, or a circuit for executing the functions of the described components which may be software units executing on the at least one processor. Multiple processors running parallel processing threads may be provided enabling parallel processing of some or all of the functions of the components. Memory302may be configured to provide computer instructions303to the at least one processor301to carry out the functionality of the components.

A storage system320may be provided with at least first and second tier storage340,350and a tier adjusting storage mechanism330. The first tier storage340may be cost-effective storage, such as HDD storage. The second tier storage350may be higher performance storage, such as SSD storage.

The event stream storage optimization system310includes a first tier storage component311for providing an entire event stream for storage in a first tier storage340and a second tier storage component312for providing dynamically determined advance portions of the event stream for storage in second tier storage350. The second tier storage component312may augment the tier adjusting storage mechanism330for the advance portions.

The event stream storage optimization system310includes an advance portion determining component313for dynamically determining advance portions of the event stream for at least some of the consumers based on a consumer's position index in the event stream.

The event stream storage optimization system310includes an advance portion optimizing component314for optimizing a size of each advance portion for each consumer or a group of consumers. The advance portion optimizing component314may provide an equal size for each consumer based on a number of current consumers. Alternatively, the advance portion optimizing component314may include a consumer behavior component315for analyzing consumer behavior to optimize the sizes based on the behavior. The consumer behavior component315may determine one or more of: a speed that a consumer is processing events; a number of events a consumer is behind the head of the event stream; and a future number of consumers that need access to the advance portion in the future. The advance portion optimizing component314may include a scoring component316for scoring advance portions for each consumer to select most commonly required portions and to accommodate overlaps of the advance portions across the consumers.

The event stream storage optimization system310may include a consumer group component317for optimizing each advance portion for a representative proportion of consumers or a group for which it is determined optimization is required.

FIG.4depicts a block diagram of components of a computing system300as used for event stream storage optimization system310, in accordance with an embodiment of the present invention. It should be appreciated thatFIG.4provides only an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.

The computing system can include one or more processors402, one or more computer-readable RAMs404, one or more computer-readable ROMs406, one or more computer readable storage media408, device drivers412, read/write drive or interface414, and network adapter or interface416, all interconnected over a communications fabric418. Communications fabric418can be implemented with any architecture designed for passing data and/or control information between processors (such as microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components within the system.

One or more operating systems410, and application programs411, such as the event stream storage optimization system310are stored on one or more of the computer readable storage media408for execution by one or more of the processors402via one or more of the respective RAMs404(which typically include cache memory). In the illustrated embodiment, each of the computer readable storage media408can be a magnetic disk storage device of an internal hard drive, CD-ROM, DVD, memory stick, magnetic tape, magnetic disk, optical disk, a semiconductor storage device such as RAM, ROM, EPROM, flash memory, or any other computer readable storage media that can store a computer program and digital information, in accordance with embodiments of the invention.

The computing system can also include a RAY drive or interface414to read from and write to one or more portable computer readable storage media426. Application programs411on the computing system can be stored on one or more of the portable computer readable storage media426, read via the respective RAY drive or interface414and loaded into the respective computer readable storage media408.

The computing system can also include a network adapter or interface416, such as a TCP/IP adapter card or wireless communication adapter. Application programs411on the computing system can be downloaded to the computing device from an external computer or external storage device via a network (for example, the Internet, a local area network or other wide area networks or wireless networks) and network adapter or interface416. From the network adapter or interface416, the programs may be loaded into the computer readable storage media408. The network may comprise copper wires, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers and edge servers.

The computing system can also include a display screen420, a keyboard or keypad422, and a computer mouse or touchpad424. Device drivers412interface to display screen420for imaging, to keyboard or keypad422, to computer mouse or touchpad424, and/or to display screen420for pressure sensing of alphanumeric character entry and user selections. The device drivers412, R/W drive or interface414, and network adapter or interface416can comprise hardware and software stored in computer readable storage media408and/or ROM406.

Cloud Computing

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

A computer program product of the present invention comprises one or more computer readable hardware storage devices having computer readable program code stored therein, said program code executable by one or more processors to implement the methods of the present invention.