Abstract:
A method and system delivers data to a batch consumer and a streaming consumer. The method and system include retrieving data from a plurality of data centers, storing the data in a first directory, bundling the data into plurality of batches in the first directory, transferring each batch of the plurality of batches to the second directory, delivering the each batch of the plurality of batches in the second directory to the batch consumer and the streaming consumer and delivering the data in the first directory to the streaming consumer.

Description:
FIELD OF THE INVENTION 
     The present invention relates to data delivery and in particular, it relates to data distribution and analytics of large data sets. 
     BACKGROUND 
     In recent years, computing systems have seen a major change as growing volumes of data and stalling processor speeds required more and more applications to scale out to distributed systems. Today, various data sources from Internet to business operations produce large volumes of data. The management and distribution of large volumes of data, from storing to long-term archiving has become a tedious task. 
     The growing number of organizations led to a huge production of data, which in turn resulted in the need of speed and sophisticated data processing systems. Batch processing and streaming analysis of new real-time data sources is required to let organizations to take timely action. 
     In traditional distributed systems, there are multiple threads pulling data from multiple remote data centers for a given time period. For example, the data provider has a predefined Service Level Agreement (SLA) to deliver data to a consumer for every 1 minute. In such a case, for the first minute the thread T 1  is pulls the data from a remote data center DC 1 . The thread T 1  waits for the thread T 2  and other such threads pulling data before proceeding to the second minute. If this flow doesn&#39;t work properly than it violates the predefined SLA, as thread T 1  will move to the second minute. However, T 2  is still working on the first minute. A consumer on looking at the data in the second minute might be deceived into believing that the data collected in the first minute is immutable and might not get complete snapshot of the data there. Hence, it does not provide the consumer a mechanism to have a clean abstraction and a clean implementation. 
     In light of the above discussion, there is a need for a method and system, which overcomes all the above stated problems. 
     BRIEF DESCRIPTION 
     The above-mentioned shortcomings, disadvantages and problems are addressed herein which will be understood by reading and understanding the following specification. 
     In various embodiments, the present invention provides a method for delivering data to a batch consumer and a streaming consumer. The method includes retrieving data from a plurality of data centers, storing the data in a first directory, bundling the data into plurality of batches in the first directory, transferring each batch of the plurality of batches to the second directory, delivering the each batch of the plurality of batches in the second directory to the batch consumer and the streaming consumer and delivering the data in the first directory to the streaming consumer. 
     In another aspect, the present invention provides a system for delivering data to a batch consumer and a streaming consumer. The system includes a data retrieval module, a storage module, a bundler, a transmitting module, and a delivery module. The data retrieval module is configured to retrieve the data from the plurality of data centers. The storage module is configured to store the data in the first directory. The bundler is configured to bundle the data into a plurality of batches in the first directory. The transmitting module is configured to transmit the each batch of the plurality of batches to the second directory. The delivery module is configured to deliver the data in the first directory to the streaming consumer and each batch of the plurality of batches to the batch consumer and the streaming consumer. 
     Systems and methods of varying scope are described herein. In addition to the aspects and advantages described in this summary, further aspects and advantages will become apparent by reference to the drawings and with reference to the detailed description that follows. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates a system for delivering data to a batch consumer and a streaming consumer, in accordance with various embodiments of the present invention; 
         FIG. 2  illustrates a block diagram of a data analytical engine, in accordance with various embodiments of the present invention; and 
         FIG. 3  illustrates a flowchart for delivering data to the batch consumer and the streaming consumer, in accordance with various embodiments of the present invention. 
     
    
    
     DETAILED DESCRIPTION 
     In the following detailed description, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific embodiments, which may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments, and it is to be understood that other embodiments may be utilized and that logical, mechanical, electrical and other changes may be made without departing from the scope of the embodiments. The following detailed description is, therefore, not to be taken in a limiting sense. 
       FIG. 1  illustrates a system  100  for delivering data to a batch consumer  130  and a streaming consumer  140 , in accordance with various embodiments of the present invention. The system  100  includes a plurality of data centers  110 , a data analysis engine  120 , the batch consumer  130 , and the streaming consumer  140 . 
     The plurality of data centers  110  refers to centralized repositories, either physical or virtual. The plurality of data centers  110  is used for the storage, management and dissemination of data pertaining to a particular organization. 
     The data analysis engine  120  refers to a processing center that collects, organizes, examines, displays and analyzes the collected data. The data analysis engine  120  processes the data retrieved from the plurality of data centers  110  in two approaches—batch analysis and streaming analysis. In batch analysis, the data analysis engine  120  collects the data retrieved from the plurality of data centers  110  over a predefined period of time. Further, the data analysis engine  120  delivers the collected data to the batch consumer  130  for analysis. In streaming analysis, the data analysis engine  120  receives a continuous stream of data from the plurality of data centers  110  and delivers the data to streaming consumer  140  for analysis. 
     The batch consumer  130  refers to any person, company or entity that analyzes the data collected over a period of time. 
     The streaming consumer  140  refers to any person, company or entity that analyzes the data received in the form of data streams. The data streams received are analyzed in a streaming or on-line fashion in real time. 
       FIG. 2  illustrates a block diagram  200  of the data analysis engine  210 , in accordance with various embodiments of the present invention. The data analysis engine  210  includes a data retrieval module  220 , a storage module  230 , a bundler  240 , a transmitting module  250 , and a delivery module  260 . 
     The data retrieval module  220  of the data analysis engine  210  retrieves the data requested by the batch consumer  130  and the streaming consumer  140  from the plurality of data centers  110 . In an embodiment, the data is continuous streamed from the plurality of data centers  110 . 
     The storage module  230  further includes a first directory  230 A and a second directory  230 B. The first directory  230 A refers to a memory location that stores the data retrieved from the plurality of data centers  110 . The second directory  230 B refers to a memory location that stores the data received from the first directory  230 A. 
     The bundler  240  bundles the data stored in the first directory  230 A into a plurality of batches. The term used herein, the plurality of batches refers to a series of data packages. The bundler  240  bundles the data in accordance with a predefined time period. The predefined time period is defined by a Service Level Agreement (SLA). The term used herein, SLA refers to a service contract where a data center of the plurality of data centers  110  may commit to provide a particular service level to a given customer. SLA governs the quality, availability, and support commitments that the provider makes to the consumer. For example, the data retrieved for every 10 minutes from the plurality of data centers  110  is bundled that is governed by a predefined SLA. 
     The transmitting module  250  transmits each batch of the plurality of batches to the secondary directory  230 B. The transmitting module transmits the each batch of the plurality of batches at the end of the predefined time period defined by the SLA. The each batch of the plurality of batches in the second directory  230 B is immutable. The content of the each batch of the plurality of batches that is immutable cannot be altered after the creation of each batch. 
     The delivery module  260  delivers the each batch of the plurality of batches to the batch consumer  130  and the streaming consumer  140 . Further, the delivery module  260  delivers the data in the first directory  230 A to the streaming consumer. In an embodiment, the data in the first directory  230 A and each batch of the plurality of batches in the second directory  230 B are delivered to the streaming consumer in parallel for analysis. 
       FIG. 3  illustrates a flowchart  300  for delivering data to the batch consumer  130  and the streaming consumer  140 , in accordance with various embodiments of the present invention. At step  310 , the flowchart  300  initiates. At step  320 , the data analysis engine  120  retrieves the data from the plurality of data centers  110 . The data is retrieved as continuous streams from the plurality of data centers  110 . 
     At step  330 , the data analysis engine  120  stores the data in the first directory  230 A. At step  340 , the data analysis engine  120  bundles the data retrieved as continuous streams into the plurality of batches. The data analysis engine  120  bundles the data as defined by the SLA. The SLA commits to the batch consumer  130  and the streaming consumer  140  to provide the data for the predefined time period. For example, the data analysis engine  120  bundles the data retrieved for every 10 minutes into the plurality of batches. 
     At step  350 , the data analysis engine  120  transfers the plurality of batches to the second directory  230 B. Each batch of the plurality of batches in the second directory  230 B becomes immutable. At step  360 , the data analysis engine  120  delivers the each batch of the plurality of batches to the batch consumer  130 . At step  370 , the data analysis engine  120  delivers the each batch of the plurality of batches in the second directory  230 B and the data in the first directory  230 A to the streaming consumer  140 . At step  380 , the flowchart  300  terminates. 
     For example, the data analysis engine  120  retrieves data in continuous streams for an interval from 13:00 to 14:00 from the plurality of data centers  110 . The data analysis engine  120  stores the data in the first directory  230 A. For every 10 minutes (the predefined time period is configurable by the SLA) the data analysis engine  120  bundles the data into the plurality of batches. At the end of first 10 minutes (13:00-13:10) the data analysis engine  120  flushes the batch B 1  to the secondary directory  230 B. The batch B 1  becomes immutable in the secondary directory  230 B when the data analysis engine  120  bundles the data for the next 10 minutes time period (13:10-13:20). The data analysis engine  120  delivers the batch B 1  in the second directory  230 B to the batch consumer  130  and the streaming consumer  140  for further analysis. Further, the data analysis engine  120  delivers the data collected in the first 10 minutes (13:00-13:10) in the first directory  230 A to the streaming consumer for analysis. 
     This written description uses examples to describe the subject matter herein, including the best mode, and also to enable any person skilled in the art to make and use the subject matter. The patentable scope of the subject matter is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal language of the claims.