Data analytics on distributed databases

Data analytics is performed on a distributed document storage database by receiving a request for initiating a data analytics job; collecting statistics from the database in response to the request; using the statistics to estimate a first cost for merging an incremental data update for the job into a first resilient distributed dataset; using the statistics to estimate a second cost for newly creating a second resilient distributed dataset for the job; when the first cost is less than the second cost, reading data updates from the database and merging the data updates into the first resilient distributed dataset; and when the first cost is not less than the second cost, newly creating the second resilient distributed dataset by reading all documents from the database.

FIELD

The present application relates generally to data analytics and, more particularly, to performing data analytics on document storage devices using incremental data updates.

BACKGROUND

Data interchange is a method for exchanging computer-readable data between two or more autonomous computer systems or servers. These computer systems may use different operating systems. JavaScript™ Object Notation (JSON) is a lightweight data interchange format that uses human-readable text to store and transmit data objects comprising attribute-value pairs. One common use of JSON is to read data from a web server, and to display the data in a web page. JSON may be used as an alternative to XML (Extendible Markup Language) for organizing data. Likewise, JSON may be used in conjunction with distributed document storage databases. JSON documents are relatively lightweight and are executed rapidly on web servers.

JSON includes “name: object” pairs and punctuation in the form of brackets, parenthesis, colons, and semicolons. Each object is defined with an operator such as “text:” or “image:” and then grouped with a value for that operator. The simple structure and absence of mathematical notation and algorithms makes JSON intuitive, easy to understand, and quickly mastered, even by those with limited formal programming experience. Moreover, JSON facilitates the development of web and mobile applications while not being affected by database schema changes. A schema is an organizational structure that represents a logical view of a database. The schema defines how data is organized, specifies relationships among the data, and formulates all constraints that are to be applied to the data.

JSON distributed document storage databases do not always provide adequate data analysis capabilities. As a result, external data analytic services, such as Spark™, have been developed to integrate data analysis capabilities with JSON distributed document storage databases. In order to leverage data analytic services, documents in a JSON document storage database must be read and transformed into a Resilient Distributed Dataset (RDD), and then an analytics job may be executed on the RDD. The RDD is an immutable, fault-tolerant, distributed collection of objects that can be operated on in parallel. The RDD can contain any type of object and is created by loading an external dataset or distributing a collection from a driver program. RDD data is resilient, in the sense that the data can be recomputed in case all or a portion of the data is lost. RDD data is distributed, such that the data can be read and processed from any of multiple nodes without having to drag the data to any particular node. RDDs are computed in memory and can be persisted in memory. RDDs can be recomputed each time an action is executed, or an RDD may be persisted in memory, in which case elements of the RDD are retained on a cluster for much faster access the next time that the elements are queried. RDDs are advantageous in terms of rearranging computations to optimize data processing.

As a practical matter, many data analytics jobs are required to be executed at regular time intervals, or on a continual basis. When a first round of a data analytics job is executed, a first set of documents from the JSON distributed document storage database is analyzed. Then, when a second round of the data analytics job is to be executed, a second set of documents from the JSON distributed document storage database needs to be analyzed. In general, the second set of documents is not identical to the first set of documents. Since the documents to be analyzed are changing dynamically, this poses challenges in terms of effectively and efficiently supporting data analytics on JSON distributed document storage databases. Thus, there exists a need to overcome at least one of the preceding deficiencies and limitations of the related art.

SUMMARY

A method for performing data analytics on a distributed document storage database, in one aspect, may comprise receiving a request for initiating a data analytics job; collecting a set of statistics from the distributed document storage database in response to the request; using the set of statistics to estimate a first cost for merging an incremental data update for the data analytics job into a first resilient distributed dataset; using the set of statistics to estimate a second cost for newly creating a second resilient distributed dataset for the data analytics job; comparing the first cost to the second cost and, when the first cost is less than the second cost, reading one or more data updates from the distributed document storage database and merging the one or more data updates into the first resilient distributed dataset; and when the first cost is not less than the second cost, newly creating the second resilient distributed dataset by reading all documents from the distributed document storage database.

A computer program product for performing data analytics on a distributed document storage database, in another aspect, may comprise a computer-readable storage medium having a computer-readable program stored therein, wherein the computer-readable program, when executed on a processor, causes the processor to receive a request for initiating a data analytics job; collect a set of statistics from the distributed document storage database in response to the request; use the set of statistics to estimate a first cost for merging an incremental data update for the data analytics job into a first resilient distributed dataset; use the set of statistics to estimate a second cost for newly creating a second resilient distributed dataset for the data analytics job; compare the first cost to the second cost and, when the first cost is less than the second cost, read one or more data updates from the distributed document storage database and merge the one or more data updates into the first resilient distributed dataset; and when the first cost is not less than the second cost, newly create the second resilient distributed dataset by reading all documents from the distributed document storage database.

An apparatus for performing data analytics on a distributed document storage database, in another aspect, may comprise a processor and a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to receive a request for initiating a data analytics job; collect a set of statistics from the distributed document storage database in response to the request; use the set of statistics to estimate a first cost for merging an incremental data update for the data analytics job into a first resilient distributed dataset; use the set of statistics to estimate a second cost for newly creating a second resilient distributed dataset for the data analytics job; compare the first cost to the second cost and, when the first cost is less than the second cost, read one or more data updates from the distributed document storage database and merge the one or more data updates into the first resilient distributed dataset; and when the first cost is not less than the second cost, newly create the second resilient distributed dataset by reading all documents from the distributed document storage database.

DETAILED DESCRIPTION

FIG. 1illustrates a processing framework in accordance with one or more embodiments of the present invention. For purposes of illustration, data analytics may be provided using an open-source data analytics framework such as Apache Spark™ Spark™ is described herein for illustrative purposes, as another type of data analytics framework may be used in lieu of, or in addition to, Spark™. Spark™ includes a core engine that functions as an application programming interface (API) layer. A set of resilient distributed datasets (RDDs)100are provided for implementing data analytics procedures. A set of related tools are provided for managing and analyzing data, including a Structured Query Language (SQL) engine, a library of machine learning algorithms (MLib), a graph processing system (GraphX), and streaming data processing software. Spark™ can process data from a variety of data repositories, including a Hadoop™ Distributed File System (HDFS), a Not Only Structured Query Language (NoSQL) database, or a relational data store such as Apache Hive™. Spark™ supports in-memory processing to boost the performance of big data analytics applications, but Spark™ can also perform conventional disk-based processing when data sets are too large to fit into available system memory.

The set of RDDs100is an abstract representation of data divided into partitions and distributed across a cluster. Each RDD in the set of RDDs100represents an immutable, partitioned collection of data elements that can be operated on in parallel. Each of the partitions may be stored in random-access memory (RAM), or on a data storage drive, or on any other type of computer-readable memory device.

Spark™ supports two basic categories of operations that may be performed on any RDD in the set of RDDs100. These categories include transformations and actions. A transformation102is used to transform data in any RDD of the set of RDDs100from one form to another. Examples of transformations include operations such as map, filter, and flatMap. When the transformation102is applied to a first RDD of the set of RDDs100, the transformation102is generally not performed immediately. Rather, a Directed Acyclic Graph (DAG) is created that incorporates the transformation102, the first RDD, and a function used to implement the transformation102. The transformation102may continue building the DAG by using zero or more additional RDDs of the set of RDDs100, until an action104is applied to a last RDD of the additional RDDs. The action104triggers execution of all transformation102operations on the first RDD using the additional RDDs, or using the DAG. An end result106of the transformation102is a new RDD in the set of RDDs100that includes transformed data. After the transformation102operations are completed, the action104operation is executed on the last RDD.

One RDD of the set of RDDs100may be dependent upon zero, one, or more than one additional RDD of the set of RDDs100. Due to the dependent nature of the set of RDDs100, eventually the set of RDDs will create a single DAG from start to end. This property is referred to as lineage. Lineage is an important aspect for fault tolerant operation in Spark™. Execution of any operation in Spark™ is distributed to various nodes. When any node goes down, or an executing process on any node crashes, then Spark™ automatically reschedules the process to another suitable node and recovers the intermediate state of the failed node using this lineage. All operations are relaunched using lineage, and any intermediate data that may have been computed in the failed node is recomputed.

FIG. 2illustrates an exemplary method for processing a data analytics job in accordance with one or more embodiments of the present invention. For purposes of illustration, the data analytics job may be processed using a Spark™ processing framework in conjunction with a Cloudant™ distributed document storage database. As mentioned previously, Spark™ is a parallel, open-source processing framework for running large-scale data analytics applications across clustered computers. Spark™ is described herein for illustrative purposes, as another type of data analytics framework may be used in lieu of, or in addition to, Spark™.

For purposes of illustration, the distributed document storage database is a JSON document store207provided by a managed database service209such as Cloudant™. Cloudant™ is a managed service for managing a distributed database such as the JSON document store207. Cloudant™ and JSON are described herein for illustrative purposes, as another type of database manager may be used in lieu of, or in addition to, Cloudant™ Similarly, another type of distributed database may be used in lieu of, or in addition to, the JSON document store207.

Cloudant™ is offered in at least three forms: Cloudant Shared™, Cloudant Enterprise database-as-a-service (DBaaS)™, and Cloudant Local™. All three forms offer an Application Program Interface (API). Cloudant Shared™ runs on a multi-tenant infrastructure. Accounts are provisioned for customers on a shared database cluster. Cloudant Enterprise DBaas™ runs on a single-tenant, dedicated infrastructure to provide a high level of performance and availability. The single-tenant architecture is provided by provisioning bare-metal hardware, or by using virtual infrastructure on providers such as SoftLayer™ (an IBM company); Rackspace™; AWS™; and Microsoft Azure™. Cloudant Local™ is an on-premises version of Cloudant™ software that companies can install locally in their own data centers to run their own DBaaS. A local cluster includes machines dedicated for either a load balancer (a minimum of one machine is required), or a database server node (a minimum of three machines are required). While Cloudant Local™ provides management tools, software, and techniques, the customer manages the infrastructure and tunes the software.

With reference toFIG. 2, a Structured Query Language (SQL) data analytics job201is received at a connector203. The connector203is configured for transforming JSON objects in the JSON document store207of the managed database service209into a resilient distributed dataset (RDD)205in order to perform data analytics. The connector203configures data source meta-information of the managed database service209for use with the data analytics framework, including configuration of data source connection Uniform Resource Locators (URLs), as well as a schema for context. The connector203reads JSON documents from the JSON document store207, creates a resilient distributed dataset (RDD)205, and submits the SQL data analytics job201to the managed database service209for execution.

The connector203is used to leverage the JSON document store207as a data source for performing massive data analytics. This capability is becoming increasingly relevant, as more and more web and mobile applications present strong data analytics requirements for JSON data. However, data analytics applications, such as Spark™, use an in-memory data structure—namely, the RDD205—for performing massive data analytics. Conventional approaches for using data analytics applications in conjunction with the JSON document store207require all data to be reloaded if any data update has occurred at the JSON document store207. This requirement places practical limitations on the usage of the JSON document store207with data analytics procedures.

Many actual data analytics jobs are not one-time jobs. Thus, submitting the SQL data analytics job201via the connector203results in performance problems when the managed database service209has performed data updates to the JSON document store207. For example, disk storage space and network input/output (I/O) are wasted for re-fetching JSON documents from the JSON document store207to the connector203. Moreover, central processing unit (CPU) capacity of the connector203is wasted for reading and transforming JSON documents into the RDD205.

In accordance with a set of exemplary embodiments disclosed herein, data analytics on the JSON document store207may be speeded up by determining whether or not a data analytics job can be performed more efficiently simply by updating the JSON document store207, as opposed to newly re-fetching all JSON documents from the JSON document store207. This determination is performed using a cost model that selects a minimum cost option from among a first cost and a second cost. The first cost represents a cost for merging an incremental update for the data analytics job into a first resilient distributed dataset (RDD). The second cost represents a cost for newly creating a second RDD for the data analytics job.

The first cost may comprise one or more of: a third cost of at least one input/output operation at the distributed document storage database attributable to merging an incremental data update for the data analytics job into the first resilient distributed dataset; a fourth cost of at least one network input/output operation from the distributed document storage database attributable to merging the incremental data update for the data analytics job into the first resilient distributed dataset; or a fifth cost of transforming one or more documents in the distributed document storage database to the first resilient distributed dataset. Thus, the fourth cost represents a cost of at least one input/output (I/O) operation at the JSON document store207which is attributable to a data analytics procedure. The fifth cost comprises a cost of at least one network I/O operation from the JSON document store207to the data analytics procedure. The sixth cost comprises a cost of transforming JSON documents in the JSON document store207to the RDD205.

The second cost may comprise one or more of: a sixth cost of at least one input/output operation at the distributed document storage database attributable to newly creating the second resilient distributed dataset for the data analytics job; a seventh cost of at least one network input/output operation from the distributed document storage database attributable to newly creating the second resilient distributed dataset for the data analytics job; or an eighth cost of transforming one or more documents in the distributed document storage database to the second resilient distributed dataset. Thus, the sixth cost represents a cost of at least one input/output (I/O) operation at the JSON document store207which is attributable to a Spark™ data analytics procedure. The seventh cost comprises a cost of at least one network I/O operation from the JSON document store207to the data analytics procedure. The eighth cost comprises a cost of transforming JSON documents in the JSON document store207to the RDD205.

Let J be an SQL job which will run repeatedly at a regular interval T, whose data source is a JSON document store207using a specific schema S. For example, the regular interval T may comprise every Monday at midnight. Let D be the total size of JSON documents for J at a last (most recent) execution time T_last. Three challenges exist for supporting J with incremental data updates at T_start=T_last+T where T_start is a starting time of a new J. A first challenge is determining how many JSON documents have been updated since T_start, and identifying these updated documents. A second challenge is determining which of the following procedures is more efficient—reading only newly updated data, or re-fetching all JSON documents from scratch. A third challenge is determining how to merge updates with a set of original RDDs, such as the RDD205, to produce new RDDs for a new J.

FIG. 3is a flowchart illustrating a first exemplary method for performing data analytics in accordance with one or more embodiments of the present invention. The method commences at block301where a request is received for initiating a data analytics job. Next, at block303, a set of statistics is collected from at least one of a distributed document storage database or a job log in response to the request. For example, the set of statistics may be collected from the JSON document store207(FIG. 2). The set of statistics is used to estimate a first cost for merging an incremental data update for the data analytics job into a first resilient distributed dataset (FIG. 3, block305). Further details are provided inFIG. 6, to be described in greater detail hereinafter. The set of statistics is also used to estimate a second cost for newly creating a second resilient distributed dataset for the data analytics job (block307). Note that blocks305and307may be performed in any order or substantially contemporaneously.

The method advances to block309where a test is performed to determine whether or not the first cost is less than the second cost. When the first cost is less than the second cost, the method advances to block311where one or more data updates are read from the distributed document storage database such as the JSON document store207(FIG. 2). For example, this step may be performed by querying the JSON document store207to collect data updates. Then, at block313(FIG. 3), the one or more data updates are merged into the first resilient distributed dataset. This step may be performed, for example, by submitting a Spark™ RDD merge job to Spark™ to produce one or more new RDDs for the RDD205(FIG. 2). The negative branch from block309(FIG. 3) leads to block315. When the first cost is not less than the second cost, the second resilient distributed dataset is newly created by reading all documents from the distributed document storage database. This step may be performed, for example, by using the original data analytics job of block301. The program advances from block313or block315to block317where the data analytics job is performed.

FIG. 4illustrates a system for performing data analytics in accordance with one or more embodiments of the present invention. For purposes of illustration, the data analytics job may be processed using a Spark™ processing framework in conjunction with a Cloudant™ distributed document storage database. As mentioned previously, Spark™ is a parallel, open-source processing framework for running large-scale data analytics applications across clustered computers. Spark™ is described herein for illustrative purposes, as another type of data analytics framework may be used in lieu of, or in addition to, Spark™.

A managed database service409is configured for managing distributed databases. One illustrative implementation of the managed database service409is Cloudant™. For purposes of illustration, the managed database service409includes a first JSON document store425, a second JSON document store427, and an Nth JSON document store429, where N is a positive integer greater than two. Cloudant™ and JSON are described herein for illustrative purposes, as another type of managed database service may be used in lieu of, or in addition to, Cloudant™. Similarly, another type of distributed database may be used in lieu of, or in addition to, the first JSON document store425, the second JSON document store427, and the Nth JSON document store429.

With reference toFIG. 4, a Structured Query Language (SQL) data analytics job201is received at an enhanced connector403. The enhanced connector403includes a data tracker module421, a cost estimator407, and a resilient distributed dataset (RDD) merger411. The enhanced connector403is configured for transforming JSON objects in any of the respective first, second, and Nth JSON document stores425,427and429into a corresponding resilient distributed dataset (RDD) of a first resilient distributed dataset (RDD)433, a second resilient distributed dataset (RDD)435, or an Nth resilient distributed dataset (RDD)437in order to perform data analytics. The first, second, and Nth RDDs433,435, and437are provided by a data analysis service431configured for performing a data analytics application such as, for example, Spark™.

The enhanced connector403configures data source meta-information of the managed database service409for use with the data analytics processing framework, including configuration of data source connection Uniform Resource Locators (URLs), as well as a schema for context. The enhanced connector403reads JSON documents from any respective JSON document store of the first, second, and Nth JSON document stores433,435, and437, creates a corresponding resilient distributed dataset (RDD) such as the first, second, or Nth RDD433,435, and437, and submits the SQL data analytics job201to the managed database service409for execution.

The enhanced connector403is used to leverage the managed database service409as a data source for performing massive data analytics. This capability is becoming increasingly relevant, as more and more web and mobile applications present strong data analytics requirements for JSON data. However, data analytics applications such as Spark™, for example, use an in-memory data structure—namely, the RDDs433,435, and437—for performing massive data analytics. Conventional approaches for using data analytics applications in conjunction with the managed database service409require all data to be reloaded if any data update has occurred to any of the first, second, or Nth JSON document stores425,427and429. This requirement places practical limitations on the usage of the first, second, or Nth JSON document stores425,427and429with data analytics procedures.

Many actual data analytics jobs are not one-time jobs. However, submitting the SQL data analytics job201via the enhanced connector403using the procedure ofFIG. 3reduces or eliminates performance problems when the managed database service409(FIG. 4) has performed data updates to any of the first, second, or Nth JSON document stores425,427and429. For example, disk storage space and network input/output (I/O) are conserved by selectively re-fetching JSON documents from the JSON document stores425,427or429to the enhanced connector403. Moreover, central processing unit (CPU) capacity of the connector403is conserved by selectively reading and transforming JSON documents into the data analysis service431.

In accordance with a set of exemplary embodiments disclosed herein, data analytics on the first, second, and Nth JSON document stores425,427and429are speeded up by determining whether or not data analytics can be performed simply by updating at least one of the first, second, and Nth JSON document stores425,427and429, as opposed to newly re-fetching all JSON documents from the JSON document stores425,427and429. This determination is performed using the cost estimator407. The cost estimator407minimizes at least one of a first cost, a second cost, or a third cost. The first cost comprises a cost of disk input/output (I/O) at one or more of the first, second, or Nth JSON document stores425,427and429with regard to the data analysis service431. The second cost comprises a cost of network I/O from one or more of the first, second, or Nth JSON document stores425,427, and429to the data analysis service431. The third cost comprises a cost of transforming JSON documents in any of the first, second, and Nth JSON document stores425,427and429to the data analysis service431.

Let J be an SQL job which will run repeatedly at a regular interval T, whose data source is a JSON document store207using a specific schema S. For example, the regular interval T may comprise every Monday at midnight. Let D be the total size of JSON documents for J at a last (most recent) execution time T_last. The system ofFIG. 4is configured for supporting J with incremental data updates at T_start=T_last+T where T_start is a starting time of a new J. The data tracker module421determines how many JSON documents have been updated in the managed database service409since T_start, and identifies these updated documents. The cost estimator407determines which of the following procedures is more efficient—reading only newly updated data, or re-fetching all JSON documents from the managed database service409from scratch. The RDD merger411determines how to merge updates with a set of original RDDs, such as the first, second, and third RDDs433,435, and437, to produce new RDDs for a new J. The RDD merger411performs this merging task by formulating an RDD merge job417and forwarding the merge job417to the data analysis service431.

The RDD merger411is configured for performing two transform operations, filter and union, to merge data updates into an existing RDD for a JSON document store, to thereby provide a first new RDD and a second new RDD for a current data analytics job. More specifically, a first set of documents is obtained from the first, second, and Nth JSON document stores425,427and429where each of the documents in the first set is either deleted or updated. Then, the filter operation is used on the existing RDD to filter out all of these deleted or updated documents to generate the first new RDD. Next, a second set of documents is obtained from the first, second, and Nth JSON document stores425,427and429where each of the documents in the second set is either updated or created. The second new RDD is generated for this second set of documents. Then, a union transformation is used to combine the first new RDD with the second new RDD to form a newly updated RDD that includes merged data updates.

The enhanced connector403supports performing data analytics on any of the first, second, or Nth JSON document stores425,427and429with incremental data updates, instead of reloading all data in the JSON document stores425,427and429from scratch. Specifically, the data tracker module421, the cost estimator407, and the RDD merger411are used to provide the enhanced connector403. The data tracker module421, the cost estimator407, and the RDD merger411, are operatively coupled between the managed database service409and the data analysis service431. The data tracker module421is configured for tracking data changes at any of the first, second, or Nth JSON document stores425,427and429. The data tracker module421is also configured for determining, in response to a re-execution of the SQL data analytics job201, whether or not a data reload is necessary. The RDD merger411is configured for merging all updated JSON objects in any of the first, second, or Nth JSON document stores425,427and429to a previous RDD of the first, second, or Nth RDDs433,435, or437for the re-executed SQL data analytics job201. Thus, the enhanced connector403reduces I/O costs associated with data transformation and improves the overall performance of a data analytics system that combines one or more JSON document stores, such as the first, second, or third JSON document stores425,427and429with the data analysis service431.

FIG. 5is a flowchart illustrating a second exemplary method for performing data analytics in accordance with one or more embodiments of the present invention. The method commences at block501where a request for a data analytics job is received. Next, at block503, one or more JSON documents in a JSON document store are transformed into a set of RDDs. For example, at least one of the respective first, second, or Nth JSON document stores425,427or429(FIG. 4) may be transformed into a corresponding set of RDDs including one or more of the first, second, or Nth RDDs433,435, or437. Next, at block505(FIG. 5), in response to the data analytics job being repeatable, recurring, or continuous, the enhanced connector403(FIG. 4) keeps the corresponding set of RDDs in memory, or persists the corresponding set of RDDs to one or more disks of a data cluster. Block505(FIG. 5) reduces I/O and network costs.

The operational sequence ofFIG. 5progresses to block507where a determination is made as to whether data in the respective JSON document store425,427and429(FIG. 4) should be reloaded, or whether the data should simply receive an incremental update. This step may be performed using the data tracker module421. For example, users may be provided with an option to instruct the data tracker module421to implement an eager option or a lazy option. The eager option performs incremental updating of the corresponding set of RDDs when an update volume at the respective JSON document store425,427or429exceeds a user or system-predefined threshold. The lazy option, reloading all documents in the JSON document stores425,427and429, is only implemented at job re-execution. Then, at block509(FIG. 5), in response to determining that the data should receive an incremental update, one or more new RDDs are merged and built for the corresponding set of RDDs which incorporate the incremental data update. This step may be performed using the RDD merger411(FIG. 4). Next, at block511(FIG. 5), an execution of the data analytics job is triggered.

FIG. 6is a flowchart illustrating an exemplary method for estimating a cost of performing data analytics in accordance with one or more embodiments of the present invention. This estimating step was previously described, for example, in conjunction with block305ofFIG. 3.

The operational sequence ofFIG. 6commences at block601where a set of meta statistics is obtained from at least one of a distributed document storage database or a database log. The distributed document storage database may comprise one or more JSON document stores425,427or429(FIG. 4). For example, the enhanced connector403is configured for gathering one or more of the following meta statistics: data_size (Ds), doc_count(Dc), document identifiers (ids) including ids_deleted(Ids_D), ids_updated(Ids_U), ids_created (Ids_C), network_throughput (MB/s)(Nt_T), and rdd_merge_throughput(Doc count/s)(rdd_m_T). The network_throughput and rdd_merge_throughput can be obtained by scheduling tests prior to fetching RDDs or by using values from an immediately preceding job execution.

The operational sequence ofFIG. 6progresses to block603where one or more meta statistics are derived from the gathered meta statistics. The derived statistics may include, for example, a network cost Nc and a resilient distributed dataset (RDD) cost Rc. It may be noted that the enhanced connector403(FIG. 4) is configured for deriving one or more of the following meta statistics from the gathered meta statistics: average_doc_size (avg_Ds)=Ds/Dc; update_size (Us)=(Ids_U=Ids_C)*avg_Ds; network_cost (Nc)=Us/Nt_T; rdd_cost(Rc)=(Ids_D+Ids_U+Ids_C))/rdd_m_T; total_cost (Tc)=Rc+Nc; total cost of fetching all documents from a JSON store (Tc_O); total cost of fetching changed documents from the JSON store (Tc_D); and a total_cost_proportion (Tc_P) ratio=Tc_O/Tc_D. Changed documents refer to documents that have been edited. Unchanged documents have not been edited after being saved to the JSON store.

Next, at block605(FIG. 6), the total_cost (Tc) is calculated as a sum of the network cost Nc and the RDD cost Rc. Then, at block607, the total_cost_proportion Tc_P ratio is calculated as the total cost of fetching all documents from the JSON store (Tc_O) divided by the total cost of fetching changed documents from the JSON store (Tc_D). A test is performed at block609to ascertain whether or not the total_cost_proportion Tc_P ratio is greater than a predetermined or specified threshold. If so, a job is submitted to fetch changed documents from the distributed document storage database (block611). The negative branch from block609leads to block613where a job is submitted to fetch all documents from the distributed document storage database.

FIG. 7is a flowchart illustrating an exemplary method for merging a resilient distributed dataset with one or more data updates in accordance with one or more embodiments of the present invention. Illustratively, the operational sequence ofFIG. 7may be used to implement block313ofFIG. 3where one or more data updates are merged into the first RDD. Block313may be performed by submitting an RDD merge job to a data analytics application, such as Spark™, to produce one or more new RDDs for the RDD205(FIG. 2). The merging procedure ofFIG. 7may be performed using the RDD merger411ofFIG. 4.

The procedure ofFIG. 7commences at block701where a first set of documents is obtained from a JSON document store (such as any of the first, second, or Nth JSON document stores425,427and429ofFIG. 4), where each of the documents in the first set is either deleted or updated. Then, at block703(FIG. 7), a filter operation is used on an existing RDD for the JSON document store to filter out all of these deleted or updated documents to generate a first new RDD. Thus, all documents whose identifier (Ids) is from a set comprising deleted Ids and updated Ids (Ids_D+Ids_U) are removed from the existing RDD to produce the first new RDD. This step may be performed by using a filter transform operation rdd.filter(func) where func returns True if a key of a current data item in Ids_D+Ids_U first RDD (rdd1)=rdd.filter(func(id)), where func(id) returns True if id is in Ids_D+Ids_u.

Next, at block705, a second set of documents is obtained from the JSON document store, where each of the documents in the second set is either updated or created. Then, at block706, a second new RDD is generated for this second set of documents. For example, let all documents in the set Ids_D+Ids_U be denoted as Doc. Read these documents to the second new RDD denoted as rdd2=spark.json.read(Doc).

The operational sequence ofFIG. 7progresses to block707where a union transformation is used to combine the first new RDD with the second new RDD to form a newly updated RDD that includes merged data updates: (rdd_new=rdd1.union(rdd2). Then, at block709, a new data analytics job is executed on the newly updated RDD.

FIG. 8illustrates an exemplary network for performing data analytics in accordance with one or more embodiments of the present invention. This computer system is only one example of a suitable processing system and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the methodology described herein. The processing system shown may be operational with numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with the processing system shown inFIG. 8may include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like.

The computer system may also communicate with one or more external devices26such as a keyboard, a pointing device, a display28, etc.; one or more devices that enable a user to interact with the computer system; and/or any devices (e.g., network card, modem, etc.) that enable the computer system to communicate with one or more other computing devices. Such communication can occur via Input/Output (I/O) interfaces20.