Processing large data sets from heterogeneous data sources using federated computing resources

Disclosed are various embodiments of systems, methods, and computer programs that can facilitate processing of large data sets from data sources. Examples of the disclosure can allow a client to perform data processing tasks in computing resources that are external to a data source from which the data is retrieved rather than relying upon the computing resources of the data source.

BACKGROUND

In various settings, data processing tasks involving large data sets can be required in various use cases. For example, email targeting campaigns can require processing data sets involving various tables of data about users from various data sources as well as performing database queries on these data sets to identify users to whom targeted emails will be sent. In this scenario, data might be needed from different data stores and from various data tables within the different data stores. Additionally, for a large scale email targeting campaign, data store queries involving billions and billions of data entries might be required in order to identify the targets of such a campaign. These operations can tax the computing resources of the data stores from which the data is being retrieved, which can hamper the performance of the data stores with respect to other operations that the data stores might be tasked to perform.

DETAILED DESCRIPTION

Embodiments of the present disclosure relate to performing data processing tasks based upon data obtained from one or more data sources. Data can be obtained from data sources that are external to a computing environment in which a query engine is executed and a query script or any other executable code is interpreted. For example, a computing environment tasked by a client or user with generating an email targeting campaign may require data from various data sources that are external to the computing environment. In some scenarios, these data sources may be operated and maintained by other users, teams, or entities. For example, in a large enterprise, a marketing team can be tasked with running an email marketing campaign based upon data that is housed in a data source that is maintained by an operations team. In such a scenario, the team or entity that is charged with maintaining the data source can prioritize the needs of external users lower than its own users. However, external users may require very large data sets that are generated using resource intensive queries or other data operations from the data source.

In some scenarios, a user, team, or entity charged with maintaining a data source, in order to prioritize its own uses and the uptime of the data source above the needs of external users, may limit the access of external users to the resources of the data source. In other scenarios, a data processing task can involve the retrieval of data from multiple data sources that are heterogeneous in nature. For example, in order to identify recipient users of an email marketing campaign, a client may require data from more than one data source. In one example, an email marketing campaign can take the form of emails that are sent to users who have viewed or purchased a particular item from a catalog. In this example, in order to identify users to whom the email marketing campaign will apply, one or more query can be executed by the data source in which data is housed regarding users who have viewed or purchased a particular item.

In a data source in which information about millions of users are housed, such a query (or queries) can consume considerable resources or considerable time. In some scenarios, an owner of the data source may restrict the ability of other users or clients to perform data processing operations that consume significant amounts of time or computing resources of the data source. Accordingly, embodiments of this disclosure facilitate performing data processing operations on potentially large data sets that can relieve much of the data processing burden from the data source from which the data sets are obtained.

With reference toFIG. 1, shown is a networked environment100according to various embodiments. The networked environment100includes a computing environment103, at least one computing instance computing environment104, at least one data source computing environment105, and a client106, which are in data communication with each other via a network109. The network109includes, for example, the Internet, intranets, extranets, wide area networks (WANs), local area networks (LANs), wired networks, wireless networks, or other suitable networks, etc., or any combination of two or more such networks. For example, such networks may comprise satellite networks, cable networks, Ethernet networks, and other types of networks.

In one embodiment, the computing environment103,104can comprise many computing devices or servers in which virtualized computing instances can be created. The virtualized computing instances correspond to virtual machines or computing resources that can perform data processing operations or any other operations that a computing device can perform. For example, a computing instance of a certain memory allocation, central processing unit (CPU) allocation, operating system, mass storage allocation, or other properties, can be created within the computing environment103,104.

Various applications and/or other functionality can be executed in the computing environment103according to various embodiments. Also, various data is stored in a data store112that is accessible to the computing environment103. The data store112may be representative of a plurality of data stores112as can be appreciated. The data stored in the data store112for example, is associated with the operation applications and/or computing instances executed by the computing environment103.

The components executed on the computing environment103, for example, include the data processing application115and the data source connector117. The data processing application115is executed to perform data processing operations on data obtained from data sources that are external to the computing environment103. In one example, the data processing application115can implement a query engine that can interpret scripts that are written in a query language. Examples of query languages that the data processing application115can interpret are structured query language (SQL) scripts, Hive scripts, or other fragments and scripts that are authored in other query languages.

The data processing application115can interpret query languages because a client or user manipulating or creating large data sets that are taken from relational or non-relational databases or data stores will often author a script in a query language to manipulate source data and generate a resultant data set in a format that can be used for a particular application. Returning again the example of an email targeting campaign in which certain users from a large population of users are identified as recipients of a marketing campaign, the initiator of such a campaign may wish to identify a subset of users from potentially millions of users and query against data that is stored about these users in multiple data sources. Additionally, generating an email targeting campaign can also involve identifying one or more products from a product catalog that can be recommended to users who are identified as a part of the targeting campaign. Oftentimes, the most efficient way to express the logic necessary to gather and sort data in this way is using a query language.

The data processing application115can interpret query language requests or commands that potentially involve large data sets and/or multiple data sources and act as an orchestrator for execution of the script. In other words, the data processing application115or computing instances121allocated in the computing instance computing environment104can interpret queries that query against external data sources and deconstruct the query into a command that requests data stored in the data source while executing data processing operations expressed by the query in computing resources that are external to the data source. For example, in the case of a data source that is a relational database, such as an ORACLE database, the data processing application115or a computing instance121can avoid requesting that the relational database perform a JOIN operation. Instead, the data processing application115or a computing instance121can individually request the tables between which the script is joining and perform the JOIN in computing resources allocated within the computing instance computing environment104that are external to the data source.

In this way, the data source129need not perform the JOIN on behalf of the data processing application115or computing instance121even though the query language script, when interpreted by other query engines, may have requested that the data source perform the JOIN and return the resultant data set. In this way, the computing resources of the external data source can avoid being taxed and the data processing application115can also allocate computing resources above and beyond those that are available to the data source, which can offer improved performance in some cases. Additionally, for queries that include other data processing tasks other than the mere retrieval of data, the data processing application115or computing instance121can request the source data from the data sources identified by the query and perform the data processing tasks in computing resources external to the data source.

The data source connector117can provide accessibility to external data sources to the data processing application115or computing instance121. The data source connector117can interpret metadata associated with various types of external data sources that specifies how a particular data source should be accessed. For example, a particular data source can provide an application programming interface (API) that specifies how data stored in the data source can be accessed. The API can also specify which data retrieval or data processing tasks are supported by the data source.

The computing instance computing environment104can represent a computing environment in which a user can create or allocate computing resources that can handle data processing tasks as directed by the data processing application115. In this sense, computing instances121that can be allocated or created within the computing environment. The computing environment104can comprise many computing devices or servers in which computing instances121can be created. Computing instances121correspond to virtual machines that can be spawned or created on one or more computing devices from which the computing environment103is devised. The computing instances121correspond to virtual machines or computing resources that can perform data processing operations or any other operations that a computing device can perform. For example, a computing instance121of a certain memory allocation, central processing unit (CPU) allocation, operating system, mass storage allocation, or other properties, can be created within the computing environment103.

In some scenarios, the data processing application115can be executed by a computing instance within the computing environment103. Additionally, data processing operations can be performed using computing instances121created within the computing environment104. In this way, a client can provide its own computing instances within the computing instance computing environment104rather than relying upon the computing resources of the data source129or the computing environment103to produce resultant data sets based upon a query script133or any other executable script or code.

The data source computing environment105corresponds to one or more computing devices associated with a data source from which data can be retrieved by the data processing application115to perform data transformation or data processing operations. The data source computing environment105can correspond, for example, to a database cluster, or any other type of data store or data warehouse from which data can be retrieved by the data processing application115. The data source computing environment105can execute one or more applications or services that provide an application programming interface (API) or other type of interface through data stored in the data source129can be retrieved. In one embodiment, the data source computing environment105is external to the computing environment103. In one scenario, the data source computing environment105can be operated by users or entities that are different from the users or entities associated with the computing environment103.

The data source computing environment105may comprise, for example, a server computer or any other system providing computing capability. Alternatively, the data source computing environment105can employ a plurality of computing devices that may be arranged, for example, in one or more server banks or computer banks or other arrangements. Such computing devices may be located in a single installation or may be distributed among many different geographical locations. For example, the data source computing environment105may include a plurality of computing devices that together may comprise a hosted computing resource, a grid computing resource and/or any other distributed computing arrangement. In some cases, the data source computing environment105may correspond to an elastic computing resource where the allotted capacity of processing, network, storage, or other computing-related resources may vary over time.

The data source computing environment105can house a data source129. The data source129corresponds to one or more data stores in which data can be stored. The data can correspond to customer data, transaction data, user data, or any other data that can be retrieved using a query language script or query language command. Additionally, the networked environment100can include multiple data source computing environments105. In one scenario, a query can reference different data sources129that are located in different data source computing environments105. The query can also specify a data processing operation that can be performed with respect to data referenced by the query to generate a resultant data set.

The data stored in the data store112includes, for example, data source data123, filter data229, a transaction history231, and user data233. The data source data123corresponds to data that identifies data sources129or data source computing environments105that are accessible by the data processing application115. In one scenario, a data source129can be registered with the data processing application115by providing metadata specifying how data housed in the data source129can be accessed by the data processing application115. In one example, the metadata can specify a particular API that is supported by the data source129or the data source computing environment105or a network address at which the data source129can be accessed. Connector data128can store the metadata that specifies how a particular data source129or data source computing environment105can be accessed. In some embodiments, the data source129or the data source computing environment105can implement a connector that supports a query language with which the data processing application115can retrieve data from the data source129.

In some scenarios, a particular data source129or data source computing environment105may require authenticated access in order to retrieve data from the data source129. Accordingly, the connector data128can also specify how an authenticated request to the data source129or data source computing environment105should be conducted. For example, the connector data128can specify that an access policy should be created that grants access to the data source129on behalf of a particular client or user account within the data source computing environment105.

Client data125represents data about clients, or users, of the data processing application115. The data processing application115can support data processing operations based upon data retrieved from various data source computing environments105for multiple users. Accordingly, the client data125can house data corresponding to user accounts of multiple users of the data processing application115who can employ the data processing application115for different data processing tasks. The client data125can include information regarding data sources131to which a particular client or user account has access. Query scripts133can store query language scripts that define data processing tasks for execution by the data processing application115. In one example, a query script133can include a Hive script, a SQL script, or any other type of code, markup language, or scripting language that can be executed by the data processing application115to generate a resultant data set. A query script133can also include code that is executable by computing environment103.

Computing instance data127represents data that corresponds to computing instances121that are created within the computing instance computing environment103. The computing instance data127identifies computing instances121that have been created within the computing instance computing environment103as well as the data processing tasks or query scripts133to which a particular computing instance121is assigned. The computing instance data127can also specify an operating system, CPU, memory, or other properties of a particular computing instance121created within the computing instance computing environment104.

The client106is representative of a plurality of client devices that may be coupled to the network109. The client109can represent a computing device used by a user initiating a data processing task that is handled by the data processing application115. The client106can include, for example, a processor-based system such as a computer system. Such a computer system may be embodied in the form of a desktop computer, a laptop computer, personal digital assistants, cellular telephones, smartphones, set-top boxes, music players, web pads, tablet computer systems, game consoles, electronic book readers, or other devices with like capability. The client106may include a display171. The display171may comprise, for example, one or more devices such as liquid crystal display (LCD) displays, gas plasma-based flat panel displays, organic light emitting diode (OLED) displays, electrophoretic ink (E ink) displays, LCD projectors, or other types of display devices, etc.

The client106may be configured to execute various applications such as a browser274and/or other applications. The browser274may be executed in a client106, for example, to access network content served up by the computing environment103and/or other servers, thereby rendering a user interface150on the display171. In particular, the user interface150can be generated by the data processing application115or other applications executed in the computing environment103and rendered by the client106. A user can configure the data processing application115or initiate a data processing task within the data processing application115via the client106.

Next, a general description of the operation of the various components of the networked environment100is provided. The data processing application115is executed to facilitate the processing of potentially large data sets that can be obtained from potentially multiple data sources129. One example of such a task is constructing a resultant data set from multiple data sets in various data sources129. A data processing task can take the form of one or more queries in a query language in a query script133that are interpreted by the data processing application115.

The data processing application115can interpret a query in a query script133and execute various queries such that data processing tasks, such as, for example, JOIN operations or other processing intensive operations, are performed in one or more computing instances121created by the data processing application115rather than by the data source computing environment105. Accordingly, in order to execute a query script133, the data processing application115or a user can create a computing instance121that can be assigned to execution of the query script133. The query script133can define one or more queries or other operations that direct the data processing application115to obtain data from one or more data sources129and generate a resultant data set, such as a list of the identity of multiple users who are the target of an email marketing campaign.

The data processing application115can interpret a query script133and execute the queries and other commands within the query script133in order to generate a resultant data set. In some examples, the data processing application115can allocate a computing instance121and assign execution of the query script133to the computing instance121. In other examples, upon reaching a query in a query script133that incorporates a request for data from a data source129, the data processing application115can create a computing instance121assigned to execution of the query script133. In either scenario, the computing instance121can retrieve the data from the data source129(and potentially multiple data sources129) and load the data into the computing instance121. The allocated computing instance121can also transform the data according to a data processing task contained within the query script133in order to generate resultant data. In some examples, the data processing task115can allocate multiple computing instances121to a data processing task depending upon the size of the data set that is associated with a task specified by a query script133.

Additionally, the data processing application115can allocate more computing instances121to execution of a query script133depending upon how quickly a particular task needs to be completed. For example, if a task requires processing of a particularly large data set or particularly fast execution, the data processing application115can execute more than one computing instance121to a data processing task or allocate a computing instance121with additional virtual computing resources, such a virtual CPUs, RAM, mass storage, etc.

In some embodiments, the data processing application115or a computing instance121generated by the data processing application115, upon reaching a query that incorporates a data processing task with respect to data stored in a data source129, can generate a command supported by the data source129to retrieve the data according to the connector data128. The data processing application115can then invoke the data source connector117to retrieve the data identified by the query. The data can be identified by a table name, a field name, and/or any other identifier by which the data can be identified within the data source129.

Accordingly, the data processing application115can initiate loading of the data retrieved from the one or more data sources129into the computing instance121assigned to execution of the query script133. The data processing application115or computing instance121can then perform a data processing task specified by the query. For example, the data retrieved from one or more data sources129may include various data tables that correspond to data that is of interest. The data processing task specified by the query can include a JOIN operation that generates a resultant data set. The data processing application115can execute the JOIN operation using a computing instance121that is external to the data source129or data source computing environment105. In this way, the data processing application115can facilitate processing of a large data set without necessarily taxing the resources of the data sources129or data source computing environments105from which the data was obtained.

In some scenarios, the data source129required by a particular query in a query script133may require authenticated access on behalf of a client or user for whom the query script133is executed within the data processing application115or computing instance121. In this scenario, a query script133interpreted by the data processing application115or computing instance121can facilitate generating an access policy authorizing access to the data source on behalf of a particular user or client. In this scenario, the access policy provides federated access to the data source on behalf of the client or user. In this way, the client or user initiating execution of the query script133provides an authentication credential for access to the data source129.

In another scenario, the query script133can include one or more commands that generate or retrieve an authentication token with which the data processing application115or computing instance121can access a data source129. The data processing application115or computing instance121can be configured to discard such an authentication token upon accessing the data source. In some examples, the authentication token can expire after a predetermined period of time.

In some scenarios, access to the data source129may be impaired due to downtime of the data source129and/or network conditions. Accordingly, in this scenario, the data processing application115can retry attempts to access data corresponding to the query. In this way, a client or user initiating execution of the query script133need not be aware of the status of the data source129, as the data processing application115handles failed operations resulting from execution of a query script133on behalf of the user or client.

Upon loading of the data from one or more data source129, the data processing application115or computing instance121can then perform a data processing task or data transformation operation embodied in a particular query or query script133. The data processing application115or computing instance121can generate a resultant data set, which can comprise a table of data that is of interest to a user or client initiating execution of the query script133. The resultant data set can be used for any purpose and is generated without taxing the sources of the data source129or data source computing environment105. In other words, the data processing application115can facilitate execution of the query in the computing instance121instead of using the computing resources of the data source129or the data source computing environment105.

Referring now toFIG. 2, shown is a depiction of a data processing pipeline facilitated by the data processing application115according to various embodiments of the disclosure.FIG. 2illustrates how the data processing application115, in response to a query in a query script133that contains a data processing task that is based upon data stored in one or more data sources129, can facilitating loading the data from the data sources129into a data processing pipeline.

As depicted inFIG. 2, data201can be retrieved or extracted from one or more data sources129by the data processing application115or a computing instance121in the computing instance computing environment104. The data201retrieved from the data sources129is associated with a query interpreted by a query engine implemented by the data processing application115. The data processing application115can identify a data processing task associated with the query and transmit a command to retrieve the data201from the data sources129. In some embodiments, the data processing application115can identify metadata from the data source data123or client data125that identifies how data from the data source129can be retrieved. The data processing application115can generate a computing instance121within the computing instance computing environment104and provide the metadata to the computing instance121. In one embodiment, the data processing application115can direct the computing instance121to retrieve the data from the data source129.

Upon receiving the data201from the data sources129, the data201can be loaded into a computing instance121assigned to execution of the data processing application115. In one embodiment, the data processing application115can create a computing instance121and direct the computing instance121to transmit a command to retrieve the data201from the data source129. In this way, the data processing application115can facilitate execution of a query in the computing instance121instead of using the resources of the data source129or a data source computing environment105from which the data201is obtained. Utilizing the resources of the computing instance121can also offer improved performance with respect to creation resultant data203based upon the query in a query script133.

With reference toFIG. 3, shown is a flowchart that provides one example of the execution of the data processing application115. Alternatively,FIG. 3shows steps of a method implemented in the computing environment103.FIG. 3illustrates an example of the data processing application115retrieving data from a data source129. Accordingly, at box301, the data processing application115can identify a query that includes a data processing task in a query script133. As noted above, the query can reference data from data sources129that are external to the computing environment103in which the data processing application115is executed.

At box303, the data processing application115can identify a data source129associated with the query. In other words, the data processing application115can determine that the query reference data that is stored in an external data source129. At box305, the data processing application115can obtain authenticated access to the data source129on behalf of a client106or user initiating execution of the query script133. Authenticated access can be obtained by obtaining an authentication token or federated access to the data source129on behalf of the user or client106. In some embodiments, authenticate access can be obtained for a computing instance121created by the data processing application115that is created to perform a data processing task represented in a query from the query script133. At box307, the data processing application115can retrieve metadata specifying how data can be retrieved from the data source129by a computing instance121in the computing instance computing environment104.

With reference toFIG. 4, shown is a flowchart that provides one example of the execution of the data processing application115. Alternatively,FIG. 4shows steps of a method implemented in the computing environment103.FIG. 4illustrates an example of the data processing application115facilitating the creation of a resultant data set from data obtained from one or more data sources129. Accordingly, at box401, the data processing application115can generate a computing instance121in which a data processing task corresponding to a query can be executed. At box403, the data processing application115can provide metadata specifying on data can be obtained by the computing instance121from the one or more data sources129. In one example, the data can be loaded into a mass storage resource allocated to the computing instance121or a memory resource of the computing instance121.

At box405, the data processing application115can initiate execution of the data processing task specified by a query in the computing instance121. At box407, the data processing application115can create a resultant data set based upon the data and the data processing task. The resultant data set can comprise a table with data resulting from the query or the query script133. At box409, the data processing application115or computing instance121created by the data processing application115can publish the resultant data generated by the computing instance121. Publishing the resultant data can take many forms. In one embodiment, publishing the resultant data can take the form saving the resultant data back to the data source129. In another embodiment, publishing the resultant data can include transmitting a resultant data set to a client device106. In yet another embodiment, publishing the resultant data can include publishing a resultant data set to another data source129that is different from the data source129from which the initial data was obtained.

With reference toFIG. 5, shown is a schematic block diagram of the computing environment103according to an embodiment of the present disclosure. The computing environment103includes one or more computing devices500. Each computing device500includes at least one processor circuit, for example, having a processor503and a memory506, both of which are coupled to a local interface509. To this end, each computing device500may comprise, for example, at least one server computer or like device. The local interface509may comprise, for example, a data bus with an accompanying address/control bus or other bus structure as can be appreciated.

Stored in the memory506are both data and several components that are executable by the processor503. In particular, stored in the memory506and executable by the processor503are the data processing application115, data source connector117, and potentially other applications. Also stored in the memory506may be a data store112and other data. In addition, an operating system may be stored in the memory506and executable by the processor503.

Also, the processor503may represent multiple processors503and/or multiple processor cores and the memory506may represent multiple memories506that operate in parallel processing circuits, respectively. In such a case, the local interface509may be an appropriate network that facilitates communication between any two of the multiple processors503, between any processor503and any of the memories506, or between any two of the memories506, etc. The local interface509may comprise additional systems designed to coordinate this communication, including, for example, performing load balancing. The processor503may be of electrical or of some other available construction.

Further, any logic or application described herein, including the data processing application115and data source connector117, may be implemented and structured in a variety of ways. For example, one or more applications described may be implemented as modules or components of a single application. Further, one or more applications described herein may be executed in shared or separate computing devices or a combination thereof. For example, a plurality of the applications described herein may execute in the same computing environment103or in multiple computing devices in the same computing environment103. Additionally, it is understood that terms such as “application,” “service,” “system,” “engine,” “module,” and so on may be interchangeable and are not intended to be limiting.