English:
[0001] As the technological capacity for organizations to create, track, and retain information continues to grow, a variety of different technologies for managing and storing the rising tide of information have been developed. Database systems, for example, provide clients with many different specialized or customized configurations of hardware and software to manage stored information. The increasing amount of data that organizations must store and manage often correspondingly increases both the size and complexity of data storage and management technologies, like database systems, which in turn escalate the cost of maintaining the information. New technologies seek to reduce both the complexity and storage requirements of maintaining data by introducing different data formats that offer different processing or maintenance capabilities. However, introducing multiple data formats is not without cost. Data is often processed by different systems which may not support the current data format of the data. Thus, the ability to perform techniques that extract, transform, and load data between different formats or locations is desirable.
[0002] FIG. 1 illustrates a logical block diagram of event driven extract, transform, load (ETL) processing, according to some embodiments.
[0003] FIG. 2 is a block diagram illustrating a provider network offering different services including an extract, transform, load (ETL) service that performs event driven ETL processing, according to some embodiments.
[0004] FIG. 3 is a block diagram illustrating an ETL service that performs event driven ETL processing, according to some embodiments.
[0005] FIG. 4 is a logical block diagram illustrating ETL job management, according to some embodiments.
[0006] FIG. 5 is a logical block diagram illustrating interactions to execute an ETL job, according to some embodiments.
[0007] FIG. 6 illustrates example interactions to register a trigger event for an ETL job, according to some embodiments.
[0008] FIG. 7 is a high-level flowchart illustrating methods and techniques to implement event driven ETL processing, according to some embodiments.
[0009] FIG. 8 illustrates an example system configured to implement the various methods, techniques, and systems described herein, according to some embodiments.
[0010] While embodiments are described herein by way of example for several embodiments and illustrative drawings, those skilled in the art will recognize that embodiments are not limited to the embodiments or drawings described. It should be understood, that the drawings and detailed description thereto are not intended to limit embodiments to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope as defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. As used throughout this application, the word "may" is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must). Similarly, the words "include," "including," and "includes" mean including, but not limited to.
[0011] It will also be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the scope of the present invention. The first contact and the second contact are both contacts, but they are not the same contact.
[0012] Various embodiments of event driven extract, transformation, and load (ETL) processing are described herein. ETL processing provides data administrators, stake holders, producers, or consumers with the ability to take advantage of different data formats, storage systems, or storage locations by facilitating the movement data across different data stores and data schemas or formats. For example, data producers that collect usage and other analytics for web site interactions (e.g., visitor clicks and other actions), may generate stored analytics data in large log files or other semi-structured data formats. However, in order to perform various analytical operations or queries over the analytics data, an ETL process may be performed to extract desired data, transformed the desired data into a format usable by an analytics engine, like a database system, and load the extracted data into the appropriate storage system in the appropriate data schema and format.
[0013] Typical ETL processing techniques rely upon chronological mechanisms for initiating the execution of ETL processing. An ETL process may be scheduled to execute nightly, or once a week at a certain time on a certain day. However, because ETL processing jobs interact with and transform data, the timing for executing ETL processing jobs may be optimally aligned with changes to data or events respecting data as data is not always created, updated, modified, or deleted upon a schedule, nor created, updated, modified, or deleted in the same way. For instance, new data objects may be created or stored in a data store to add another day's worth of sales data or may be stored as corrections to a larger data set. Event driven ETL processing may discriminate between optimal ETL execution scenarios and non-optimal ETL execution scenarios to account for the differing nature of changes to data.
[0014] FIG. 1 illustrates a logical block diagram of event driven extract, transform, load (ETL) processing, according to some embodiments. Data store(s) 110 may store data objects on behalf of one or multiple clients. Data objects may be may any form of data or data structure, (e.g., file, directory, database table, log, block, partition, chunk, or range of bytes). Various data events 120 may occur in a data store with respect to stored data objects. For example, data creation events 122 may result from the storing of new data objects. A data producer, for example, may upload a new log file into data store(s) 120, creating a new data object. In another example, a new partition for database table may be added. Data modification event(s) 124 may occur when data is updated or transformed to create other data objects. For instance, an aggregation operation may be performed to access raw data stored in a data object, aggregate the values of different fields, and store the aggregated values in another data object (e.g., calculate total sales from a data object storing daily transactions and store the total sales for the day in a new entry in a revenue table). Data deletion events 126 may occur to archive, move, or remove data objects from data store(s) 110. For example, raw data objects may be discarded once a cleaned version of the data object is created.
[0015] As illustrated in FIG. 1, different data events 120 may serve as trigger events for ETL process 150 that are executed by ETL job processing 130. For example, a trigger event (e.g., triggered by a specified data event 120) may be associated with an ETL process (e.g. in a trigger event registry as discussed below with regard to FIG. 4). ETL job processing 130 may detect the occurrence of the trigger event 150 and determine whether the ETL process should be executed using execution criteria specified for the ETL process. For example, ETL job processing may include execution criteria analysis 140 to compare various attributes of a source data object for the ETL process with different criteria (e.g., size of the source data object with a threshold execution size, number of items, such as rows, within the source data object, with a row threshold, etc.). Other execution criteria, such as those discussed below with regard to FIGS. 4, 6, and 7 may be evaluated, including criteria for evaluating the status of other ETL processes.
[0016] If the execution criteria are satisfied, then ETL job processing 130 may execute the triggered ETL process 160. For example, ETL job processing 130 may obtain portions of a source data object in data store(s) 110, apply different transformations as part of the ETL processing, such as transformations that convert, combine or otherwise modify data values obtained from the source data object, remove, filter or drop data values obtained from the source data object, or restructure, rearrange, or relocate data values obtained from the source data object. The results of the applied transformations may be included in one or more transformed data objects which may be stored in data store(s) 110. ETL job processing 130 may create different data events as a result of executing ETL processes and thus may initiate subsequent ETL processing.
[0017] Please note that the previous description of event driven ETL processing is a logical illustration and thus is not to be construed as limiting as to the architecture for implementing a data store or ETL job processing.
[0018] This specification begins with a general description of a provider network that implements an extract, transform, load (ETL) service that identifies, transforms, and moves data stored in the provider network or in external data stores. Then various examples of the ETL service including different components/modules, or arrangements of components/module that may be employed as part of implementing the ETL service are discussed. A number of different methods and techniques to implement event driven ETL processing are then discussed, some of which are illustrated in accompanying flowcharts. Finally, a description of an example computing system upon which the various components, modules, systems, devices, and/or nodes may be implemented is provided. Various examples are provided throughout the specification.
[0019] FIG. 2 is a block diagram illustrating a provider network offering different services including an extract, transform, load (ETL) service that performs event driven ETL processing, according to some embodiments. Provider network 200 may be a private or closed system or may be set up by an entity such as a company or a public sector organization to provide one or more services (such as various types of cloud-based storage) accessible via the Internet and/or other networks to clients 250. Provider network 200 may be implemented in a single location or may include numerous data centers hosting various resource pools, such as collections of physical and/or virtualized computer servers, storage devices, networking equipment and the like (e.g., computing system 1000 described below with regard to FIG. 8), needed to implement and distribute the infrastructure and storage services offered by the provider network 200. In some embodiments, provider network 200 may implement various computing resources or services, such as a data storage service(s) 210 (e.g., object storage services, block-based storage services, or data warehouse storage services), ETL service 220, as well as other service(s) 230, which may include a virtual compute service, data processing service(s) (e.g., map reduce, data flow, and/or other large scale data processing techniques), and/or any other type of network based services (which may include various other types of storage, processing, analysis, communication, event handling, visualization, and security services not illustrated).
[0020] In various embodiments, the components illustrated in FIG. 2 may be implemented directly within computer hardware, as instructions directly or indirectly executable by computer hardware (e.g., a microprocessor or computer system), or using a combination of these techniques. For example, the components of FIG. 2 may be implemented by a system that includes a number of computing nodes (or simply, nodes), each of which may be similar to the computer system embodiment illustrated in FIG. 8 and described below. In various embodiments, the functionality of a given system or service component (e.g., a component of data storage service 230) may be implemented by a particular node or may be distributed across several nodes. In some embodiments, a given node may implement the functionality of more than one service system component (e.g., more than one data store component).
[0021] Data storage service(s) 210 may implement different types of data stores for storing, accessing, and managing data on behalf of clients 250 as a network-based service that enables clients 250 to operate a data storage system in a cloud or network computing environment. For example, data storage service(s) 210 may include various types of database storage services (both relational and non-relational) or data warehouses for storing, querying, and updating data. Such services may be enterprise-class database systems that are scalable and extensible. Queries may be directed to a database or data warehouse in data storage service(s) 210 that is distributed across multiple physical resources, and the database system may be scaled up or down on an as needed basis. The database system may work effectively with database schemas of various types and/or organizations, in different embodiments. In some embodiments, clients/subscribers may submit queries in a number of ways, e.g., interactively via an SQL interface to the database system. In other embodiments, external applications and programs may submit queries using Open Database Connectivity (ODBC) and/or Java Database Connectivity (JDBC) driver interfaces to the database system.
[0022] Data storage service(s) 210 may also include various kinds of object or file data stores for putting, updating, and getting data objects or files, which may include data files of unknown file type. Such data storage service(s) 210 may be accessed via programmatic interfaces (e.g., APIs) or graphical user interfaces. Data storage service(s) 210 may provide virtual block-based storage for maintaining data as part of data volumes that can be mounted or accessed similar to local block-based storage devices (e.g., hard disk drives, solid state drives, etc.) and may be accessed utilizing block-based data storage protocols or interfaces, such as internet small computer interface (iSCSI).
[0023] In some embodiments, ETL service 220 may create and dynamically update a catalog of data stored on behalf of clients in provider network 200 across the various data storage services 210, as discussed in detail below with regard to FIG. 3. For example, a database stored in a non-relational database format may be identified along with container storing objects in an object-based data store as both being stored on behalf of a same customer of provider network 200. ETL service 220 may also perform ETL jobs that extract, transform, and load from one or more of the various data storage service(s) 210 to another location. For example, ETL service 220 may provide clients with the resources to create, maintain, and orchestrate data loading jobs that take one or more data sets, perform various transformation operations, and store the transformed data for further processing (e.g., by one or more of data processing service(s)). ETL service 220 may access a data catalog generated by ETL service 220 in order to perform an ETL operation (e.g., a job to convert a data object from one file type into one or more other data objects of a different file type). As discussed in detail below with regard to FIGS. 3 - 5, ETL service 220 may generate transformation workflows on behalf of clients automatically.
[0024] Other service(s) 230 may include various types of data processing services to perform different functions (e.g., anomaly detection, machine learning, querying, or any other type of data processing operation). For example, in at least some embodiments, data processing services may include a map reduce service that creates clusters of processing nodes that implement map reduce functionality over data stored in one of data storage services 210. Various other distributed processing architectures and techniques may be implemented by data processing services (e.g., grid computing, sharding, distributed hashing, etc.). Note that in some embodiments, data processing operations may be implemented as part of data storage service(s) 210 (e.g., query engines processing requests for specified data). Data processing service(s) may be clients of ETL service 220 in order to invoke the execution of an ETL job to make data available for processing in a different location or data format for performing various processing operations with respect to data sets stored in data storage service(s) 210.
[0025] Generally speaking, clients 250 may encompass any type of client configurable to submit network-based requests to provider network 200 via network 260, including requests for storage services (e.g., a request to create, read, write, obtain, or modify data in data storage service(s) 210, a request to generate an ETL job at ETL service 220, etc.). For example, a given client 250 may include a suitable version of a web browser, or may include a plug-in module or other type of code module configured to execute as an extension to or within an execution environment provided by a web browser. Alternatively, a client 250 may encompass an application such as a database application (or user interface thereof), a media application, an office application or any other application that may make use of storage resources in data storage service(s) 210 to store and/or access the data to implement various applications. In some embodiments, such an application may include sufficient protocol support (e.g., for a suitable version of Hypertext Transfer Protocol (HTTP)) for generating and processing network-based services requests without necessarily implementing full browser support for all types of network- based data. That is, client 250 may be an application configured to interact directly with provider network 200. In some embodiments, client 250 may be configured to generate network-based services requests according to a Representational State Transfer (REST)-style network-based services architecture, a document- or message-based network-based services architecture, or another suitable network-based services architecture.
[0026] In some embodiments, a client 250 may be configured to provide access to provider network 200 to other applications in a manner that is transparent to those applications. For example, client 250 may be configured to integrate with an operating system or file system to provide storage on one of data storage service(s) 210 (e.g., a block-based storage service). However, the operating system or file system may present a different storage interface to applications, such as a conventional file system hierarchy of files, directories and/or folders. In such an embodiment, applications may not need to be modified to make use of the storage system service model. Instead, the details of interfacing to the data storage service(s) 210 may be coordinated by client 250 and the operating system or file system on behalf of applications executing within the operating system environment.
[0027] Clients 250 may convey network-based services requests (e.g., access requests directed to data in data storage service(s) 210, operations, tasks, or jobs, being performed as part of other service(s) 230, or to interact with ETL service 220) to and receive responses from provider network 200 via network 260. In various embodiments, network 260 may encompass any suitable combination of networking hardware and protocols necessary to establish network- based-based communications between clients 250 and provider network 200. For example, network 260 may generally encompass the various telecommunications networks and service providers that collectively implement the Internet. Network 260 may also include private networks such as local area networks (LANs) or wide area networks (WANs) as well as public or private wireless networks. For example, both a given client 250 and provider network 200 may be respectively provisioned within enterprises having their own internal networks. In such an embodiment, network 260 may include the hardware (e.g., modems, routers, switches, load balancers, proxy servers, etc.) and software (e.g., protocol stacks, accounting software, firewall/security software, etc.) necessary to establish a networking link between given client 250 and the Internet as well as between the Internet and provider network 200. It is noted that in some embodiments, clients 250 may communicate with provider network 200 using a private network rather than the public Internet.
[0028] FIG. 3 is a block diagram illustrating an ETL service that performs event driven ETL processing, according to some embodiments. ETL service 220 may provide access to data catalogs 360 and ETL jobs (for creation, management, and execution) via interface 310, which may be a programmatic interface (e.g., Application Programming Interface (API)), command line interface, and/or graphical user interface, in various embodiments.
[0029] ETL Service 220 may implement ETL job creation 320 to handle the creation of ETL jobs through manual job creation (e.g., creating, edit, or uploading ETL code or creating or editing graphs of ETL jobs) or through automated job creation. ETL job creation 320 may handle requests for automated ETL job creation and manual ETL job creation. For example, ETL job creation 320 may receive job generation request which may specify the data object and target data format for the ETL job. Other job information, such as access credentials, triggering events, or any other information to execute the ETL job may be included as part of the creation request or as part of a trigger event registration request, discussed below with regard to FIG. 6. ETL job creation 320 may automatically generate ETL code to perform an ETL job by determining the source data format of the data object and the target data format of the data object. For example, in one embodiment, the source and target data formats from data catalog 360. In another embodiment, data format identification may perform data format recognition techniques, or access other data stores (e.g., such as a data catalog stored in relational database) to retrieve the data format information. ETL job creation 320 may then compare the source data format and target data format or schema to select transformations to apply to the source data object to achieve the target data format. ETL job creation 320 may then generate code for selected transformations and construct the source code for executing the selected transformations. The code for the ETL job may be stored in ETL job store 350 for subsequent execution.
[0030] ETL job creation 320 may also implement manual creation of ETL jobs. For example, transformation operations may be manually selected, combined, or assembled via graphical user interface to define a workflow of transformations to apply. Code corresponding to the workflow may be generated (or supplied by a user), edited, and stored for subsequent execution as part of ETL job store 350.
[0031] ETL service 220 may implement ETL job management 330 to provide clients with the ability to manage, edit, delete, or otherwise change ETL jobs. Trigger events, may also be defined for ETL jobs (as discussed below with regard to FIG. 6). ETL job management 330 may monitor for trigger events and request execution of ETL jobs, as discussed below with regard to FIG. 4.
[0032] ETL service 220 may implement ETL job execution 340 to provide an execution platform ETL jobs. In some embodiments, ETL job execution 340 may provide a serverless architecture (from the perspective of clients) so that the appropriate number of resources are provisioned (e.g., virtual compute instances from a virtual compute service executing the ETL job code) in order to satisfy performance requirements, objectives, or goals provided by a client or by ETL service 220. ETL job execution 340 may execute jobs, in some embodiments, automatically without any user editing changes to the automatically generated ETL code from ETL job creation 320. In some embodiments, ETL job execution 340 may execute automatically generated ETL jobs that were modified. ETL job execution 340 may execute jobs in response to detected triggering events for ETL jobs (which may be detected by ETL job management or another system or service monitoring for triggering event conditions), as discussed below with regard to FIG. 5.
[0033] ETL service 220 may maintain data catalogs 360 that describe data sets (stored in provider network 200 or in external storage locations). ETL service 220 may identify unknown data objects, identify a data format for the unknown data objects and store the data format in a data catalog for the unknown data objects. ETL service 220 allow for catalog users, owners, or other stakeholders, to modify or otherwise manage data catalogs. For instance, ETL service 220 may process and execute access requests directed to data catalog(s) 360 (e.g., requests to combine, delete, or split tables of metadata in the catalog or edit the metadata determined for a data catalog). ETL service 220 may implement access or control policies for data catalogs (e.g., to limit access to a data catalog to authorized users). ETL service 220 may implement data retention or life cycle policies to determine how long data catalogs (or older versions of data catalogs) are maintained. ETL service 220 may handle the provisioning of storage resources in data for creating new data catalogs. ETL service 220 may also perform load balancing, heat management, failure recovery, and other resource management techniques (e.g., implement durability requirements) to ensure the availability of data catalogs for clients.
[0034] Storage for data catalog(s) 360 may be implemented by one or more storage nodes, services, or computing devices (e.g., system 1000 discussed below with regard to FIG. 8) to provide persistent storage for data catalogs generated by data catalog service 200. Such storage nodes (or other storage components of storage for data catalog(s) 360) may implement various query processing engines or other request handling components to provide access to data catalogs according to requests received via interface 310. For example, data catalog storage may be implemented as a non-relational database, in one embodiment, that stores file types and other metadata for data objects in table. In some embodiments, collections of metadata for various data objects stored across different storage service(s) 210 on behalf a single user account may be stored together in a single catalog of metadata that may be made accessible to clients.
[0035] FIG. 4 is a logical block diagram illustrating ETL job management, according to some embodiments. ETL job management 330 may monitor for trigger events for ETL jobs and request the execution of ETL jobs that satisfy execution criteria for the ETL job. ETL job management 330 may implement trigger event registry 430, in various embodiments. Trigger event registry 430 may be a data store, such as database, or other data storage system that tracks the trigger events 432 for which monitoring is enabled, as well as the execution criteria 434 for evaluating whether an ETL job may be performed. Trigger event registry 430 may be updated by requests 470 to register or update trigger events, as discussed in detail below with regard to FIG. 6. For example, a request 470 to register a trigger event may identify the ETL job, type of trigger event, and execution criteria to be applied for determining whether execution even of the ETL job when the trigger event is detected can proceed.
[0036] ETL job management 330 may implement trigger event monitoring 410 to detect trigger events for ETL jobs. For example, trigger event monitoring 410 may implement listeners or other processes that wait and identify events for a specified ETL job. For example, trigger event monitoring 410 may enable a listener for an ETL job that operates upon data source X by configuring the listener to identify updates 440 to data source X amongst update notifications provided to ETL job management 330. In some embodiments, trigger event monitoring 410 may monitor, get, or evaluate data object updates 450 provided 440 to data catalog 360. For example, ETL service 220 may make an interface to data catalog 360 available so that when a data producer, modifier, or other process that accesses a data object in a data store performs an action, a description of the update or a resulting update to the data objects metadata or data schema is provided 440 to data catalog 360 via the interface (e.g., via an API request to report the update). Trigger event monitoring 410 may subscribe to a feed or stream of updates (e.g., particular to a data object in the data catalog or the data catalog as a whole) in order to monitor the data object updates for trigger events. For other types of trigger events, trigger event monitoring may implement other listeners, scanners, or handlers to retrieve information about ETL process execution status, for example, or to implement a schedule for time-based trigger events.
[0037] Execution criteria evaluation 420 may lookup execution criteria 434 for detected trigger events 454 to determine whether the ETL job for the trigger even should be executed. For example, in various embodiments, execution criteria may include criteria with respect to a source data object for the ETL job (e.g., size, data values, or other metadata or attributes of the data object). Execution criteria evaluation 420 may obtain the needed information from data catalog 360, such as data object attributes 460 (e.g., data object size, number of rows added, etc.) to evaluate execution criteria. Although not illustrated, information to evaluate other execution criteria (e.g., by accessing ETL job status information or data objects directly) may be obtained, in different embodiments, to evaluate execution criteria. For those trigger events that are evaluated and satisfy the execution criteria, an ETL job execution request 480 may be provided to ETL job execution service 340 to execute the job identified by the request 480.
[0038] FIG. 5 is a logical block diagram illustrating interactions to execute an ETL job, according to some embodiments. ETL job execution 340 may implement an ETL job execution coordinator 510 that assigns ETL jobs 512 to one or more ETL job execution workers(s) 520 which may access source data store 530 and target data store 540 to obtain data 524, apply transformations, and store transformed data 526. ETL job execution request 502 may identify the job to execute (e.g., by include a job name or other identifier). ETL job execution coordinator 510 may determine the resources needed to execute the ETL job and assign the ETL job to one or more ETL job execution worker(s) 520.
[0039] ETL Job execution worker(s) 520 may get information 522 (including executable code, invoked operations or transformations, and other information to execute the identified ETL job) from ETL job store 350 for the ETL job. ETL job execution worker(s) 520 may then perform the ETL job in parallel or serialized fashion, obtaining data 524 from the source data store 530 (which may be a data storage service 210 of provider network 200). For example, ETL job execution worker(s) 520 may establish a connection to transfer data from source data store 530 and send one or more requests to obtain some or all of a source data object (e.g., via API requests for the source data store or via storage or transfer protocol, like secure file transfer protocol (SFTP) or an internet Small Computer Systems Interface (iSCSI)). Job execution worker(s) 520 may then apply the various transformation(s) or other operations specified by the ETL j ob to the obtained data.
[0040] Various transformations may be applied by ETL job execution worker(s) 520 and may include, but are not limited to, dropping one or more fields, values, or items from the source data object, converting data into a relational data format (e.g., converting lists, items or attributes, into row entries with respective column values), renaming a column, field, or attribute, selecting particular fields from the data object, or splitting fields into two different frames, locations, fields, or attributes, splitting rows, entries, or items into separate rows, entries, or items, unboxing or boxing data values, like strings, aggregating or combining data values, reducing the fidelity of data values by converting, rounding, truncating or modifying data values, or recognizing and generating custom data values (e.g., that combine values of multiple different types).
[0041] ETL job execution worker(s) 520 may establish a connection to store transformed data 526 into target data store 540 (which may be the same or different as source data store 530) via API requests for target data store 540 or via storage or a transfer protocol, like SFTP or iSCSI. ETL job execution worker(s) 520 may also access and update an ETL job execution log 550 with job execution status 528. For example, ETL job execution workers may identify the progress of the ETL job (e.g., X rows out of Y total rows in a source table processed). In this way, failures of ETL job execution worker(s) 520 may be recovered from by access ETL job execution log 550 to determine the last processed portion of a source data object. Errors and other events may be recorded for the ETL job in ETL job execution log 550, in some embodiments. ETL job execution worker(s) 520 may send trigger events 562 indicating completion of the ETL job to ETL job management 330, in some embodiments, which may be a trigger event or execution criteria for other ETL jobs.
[0042] FIG. 6 illustrates example interactions to register a trigger event for an ETL job, according to some embodiments. As noted above, interface 310 may be a network-based, graphical, and/or programmatic interface (e.g., API), that allows clients, such as client 610 (which may be similar to clients 250 discussed above with regard to FIG. 2), access to ETL service 220. Client 610 may submit a request to register a trigger event 610 to ETL job management 330 via interface 310. The register trigger event request 610 may be sent as part of a request to create an ETL job or process for execution, or separately. Different information may be included in registration request 610 (or may be included in separate requests, not illustrated).
[0043] For example, register trigger event request 620 may include information to identify a source data object from which an ETL process may execute upon. An object identifier, name, file path, or other location may be used to specify the source data object. In some embodiments, multiple source data object(s) may be identified. Register trigger event request 620 may identify the target data object(s) to store transformed data objects as part of executing the ETL process. For example, target data object(s) may be identified by pointing a location, such as file directory, or a data object to add the target data object(s) to (e.g., as new partitions).
[0044] Register trigger event request 620 may identify the ETL process for which the trigger event applies, in various embodiments. For instance, the registration request may include a job identifier, such as a job name, pointer, index entry, or other location where the ETL process to execute may be retrieved. Register trigger event 620 may identify a type of trigger event for the ETL process. For example, a trigger event may be identified as an event triggered by events associated with data (whether the data is the source data object or another data object), including the creation, modification, or deletion of data. For example, when a new data object is stored, the arrival of the new data may trigger an event. Another trigger event type may be based on the performance or completion of another ETL job or process. For example, the completion of an ETL process to create a filtered data set may trigger the execution of an ETL process that uses the filtered data set as a source data set. Trigger events may also be time-based events. For example, a time-based trigger event may schedule the execution of the ETL process at a particular time of day, day of the week, day of the month, or day of the year.
[0045] Register trigger event request 620 may also include the various kinds of execution criteria that may be evaluated with respect to the trigger event. For example, execution criteria may evaluate the source data object upon which the ETL process performs by specifying data values, fields, rows, metadata (including tags, labels, size, or other data object attributes), as well as different calculations and/or comparisons to evaluate the execution criteria. Data size thresholds, for instance, may be evaluated, permitting or denying execution of the ETL process depending on the size of the source data object. In another example, an execution criterion may determine and compare the number of rows or attributes added as part of the source data object with row threshold. Execution criteria may include other criteria, such as evaluations of the execution status of other ETL processes, time-based evaluations (e.g., time elapsed since last execution of the ETL process), or any other evaluation to permit or deny execution of the ETL process.
[0046] In some embodiments, register trigger event request 620 may provide further information for the execution of the ETL process, such as access credentials (e.g., to obtain access to data stores), configuration of logging for the ETL process (e.g., identifying a log store or file for storing an execution for the ETL process), as well as other execution configuration information (e.g., retry attempts if the ETL process fails, timeout period for attempting to access data stores, etc.).
[0047] Client 610 may also send a request to modify a trigger event 630 to ETL job management 330 via interface 310. For example, the trigger event type may be changed (from a data-based trigger event to a time-based trigger event or a process-based trigger event). Changes may be made to the execution criteria. For example, thresholds or conditions may be altered, additional execution criteria added or execution criteria removed. Execution criteria may be modified to change which execution criteria have to be satisfied in order to execute the ETL job. For example, execution criteria may originally require that criteria A, B, and C all be satisfied to execute the ETL job. The modification request 630 may change the execution criteria to require that criteria A be satisfied along with at least one of criteria B or C.
[0048] Client 610 may also send a request to delete a trigger event 640 to ETL job management 330 via interface 310. Deletion of a trigger event may not delete the ETL job, source data object, or target data objects. But may disable monitoring for a trigger event for the ETL job.
[0049] Although FIGS. 2 - 6 have been described and illustrated in the context of an ETL service, the various techniques and components illustrated and described in FIGS. 2 - 6 may be easily applied to other data access or management systems in different embodiments that may facilitate ETL processing operations. Stand-alone ETL processing systems are an example of another embodiment that may be implemented in private networks or systems to perform similar techniques to those described above. As such, FIGS. 2 - 6 are not intended to be limiting as to other embodiments of a system that may implement event driven ETL processing. FIG. 7 is a high-level flowchart illustrating methods and techniques to implement event driven ETL processing, according to some embodiments. Various different systems and devices may implement the various methods and techniques described below, either singly or working together. For example, an ETL service such as described above with regard to FIGS. 2 - 6 may be configured to implement the various methods. Alternatively, a combination of different systems and devices, such as a storage subsystem that manages access to data stored in directly attached storage devices may transform data using ETL processing the below techniques. Therefore, the above examples and or any other systems or devices referenced as performing the illustrated method, are not intended to be limiting as to other different components, modules, systems, or configurations of systems and devices.
[0050] As indicated at 710, a request may be received to register a trigger event for an Extract, Transform, Load (ETL) process that applies transformation(s) to a source data object in a first data store, in some embodiments. The request may be received via a programmatic interface, such as discussed above according to FIG. 6 and may indicate various information to associate with or configure the trigger event. For example, the request may indicate or specify the source data object (or multiple source data objects) from which data is obtained upon which ETL processing is performed, the target data object(s) generated as a result of the ETL processing (e.g., new partitions of table or new files in a directory), the type of trigger event (e.g., data arrival, data creation, data update, etc.), execution criteria (e.g., criteria with respect to the source data object, criteria with respect to the timing of the trigger event, criteria with respect to the execution status of other ETL processing jobs), access credentials for executing the ETL process (e.g., identity tokens, username/password information, security role or other information to access source or target data objects), logging configuration (e.g., identifying a location for an event log for the ETL process), or execution configuration (e.g., indicating the number of attempts to execute the ETL process in the event of failure).
[0051] As indicated at 720, in response to receiving the request monitoring may be enabled for the trigger event. For example, in one embodiment, a monitoring agent or listener may be created to receive updates or status changes to the source data object (e.g., such as updates to a data catalog for the source data object, as discussed above with regard to FIG. 4). The updates may be monitored to detect a trigger event. In another embodiment, a periodic (or aperiodic) scan, poll, or other check may be performed with respect to the source data object. For example, a header or other metadata for the source data object may be read to determine whether the source data object has been modified since a last check. Monitoring for trigger events may be enabled differently for different types of triggering events. For trigger events that are triggered by a time (e.g., daily, weekly, or monthly process schedule, or time elapsed since a last ETL process was initiated, attempted, or completed), monitoring may include implementing a timer or other mechanism to determine whether the trigger event occurred.
[0052] As indicated at 730, a trigger event may be detected, in some embodiments. For example, a new data object may be stored in a location in the data store or an addition to or modification of data may be performed with respect to a data object in the data store. In at least some embodiments, the source data object may be a table or other data object that includes multiple partitions and the detected trigger event may be the arrival of an additional partition for the table.
[0053] As indicated at 740, the trigger event may be evaluated with respect to execution criteria for the ETL process. As noted above, execution criteria may allow a data administrator to tailor the execution of the ETL process to different circumstances or events for data, such as various stages in a data life cycle. Execution criteria may include any condition permitting or denying execution of the ETL process. For example, execution criteria may be used to evaluate the source data object, such as the size, format, or other attribute. An execution criterion may deny execution of the ETL process for source data objects less than 512 megabytes in size, in one embodiment.
[0055] Different combinations of execution criteria may be specified for a trigger event. In some embodiments, execution criteria may be specified in a request to register the trigger event. In other embodiments, default execution criteria (e.g., dependent on the trigger event type) may be assigned to the trigger event. In some embodiments, combinations of execution criteria may be satisfied or not satisfied, and execution of the ETL process may still be allowed. For example, execution criteria may include a further criterion that permits execution of the ETL process if 2 or more execution criteria out of a larger group of execution criteria are satisfied.
[0056] If execution criteria are not satisfied, as indicated by the negative exit from 750, then monitoring for the trigger event may continue. For trigger events that satisfy the execution criteria, the ETL process may be executed. For example, as indicated at 760, at least a portion of the source data object may be obtained from the first data store. Different records, files, partitions or other portions may be read from the first data store for processing. As indicated at 770, the transformation(s) may be applied to the obtained portion according to the ETL process in order to generate one or more transformed data object(s). For example, different transformations to filter, modify, rearrange, combine, delete, convert, other any other change to the portion may be performed to generate the transformed data objects. Transformed data may be split into multiple data objects or stored in a single transformed data objects. In some embodiments, transformed data object(s) may be a partition or other addition to an existing data set (e.g., a partition for an existing table).
[0057] Once the transformed data object(s) are generated, the transformed data objects may be stored in a second data store, as indicated at 780. For example, the transformed data objects may be stored in a different storage system or service (e.g., a different database system than the first data store). In some embodiments, the second data store may be the same as the first data store. For example, the transformed data object(s) may be stored in a different location in the first data store (e.g., in a different file directory). In at least some embodiments, an event may be triggered by storing the transformed data object(s) in the second data store. For example, the event may be another trigger event for another ETL process. A request may be sent to update a data catalog for the transformed data object(s) which may be detected, triggering the other trigger event.
[0058] In some embodiments, a retry configuration or threshold may be implemented for the ETL process. For example, if an ETL process is allowed to execute (after satisfying the execution criteria) but fails (e.g., in attempting to obtain data from the source data object, to apply transformations to the source data object, or to store transformed data objects), the retry configuration may direct one ear more subsequent attempts to execute the ETL process. In this way, the ETL process may be executed without waiting for the detection of another trigger event, which may be beneficial in scenarios where a chain of ETL processes are dependent upon the completion of prior ETL processes.
[0059] The methods described herein may in various embodiments be implemented by any combination of hardware and software. For example, in one embodiment, the methods may be implemented by a computer system (e.g., a computer system as in FIG. 8) that includes one or more processors executing program instructions stored on a computer-readable storage medium coupled to the processors. The program instructions may be configured to implement the functionality described herein (e.g., the functionality of various servers and other components that implement the network-based virtual computing resource provider described herein). The various methods as illustrated in the figures and described herein represent example embodiments of methods. The order of any method may be changed, and various elements may be added, reordered, combined, omitted, modified, etc.
[0060] Embodiments of event driven ETL processing as described herein may be executed on one or more computer systems, which may interact with various other devices. One such computer system is illustrated by FIG. 8. In different embodiments, computer system 1000 may be any of various types of devices, including, but not limited to, a personal computer system, desktop computer, laptop, notebook, or netbook computer, mainframe computer system, handheld computer, workstation, network computer, a camera, a set top box, a mobile device, a consumer device, video game console, handheld video game device, application server, storage device, a peripheral device such as a switch, modem, router, or in general any type of compute node, computing device, or electronic device.
[0061] In the illustrated embodiment, computer system 1000 includes one or more processors 1010 coupled to a system memory 1020 via an input/output (I/O) interface 1030. Computer system 1000 further includes a network interface 1040 coupled to I/O interface 1030, and one or more input/output devices 1050, such as cursor control device 1060, keyboard 1070, and display(s) 1080. Display(s) 1080 may include standard computer monitor(s) and/or other display systems, technologies or devices. In at least some implementations, the input/output devices 1050 may also include a touch- or multi -touch enabled device such as a pad or tablet via which a user enters input via a stylus-type device and/or one or more digits. In some embodiments, it is contemplated that embodiments may be implemented using a single instance of computer system 1000, while in other embodiments multiple such systems, or multiple nodes making up computer system 1000, may be configured to host different portions or instances of embodiments. For example, in one embodiment some elements may be implemented via one or more nodes of computer system 1000 that are distinct from those nodes implementing other elements.
[0062] In various embodiments, computer system 1000 may be a uniprocessor system including one processor 1010, or a multiprocessor system including several processors 1010 (e.g., two, four, eight, or another suitable number). Processors 1010 may be any suitable processor capable of executing instructions. For example, in various embodiments, processors 1010 may be general-purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA. In multiprocessor systems, each of processors 1010 may commonly, but not necessarily, implement the same ISA.
[0063] In some embodiments, at least one processor 1010 may be a graphics processing unit. A graphics processing unit or GPU may be considered a dedicated graphics-rendering device for a personal computer, workstation, game console or other computing or electronic device. Modern GPUs may be very efficient at manipulating and displaying computer graphics, and their highly parallel structure may make them more effective than typical CPUs for a range of complex graphical algorithms. For example, a graphics processor may implement a number of graphics primitive operations in a way that makes executing them much faster than drawing directly to the screen with a host central processing unit (CPU). In various embodiments, graphics rendering may, at least in part, be implemented by program instructions configured for execution on one of, or parallel execution on two or more of, such GPUs. The GPU(s) may implement one or more application programmer interfaces (APIs) that permit programmers to invoke the functionality of the GPU(s). Suitable GPUs may be commercially available from vendors such as NVIDIA Corporation, ATI Technologies (AMD), and others.
[0064] System memory 1020 may be configured to store program instructions and/or data accessible by processor 1010. In various embodiments, system memory 1020 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or any other type of memory. In the illustrated embodiment, program instructions and data implementing desired functions, such as those described above are shown stored within system memory 1020 as program instructions 1025 and data storage 1035, respectively. In other embodiments, program instructions and/or data may be received, sent or stored upon different types of computer-accessible media or on similar media separate from system memory 1020 or computer system 1000. Generally speaking, a non-transitory, computer-readable storage medium may include storage media or memory media such as magnetic or optical media, e.g., disk or CD/DVD-ROM coupled to computer system 1000 via I/O interface 1030. Program instructions and data stored via a computer-readable medium may be transmitted by transmission media or signals such as electrical, electromagnetic, or digital signals, which may be conveyed via a communication medium such as a network and/or a wireless link, such as may be implemented via network interface 1040.
[0065] In one embodiment, I/O interface 1030 may be configured to coordinate I/O traffic between processor 1010, system memory 1020, and any peripheral devices in the device, including network interface 1040 or other peripheral interfaces, such as input/output devices 1050. In some embodiments, I/O interface 1030 may perform any necessary protocol, timing or other data transformations to convert data signals from one component (e.g., system memory 1020) into a format suitable for use by another component (e.g., processor 1010). In some embodiments, I/O interface 1030 may include support for devices attached through various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard, for example. In some embodiments, the function of I/O interface 1030 may be split into two or more separate components, such as a north bridge and a south bridge, for example. In addition, in some embodiments some or all of the functionality of I/O interface 1030, such as an interface to system memory 1020, may be incorporated directly into processor 1010.
[0066] Network interface 1040 may be configured to allow data to be exchanged between computer system 1000 and other devices attached to a network, such as other computer systems, or between nodes of computer system 1000. In various embodiments, network interface 1040 may support communication via wired or wireless general data networks, such as any suitable type of Ethernet network, for example; via telecommunications/telephony networks such as analog voice networks or digital fiber communications networks; via storage area networks such as Fibre Channel SANs, or via any other suitable type of network and/or protocol.
[0067] Input/output devices 1050 may, in some embodiments, include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices suitable for entering or retrieving data by one or more computer system 1000. Multiple input/output devices 1050 may be present in computer system 1000 or may be distributed on various nodes of computer system 1000. In some embodiments, similar input/output devices may be separate from computer system 1000 and may interact with one or more nodes of computer system 1000 through a wired or wireless connection, such as over network interface 1040.
[0068] As shown in FIG. 8, memory 1020 may include program instructions 1025, configured to implement the various methods and techniques as described herein, and data storage 1035, comprising various data accessible by program instructions 1025. In one embodiment, program instructions 1025 may include software elements of embodiments as described herein and as illustrated in the Figures. Data storage 1035 may include data that may be used in embodiments. In other embodiments, other or different software elements and data may be included.
[0069] Those skilled in the art will appreciate that computer system 1000 is merely illustrative and is not intended to limit the scope of the techniques as described herein. In particular, the computer system and devices may include any combination of hardware or software that can perform the indicated functions, including a computer, personal computer system, desktop computer, laptop, notebook, or netbook computer, mainframe computer system, handheld computer, workstation, network computer, a camera, a set top box, a mobile device, network device, internet appliance, PDA, wireless phones, pagers, a consumer device, video game console, handheld video game device, application server, storage device, a peripheral device such as a switch, modem, router, or in general any type of computing or electronic device. Computer system 1000 may also be connected to other devices that are not illustrated, or instead may operate as a stand-alone system. In addition, the functionality provided by the illustrated components may in some embodiments be combined in fewer components or distributed in additional components. Similarly, in some embodiments, the functionality of some of the illustrated components may not be provided and/or other additional functionality may be available.
[0070] Those skilled in the art will also appreciate that, while various items are illustrated as being stored in memory or on storage while being used, these items or portions of them may be transferred between memory and other storage devices for purposes of memory management and data integrity. Alternatively, in other embodiments some or all of the software components may execute in memory on another device and communicate with the illustrated computer system via inter-computer communication. Some or all of the system components or data structures may also be stored (e.g., as instructions or structured data) on a computer-accessible medium or a portable article to be read by an appropriate drive, various examples of which are described above. In some embodiments, instructions stored on a non-transitory, computer-accessible medium separate from computer system 1000 may be transmitted to computer system 1000 via transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as a network and/or a wireless link. Various embodiments may further include receiving, sending or storing instructions and/or data implemented in accordance with the foregoing description upon a computer-accessible medium. Accordingly, the present invention may be practiced with other computer system configurations.
[0071] It is noted that any of the distributed system embodiments described herein, or any of their components, may be implemented as one or more web services. For example, nodes within an ETL system may present ETL services to clients as network-based services. In some embodiments, a network-based service may be implemented by a software and/or hardware system designed to support interoperable machine-to-machine interaction over a network. A network-based service may have an interface described in a machine-processable format, such as the Web Services Description Language (WSDL). Other systems may interact with the web service in a manner prescribed by the description of the network-based service's interface. For example, the network-based service may define various operations that other systems may invoke, and may define a particular application programming interface (API) to which other systems may be expected to conform when requesting the various operations.
[0072] In various embodiments, a network-based service may be requested or invoked through the use of a message that includes parameters and/or data associated with the network- based services request. Such a message may be formatted according to a particular markup language such as Extensible Markup Language (XML), and/or may be encapsulated using a protocol such as Simple Object Access Protocol (SOAP). To perform a web services request, a network-based services client may assemble a message including the request and convey the message to an addressable endpoint (e.g., a Uniform Resource Locator (URL)) corresponding to the web service, using an Internet-based application layer transfer protocol such as Hypertext Transfer Protocol (HTTP).
[0073] In some embodiments, web services may be implemented using Representational State Transfer ("RESTful") techniques rather than message-based techniques. For example, a web service implemented according to a RESTful technique may be invoked through parameters included within an HTTP method such as PUT, GET, or DELETE, rather than encapsulated within a SOAP message.
[0074] The various methods as illustrated in the FIGS, and described herein represent example embodiments of methods. The methods may be implemented in software, hardware, or a combination thereof. The order of method may be changed, and various elements may be added, reordered, combined, omitted, modified, etc.
[0075] Various modifications and changes may be made as would be obvious to a person skilled in the art having the benefit of this disclosure. It is intended that the invention embrace all such modifications and changes and, accordingly, the above description to be regarded in an illustrative rather than a restrictive sense.

中文:
[0001]随着组织创建、跟踪和保留信息的技术能力不断提高,已经开发出用于管理和存储不断增长的信息浪潮的各种不同技术。例如,数据库系统为客户端提供许多不同的专用或定制的硬件和软件配置来管理存储的信息。组织必须存储和管理的数据量越来越多,常常对应地增加了数据存储和管理技术(如数据库系统)的规模和复杂性,这继而增加了维护信息的成本。新技术试图通过引入提供不同处理或维护能力的不同数据格式来降低维护数据的复杂性和存储要求。然而,引入多种数据格式并非没有代价。数据通常由不同系统处理,所述系统可不支持数据的当前数据格式。因此,期望执行在不同格式或位置之间提取、变换和加载数据的技术的能力。
[0002]图1示出根据一些实施例的事件驱动提取、变换、加载(ETL)处理的逻辑框图。
[0003]图2是根据一些实施例的示出提供不同服务的提供商网络的框图,所述服务包含执行事件驱动提取、变换、加载(ETL)处理的ETL服务。
[0004]图3是根据一些实施例的示出执行事件驱动ETL处理的ETL服务的框图。
[0005]图4是根据一些实施例的示出ETL作业管理的逻辑框图。
[0006]图5是根据一些实施例的示出用于执行ETL作业的交互的逻辑框图。
[0007]图6示出根据一些实施例的用于注册ETL作业的触发事件的实例交互。
[0008]图7是根据一些实施例的示出用于实施事件驱动ETL处理的方法和技术的高级流程图。
[0009]图8示出根据一些实施例的配置成实施本文描述的各种方法、技术和系统的实例系统。
[0010]尽管本文通过实例的方式针对若干实施例和说明性附图描述了实施例,但是本领域技术人员将认识到,实施例不限于所描述的实施例或附图。应当理解,附图和对其的详细描述并非旨在将实施例限制于所公开的特定形式,而是相反,其意图为覆盖落入由所附权利要求限定的精神和范围内的所有修改、等同物和替代物。本文使用的标题仅用于组织目的,并非意指用于限制说明书或权利要求的范围。如在整个本申请中所使用的,词语“可”以允许的意义使用(即,意指具有潜在的意义),而不是强制意义(即,意指必须)。类似地,词语“包含(include)”、“包含(including)”和“包含(includes)”意指包含但不限于。
[0011]还应当理解,尽管术语第一、第二等可在本文中用于描述各种元件,但是这些元件不应受这些术语的限制。这些术语仅用于区分一种元件与另一种元件。举例来说,在不脱离本发明的范围的情况下,第一触点可被称为第二触点,并且类似地,第二触点可被称为第一触点。第一触点和第二触点两者都为触点,但它们不是相同的触点。
[0012]本文描述了事件驱动提取、变换和加载(ETL)处理的各种实施例。ETL处理通过促进跨不同数据存储库和数据模式或格式的移动数据,为数据管理员、利益相关者、生产者或消费者提供利用不同数据格式、存储系统或存储位置的能力。举例来说,收集网站交互的使用和其它分析(例如,访问者点击和其它动作)的数据产生器可生成呈大日志文件或其它半结构化数据格式的存储的分析数据。然而,为了对分析数据执行各种分析操作或查询,可执行ETL过程以提取期望的数据,将期望的数据变换为分析引擎(如数据库系统)可用的格式,并将提取的数据以适当的数据模式和格式加载到适当的存储系统中。
[0013]典型的ETL处理技术依赖于按时间顺序的机制来启动ETL处理的执行。ETL过程可被安排为每晚执行,或在某一天的某个时间每周执行一次。然而,由于ETL处理作业与数据交互并变换数据,因此可最佳地使执行ETL处理作业的定时与数据或关于数据的事件的改变对准,因为数据并非始终按调度创建、更新、修改或删除,也不会以相同的方式创建、更新、修改或删除。举例来说,可在数据存储库中创建或存储新数据对象以添加另一天的销售价值数据,或可将新数据对象存储为较大数据集的校正。事件驱动ETL处理可区分最佳ETL执行情境和非最佳ETL执行情境,以解决数据变化的不同性质。
[0014]图1示出根据一些实施例的事件驱动提取、变换、加载(ETL)处理的逻辑框图。数据存储库110可代表一个或多个客户端存储数据对象。数据对象可以是任何形式的数据或数据结构(例如,文件、目录、数据库表、日志、块、分区、组块或字节范围)。各种数据事件120可相对于存储的数据对象在数据存储库中发生。举例来说,数据创建事件122可由存储新数据对象产生。例如,数据产生器可将新的日志文件上载到数据存储库120中,从而创建新数据对象。在另一个实例中,可添加用于数据库表的新分区。当数据被更新或变换以创建其它数据对象时,可发生数据修改事件124。举例来说,可执行聚合操作以访问存储在数据对象中的原始数据,聚合不同字段的值,并将聚合值存储在另一个数据对象中(例如,从存储每日交易的数据对象计算总销售额并且存储收入表中新条目中当天的总销售额)。可发生数据删除事件126以从数据存储库110存档、移动或移除数据对象。举例来说,一旦创建了数据对象的清理版本,就可丢弃原始数据对象。
[0015]如图1所示,不同的数据事件120可用作用于由ETL作业处理130执行的ETL过程150的触发事件。举例来说,触发事件(例如,由指定的数据事件120触发)可与ETL过程相关联(例如,在触发事件注册表中,如下面关于图4所讨论的)。ETL作业处理130可检测触发事件150的发生,并确定是否应该使用为ETL过程指定的执行标准来执行ETL过程。举例来说,ETL作业处理可包含执行标准分析140,以将用于ETL过程的源数据对象的各种属性与不同标准进行比较(例如,具有阈值执行大小的源数据对象的大小、在源数据对象内具有行阈值的项目(如行)数等)。可评估其它执行标准,如下面关于图4、图6和图7所讨论的标准,包含用于评估其它ETL过程的状态的标准。
[0016]如果满足执行标准,则ETL作业处理130可执行触发的ETL过程160。举例来说,ETL作业处理130可获得数据存储库110中的源数据对象的部分;应用不同的变换作为ETL处理的一部分,如将从源数据获得的数据值转换、组合或以其它方式修改的变换、将从源数据对象获得的数据值移除、过滤或略除,或将从源数据对象获得的数据值重建、重新排列或重定位。所应用的变换的结果可包含在一个或多个变换的数据对象中,所述变换的数据对象可存储在数据存储库110中。ETL作业处理130可创建不同的数据事件作为执行ETL过程的结果,因此可启动后续的ETL处理。
[0017]请注意,事件驱动ETL处理的先前描述是逻辑说明,因此不应被解释为限制用于实施数据存储库或ETL作业处理的架构。
[0018]本说明书以提供商网络的一般描述开始,所述提供商网络实施提取、变换、加载(ETL)服务,所述服务识别、变换和移动存储在提供商网络或外部数据存储库中的数据。接着讨论包含可作为实施ETL服务的一部分采用的不同组件/模块或组件/模块的布置的ETL服务的各种实例。然后讨论用于实施事件驱动ETL处理的多种不同方法和技术,其中一些在所附流程图中说明。最后,提供可在其上实施各种组件、模块、系统、装置和/或节点的实例计算系统的描述。在整个说明书中提供各种实例。
[0019]图2是根据一些实施例的示出提供不同服务的提供商网络的框图,所述服务包含执行事件驱动ETL处理的提取、变换、加载(ETL)服务。提供商网络200可为专用或封闭系统,或可由如公司或公共部门组织的实体建立,以向客户端250提供经由因特网和/或其它网络可访问的一种或多种服务(如各种类型的基于云的存储)。提供商网络200可在单个位置实施,或可包含托管各种资源池的众多数据中心,如物理和/或虚拟化计算机服务器、存储装置、网络设备等的集合(例如,下文关于图8描述的计算系统1000),需要这些数据中心以实施和分布由提供商网络200提供的基础设施和存储服务。在一些实施例中,提供商网络200可实施各种计算资源或服务,如数据存储服务210(例如,对象存储服务、基于块的存储服务或数据仓库存储服务)、ETL服务220,以及其它服务230,所述其它服务可包含虚拟计算服务、数据处理服务(例如,地图缩减、数据流和/或其它大规模数据处理技术),和/或任何其它类型的基于网络的服务(其可包含未示出的各种其它类型的存储、处理、分析、通信、事件操纵、可视化和安全服务)。
[0020]在各种实施例中,图2所示出的组件可直接在计算机硬件内作为可由计算机硬件(例如,微处理器或计算机系统)直接或间接执行的指令实施,或使用这些技术的组合实施。举例来说,图2的组件可由包含多个计算节点(或简称为节点)的系统实施,每个计算节点可类似于图8中示出并在下文描述的计算机系统实施例。在各种实施例中,给定系统或服务组件(例如,数据存储服务230的组件)的功能可由特定节点实施,或可分布在若干节点上。在一些实施例中,给定节点可实施多于一个服务系统组件(例如,多于一个数据存储组件)的功能。
[0021]数据存储服务210可实施用于代表客户端250存储、访问和管理数据的不同类型的数据存储库,作为基于网络的服务,其使得客户端250能够在云或网络计算环境中操作数据存储系统。举例来说,数据存储服务210可包含各种类型的数据库存储服务(关系和非关系)或用于存储、查询和更新数据的数据仓库。这些服务可以是可缩放和可扩展的企业级数据库系统。查询可针对分布在多个物理资源上的数据存储服务210中的数据库或数据仓库,并且数据库系统可根据需要按比例放大或缩小。在不同的实施例中,数据库系统可有效地与各种类型和/或组织的数据库模式一起工作。在一些实施例中,客户端/订户可以多种方式提交查询,例如,通过SQL接口以交互方式向数据库系统提交。在其它实施例中,外部应用和程序可使用开放式数据库连接(ODBC)和/或Java数据库连接(JDBC)驱动程序接口向数据库系统提交查询。
[0022]数据存储服务210还可包含用于放置、更新和获取数据对象或文件(其可包含未知文件类型的数据文件)的各种对象或文件数据存储库。可经由编程接口(例如,API)或图形用户接口来访问这类数据存储服务210。数据存储服务210可提供基于虚拟块的存储,用于将数据维护为数据卷的一部分,所述数据卷可类似于基于本地块的存储装置(例如,硬盘驱动器、固态驱动器等)被安装或访问,并且可利用基于块的数据存储协议或接口(如因特网小型计算机接口(iSCSI))来访问。
[0023]在一些实施例中,ETL服务220可跨各种数据存储服务210创建并动态地更新代表提供商网络200中的客户端存储的数据目录,如下文关于图3详细讨论的。举例来说,以非关系数据库格式存储的数据库可与在基于对象的数据存储库中存储对象的容器一起被识别为两者都代表提供商网络200的同一客户存储。ETL服务220还可执行从各种数据存储服务210中的一个或多个提取、变换和加载到另一位置的ETL作业。举例来说,ETL服务220可向客户端提供资源以创建、维护和编排数据加载作业,所述数据加载作业采取一种或多种数据集,执行各种变换操作,并且存储变换的数据以供进一步处理(例如,通过数据处理服务中的一种或多种)。ETL服务220可访问由ETL服务220生成的数据目录,以便执行ETL操作(例如,将数据对象从一种文件类型转换为一种或多种不同文件类型的其它数据对象的作业)。如下文关于图3至图5详细讨论的,ETL服务220可代表客户端自动生成变换工作流。
[0024]其它服务230可包含各种类型的数据处理服务以执行不同功能(例如,异常检测、机器学习、查询或任何其它类型的数据处理操作)。举例来说,在至少一些实施例中,数据处理服务可包含地图缩减服务,其创建对存储在数据存储服务210中的一个中的数据实施地图缩减功能的处理节点的集群。各种其它分布式处理架构和技术可通过数据处理服务(例如,网格计算、分片、分布式散列等)来实施。注意,在一些实施例中,数据处理操作可实施为数据存储服务210的一部分(例如,查询引擎处理对指定数据的请求)。数据处理服务可以是ETL服务220的客户端,以便调用ETL作业的执行以使数据可用于在不同位置或数据格式中进行处理,从而执行各种关于存储在数据存储服务210中的数据集的处理操作。
[0025]一般来说,客户端250可涵盖可配置为经由网络260向提供商网络200提交基于网络的请求的任何类型的客户端,所述基于网络的请求包含对存储服务的请求(例如,创建、读取、写入、获得或修改数据存储服务210中的数据的请求、在ETL服务220处生成ETL作业的请求等)。举例来说,给定客户端250可包含合适版本的web浏览器,或可包含插件模块或其它类型的代码模块,其配置成作为web浏览器提供的执行环境的扩展或在所述执行环境内执行。替代地,客户端250可涵盖应用,如数据库应用(或其用户接口)、媒体应用、办公室应用或可利用数据存储服务210中的存储资源来存储和/或访问数据以实施各种应用的任何其它应用。在一些实施例中,这类应用可包含足够的协议支持(例如,用于合适版本的超文本传输协议(HTTP)),用于生成和处理基于网络的服务请求,而不必实施对所有类型的基于网络的数据的完全浏览器支持。也就是说,客户端250可为配置成直接与提供商网络200交互的应用。在一些实施例中,客户端250可配置成根据代表性状态传输(REST)风格的基于网络的服务架构、基于文档或消息的基于网络的服务架构或另一合适的基于网络的服务架构来生成基于网络的服务请求。
[0026]在一些实施例中,客户端250可配置成以对其它应用透明的方式向那些应用提供对提供商网络200的访问。举例来说,客户端250可配置成与操作系统或文件系统集成,以在数据存储服务210中的一个(例如,基于块的存储服务)上提供存储。然而,操作系统或文件系统可向应用提供不同的存储接口,如文件、目录和/或文件夹的传统文件系统层次结构。在这类实施例中,可不需要修改应用来利用存储系统服务模型。相反,介接到数据存储服务210的细节可由客户端250和操作系统或文件系统代表在操作系统环境内执行的应用来协调。
[0027]客户端250可经由网络260向提供商网络200传达基于网络的服务请求(例如,针对数据存储服务210中的数据的访问请求、操作、任务或作业,作为其它服务230的一部分执行,或与ETL服务220交互)且接收来自提供商网络200的响应。在各种实施例中,网络260可涵盖在客户端250和提供商网络200之间建立基于网络的通信所必需的联网硬件和协议的任何合适的组合。举例来说,网络260通常可涵盖共同实施因特网的各种电信网络和服务提供商。网络260还可包含专用网络,如局域网(LAN)或广域网(WAN)以及公共或专用无线网络。举例来说,给定客户端250和提供商网络200两者可分别在具有它们自身内部网络的企业内提供。在这类实施例中,网络260可包含在给定客户端250和因特网之间以及因特网和提供商网络200之间建立网络链路所必需的硬件(例如,调制解调器、路由器、交换机、负载平衡器、代理服务器等)和软件(例如,协议栈、计费软件、防火墙/安全软件等)。注意,在一些实施例中,客户端250可使用专用网络而不是公共因特网与提供商网络200通信。
[0028]图3是根据一些实施例的示出执行事件驱动ETL处理的ETL服务的框图。ETL服务220可经由接口310提供对数据目录360和ETL作业(用于创建、管理和执行)的访问,在各种实施例中,接口310可以是编程接口(例如,应用编程接口(API))、命令行接口和/或图形用户接口。
[0029]ETL服务220可实施ETL作业创建320以通过手动作业创建(例如,创建、编辑或上载ETL代码或创建或编辑ETL作业的图形)或通过自动作业创建来操纵ETL作业的创建。ETL作业创建320可操纵自动ETL作业创建和手动ETL作业创建的请求。举例来说,ETL作业创建320可接收作业生成请求,所述作业生成请求可指定用于ETL作业的数据对象和目标数据格式。其它作业信息(如访问凭证,触发事件或执行ETL作业的任何其它信息)可作为创建请求的一部分或作为触发事件注册请求的一部分包含在内,如下面关于图6所讨论的。ETL作业创建320可通过确定数据对象的源数据格式和数据对象的目标数据格式来自动生成ETL代码以执行ETL作业。举例来说,在一个实施例中,源和目标数据格式来自数据目录360。在另一实施例中,数据格式识别可执行数据格式辨识技术,或访问其它数据存储库(例如,如存储在关系数据库中的数据目录)以检索数据格式信息。然后,ETL作业创建320可比较源数据格式和目标数据格式或模式,以选择应用于源数据对象的变换来实现目标数据格式。然后,ETL作业创建320可生成用于所选择变换的代码并构造用于执行所选择变换的源代码。用于ETL作业的代码可存储在ETL作业存储库350中以供后续执行。
[0030]ETL作业创建320还可实施ETL作业的手动创建。举例来说,可通过图形用户接口手动选择、组合或组装变换操作,以定义要应用的变换的工作流。对应于工作流的代码可被生成(或由用户供应)、编辑和存储以供随后作为ETL作业存储库350的一部分执行。
[0031]ETL服务220可实施ETL作业管理330以向客户端提供管理、编辑、删除或以其它方式改变ETL作业的能力。还可为ETL作业定义触发事件(如下面关于图6所讨论的)。ETL作业管理330可监测触发事件并请求ETL作业的执行,如下面关于图4所讨论的。
[0032]ETL服务220可实施ETL作业执行340以提供执行平台ETL作业。在一些实施例中,ETL作业执行340可提供无服务器架构(从客户端的角度),以便供应适当数量的资源(例如,来自执行ETL作业代码的虚拟计算服务的虚拟计算实例)以便满足客户端或ETL服务220提供的性能要求、目的或目标。在一些实施例中,ETL作业执行340可自动执行作业,而无需任何用户编辑来自ETL作业创建320的自动生成的ETL代码的改变。在一些实施例中,ETL作业执行340可执行被修改的自动生成的ETL作业。ETL作业执行340可响应于检测到的ETL作业的触发事件(其可由ETL作业管理或用于触发事件条件的另一系统或服务监测来检测)来执行作业,如下面关于图5所讨论的。
[0033]ETL服务220可维护描述数据集(存储在提供商网络200中或外部存储位置中)的数据目录360。ETL服务220可识别未知数据对象,识别未知数据对象的数据格式,并将数据格式存储在未知数据对象的数据目录中。ETL服务220允许目录用户、所有者或其它利益相关者修改或以其它方式管理数据目录。举例来说,ETL服务220可处理和执行针对数据目录360的访问请求(例如,组合、删除或拆分目录中的元数据表或编辑为数据目录确定的元数据的请求)。ETL服务220可实施数据目录的访问或控制策略(例如,针对经授权用户限制对数据目录的访问)。ETL服务220可实施数据保留或生命周期策略以确定维护数据目录(或较旧版本的数据目录)的时间。ETL服务220可操纵用于创建新数据目录的数据中的存储资源的供应。ETL服务220还可执行负载平衡、热管理、故障恢复和其它资源管理技术(例如,实施持久性要求)以确保客户端的数据目录的可用性。
[0034]用于数据目录360的存储可由一个或多个存储节点、服务或计算装置(例如,下文关于图8讨论的系统1000)来实施,以为由数据目录服务200生成的数据目录提供持久存储。这类存储节点(或用于数据目录360的存储的其它存储组件)可实施各种查询处理引擎或其它请求操纵组件,以根据经由接口310接收的请求提供对数据目录的访问。举例来说,在一个实施例中,数据目录存储可实施为非关系数据库,其存储表中的数据对象的文件类型和其它元数据。在一些实施例中,代表单个用户帐户跨不同存储服务210存储的各种数据对象的元数据集合可一起存储在可使客户端可访问的单个元数据目录中。
[0035]图4是根据一些实施例的示出ETL作业管理的逻辑框图。ETL作业管理330可监测ETL作业的触发事件并请求执行满足ETL作业的执行标准的ETL作业。在各种实施例中,ETL作业管理330可实施触发事件注册表430。触发事件注册表430可以是数据存储库(如数据库),或跟踪启用了监测的触发事件432的其它数据存储系统,以及用于评估是否可执行ETL作业的执行标准434。触发事件注册表430可由请求470更新以注册或更新触发事件,如下面关于图6详细讨论的。举例来说,用于注册触发事件的请求470可识别ETL作业、触发事件的类型,以及要应用于确定在检测到触发事件时甚至执行ETL作业是否继续的执行标准。
[0036]ETL作业管理330可实施触发事件监测410以检测ETL作业的触发事件。举例来说,触发事件监测410可实施等待和识别指定ETL作业的事件的监听器或其它进程。举例来说,触发事件监测410可通过将监听器配置为在提供给ETL作业管理330的更新通知中识别对数据源X的更新440来启用对数据源X进行操作的ETL作业的监听器。在一些实施例中,触发事件监测410可监测、获取或评估向数据目录360提供440的数据对象更新450。举例来说,ETL服务220可使到数据目录360的接口可用,使得当访问数据存储库中的数据对象的数据产生器、修改器或其它进程执行动作时,更新的描述或对数据对象元数据或数据模式所产生的更新经由接口被提供440到数据目录360(例如,通过API请求来报告更新)。触发事件监测410可订阅更新的馈送或流(例如,特定于数据目录中的数据对象或整个数据目录),以便监测触发事件的数据对象更新。对于其它类型的触发事件,触发事件监测可实施其它监听器、扫描器或操纵器,以例如检索有关ETL过程执行状态的信息,或实施基于时间的触发事件的调度。
[0037]执行标准评估420可查找用于检测到的触发事件454的执行标准434,以确定是否应该执行用于触发的ETL作业。举例来说,在各种实施例中,执行标准可包含关于ETL作业的源数据对象的标准(例如,数据对象的大小、数据值或其它元数据或属性)。执行标准评估420可从数据目录360获得所需信息,如数据对象属性460(例如,数据对象大小、添加的行数等)以评估执行标准。尽管未示出,但是在不同的实施例中,可获得用于评估其它执行标准(例如,通过直接访问ETL作业状态信息或数据对象)的信息,以评估执行标准。对于那些被评估并满足执行标准的触发事件,可向ETL作业执行服务340提供ETL作业执行请求480以执行由请求480识别的作业。
[0038]图5是根据一些实施例的示出用于执行ETL作业的交互的逻辑框图。ETL作业执行340可实施ETL作业执行协调器510,所述ETL作业执行协调器510将ETL作业512分配给一个或多个ETL作业执行工作器520,所述ETL作业执行工作器520可访问源数据存储库530和目标数据存储库540以获得数据524,应用变换,并且存储变换的数据526。ETL作业执行请求502可识别要执行的作业(例如,通过包含作业名称或其它标识符)。ETL作业执行协调器510可确定执行ETL作业所需的资源并将ETL作业分配给一个或多个ETL作业执行工作器520。
[0039]ETL作业执行工作器520可从ETL作业存储库350获得用于ETL作业的信息522(包含可执行代码、调用的操作或变换,以及执行所识别的ETL作业的其它信息)。然后,ETL作业执行工作器520可以并行或串行方式执行ETL作业,从而从源数据存储库530(其可以是提供商网络200的数据存储服务210)获得数据524。举例来说,ETL作业执行工作器520可建立连接以从源数据存储库530传输数据并发送一个或多个请求以获得一些或全部源数据对象(例如,通过对源数据存储库的API请求,或通过存储或传输协议,如安全文件传输协议(SFTP)或互联网小型计算机系统接口(iSCSI))。然后,作业执行工作器520可将由ETL作业指定的各种变换或其它操作应用于所获得的数据。
[0040]各种变换可由ETL作业执行工作器520应用,并且可包含但不限于:从源数据对象中略除一个或多个字段、值或项目;将数据变换为关系数据格式(例如,将列表、项目或属性转换为具有相应列值的行条目);重命名列、字段或属性;从数据对象中选择特定字段,或将字段拆分为两个不同的帧、位置、字段或属性;将行、条目或项目拆分成不同的行、条目或项目;拆箱或装箱数据值,如字符串;聚合或组合数据值;通过转换、舍入、舍项或修改数据值来降低数据值的保真度,或辨识和生成自定义数据值(例如,组合多种不同类型的值)。
[0041]ETL作业执行工作器520可建立连接以通过对目标数据存储库540的API请求或通过存储或传输协议(如SFTP或iSCSI)将变换的数据526存储到目标数据存储库540(其可与源数据存储库530相同或不同)中。ETL作业执行工作器520还可访问和更新具有作业执行状态528的ETL作业执行日志550。举例来说,ETL作业执行工作器可识别ETL作业的进度(例如,处理的源表中的Y行总数中的X行)。以这种方式,可通过访问ETL作业执行日志550恢复ETL作业执行工作器520的故障,以确定源数据对象的最后处理的部分。在一些实施例中,可在ETL作业执行日志550中记录ETL作业的错误和其它事件。在一些实施例中,ETL作业执行工作器520可向ETL作业管理330发送指示ETL作业完成的触发事件562,其可以是用于其它ETL作业的触发事件或执行标准。
[0042]图6示出根据一些实施例的用于注册ETL作业的触发事件的实例交互。如上所述,接口310可以是基于网络的、图形的和/或编程接口(例如,API),其允许客户端,如客户端610(其可类似于上面关于图2讨论的客户端250)访问ETL服务220。客户端610可经由接口310向ETL作业管理330提交注册触发事件610的请求。注册触发事件请求610可作为创建ETL作业或过程以执行的请求的一部分发送,或单独发送。不同的信息可包含在注册请求610中(或可包含在单独的请求中,未示出)。
[0043]举例来说,注册触发事件请求620可包含用于识别ETL过程可针对而执行的源数据对象的信息。可使用对象标识符、名称、文件路径或其它位置来指定源数据对象。在一些实施例中,可识别多个源数据对象。注册触发事件请求620可识别目标数据对象以存储变换的数据对象,作为执行ETL过程的一部分。举例来说,可通过指向如文件目录的位置或数据对象来识别目标数据对象,以将目标数据对象添加到(例如,作为新分区)。
[0044]在各种实施例中,注册触发事件请求620可识别触发事件应用的ETL过程。举例来说,注册请求可包含作业标识符,如作业名称、指针、索引条目或可检索要执行的ETL过程的其它位置。注册触发事件620可识别ETL过程的触发事件的类型。举例来说,触发事件可被识别为由与数据(无论数据是源数据对象还是另一数据对象)相关联的事件触发的事件,包含数据的创建、修改或删除。举例来说,当存储新数据对象时,新数据的到达可触发事件。另一触发事件类型可基于另一ETL作业或过程的执行或完成。举例来说,完成创建过滤数据集的ETL过程可触发执行使用过滤数据集作为源数据集的ETL过程。触发事件也可以是基于时间的事件。举例来说,基于时间的触发事件可在一天的特定时间、一周中的某天、一个月中的某天或一年中的某天调度ETL过程的执行。
[0045]注册触发事件请求620还可包含可针对触发事件评估的各种执行标准。举例来说,执行标准可通过指定数据值、字段、行、元数据(包含标记、标签、大小或其它数据对象属性)以及用于评估执行标准的不同的计算和/或比较来评估ETL过程所针对而执行的源数据对象。举例来说,可评估数据大小阈值,允许或拒绝根据源数据对象的大小执行ETL过程。在另一实例中,执行标准可确定并比较作为源数据对象的一部分添加的行或属性的数量与行阈值。执行标准可包含其它标准,如其它ETL过程的执行状态的评估、基于时间的评估(例如,自上次执行ETL过程以来经过的时间),或允许或拒绝执行ETL过程的任何其它评估。
[0046]在一些实施例中,注册触发事件请求620可提供用于执行ETL过程的另外的信息,如访问凭证(例如,以获得对数据存储库的访问)、用于ETL过程的日志记录的配置(例如,识别日志存储库或用于存储ETL过程的执行的文件),以及其它执行配置信息(例如,如果ETL过程失败则重试尝试,尝试访问数据存储库的超时时间段等)。
[0047]客户端610还可经由接口310向ETL作业管理330发送修改触发事件630的请求。举例来说,可改变触发事件类型(从基于数据的触发事件到基于时间的触发事件或基于过程的触发事件)。可对执行标准进行改变。举例来说,可改变阈值或条件,添加附加执行标准或移除执行标准。可修改执行标准以改变必须满足哪些执行标准以便执行ETL作业。举例来说,执行标准最初可要求满足标准A、B和C以执行ETL作业。修改请求630可改变执行标准以要求满足标准A以及标准B或C中的至少一个。
[0048]客户端610还可经由接口310向ETL作业管理330发送删除触发事件640的请求。触发事件的删除可不删除ETL作业、源数据对象或目标数据对象。但是可禁用对ETL作业的触发事件的监测。
[0049]尽管已经在ETL服务的上下文中描述和示出了图2至图6,但在不同的实施例中,在图2至图6中示出和描述的各种技术和组件可容易地应用于其它数据访问或管理系统,这可促进ETL处理操作。独立ETL处理系统是可在专用网络或系统中实施以执行与上述类似的技术的另一实施例的实例。因此,图2至图6不希望相对于可实施事件驱动ETL处理的系统的其它实施例具有限制性。图7是根据一些实施例的示出用于实施事件驱动ETL处理的方法和技术的高级流程图。各种不同的系统和装置可单独地或一起工作地实施下面描述的各种方法和技术。举例来说,如上文关于图2至图6所描述的ETL服务可配置成实施各种方法。替代地,不同系统和装置的组合(如管理对存储在直接附接的存储装置中的数据的访问的存储子系统)可使用ETL处理(以下技术)来变换数据。因此,上面的实例和/或被称为执行所说明的方法的任何其它系统或装置并不旨在限制其它不同的组件、模块、系统或系统和装置的配置。
[0050]如在710处所指示,在一些实施例中,可接收请求以注册将变换应用于第一数据存储库中的源数据对象的提取、变换、加载(ETL)过程的触发事件。请求可通过编程接口(如上面根据图6所讨论的)接收,并可指示与触发事件相关联或配置触发事件的各种信息。举例来说,请求可指示或指定从其获得数据并且所针对而执行ETL处理的源数据对象(或多个源数据对象)、作为ETL处理的结果而生成的目标数据对象(例如,表的新分区或目录中的新文件)、触发事件的类型(例如,数据到达、数据创建、数据更新等)、执行标准(例如,关于源数据对象的标准、关于触发事件的定时的标准、关于其它ETL处理作业的执行状态的标准)、用于执行ETL过程的访问凭证(例如,身份令牌、用户名/密码信息、安全角色或访问源或目标数据对象的其它信息)、日志记录配置(例如,识别ETL过程的事件日志的位置),或执行配置(例如,指示在发生故障时执行ETL过程的尝试次数)。
[0051]如在720处所指示,响应于接收到请求,可针对触发事件启用监测。举例来说,在一个实施例中,可创建监测代理或监听器以接收对源数据对象的更新或状态改变(例如,如对源数据对象的数据目录的更新,如上面关于图4所讨论的)。可监测更新以检测触发事件。在另一个实施例中,可关于源数据对象执行周期性(或非周期性)扫描、轮询或其它检查。举例来说,可读取源数据对象的头部或其它元数据以确定自上次检查以来源数据对象是否已被修改。对于不同类型的触发事件,可不同地启用对触发事件的监测。对于由时间触发的触发事件(例如,每日,每周或每月过程调度,或自启动、尝试或完成上次ETL过程以来经过的时间),监测可包含实施计时器或其它机制以确定触发事件是否发生。
[0052]如在730处所指示,在一些实施例中可检测到触发事件。举例来说,新数据对象可存储在数据存储库中的位置中,或可相对于数据存储库中的数据对象执行数据的添加或修改。在至少一些实施例中,源数据对象可以是包含多个分区的表或其它数据对象,并且检测到的触发事件可以是表的附加分区的到达。
[0053]如在740处所指示,可相对于ETL过程的执行标准评估触发事件。如上所述,执行标准可允许数据管理员将ETL过程的执行定制到数据的不同环境或事件,如数据生命周期中的各个阶段。执行标准可包含允许或拒绝执行ETL过程的任何条件。举例来说,执行标准可用于评估源数据对象,如大小、格式或其它属性。在一个实施例中,执行标准可拒绝对大小小于512兆字节的源数据对象执行ETL过程。
[0055]可为触发事件指定执行标准的不同组合。在一些实施例中,可在注册触发事件的请求中指定执行标准。在其它实施例中,可将默认执行标准(例如,取决于触发事件类型)分配给触发事件。在一些实施例中,可满足或不满足执行标准的组合,并且仍然可允许执行ETL过程。举例来说,如果满足更大的执行标准组中的两个或更多个执行标准,则执行标准可包含允许执行ETL过程的另外的标准。
[0056]如果不满足执行标准,如从750的否定退出所指示的,则可继续监测触发事件。对于满足执行标准的触发事件,可执行ETL过程。举例来说,如在760处所指示,可从第一数据存储库获得源数据对象的至少一部分。可从第一数据存储库读取不同的记录、文件、分区或其它部分以进行处理。如在770处所指示,可根据ETL过程将变换应用于所获得的部分,以便生成一个或多个变换的数据对象。举例来说,可执行用于过滤、修改、重新排列、组合、删除、转换以及对所述部分的任何其它改变的不同变换以生成变换的数据对象。变换的数据可拆分成多个数据对象或存储在单个变换的数据对象中。在一些实施例中,变换的数据对象可以是对现有数据集的分区或其它添加(例如,现有表的分区)。
[0057]一旦生成了变换的数据对象,就可将变换的数据对象存储在第二数据存储库中,如在780处所指示。举例来说,变换的数据对象可存储在不同的存储系统或服务中(例如,与第一数据存储库不同的数据库系统)。在一些实施例中,第二数据存储库可与第一数据存储库相同。举例来说,变换的数据对象可存储在第一数据存储库中的不同位置中(例如,在不同的文件目录中)。在至少一些实施例中,可通过将变换的数据对象存储在第二数据存储库中来触发事件。举例来说,事件可以是另一个ETL过程的另一个触发事件。可发送请求以更新可检测到的变换的数据对象的数据目录,从而触发另一个触发事件。
[0058]在一些实施例中,可为ETL过程实施重试配置或阈值。举例来说,如果允许ETL过程执行(在满足执行标准之后)但是失败(例如,尝试从源数据对象获得数据,将变换应用于源数据对象,或存储变换的数据对象),则重试配置可指示一个或多个后续尝试来执行ETL过程。以这种方式,可在不等待检测到另一个触发事件的情况下执行ETL过程,这在ETL过程链取决于先前ETL过程的完成的情境下可以是有益的。
[0059]在各种实施例中,本文描述的方法可通过硬件和软件的任何组合来实施。举例来说,在一个实施例中,所述方法可由计算机系统(例如,如图8中的计算机系统)实施,所述计算机系统包含执行存储在联接到处理器的计算机可读存储介质上的程序指令的一个或多个处理器。程序指令可配置成实施本文描述的功能(例如,实施本文描述的基于网络的虚拟计算资源提供者的各种服务器和其它组件的功能)。如附图中所说明和本文描述的各种方法表示方法的实例实施例。可改变任何方法的顺序,并且可添加、重新排序、组合、省略、修改等各种元件。
[0060]如本文描述的事件驱动ETL处理的实施例可在一个或多个计算机系统上执行,所述计算机系统可与各种其它装置交互。图8示出了一种这类计算机系统。在不同的实施例中,计算机系统1000可是各种类型的装置中的任何一种,包含但不限于:个人计算机系统、台式计算机、膝上型计算机、笔记本或上网本计算机、大型计算机系统、手持式计算机、工作站、网络计算机、相机、机顶盒、移动装置、消费者装置、视频游戏控制台、手持式视频游戏装置、应用服务器、存储装置、如交换机、调制解调器、路由器之类的外围装置,或通常任何类型的计算节点、计算装置或电子装置。
[0061]在所说明的实施例中,计算机系统1000包含经由输入/输出(I/O)接口1030耦合到系统存储器1020的一个或多个处理器1010。计算机系统1000另外包含耦合到I/O接口1030的网络接口1040,以及一个或多个输入/输出装置1050,如光标控制装置1060、键盘1070和显示器1080。显示器1080可包含标准计算机监测器和/或其它显示系统、技术或装置。在至少一些实施方式中,输入/输出装置1050还可包含触摸或多触摸使能装置,如触摸板或平板电脑,经由所述触摸板或平板电脑用户经由触控笔型装置和/或一个或多个数字进入输入。在一些实施例中,预期可使用计算机系统1000的单个实例来实施实施例,而在其它实施例中,多个这类系统或构成计算机系统1000的多个节点可配置成托管实施例的不同部分或实例。举例来说,在一个实施例中,可经由计算机系统1000的一个或多个节点来实施一些元件,所述节点不同于实施其它元件的那些节点。
[0062]在各个实施例中,计算装置1000可为包含一个处理器1010的单处理器系统,或包含若干处理器1010(例如,两个、四个、八个或另一合适数量)的多处理器系统。处理器1010可为能够执行指令的任何合适的处理器。举例来说,在各个实施例中,处理器1010可为实施各种指令集架构(ISA)中的任何一种的通用或嵌入式处理器,如x86、PowerPC、SPARC或MIPSISA或任何其它合适的ISA。在多处理器系统中,处理器1010中的每一个可共同但不一定实施相同的ISA。
[0063]在一些实施例中,至少一个处理器1010可为图形处理单元。图形处理单元或GPU可被视为个人计算机、工作站、游戏控制台或其它计算或电子装置的专用图形渲染装置。现代GPU在操纵和显示计算机图形方面可非常有效,并且它们高度并行的结构可使它们比一系列复杂图形算法的典型CPU更有效。举例来说,图形处理器可比用主机中央处理单元(CPU)直接绘制到屏幕上更快地执行它们的方式实施多个图形基元操作。在各种实施例中,图形渲染可至少部分地由配置用于在这类GPU中的一个上执行或在这类GPU中的两个或更多个上并行执行的程序指令来实施。GPU可实施允许程序员调用GPU的功能性的一个或多个应用编程接口(API)。合适的GPU可从如辉达公司(NVIDIA Corporation)、治天科技公司(现美国超微半导体公司)(ATI Technologies(AMD))等供应商商购获得。
[0064]系统存储器1020可配置成存储由处理器1010可访问的程序指令和/或数据。在各个实施例中,系统存储器1020可使用任何合适的存储器技术来实施,如静态随机存取存储器(SRAM)、同步动态RAM(SDRAM)、非易失性/闪存型存储器或任何其它类型的存储器。在所说明的实施例中,实施如上面描述的那些的所期望功能的程序指令和数据被示为分别作为程序指令1025和数据存储件1035存储在系统存储器1020中。在其它实施例中,可在不同类型的计算机可访问介质上或与系统存储器1020或计算机系统1000分离的类似介质上接收、发送或存储程序指令和/或数据。一般来说,非暂时性计算机可读存储介质可包含存储介质或存储器介质(如磁性或光学介质),例如经由I/O接口1030耦合到计算机系统1000的磁盘或DVD/CD-ROM。经由计算机可读介质存储的程序指令和数据可由传送介质或信号,如电信号、电磁信号或数字信号传送,所述信号可经由通信介质,如网络和/或无线链路传达,如可经由网络接口1040实施。
[0065]在一个实施例中,I/O接口1030可配置成协调处理器1010、系统存储器1020和装置中的任何外围装置之间的I/O业务量,包含网络接口1040或如输入/输出装置1050的其它外围接口。在一些实施例中,I/O接口1030可执行任何必要的协议、定时或其它数据变换,以将来自一个组件(例如,系统存储器1020)的数据信号变换成适合于由另一组件(例如,处理器1010)使用的格式。在一些实施例中,I/O接口1030可包含对通过各种类型的外围总线附接的装置的支持,例如,如外围组件互连(PCI)总线标准或通用串行总线(USB)标准的变体。在一些实施例中,I/O接口1030的功能可分成两个或更多个单独的组件,例如北桥和南桥。另外,在一些实施例中,I/O接口1030的一些或全部功能性,如到系统存储器1020的接口可直接并入到处理器1010中。
[0066]网络接口1040可配置成允许在计算机系统1000和附接到网络的其它装置(如其它计算机系统)之间或在计算机系统1000的节点之间交换数据。在各种实施例中,网络接口1040可支持经由有线或无线通用数据网络,例如,如任何类型的以太网;经由如模拟语音网络或数字光纤通信网络的电信/电话网络;经由如光纤通道SAN的存储区域网络或经由任何其它合适类型的网络和/或协议的通信。
[0067]在一些实施例中,输入/输出装置1050可包含一个或多个显示终端、键盘、小键盘、触摸板、扫描装置、语音或光学辨识装置,或适合于由一个或多个计算机系统1000输入或检索数据的任何其它装置。多个输入/输出装置1050可存在于计算机系统1000中,或可分布在计算机系统1000的各个节点上。在一些实施例中,类似的输入/输出装置可与计算机系统1000分离,并且可通过有线或无线连接(例如通过网络接口1040)与计算机系统1000的一个或多个节点交互。
[0068]如图8所示,存储器1020可包含:程序指令1025,其配置成实施如本文所描述的各种方法和技术;以及数据存储件1035,其包括由程序指令1025可访问的各种数据。在一个实施例中,程序指令1025可包含如本文所描述并如图中所示的实施例的软件要素。数据存储件1035可包含可在实施例中使用的数据。在其它实施例中,可包含其它或不同的软件元件和数据。
[0069]本领域技术人员将理解,计算机系统1000仅为说明性的,并不旨在限制本文所描述的技术的范围。具体地,计算机系统和装置可包含可执行所指示功能的硬件或软件的任何组合,包含计算机、个人计算机系统、台式计算机、膝上型计算机、笔记本或上网本计算机、大型计算机系统、手持式计算机、工作站、网络计算机、相机、机顶盒、移动装置、网络装置、因特网器具、PDA、无线手机、寻呼机、消费者装置、视频游戏控制台、手持视频游戏装置、应用服务器、存储装置、外围装置(如交换机、调制解调器、路由器)或通常任何类型的计算或电子装置。计算机系统1000还可连接到未说明的其它装置,或替代地可作为独立系统操作。另外,在一些实施例中,由所说明的组件提供的功能性可组合在更少的组件中或分布在附加组件中。类似地,在一些实施例中,可不提供所说明的组件中的一些的功能性和/或可使用其它附加功能。
[0070]本领域技术人员还将理解,虽然各种项目被说明为在使用时存储在存储器中或存储库上,但是为了存储器管理和数据完整性的目的,这些项目或它们的一部分可在存储器和其它存储装置之间传输。替代地,在其它实施例中,软件组件中的一些或所有可在另一装置上的存储器中执行,并经由计算机间通信与所说明的计算机系统通信。系统组件中的一些或所有或数据结构也可存储(例如,作为指令或结构化数据)在计算机可访问介质或便携式物品上,以由适当的驱动器读取,其各种实例在上面进行描述。在一些实施例中,存储在与计算机系统1000分离的非暂时性计算机可访问介质上的指令可经由传送介质或经由通信介质(如网络和/或无线链路)传达的信号(如电信号、电磁信号或数字信号)传送到计算机系统1000上。各个实施例可另外包含在计算机可访问介质上接收、发送或存储根据前述描述实施的指令和/或数据。因此,可用其它计算机系统配置来实施本发明。
[0071]注意,本文描述的任何分布式系统实施例或它们的组件中的任何可实施为一个或多个web服务。举例来说,ETL系统内的节点可将ETL服务作为基于网络的服务呈现给客户端。在一些实施例中,基于网络的服务可由设计用于支持网络上的可互操作的机器到机器交互的软件和/或硬件系统来实施。基于网络的服务可具有以机器-可处理格式描述的接口,如Web服务描述语言(WSDL)。其它系统可基于网络的服务的接口的描述所规定的方式与web服务交互。举例来说,基于网络的服务可定义其它系统可调用的各种操作,并且可定义特定应用编程接口(API),当请求各种操作时,可期望其它系统以符合所述特定应用编程接口。
[0072]在各种实施例中,可通过使用包含与基于网络的服务请求相关联的参数和/或数据的消息来请求或调用基于网络的服务。可根据如可扩展标记语言(XML)的特定标记语言来格式化这类消息,和/或可使用如简单对象访问协议(SOAP)的协议来封装这类消息。为了执行web服务请求,基于网络的服务客户端可使用基于因特网的应用层传输协议(如超文本传输协议(HTTP))汇编包含所述请求的消息,并将所述消息传达到对应于web服务的可寻址端点(如,统一资源定位符(URL))。
[0073]在一些实施例中,可使用代表性状态传输(“RESTful”)技术而不是基于消息的技术来实施web服务。举例来说,可通过HTTP方法(如PUT、GET或DELETE)内包含的参数来调用根据RESTful技术实施的web服务,而不是封装在SOAP消息中。
[0074]如附图中所说明和本文描述的各种方法表示方法的实例实施例。所述方法可用软件、硬件或其组合来实施。可改变方法的顺序,并且可添加、重新排序、组合、省略、修改等各种元件。
[0075]可进行各种修改和改变,这对于受益于本公开的本领域技术人员来说为显而易见的。本发明旨在涵盖所有这类修改和变化,并且因此,以上描述被认视为说明性的而不是限制性的。