Patent Description:
Event streaming platforms (ESP) play a central role in ingesting, storing, and processing real-time data in a scalable and resilient manner. Example environments that need ESPs for a big data pipeline include the travel industry, financial transactions, application logs, website clickstreams, loT telemetry data, and the like. An ESP is a highly scalable and durable system capable of continuously ingesting gigabytes of events per second from various sources. The data collected is available in milliseconds for intelligent applications that can react to events as they happen. The ultimate goal of an ESP is to capture business events as they happen and react to them in real-time to deliver responsive and personalized customer experiences. An ESP can ingest data from various data sources such as website clickstreams, database event streams, financial transactions, social media feeds, server logs, and events coming from loT devices. The captured data must be made available to applications that know how to process them to gain real-time actionable insights.

For example, controlled experiments, such as A|B testing, may utilize an ESP. A|B testing is used to enable updates to current systems or new features to be objectively evaluated while minimizing the risks associated with full deployment. In A|B testing, for example, an update or a new feature of a system being tested (hereinafter "version B") may be objectively evaluated against a current version (hereinafter "version A"). Users interacting with the system under test may be partitioned such that a first subset of the users are presented with version A while a second subset of the users are presented with version B. Evaluation metrics may be determined for each version using data collected regarding responses or behaviors of users in the first and second subsets. Such evaluation metrics facilitate objectively evaluating version B against version A.

Some conventional techniques for automating big data pipelines for processing use static configuration files and an orchestrator that can control each run sequence based on the predefined static configuration file. The static configuration file includes a defined run sequence, including run frequencies for each job in the run sequence. Issues with using static configuration files include the use and dependency of a centralized entity (e.g., an orchestrator). Additionally, the static configuration files can be too big, making them hard to manage, more expensive, and with limited scalability. Therefore, complex jobs, such as A|B testing with multiple dependencies, will make the static configuration files very big, and hard to maintain.

For data processing services that involve a big data pipeline, fulfilling service requests and run sequences in a big data environment with a massive influx of real-time data may include several hurdles to maintain efficient processes because of the complexities of the building and maintaining of heavy batch big data processes. Thus, improved methods, systems, and computer program products for simplification of the building and maintaining of heavy batch big data processes are needed.

<CIT> relates to methods and apparatus for managing file distribution between publishing and subscribing devices. For example, at a transporter system, a publish request for publishing a version of the dataset is received from a publisher device, wherein the publish request has a predefined format that includes an identity of the dataset, an identity of the version of the dataset, and an identity of a location for the version of the dataset. The transporter system updates a registry to associate the dataset with the identity of the dataset, version, and location, and the transporter system automatically publishes the dataset to any subscribers devices that have previously sent subscribe requests for such dataset to the transporter system.

In embodiments of the invention, a method for implementing an asynchronous communication protocol between event publishers and an asynchronous communication repository. The method includes receiving, at a first event publisher, a first plurality of publication instructions to publish a first dataset. The first plurality of publication instructions include a directive to retrieve a second dataset published by a second event publisher, a → 2a directive to determine a first metadata structure including a plurality of first attributes describing the first dataset, and a directive to send the first metadata structure to an asynchronous communication repository for storage. The asynchronous communication repository is configured to store a second metadata structure including a plurality of second attributes describing the second dataset. The method further includes receiving, at the first event publisher via an event application program interface (API), the second metadata structure from the asynchronous communication repository using an asynchronous communication protocol. The method further includes retrieving, at the first event publisher and based on the second attributes describing the second metadata structure, the second dataset from the second event publisher. The method further includes generating, at the first event publisher, the first dataset based on the second dataset. The method further includes determining, at the first event publisher, the first metadata structure based on the first dataset. The method further includes sending the first metadata structure from the first event publisher to the asynchronous communication repository via the event API.

These and other embodiments can each optionally include one or more of the following features.

In some embodiments of the invention, the asynchronous communication protocol includes a directive to only store, at the asynchronous communication repository, the first metadata structure determined based on a most-recently published first dataset.

In some embodiments of the invention, the first metadata structure and the second metadata structure each include attributes including a version, a time period, a dataset name, a path location, a dataset format, or a combination thereof. In some embodiments of the invention, the attributes of the first metadata structure and the attributes of the second metadata structure are different. In some embodiments of the invention, the first dataset is published multiple times based on a run frequency, and the method further includes updating the attributes of the first metadata structure based on a most-recently published first dataset.

In some embodiments of the invention, the first plurality of publication instructions further include a directive to retrieve a third dataset published by a third event publisher, the asynchronous communication repository is further configured to store a third metadata structure including a plurality of third attributes describing the third dataset, and the method further including receiving, at the first event publisher via the API, the third metadata structure from the asynchronous communication repository using the asynchronous communication protocol, retrieving, at the first event publisher and based on the third attributes describing the third metadata structure, the third dataset from the third event publisher, where the first dataset is generated based on the second dataset and the third dataset.

In some embodiments of the invention, the first plurality of publication instructions further include a first run frequency at which to publish the first dataset. In some embodiments of the invention, the publication instructions are a first set of publication instructions, the run frequency is a first run frequency, and the method further including receiving, at the first event publisher, a second set of publication instructions to publish an updated first dataset, wherein the second set of publication instructions include a second run frequency that is different from the first run frequency. In some embodiments of the invention, the first metadata structure includes attributes including a version, a time period, a dataset name, a path location, a dataset format, or a combination thereof, and the method further including updating the attributes of the first metadata structure based on an updated first dataset.

In embodiments of the invention, a computing apparatus for implementing an asynchronous communication protocol between event publishers and an asynchronous communication repository. The computing apparatus includes one or more processors, at least one memory device coupled with the one or more processors, and a data communications interface operably associated with the one or more processors. The at least one memory device contains a plurality of program instructions that, when executed by the one or more processors, cause the including apparatus to perform operations. The operations include receive, at a first event publisher, publication instructions to publish a first dataset. The publication instructions include a directive to retrieve a second dataset published by a second event publisher, a directive to determine a first metadata structure including a plurality of first attributes describing the first dataset, and a directive to send the first metadata structure to an asynchronous communication repository for storage. The asynchronous communication repository is configured to store a second metadata structure including a plurality of second attributes describing the second dataset. The operations further include receive, at the first event publisher via an event application program interface (API), the second metadata structure from the asynchronous communication repository using an asynchronous communication protocol. The operations further include retrieve, at the first event publisher and based on the second attributes describing the second metadata structure, the second dataset from the second event publisher. The operations further include generating, at the first event publisher, the first dataset based on the second dataset. The operations further include determine, at the first event publisher, the first metadata structure based on the first dataset. The operations further include send the first metadata structure from the first event publisher to the asynchronous communication repository via the event API.

In embodiments of the invention, a non-transitory computer storage medium encoded with a computer program, the computer program including a plurality of program instructions that when executed by one or more processors cause the one or more processors to perform operations. The operations include receive, at a first event publisher, publication instructions to publish a first dataset. The publication instructions include a directive to retrieve a second dataset published by a second event publisher, a directive to determine a first metadata structure including a plurality of first attributes describing the first dataset, and a directive to send the first metadata structure to an asynchronous communication repository for storage. The asynchronous communication repository is configured to store a second metadata structure including a plurality of second attributes describing the second dataset. The operations further include receive, at the first event publisher via an event application program interface (API), the second metadata structure from the asynchronous communication repository using an asynchronous communication protocol. The operations further include retrieve, at the first event publisher and based on the second attributes describing the second metadata structure, the second dataset from the second event publisher. The operations further include generating, at the first event publisher, the first dataset based on the second dataset. The operations further include determine, at the first event publisher, the first metadata structure based on the first dataset. The operations further include send the first metadata structure from the first event publisher to the asynchronous communication repository via the event API.

This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used in isolation as an aid in determining the scope of the claimed subject matter.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various embodiments of the invention and, together with a general description of the invention given above and the detailed description of the embodiments given below, serve to explain the embodiments of the invention. In the drawings, like reference numerals refer to like features in the various views.

For data processing services that involve a big data pipeline, fulfilling service requests and run sequences in a big data environment with a massive influx of real-time data may include several hurdles to maintain efficient processes because of the complexities of the building and maintaining of heavy batch big data processes. Thus, according to embodiments of the invention, simplification of the building and maintaining of heavy batch big data processes may be achieved by removing the complexity of a fixed configuration file and replacing the fixed configuration file with dynamic metadata shared storage that acts as an event stream. The processes themselves can maintain their batching properties and will not process data in real time because the processes described herein are focused on the way they are launched.

Some conventional techniques for automating big data pipelines for processing use static configuration files and an orchestrator that can control each run sequence based on the predefined static configuration file. The static configuration file includes a defined run sequence, including run frequencies for each job in the run sequence. Issues with using static configuration files include the use and dependency of a centralized entity (e.g., an orchestrator). Additionally, the static configuration files can be too big, making them hard to manage, more expensive, and with limited scalability. Therefore, complex jobs, such as A|B testing with multiple dependencies, will make the static configuration files very big, and hard to maintain. Thus, the example embodiments of the invention described herein, utilize an asynchronous communication repository as a common interface that stores small metadata structures that contain only information describing what has been produced by the issuing process.

<FIG> is an example environment <NUM> of an example environment <NUM> utilizing asynchronous communication repository, according to embodiments of the invention. The example environment <NUM> includes a client device <NUM>, one or more gateway server(s) <NUM>, one or more travel reservation server(s) <NUM>, one or more travel provider server(s) <NUM>, one or more payment processor server(s) <NUM>, one or more recommendation server(s) <NUM>, and an asynchronous communication repository <NUM> that communicate over a data communication network <NUM>, e.g., a local area network (LAN), a wide area network (WAN), the Internet, a mobile network, or a combination thereof.

The client device <NUM> (e.g., a device used by a requestor) can include a desktop, a laptop, a server, or a mobile device, such as a smartphone, tablet computer, wearable device (e.g., smartwatch), in-car computing device, and/or other types of mobile devices. The client device <NUM> includes applications, such as the application <NUM>, for managing the travel request to the one or more travel reservation server(s) <NUM> via the gateway server(s) <NUM>. The client device <NUM> can include other applications. The client device <NUM> initiates a travel request by a requestor via application <NUM>. The travel request may include availability search queries by requesting entities (such as clients, applications, browsers installed on user terminals, etc.) in the course of a search (e.g., airline booking search). A requestor of a travel request may include an airline agency, travel agency, metasearch engine, other dedicated global distribution systems (GDS), as for example airlines reservation systems which provide flight search applications for shopping business like flight booking, and the like.

The management protocols of gateway server <NUM> may be based on a redundant load-balancing system by managing multiple clients (e.g., client device(s) <NUM>) so that a travel booking and/or an availability travel request is handled by one of the one or more of the travel reservation server(s) <NUM> or the one or more of the travel recommendation server(s) <NUM> that provide travel recommendations and stores prior recommendations in a recommendations database <NUM>. For example, there may be multiple travel reservation server(s) <NUM> that are able to service the request, and the redundant load-balancing system of the gateway server(s) <NUM> is responsible for ensuring that a travel request is performed by one of the capable travel reservation server(s) <NUM>.

The one or more travel provider server(s) <NUM> receives and processes travel inventory data such as revenue data stored in a revenues management database <NUM> from one or more revenue management system(s), bookings data stored in a bookings database <NUM> from one or more bookings management system(s), airlines/system configurations data from a configurations database <NUM> from one or more airlines/system configurations management system(s), and the like. The one or more travel recommendation server(s) <NUM> generates travel recommendations based on the travel inventory data from the multiple sources (e.g., revenues management database <NUM>, bookings database <NUM>, configurations database <NUM>, etc.), and stores the travel recommendations in a recommendations database <NUM>.

One or more gateway server(s) <NUM> may be used as front end server(s) for managing, collecting, processing, and communicating availability queries, resource information, revenues management data, bookings data, airlines/system configurations data, etc., that is stored in the travel databases (e.g., revenues management database <NUM>, bookings database <NUM>, configurations database <NUM>, etc.). Further, the gateway server(s) may be front end server(s) for managing, collecting, processing, and communicating availability results from one or more travel reservation server(s) <NUM> to the client devices <NUM> via application <NUM>. In an exemplary embodiment, for an airline booking example, the gateway server(s) may be front end server(s) for collecting, processing, and storing travel information (e.g., flight schedules, flight information such as such as departure and destination airport, airline, departure and return dates, fares, booking classes, passenger information, and the like) from a plurality of external travel systems (e.g., airport information systems, airline information systems, third-party intermediator systems, etc.) via the one or more travel reservation server(s) <NUM>, the one or more travel provider server(s) <NUM>, and/or the one or more travel recommendation server(s) <NUM>.

Additionally, the management protocols of the one or more gateway server(s) <NUM> may be based on a redundant load-balancing system by managing multiple clients (e.g., client device(s) <NUM>) so that a payment associated with a travel booking request is handled by one of the one or more payment processor server(s) <NUM>. For example, there may be multiple payment processor server(s) <NUM> that are able to service the travel booking payment, and the redundant load-balancing system of the payment gateway server(s) <NUM> is responsible for ensuring that the travel booking request is performed by one of the capable payment processor server(s) <NUM>. Payment processors include for example, a credit/debit card issuer, a bank, digital payment service, etc., and store payment information in the payment record database <NUM>. The one or more servers for each payment processor generally facilitate remote communication therewith to authenticate and/or authorize a payment via a payment gateway server.

The asynchronous communication repository <NUM> is a data repository that stores meta data for each "event" generated by the one or more event publishers (e.g., the one or more travel reservation server(s) <NUM>, one or more travel provider server(s) <NUM>, one or more payment processor server(s) <NUM>, one or more recommendation server(s) <NUM>, etc.). The asynchronous communication repository <NUM> provides a common interface to each event publisher to communicate each entities' metadata regarding each published event. For example, the asynchronous communication repository <NUM> is an example event streaming platform (ESP) that plays a central role in ingesting, storing, and processing real-time data for the example environment <NUM>. The asynchronous communication repository <NUM> is utilized as a common interface that stores small metadata structures that contain only information describing what has been produced by the issuing process. For example, the travel recommendations server(s) <NUM> may rely on updated batches of travel information from the travel provider server(s) <NUM> from each of the databases. Instead of directly communicating with a travel provider server <NUM>, a travel recommendation server can pull the metadata structure associated with the relevant dataset needed to generate a travel recommendation (e.g., bookings data from the bookings database <NUM>).

The metadata structures stored in the asynchronous communication repository <NUM> may include a version, a time period (e.g., start time/end time), a dataset name, a path location, and a dataset format. According to embodiments of the invention, these attributes are a minimum version of an example metadata structure. Alternatively, the metadata structure can contain more information, and the metadata structure can evolve over time (e.g., add or remove some attributes). According to embodiments of the invention, metadata structure is a relatively small data packet that contains only information describing what has been produced by the issuing process, and does not contain any actual business data (e.g., data related to the functional scope of where the system is being used).

According to embodiments of the invention, the asynchronous communication repository <NUM> stores events in a provided order, but the asynchronous communication repository <NUM> does not control the process. The asynchronous communication repository <NUM> only stores the metadata structure from the various publishing entities in order to streamline and the simplify the communication processes between each entity. For example, each publishing entity (e.g., each travel server, also referred to herein as an event publisher) receives access to each metadata structure from the asynchronous communication repository <NUM> to decide what to do with the information contained with the metadata structures (if anything). Each event publisher (publishing entity) has access to all the metadata structures stored in the asynchronous communication repository <NUM>. Thus, data may be shared with the asynchronous communication repository <NUM> through a common event application program interface (API) based on an asynchronous communication protocol (e.g., data can be transmitted intermittently rather than in a steady stream).

Example illustrations for event processing between event publishers and an asynchronous communication repository <NUM> are illustrated in <FIG> and <FIG>. Example processes for providing an asynchronous communication protocol between event publishers and an asynchronous communication repository that are illustrated in <FIG> are further discussed herein with reference to process <NUM> of <FIG>.

<FIG> illustrates an example environment <NUM> for asynchronous communication between two event publishers and an asynchronous communication repository, according to embodiments of the invention. In particular, <FIG> illustrates a first event publisher (travel provider server <NUM>) and a second event publisher (travel recommendation server <NUM>) processing datasets based on publication instructions (e.g., job instructions <NUM>, <NUM>, respectively) and communicating metadata regarding each respective produced dataset via the asynchronous communication repository <NUM>. For example, travel provider server <NUM>, based on received job instructions <NUM>, generates dataset(s) 236a-n that are stored in the bookings database <NUM>, and travel recommendation server <NUM>, based on received job instructions <NUM>, generates dataset(s) 266a-n that are stored in the recommendations database <NUM>.

Additionally, each event publisher (travel provider server <NUM> and travel recommendation server <NUM>) sends and receives metadata structures to and from the asynchronous communication repository <NUM> based on the published datasets. For example, travel provider server <NUM> may receive metadata structures from other event publishers regarding published datasets via communication link <NUM>, and travel provider server <NUM> may upload a metadata structure for each published dataset (e.g., dataset 236a) via communication link <NUM>. Additionally, travel recommendation server <NUM> may receive metadata structures from other event publishers regarding published datasets via communication link <NUM>, and travel recommendation server <NUM> may upload a metadata structure for each published dataset (e.g., dataset 266a) via communication link <NUM>. As discussed herein, the metadata structures stored in the asynchronous communication repository <NUM> may include a version, a time period (e.g., start time/end time), a dataset name, a path location, and a dataset format. According to embodiments of the invention, these attributes are a minimum version of an example metadata structure. Alternatively, the metadata structure can contain more information, and the metadata structure can evolve over time (e.g., add or remove some attributes).

The example environment <NUM> provides an example for asynchronous communication between two event publishers that allows each event publisher to be free to scan through all events and finds relevant ones to process. For example, each event publisher (travel provider server <NUM> and travel recommendation server <NUM>) can choose which dataset is needed to process their own datasets. For example, as illustrated at communication link <NUM>, travel recommendation server <NUM> determines, based on the acquired metadata structure received at communication link <NUM>, which dataset of datasets 236a-n to acquire from the bookings database <NUM> based on the metadata structure. Thus, each event publisher is responsible of emitting a metadata structure describing the produced dataset, so that other event publishers know they can use that dataset when they acquire the associated metadata structure stored in the asynchronous communication repository <NUM>.

According to embodiments of the invention, the job instructions <NUM>, <NUM> may include a directive to retrieve a second dataset published by a second event publisher, a directive to determine a first metadata structure including a plurality of first attributes describing the first dataset, and a directive to send the first metadata structure to the asynchronous communication repository <NUM> for storage. For example, a directive in the job instructions <NUM>, may request the travel recommendation server <NUM> to pull the most recent dataset from datasets 236a-n published by the travel provider server <NUM> by accessing metadata structures via the asynchronous communication repository <NUM>.

Moreover, according to embodiments of the invention, the job instructions <NUM>, <NUM> may include a run frequency such that each event publisher would know when to publish each dataset of the plurality of datasets at a determined time interval. For example, each dataset is generated based on the run frequency, independent of waiting for an updated job/dataset to be completed by another publisher. Each dataset instance needs its dependencies to be generated, only then the job can successfully build the new dataset instance. In some implementations, the run sequence may be independent from the other publishers, but it does not define the generated dataset frequency. For example, a process can depend on two daily datasets, to generate a third one, but the frequency can be hourly if the system architect wanted to have faster reaction time.

<FIG> illustrates an example environment <NUM> for asynchronous communication between a plurality of event publishers and an asynchronous communication repository, according to embodiments of the invention. Example environment <NUM> is similar to example environment <NUM> regarding the asynchronous communication protocols but includes additionally event publishers to illustrate an example data process flow.

In particular, <FIG> illustrates five different event publishers (<NUM>, <NUM>, <NUM>, <NUM>, and <NUM>) processing respective datasets (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) based on respective instructions sets, and communicating metadata regarding each respective produced dataset via the asynchronous communication repository <NUM>. For example, event publisher-<NUM><NUM>, per job instructions received, generates dataset-<NUM><NUM>, and sends an associated metadata structure via communication link 315a to the asynchronous communication repository <NUM> that specifies details associated with the generated dataset-<NUM><NUM> (e.g., a version, a time period, a dataset name, a path location, a dataset format, etc.). Similarly, event publisher-<NUM><NUM>, per job instructions received, generates dataset-<NUM><NUM>, and sends an associated metadata structure via communication link 325a to the asynchronous communication repository <NUM> that specifies details associated with the generated dataset-<NUM><NUM> (e.g., a version, a time period, a dataset name, a path location, a dataset format, etc.). Similarly, event publisher-<NUM><NUM>, per job instructions received, generates dataset-<NUM><NUM>, and sends an associated metadata structure via communication link 355a to the asynchronous communication repository <NUM> that specifies details associated with the generated dataset-<NUM><NUM> (e.g., a version, a time period, a dataset name, a path location, a dataset format, etc.).

Event publisher-<NUM><NUM>, per job instructions received, determines that event publisher-<NUM><NUM> will need datasets from event publisher-<NUM><NUM> and event publisher-<NUM><NUM>. Thus, event publisher-<NUM><NUM> acquires the associated metadata structures from asynchronous communication repository <NUM> via communication link 315b (associated with event publisher-<NUM><NUM>) and communication link 325b (associated with event publisher-<NUM><NUM>), and based on the associated metadata structures, the event publisher-<NUM><NUM> determines which datasets to acquire from event publisher-<NUM><NUM> and event publisher-<NUM><NUM>. After acquiring the latest datasets as specified by the metadata structures from event publisher-<NUM><NUM> and event publisher-<NUM><NUM>, event publisher-<NUM><NUM> generates dataset-<NUM><NUM>, and sends an associated metadata structure via communication link 335a to the asynchronous communication repository <NUM> that specifies details associated with the generated dataset-<NUM><NUM> (e.g., a version, a time period, a dataset name, a path location, a dataset format, etc.).

Event publisher-<NUM><NUM>, per job instructions received, determines that event publisher-<NUM><NUM> will need datasets from event publisher-<NUM><NUM> and event publisher-<NUM><NUM>. Thus, event publisher-<NUM><NUM> acquires the associated metadata structures from asynchronous communication repository <NUM> via communication link 335b (associated with event publisher-<NUM><NUM>) and communication link 355b (associated with event publisher-<NUM><NUM>), and based on the associated metadata structures, the event publisher-<NUM><NUM> determines which datasets to acquire from event publisher-<NUM><NUM> and event publisher-<NUM><NUM>. After acquiring the latest datasets as specified by the metadata structures from event publisher-<NUM><NUM> and event publisher-<NUM><NUM>, event publisher-<NUM><NUM> generates dataset-<NUM><NUM>, and sends an associated metadata structure via communication link 345a to the asynchronous communication repository <NUM> that specifies details associated with the generated dataset-<NUM><NUM> (e.g., a version, a time period, a dataset name, a path location, a dataset format, etc.).

For example, event publisher-<NUM><NUM> may be an A|B testing server for controlled experiments, such as A|B testing, that may utilize the event streaming platform (ESP) such as environment <NUM> utilizing the asynchronous communication repository <NUM> via an asynchronous communication protocol. A|B testing is used to enable updates to current systems or new features to be objectively evaluated while minimizing the risks associated with full deployment. Thus, instead of using conventional techniques for automating big data pipelines as illustrated in environment <NUM> for processing data events using static configuration files and an orchestrator to control each run sequence based on the predefined static configuration file, the environment <NUM> utilizes an asynchronous communication protocol. Thus, static configuration files which may include a defined run sequence, including run frequencies for each job in the run sequence, are not needed. Issues with using static configuration files include the use and dependency of a centralized entity (e.g., an orchestrator). Additionally, the static configuration files can be too big, making them hard to manage, more expensive, and with limited scalability. Therefore, complex jobs, such as A|B testing with multiple dependencies (e.g. depending on dataset-<NUM><NUM> from event publisher-<NUM><NUM>, and dataset-<NUM><NUM> from event publisher-<NUM><NUM>), would make the static configuration files very big, and hard to maintain. Thus, the asynchronous communication protocol between multiple event publishers allows each event publisher to be free to scan through all events and find relevant datasets to process based on the metadata structures stored in the asynchronous communication repository <NUM>.

<FIG> illustrates a flowchart of an example process <NUM> for implementing an asynchronous communication protocol between event publishers and an asynchronous communication repository, according to embodiments of the invention. Operations of the process <NUM> can be implemented, for example, by a system that includes one or more event publishers (e.g., travel recommendation server <NUM>, or the like) and an asynchronous communication repository <NUM> of <FIG>. The process <NUM> can also be implemented by instructions stored on computer storage medium, where execution of the instructions by a system that includes a data processing apparatus cause the data processing apparatus to perform the operations of the process <NUM>.

The system receives, at a first event publisher, a first plurality of publication instructions to publish a first dataset, the first plurality of publication instructions including a directive to retrieve a second dataset published by a second event publisher, a directive to determine a first metadata structure including a plurality of first attributes describing the first dataset, and a directive to send the first metadata structure to an asynchronous communication repository for storage (<NUM>). In embodiments of the invention, the asynchronous communication repository is configured to store a second metadata structure including a plurality of second attributes describing the second dataset. For example, as illustrated in <FIG>, the travel recommendation server <NUM>, based on received job instructions <NUM>, generates dataset(s) 266a-n that are stored in the recommendations database <NUM>. According to embodiments of the invention, the job instructions <NUM>, <NUM> may include a run frequency such that each event publisher would know when to publish each dataset of the plurality of datasets at a determined time interval. For example, each dataset is generated based on the run frequency, independent of waiting for an updated job/dataset to be completed by another publisher. Moreover, according to embodiments of the invention, the instructions <NUM>, <NUM> may further include directives to access a dataset from one or more of the event publishers.

The system receives, via an event application program interface (API), the second metadata structure from the asynchronous communication repository using an asynchronous communication protocol (<NUM>). For example, as illustrated in <FIG>, the travel recommendation server <NUM> may access a metadata structure associated with the travel provider server <NUM> via communication link <NUM> from the asynchronous communication repository <NUM> based on an asynchronous communication protocol (e.g., data can be transmitted intermittently rather than in a steady stream based on waiting for prior events to occur).

The system retrieves, based on the second attributes describing the second metadata structure, the second dataset from the second event publisher (<NUM>). For example, as illustrated in <FIG> at communication link <NUM>, travel recommendation server <NUM> determines, based on the acquired metadata structure received at communication link <NUM>, which dataset of datasets 236a-n to acquire from the bookings database <NUM> based on the metadata structure. Thus, each event publisher is responsible of emitting a metadata structure describing the produced dataset, so that other event publishers know they can use that dataset when they acquire the associated metadata structure stored in the asynchronous communication repository <NUM>.

The system generates the first dataset based on the second dataset (<NUM>). For example, as illustrated in <FIG>, the travel recommendation server <NUM>, based on the received dataset from the travel provider server <NUM> (e.g., dataset 236a), generates a dataset (e.g., dataset 266a) and stores the dataset in the recommendations database <NUM>.

The system determines the first metadata structure based on the first dataset (<NUM>) and sends the first metadata structure from the first event publisher to the asynchronous communication repository via the event API (<NUM>). For example, as illustrated in <FIG>, after the travel recommendation server <NUM> generates the dataset 266a, the travel recommendation server <NUM> determines an associated metadata structure for that dataset, and sends an updated metadata structure via communication link <NUM> to the asynchronous communication repository <NUM>.

In some implementations of the invention, the asynchronous communication protocol includes a directive to only store, at the asynchronous communication repository, the first metadata structure determined based on a most-recently published first dataset.

In some implementations of the invention, the first metadata structure and the second metadata structure each include attributes including a version, a time period, a dataset name, a path location, a dataset format, or a combination thereof. For example, attributes of a metadata structure may include at least one of a version, a time period (e.g., start time/end time), a dataset name, a path location, and a dataset format. According to embodiments of the invention, these attributes are a minimum version of an example metadata structure. Alternatively, the metadata structure can contain more information, and the metadata structure can evolve over time (e.g., add or remove some attributes). According to embodiments of the invention, metadata structure is a relatively small data packet that contains only information describing what has been produced by the issuing process, and does not contain any actual business data (e.g., data related to the functional scope of where the system is being used).

In some implementations of the invention, the attributes of the first metadata structure and the attributes of the second metadata structure are different. For example, each event publisher may include different attributes within the metadata structure. For example, a metadata message may require a minimal standard set of attributes to enable universal compatibility. For example, a version, a dataset name, a dataset type, and the like, can be an example of mandatory attributes. Then depending on the value of the dataset type, different additional attributes may be used to describe some specific dataset properties. For example, a dataset type may include a particular type of file name extension. Or, some additional attributes for dataset type file may include a format, a file name, and a file location, and the like. For a dataset type that is a table, some additional attributes or a different sets of attributes may include a database name, a table name, partition information, and the like.

In some implementations of the invention, the system updates the attributes of the first metadata structure based on a most-recently published first dataset. For example, as illustrated in <FIG>, after the travel recommendation server <NUM> generates the dataset 266a, the travel recommendation server <NUM> determines an associated metadata structure for that dataset, and sends an updated metadata structure via communication link <NUM> to the asynchronous communication repository <NUM>. In some implementations of the invention, the first metadata structure includes attributes including a version, a time period, a dataset name, a path location, a dataset format, or a combination thereof, and the process <NUM> further includes updating the attributes of the first metadata structure based on an updated first dataset.

In some implementations of the invention, the first plurality of publication instructions include a directive to retrieve a third dataset published by a third event publisher, and the asynchronous communication repository is further configured to store a third metadata structure including a plurality of third attributes describing the third dataset. For example, as illustrated in <FIG>, event publisher-<NUM><NUM>, per job instructions received, determines that event publisher-<NUM><NUM> will need datasets from event publisher-<NUM><NUM> and event publisher-<NUM><NUM>. Thus, the process <NUM> may include receiving the third metadata structure from the asynchronous communication repository using the asynchronous communication protocol. Additionally, the process <NUM> may include retrieving, based on the third attributes describing the third metadata structure, the third dataset from the third event publisher, where Moreover, the first dataset is generated based on the second dataset and the third dataset. For example, as illustrated in <FIG>, event publisher-<NUM><NUM>, per job instructions received, determines that event publisher-<NUM><NUM> will need datasets from event publisher-<NUM><NUM> and event publisher-<NUM><NUM>. Thus, event publisher-<NUM><NUM> acquires the associated metadata structures from asynchronous communication repository <NUM> via communication link 335b (associated with event publisher-<NUM><NUM>) and communication link 355b (associated with event publisher-<NUM><NUM>), and based on the associated metadata structures, the event publisher-<NUM><NUM> determines which datasets to acquire from event publisher-<NUM><NUM> and event publisher-<NUM><NUM>. After acquiring the latest datasets as specified by the metadata structures from event publisher-<NUM><NUM> and event publisher-<NUM><NUM>, event publisher-<NUM><NUM> generates dataset-<NUM><NUM>, and sends an associated metadata structure via communication link 345a to the asynchronous communication repository <NUM> that specifies details associated with the generated dataset-<NUM><NUM> (e.g., a version, a time period, a dataset name, a path location, a dataset format, etc.).

In some implementations of the invention, the first plurality of publication instructions further include a first run frequency at which to publish the first dataset. In some implementations of the invention, the process <NUM> may further include receiving a second plurality of publication instructions to publish an updated first dataset, where the second plurality of publication instructions includes a second run frequency that is different from the first run frequency. For example, an event publisher may receive publication instructions to compile a new dataset every two hours, but later receive new publication instructions to compile a new dataset every hour. This change would not need to be resent to each subsequent event publishers that relies on that dataset. Instead, the other event publishers would determine that there may be an updated dataset (e.g., based on the new run frequency) using the acquired metadata structure from the asynchronous communication repository <NUM>.

<FIG> illustrates an example computer architecture <NUM> for a computer <NUM> capable of executing the software components described herein for the sending/receiving and processing of tasks for the CA components. The computer architecture <NUM> (also referred to herein as a "server") shown in <FIG> illustrates a server computer, workstation, desktop computer, laptop, or other computing device, and may be utilized to execute any aspects of the software components presented herein described as executing on a host server, or other computing platform. The computer <NUM> preferably includes a baseboard, or "motherboard," which is a printed circuit board to which a multitude of components or devices may be connected by way of a system bus or other electrical communication paths. In one illustrative embodiment, one or more central processing units (CPUs) <NUM> operate in conjunction with a chipset <NUM>. The CPUs <NUM> can be programmable processors that perform arithmetic and logical operations necessary for the operation of the computer <NUM>.

The CPUs <NUM> preferably perform operations by transitioning from one discrete, physical state to the next through the manipulation of switching elements that differentiate between and change these states. Switching elements may generally include electronic circuits that maintain one of two binary states, such as flip-flops, and electronic circuits that provide an output state based on the logical combination of the states of one or more other switching elements, such as logic gates. These basic switching elements may be combined to create more complex logic circuits, including registers, adders-subtractors, arithmetic logic units, floating-point units, or the like.

The chipset <NUM> provides an interface between the CPUs <NUM> and the remainder of the components and devices on the baseboard. The chipset <NUM> may provide an interface to a memory <NUM>. The memory <NUM> may include a random access memory (RAM) used as the main memory in the computer <NUM>. The memory <NUM> may further include a computer-readable storage medium such as a read-only memory (ROM) or non-volatile RAM (NVRAM) for storing basic routines that that help to startup the computer <NUM> and to transfer information between the various components and devices. The ROM or NVRAM may also store other software components necessary for the operation of the computer <NUM> in accordance with the embodiments described herein.

According to various embodiments, the computer <NUM> may operate in a networked environment using logical connections to remote computing devices through one or more networks <NUM>, a local-area network (LAN), a wide-area network (WAN), the Internet, or any other networking topology known in the art that connects the computer <NUM> to the devices and other remote computers. The chipset <NUM> includes functionality for providing network connectivity through one or more network interface controllers (NICs) <NUM>, such as a gigabit Ethernet adapter. For example, the NIC <NUM> may be capable of connecting the computer <NUM> to other computer devices in the utility provider's systems. It should be appreciated that any number of NICs <NUM> may be present in the computer <NUM>, connecting the computer to other types of networks and remote computer systems beyond those described herein.

The computer <NUM> may be connected to at least one mass storage device <NUM> that provides non-volatile storage for the computer <NUM>. The mass storage device <NUM> may store system programs, application programs, other program modules, and data, which are described in greater detail herein. The mass storage device <NUM> may be connected to the computer <NUM> through a storage controller <NUM> connected to the chipset <NUM>. The mass storage device <NUM> may consist of one or more physical storage units. The storage controller <NUM> may interface with the physical storage units through a serial attached SCSI (SAS) interface, a serial advanced technology attachment (SATA) interface, a fiber channel (FC) interface, or other standard interface for physically connecting and transferring data between computers and physical storage devices.

The computer <NUM> may store data on the mass storage device <NUM> by transforming the physical state of the physical storage units to reflect the information being stored. The specific transformation of physical state may depend on various factors, in different embodiments of the invention of this description. Examples of such factors may include, but are not limited to, the technology used to implement the physical storage units, whether the mass storage device <NUM> is characterized as primary or secondary storage, or the like. For example, the computer <NUM> may store information to the mass storage device <NUM> by issuing instructions through the storage controller <NUM> to alter the magnetic characteristics of a particular location within a magnetic disk drive unit, the reflective or refractive characteristics of a particular location in an optical storage unit, or the electrical characteristics of a particular capacitor, transistor, or other discrete component in a solid-state storage unit. Other transformations of physical media are possible without departing from the scope and spirit of the present description, with the foregoing examples provided only to facilitate this description. The computer <NUM> may further read information from the mass storage device <NUM> by detecting the physical states or characteristics of one or more particular locations within the physical storage units.

The mass storage device <NUM> may store an operating system <NUM> utilized to control the operation of the computer <NUM>. According to some embodiments, the operating system includes the LINUX operating system. According to another embodiment, the operating system includes the WINDOWS® SERVER operating system from MICROSOFT Corporation of Redmond, Wash. According to further embodiments, the operating system may include the UNIX or SOLARIS operating systems. It should be appreciated that other operating systems may also be utilized. The mass storage device <NUM> may store other system or application programs and data utilized by the computer <NUM>, such as a communication module <NUM> to manage the communications between the even publishers and the asynchronous communication repository via an asynchronous communication protocol, a metadata module <NUM> to manage the metadata structures, and a job execution module <NUM> to execute a task, according to embodiments described herein.

In some embodiments, the mass storage device <NUM> may be encoded with computer-executable instructions that, when loaded into the computer <NUM>, transforms the computer <NUM> from being a general-purpose computing system into a special-purpose computer capable of implementing the embodiments described herein. These computer-executable instructions transform the computer <NUM> by specifying how the CPUs <NUM> transition between states, as described above. According to some embodiments, from the availability determination server(s) <NUM> perspective, the mass storage device <NUM> stores computer-executable instructions that, when executed by the computer <NUM>, perform portions of the process <NUM> for implementing an asynchronous communication protocol between event publishers and an asynchronous communication repository, as described herein. In further embodiments, the computer <NUM> may have access to other computer-readable storage medium in addition to or as an alternative to the mass storage device <NUM>.

The computer <NUM> may also include an input/output controller <NUM> for receiving and processing input from a number of input devices, such as a keyboard, a mouse, a touchpad, a touch screen, an electronic stylus, or other type of input device. Similarly, the input/output controller <NUM> may provide output to a display device, such as a computer monitor, a flat-panel display, a digital projector, a printer, a plotter, or other type of output device. It will be appreciated that the computer <NUM> may not include all of the components shown in <FIG>, may include other components that are not explicitly shown in <FIG>, or may utilize an architecture completely different than that shown in <FIG>.

In general, the routines executed to implement the embodiments of the invention, whether implemented as part of an operating system or a specific application, component, program, object, module or sequence of instructions, or even a subset thereof, may be referred to herein as "computer program code," or simply "program code. " Program code typically includes computer readable instructions that are resident at various times in various memory and storage devices in a computer and that, when read and executed by one or more processors in a computer, cause that computer to perform the operations necessary to execute operations and/or elements embodying the various aspects of the embodiments of the invention. Computer readable program instructions for carrying out operations of the embodiments of the invention may be, for example, assembly language or either source code or object code written in any combination of one or more programming languages.

The program code embodied in any of the applications/modules described herein is capable of being individually or collectively distributed as a program product in a variety of different forms. In particular, the program code may be distributed using a computer readable storage medium having computer readable program instructions thereon for causing a processor to carry out aspects of the embodiments of the invention.

Computer readable storage media, which is inherently non-transitory, may include volatile and non-volatile, and removable and non-removable tangible media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. Computer readable storage media may further include random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid state memory technology, portable compact disc read-only memory (CD-ROM), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and which can be read by a computer. A computer readable storage medium should not be construed as transitory signals per se (e.g., radio waves or other propagating electromagnetic waves, electromagnetic waves propagating through a transmission media such as a waveguide, or electrical signals transmitted through a wire). Computer readable program instructions may be downloaded to a computer, another type of programmable data processing apparatus, or another device from a computer readable storage medium or to an external computer or external storage device via a network.

Computer readable program instructions stored in a computer readable medium may be used to direct a computer, other types of programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions that implement the functions/acts specified in the flowcharts, sequence diagrams, and/or block diagrams. The computer program instructions may be provided to one or more processors of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the one or more processors, cause a series of computations to be performed to implement the functions and/or acts specified in the flowcharts, sequence diagrams, and/or block diagrams.

In certain alternative embodiments, the functions and/or acts specified in the flowcharts, sequence diagrams, and/or block diagrams may be re-ordered, processed serially, and/or processed concurrently without departing from the scope of the embodiments of the invention. Moreover, any of the flowcharts, sequence diagrams, and/or block diagrams may include more or fewer blocks than those illustrated consistent with embodiments of the invention.

Claim 1:
A computer-implemented method comprising:
receiving, at a first event publisher (<NUM>), a first plurality of publication instructions (<NUM>) to publish a first dataset (<NUM>), the first plurality of publication instructions (<NUM>) comprising a directive to retrieve a second dataset (<NUM>) published by a second event publisher (<NUM>), a directive to determine a first metadata structure including a plurality of first attributes describing the first dataset (<NUM>), and a directive to send the first metadata structure to an asynchronous communication repository (<NUM>) for storage, wherein the asynchronous communication repository (<NUM>) is configured to store a second metadata structure including a plurality of second attributes describing the second dataset (<NUM>);
receiving, at the first event publisher (<NUM>) via an event application program interface, API, the second metadata structure from the asynchronous communication repository (<NUM>) using an asynchronous communication protocol;
retrieving, at the first event publisher (<NUM>) and based on the second attributes describing the second metadata structure, the second dataset (<NUM>) from the second event publisher (<NUM>);
generating, at the first event publisher (<NUM>), the first dataset (<NUM>) based on the second dataset (<NUM>);
determining, at the first event publisher (<NUM>), the first metadata structure based on the first dataset (<NUM>); and
sending the first metadata structure from the first event publisher (<NUM>) to the asynchronous communication repository (<NUM>) via the event API.