Patent ID: 12189502

DETAILED DESCRIPTION

The following detailed description of example implementations refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements.

Data center resiliency is the ability of a data center (or data center components) to recover quickly and continue operating despite a disruption, such as an equipment failure or a power outage. Data center resilience is important to prevent or reduce interruptions to data processing, to provide continuous (or near-continuous) service availability, and to avoid data loss. Data center resiliency can be achieved in a variety of ways, and may depend on a system architecture of a data center or multiple data centers. Different techniques for providing data center resiliency may result in different outcomes in terms of service availability, data loss, or preventing data processing interruptions. Failover is one technique that can be used to switch from a failed data center to a backup data center.

Some techniques described herein provide data center resilience, and particularly resilience in an event-driven system, using event-driven system failover and failback techniques. These techniques may be applied to a data center architecture used for event processing, sometimes called an event-driven architecture. In an event-driven architecture, an event notification may be triggered based on occurrence of an event, which is a change in state of information relevant to an event processing system. The event notification may be produced, published, propagated, detected, and/or consumed in connection with the event processing system. Systems and methods described herein enable failover in an event processing system (e.g., having an event-driven architecture) to improve service availability, reduce delays associated with event processing, and/or prevent data loss.

FIG.1is a diagram of an example system100associated with event-driven system failover and failback. As shown inFIG.1, the example system100includes a first data center102(e.g., that includes a first event-driven system) and a second data center104(e.g., that includes a second event-driven system). In the example system100, a failure occurs in the second data center104that triggers failover from the second data center104to the first data center102for event processing. In some implementations, the first data center102may be located in a first geographic region (and/or may be housed in a first building or a first group of buildings), and the second data center104may be located in a second (e.g., different or separate) geographic region (and/or may be housed in a second building or a second group of buildings).

As shown, the first data center102may include a main event store106(e.g., a main database), a replica event store108(e.g., a replica database), one or more event processors110, a computing instance112, and an event retrieval component114. Because these components are part of the first data center102, these components are sometimes referred to herein as a first main event store, a first replica event store, one or more first event processors, a first computing instance, and a first event retrieval component, respectively.

In some implementations, the second data center104may include a main event store116(e.g., a main database), a replica event store118(e.g., a replica database), one or more event processors120, a computing instance122, and an event retrieval component124. Because these components are part of the second data center104, these components are sometimes referred to herein as a second main event store, a second replica event store, one or more second event processors, a second computing instance, and a second event retrieval component, respectively.

In some cases, the data center that experiences a failure (e.g., the second data center104in the example system100) may be called a “primary data center,” and the data center that begins processing events that would otherwise be processed by the primary data center if the failure had not occurred (e.g., the first data center102in the example system100) may be called a “secondary data center.” Similarly, an event processor in the primary data center may be called a “primary event processor,” and an event processor in the secondary data center may be called a “secondary event processor.” Similarly, a main event store in the primary data center may be called a “primary main event store,” and a main event store in the secondary data center may be called a “secondary main event store.” Similarly, a replica event store in the primary data center may be called a “primary replica event store,” and a replica event store in the secondary data center may be called a “secondary replica event store.” Similarly, an event stored in the main event store of the primary data center may be called a “primary event,” and an event stored in the main event store of the secondary data center may be called a “secondary event.”

As further shown, the example system100may include a management system126. The management system126may manage the first data center102and/or the second data center104to assist with providing data center resilience, such as by assisting with failover from a primary data center (e.g., the second data center104in the example system100) to a secondary data center (e.g., the first data center102in the example system100). The management system126may include one or more components that are part of the first data center102and one or more components that are part of the second data center104. Additionally, or alternatively, the management system126may communicate with respective components of the first data center102and the second data center104via a network.

In the first data center102, the main event store106may store first events to be processed in the first data center102. As used herein in connection with the example implementations, the term “event” refers to information indicative of occurrence of an event and/or a state of an event, and may be used interchangeably with “event notification.” In some implementations, the first events stored in the main event store106are processed only in the first data center102(and not any other data centers) unless the first data center102experiences a failure. Similarly, in the second data center104, the main event store116may store second events to be processed in the second data center104. In some implementations, the second events stored in the main event store116are processed only in the second data center104(and not any other data centers) unless the second data center104experiences a failure.

In some implementations, the main event store106and/or the main event store116are commit databases. In a commit database, after a record (e.g., an event) in the commit database is changed (e.g., processed), that change is permanently saved in the commit database and replaces the information that was previously stored in connection with the record. For example, if an event stored in a main event store has been processed (e.g., by an event processor), then the main event store may be updated with an indication that the event has been processed. This assists with synchronization of databases (e.g., event stores) across data centers and tracking of processed events to prevent duplicate processing and enable processing of unprocessed events in case of a failure, as described in more detail below.

In the first data center102, the replica event store108may be synchronized with the main event store116of the second data center104. In other words, the replica event store108of the first data center102may mirror the main event store116of the second data center104. For example, the replica event store108and the main event store116may be synchronized using insert synchronization, update synchronization, drop synchronization, mixed synchronization, or any other database synchronization technique. Synchronization between the replica event store108and the main event store116may be an ongoing process where changes are updated automatically between the event stores to maintain consistency, such as by a message exchange when the main event store116is updated and/or a periodic message exchange. As a result, the replica event store108may store second events (e.g., received from the main event store116) that are to be processed in the first data center102only if a failure occurs in the second data center104.

Similarly, in the second data center104, the replica event store118may be synchronized with the main event store106of the first data center102. In other words, the replica event store118of the second data center104may mirror the main event store106of the first data center102. For example, the replica event store118and the main event store106may be synchronized using insert synchronization, update synchronization, drop synchronization, mixed synchronization, or any other database synchronization technique. Synchronization between the replica event store118and the main event store106may be an ongoing process where changes are updated automatically between the event stores to maintain consistency, such as by a message exchange when the main event store106is updated and/or a periodic message exchange. As a result, the replica event store118may store first events (e.g., received from the main event store106) that are to be processed in the second data center104only if a failure occurs in the first data center102.

The first data center102may include multiple event processors110. An event processor110may be configured to process first events stored in the main event store106. In some implementations, an event processor110may be configured to process first events having a particular event type (e.g., which may be indicated and/or stored in connection with the event, such as in the main event store106). For example, an event processor110may include software code and hardware on which the software code executes. The software code may include instructions for processing an event of a particular event type. One event processor110may process events having a first event type, another event processor110may process events having a second event type, and so on.

In some implementations, each event processor110includes or is associated with a computing instance112. The computing instance112may be used by an event processor110to transmit heartbeat messages128-1to the second data center104. Additionally, or alternatively, the computing instance112may be used by the event processor110to receive and/or process heartbeat messages128-2from the second data center104. In some implementations, a computing instance112of an event processor110may transmit heartbeat messages128-1to a corresponding computing instance122of a corresponding event processor120in the second data center104(e.g., an event processor120that processes the same type of events as the event processor110). Additionally, or alternatively, the computing instance112may transmit heartbeat messages128-1to the management system126.

In some implementations, each event processor110includes or is associated with an event retrieval component114. An event retrieval component114may retrieve events to be processed by a corresponding event processor110. For example, the event retrieval component114may retrieve events having an event type that matches an event type processed by the event processor110. In some implementations, the event retrieval component114may identify appropriate events for retrieval based on an event type tag associated with an event stored in an event store. Thus, an event processor110may be configured to select (e.g., using a corresponding event retrieval component114and/or a corresponding computing instance112) which events to consume (e.g., retrieve) and/or which events to process. When the second data center104has not failed, the event retrieval component114may retrieve events only from the main event store106. When the second data center104fails, the event retrieval component114may retrieve events from either the replica event store108or the main event store116based on a configuration and/or an instruction, as described in more detail elsewhere herein.

Similarly, the second data center104may include multiple event processors120. An event processor120may be configured to process second events stored in the main event store116. In some implementations, an event processor120may be configured to process second events having a particular event type (e.g., which may be indicated and/or stored in connection with the event, such as in the main event store116). For example, an event processor120may include software code and hardware on which the software code executes. The software code may include instructions for processing an event of a particular event type. One event processor120may process events having a first event type, another event processor120may process events having a second event type, and so on.

In some implementations, each event processor120includes or is associated with a computing instance122. The computing instance122may be used by an event processor120to transmit heartbeat messages128-2to the first data center102. Additionally, or alternatively, the computing instance122may be used by the event processor120to receive and/or process heartbeat messages128-1from the first data center102. In some implementations, a computing instance122of an event processor120may transmit heartbeat messages128-2to a corresponding computing instance112of a corresponding event processor110in the first data center102(e.g., an event processor110that processes the same type of events as the event processor120). Additionally, or alternatively, the computing instance122may transmit heartbeat messages128-2to the management system126.

In some implementations, each event processor120includes or is associated with an event retrieval component124. An event retrieval component124may retrieve events to be processed by a corresponding event processor120. For example, the event retrieval component124may retrieve events having an event type that matches an event type processed by the event processor120. In some implementations, the event retrieval component124may identify appropriate events for retrieval based on an event type tag associated with an event stored in an event store. Thus, an event processor120may be configured to select (e.g., using a corresponding event retrieval component124and/or a corresponding computing instance122) which events to consume (e.g., retrieve) and/or which events to process. When the first data center102has not failed, the event retrieval component124may retrieve events only from the main event store116. When the first data center102fails, the event retrieval component124may retrieve events from either the replica event store118or the main event store106based on a configuration and/or an instruction, as described in more detail elsewhere herein.

The example system100has a system architecture that supports two-way failover, including both failover from the first data center102to the second data center104and failover from the second data center104to the first data center102. However, in some implementations, systems described herein may support only one-way failover, such as only failover from the first data center102to the second data center104or only failover from the second data center104to the first data center102. In an example where one-way failover from only the first data center102to the second data center104is supported, the first data center102may not include the replica event store108, the event retrieval component114may retrieve events from only the main event store106, and/or the computing instance112may only transmit (and not receive) heartbeat messages128. Similarly, in an example where one-way failover from only the second data center104to the first data center102is supported, the second data center104may not include the replica event store118, the event retrieval component124may retrieve events from only the main event store116, and/or the computing instance122may only transmit (and not receive) heartbeat messages128. Additionally, or alternatively, the system100may include more than two data centers that include the components described herein and operate according to the techniques described herein.

As shown by reference number130, the system100(e.g., using the management system126) may detect a failure of a second event processor120. For example, a particular event processor120, of the multiple event processors120included in the second data center104, may fail. The failed event processor120may be configured to process a particular type of event (e.g., one of multiple event types for which the second event processors120are configured to process). The particular event processor120may fail due to a software error (e.g., a bug), a hardware error (e.g., a misconfiguration, a disk crash, or the like), a communication error (e.g., a failure to retrieve events from the main event store116, a network error, or the like), a database error (e.g., corrupted data), and/or a power failure, among other examples. In some implementations, heartbeat messages128are transmitted to the management system126, and the management system126may detect the failure based on failing to receive a heartbeat message128. Alternatively, heartbeat messages128may be transmitted to event processors (e.g., to a computing instance), and an event processor may notify the management system126upon failure to receive a heartbeat message128.

In some implementations, the system100may detect the failure of the second event processor120based on a failure to receive a heartbeat message128-2associated with the second event processor120, based on reception of a heartbeat message128-2that indicates a failure status associated with the second event processor120, and/or based on one or more metrics associated with the second event processor120. As described above, each event processor120may include or may be associated with a computing instance122that is used to transmit heartbeat messages128-2to the first data center102(e.g., to a corresponding event processor110in the first data center102and/or to the management system126). In some implementations, each event processor120may transmit independent heartbeat messages128to the first data center102(e.g., independent of other heartbeat messages128transmitted by other event processors120). This enables customized configuration of heartbeat messages128for each event processor120and customized handling of heartbeat messages128by the first data center102, as described in more detail below.

A heartbeat message128may indicate an event processor120with which the heartbeat message128is associated. In some implementations, the heartbeat message128may explicitly indicate the event processor120, such as by including information that identifies the event processor120with which the heartbeat message128is associated. For example, the heartbeat message128may include an event processor identifier, such as an event processor filename, an event processor function name, or the like. In some implementations, a heartbeat message128may only include the event processor identifier (and not any other information). In this example, presence or absence of a heartbeat message128(e.g., successful or unsuccessful transmission and reception of the heartbeat message128) may indicate a status of a corresponding event processor120(e.g., whether the event processor120is operating normally or is associated with a failure). In this example, the event processor120may refrain from transmitting the heartbeat message128, or may attempt to transmit the heartbeat message128but experience a failure in the transmission, when the event processor120experiences a failure. This conserves network resources and reduces latency of transmission of heartbeat messages128due to a smaller message size.

In some implementations, the heartbeat message128may implicitly indicate the event processor120associated with the heartbeat message128(e.g., without including an event processor identifier). For example, the event processor120may be configured to transmit the heartbeat message128to a corresponding event processor110(e.g., to a particular network address associated with the corresponding event processor110) that processes the same event types as the event processor120. In this case, the management system126and/or the event processor110may determine the event processor120associated with the heartbeat message128based on the event processor110that received the heartbeat message128. This may further conserve network resources and reduce latency of transmission of heartbeat messages128due to a smaller message size.

In some implementations, the heartbeat message128may explicitly indicate the status of a corresponding event processor120(e.g., using a field in the heartbeat message128). For example, a first value (e.g., 1) of a bit may indicate that the corresponding event processor120is operating normally (e.g., a normal status), and a second value (e.g., 0) of a bit may indicate that the corresponding event processor120is associated with a failure (e.g., a failure status). In some implementations, the system100may detect a failure of the second event processor120if either a heartbeat message128is not received (e.g., for a threshold time or a threshold quantity of periods, as described elsewhere herein) or if a heartbeat message128that explicitly indicates a failure is received. This may improve robustness in failure detection as compared to using only one of these techniques.

In some implementations, each event processor120may periodically transmit corresponding heartbeat messages128-1according to a periodicity (e.g., every 10 milliseconds, every 100 milliseconds, every 1 second, every 5 seconds, and so on). The periodicity of transmission of heartbeat messages128associated with a particular event processor120may be indicated in a configuration associated with that event processor120. In some implementations, different event processors120may be associated with different configurations, and may transmit corresponding heartbeat messages128according to different periodicities. This enables customization and prioritization based on a criticality of events being processed by a particular event processor120. In some implementations, the system100(e.g., using the management system126) may determine the periodicity for an event processor120(e.g., rather than the periodicity being input by an operator associated with the event processor120), such as based on the quantity of events stored in the main event store116that have an event type to be processed by the event processor120. For example, an event type for which a greater quantity of events is stored in the main event store116may be associated with a shorter periodicity than another event type for which a lesser quantity of events is stored in the main event store116. This enables prioritization of more important event types, as indicated by the quantity of events of that event type. In some implementations, each event processor120may transmit heartbeat messages128with the same periodicity to reduce system complexity.

In some implementations, the management system126and/or the event processor110may detect the failure associated with the event processor120based on a heartbeat message128not being received for a threshold time and/or for a threshold quantity of periods. For example, if the first data center102does not receive any heartbeat messages associated with a particular event processor120for a threshold amount of time (e.g., 10 milliseconds, 100 milliseconds, 1 second, 5 seconds, or the like), then the first data center102(e.g., a corresponding event processor110and/or the management system126) may determine that the particular event processor120is associated with a failure. As another example, if the first data center102does not receive any heartbeat messages associated with a particular event processor120for a threshold quantity of periods (e.g., 1 period, 2 periods, 3 periods, or the like), then the first data center102(e.g., a corresponding event processor110and/or the management system126) may determine that the particular event processor120is associated with a failure. The duration of a period to be used for an event processor120may be determined based on (or equal to) the periodicity associated with that event processor120.

In some implementations, the threshold time and/or the threshold quantity of periods for a particular event processor120may be indicated in a configuration associated with that event processor120. In some implementations, different event processors120may be associated with different threshold times and/or threshold quantities of periods. This enables customization and prioritization based on a criticality of events being processed by a particular event processor120. In some implementations, the system100(e.g., using the management system126) may determine the threshold time and/or the threshold quantity of periods for an event processor120(e.g., rather than those values being input by an operator associated with the event processor120), such as based on the quantity of events stored in the main event store116that have an event type to be processed by the event processor120. For example, an event type for which a greater quantity of events is stored in the main event store116may be associated with a shorter threshold time and/or a smaller quantity of periods than another event type for which a lesser quantity of events is stored in the main event store116. This enables prioritization of, and faster failover for, more important event types, as indicated by the quantity of events of that event type. In some implementations, each event processor120may be associated with the same threshold time and/or the same threshold quantity of periods to reduce system complexity.

In some implementations, the management system126may detect the failure associated with the event processor120based on one or more metrics (e.g., one or more data center metrics) associated with the event processor120. A metric may indicate an operational status and/or performance of a data center (e.g., the second data center104in system100), an event store (e.g., the main event store116in system100), and/or an event processor (e.g., an event processor120in system100). For example, a metric associated with an event processor120may indicate a processor utilization of the event processor120(e.g., central processing unit (CPU) utilization), a memory utilization of the event processor120, a rate at which the event processor120is processing events, network traffic associated with the event processor120and/or the second data center104(e.g., a volume or rate of network traffic into or out of the second data center104), and/or whether the event processor120is actively processing events, among other examples. As shown, the metrics may be used by a computing instance (e.g., the computing instance112and/or the computing instance122) and/or the management system126to detect a failure. In some implementations, the metrics may be stored in a metrics database accessible by a computing instance and/or the management system126. For example, the first data center102may include a first metrics database, and the second data center104may include a second metrics database.

In some implementations, the management system126may detect the failure associated with the event processor120if the processor utilization satisfies a threshold, if the memory utilization satisfies a threshold, if the rate at which the event processor120is processing events fails to satisfy a threshold, if the network traffic satisfies a threshold, and/or if the event processor120is not actively processing events, among other examples. In a similar manner as described elsewhere herein, one or more of these thresholds, for a particular event processor120, may be indicated in a configuration associated with that event processor120(e.g., based on operator input or a determination by the management system126based on a quantity of events of a particular event type stored in the main event store116).

In some implementations, based on detecting the failure associated with the event processor120, the management system126may transmit a notification to one or more event producers (not shown), instructing the one or more event producers (e.g., one or more event sources) to produce events for the first data center102. For example, a first set of event producers may produce events for the first data center102(e.g., to populate the main event store106with first events), and a second set of event producers may produce events for the second data center104(e.g., to populate the main event store116with second events). Based on detecting the failure associated with the second data center104(e.g., the event processor120), the management system126may notify the second set of event producers to begin producing events to the first data center102(e.g., to populate the main event store106). Additionally, or alternatively, the second set of event producers may autonomously being producing events to the first data center102(e.g., without a notification or instruction from the messaging system126) based on detecting a failure in the second data center and/or a threshold quantity of failed attempts to produce an event to the second data center104.

As shown by reference number132, the system100(e.g., using the management system126) may identify an event processor110, in the first data center102, that is to process one or more second events (e.g., associated with the failed event processor120). For example, the management system126may identify the event processor110after detecting the failure of the event processor120. The identified event processor110may be configured to process the same type of events as the failed event processor120(e.g., the identified event processor110may include code that is also included in the failed event processor120). In some implementations, the management system126may identify an event processor110corresponding to the failed event processor120based on an event processor identifier associated with the failed event processor120. For example, the management system126may identify an event processor110having the same event processor identifier as the failed event processor120. As another example, the management system126may store a data structure (e.g., a table) that indicates a mapping between event processors110in the first data center102and corresponding event processors120in the second data center104. The management system126may use this data structure to look up (e.g., using an event processor identifier of the failed event processor120) an event processor110corresponding to the failed event processor120.

In some implementations, a configuration associated with the failed event processor120may indicate whether a corresponding event processor110is to be booted up or is to be selected from multiple event processors110that are actively processing first events in the first data center102. This enables flexible configuration of different event processors. In the example ofFIG.1, the management system126selects the event processor110from multiple event processors110that are actively processing first events in the first data center102(e.g., first events stored in the main event store106). In this example, the identified event processor110and the failed event processor120may be configured for identical processing of events based on execution of identical code, which enables use of an event processor110that is already actively processing events having the same event type as events that the failed event processor120is configured to process (e.g., because the event processor110will execute the same code for first events and second events having the same event type). In other examples, described in more detail below in connection withFIG.2andFIG.3, the management system126may boot up an event processor110to process the one or more second events.

In some implementations, the management system126may determine whether to boot up or select the event processor110based on the quantity of events stored in the main event store116that have an event type to be processed by the event processor110(e.g., the same event type of events processed by the failed event processor120). For example, the management system126may boot up an event processor110to process second events having an event type for which a quantity of events stored in the main event store116(or the replica event store108) satisfies a threshold, and may select an active event processor110(e.g., that processes the same type of events processed by the failed event processor120) to process second events having an event type for which a quantity of events stored in the main event store116(or the replica event store108) does not satisfy the threshold. Additionally, or alternatively, the management system126may boot up an event processor110if a sum of first events and second events having the event type to be processed satisfies a threshold, and may select an active event processor110if a sum of first events and second events having the event type to be processed does not satisfy the threshold. This enables customization and prioritization based on a criticality and/or volume of events that are being failed over to the first data center102.

In some implementations, the management system126may determine whether an event processor110, corresponding to the failed event processor120, is already booted up. If the corresponding event processor110is booted up in the first data center102(e.g., as shown inFIG.1), then the management system126may select that event processor110to process second events for the failed event processor120, and may instruct that event processor110to process the second events. If the corresponding event processor110is not booted up in the first data center102, then the management system126may boot up that event processor110prior to instructing that event processor110to process the second events, as described in more detail below in connection withFIG.2andFIG.3.

As shown by reference number134, the management system126may identify a configuration associated with the failed event processor120and/or the identified event processor110. For example, the management system126may store a data structure that identifies event processors110and/or event processors120and corresponding configurations for those event processors. In some implementations, the data structure may indicate a correspondence between an event processor110and an event processor120(e.g., using one or more event processor identifiers), and may also indicate a configuration to be used for those event processors110,120(e.g., a single configuration for both event processors110,120or different configurations for each event processor110,120). As described above, the configuration may indicate a periodicity for heartbeat messages, a failure detection condition (e.g., a threshold time for detecting failure, a threshold quantity of periods for detecting failure, one or more metric thresholds), and/or whether to boot up or select an event processor upon detecting a failure.

Additionally, or alternatively, the configuration may indicate whether the identified event processor110is to process the one or more second events from the replica event store108in the first data center102or the main event store116of the second data center104. Retrieval and processing of events from the replica event store108may have lower latency than retrieval and processing of events from the main event store116because the replica event store108is located nearer to the event processor110. However, retrieval and processing of events from the main event store116may result in more accurate processing (e.g., less duplicate processing of events that have already been processed in the second data center104, more accuracy in the stored events that are to be processed, or the like) due to synchronization issues between the main event store116and the replica event store108. In some implementations, different event processors110may be associated with different configurations depending on whether latency or accuracy (and/or other factors) are to be prioritized.

In some implementations, the management system126may determine whether the identified event processor110is to process the one or more second events from the replica event store108or the main event store116(e.g., rather than this configuration being input by an operator associated with the event processor110), such as based on the quantity of events stored in the main event store116and/or the replica event store108that have an event type to be processed by the identified event processor110. For example, the management system126may instruct the event processor110(e.g., the event retrieval component114) to process second events from the main event store116if the second events have an event type for which a quantity of events stored in the main event store116(and/or the replica event store108) satisfies a threshold, and may instruct the event processor110(e.g., the event retrieval component114) to process second events from the replica event store108if the second events have an event type for which a quantity of events stored in the main event store116(and/or the replica event store108) does not satisfy the threshold. This enables more accurate event processing for more important event types, as indicated by the quantity of events of that event type.

In some implementations, the management system126may determine whether the identified event processor110is to process the one or more second events from the replica event store108or the main event store116based on applying a trained machine learning model. For example, the management system126may train a machine learning model based on a set of observations that indicate one or more conditions that triggered a failure, an event type to be failed over, whether events were retrieved from the replica event store108or the main event store116after the failure, and/or a success rate of processing failed-over events. The one or more conditions may include, for example, a condition that caused the management system126to detect the failure and trigger failover, one or more metrics at the time of the failure, whether the failure was triggered based on a heartbeat message or a metric, and/or one or more thresholds associated with the failure (e.g., as described above). The success rate of processing failed-over events may indicate the percentage, ratio, or another indicator of success for a quantity of failed-over events that were successfully processed (e.g., out of a total quantity of failed-over events where processing was attempted, or compared to a quantity of failed-over events that were not successfully processed by the event processor110).

For example, the management system126may train the machine learning model based on a set of historical inputs that indicate one or more conditions, an event type to be failed over, and/or whether events were retrieved from a main event store or a replica event store, as well as a corresponding success rate associated with those historical inputs. Then, when the management system126detects a failure, the management system126may input one or more current conditions (e.g., at the time of detecting the failure) and an event type associated with the failure. The trained machine learning model may use these inputs to predict a first success rate associated with event retrieval of the failed event type from the main event store116and to predict a second success rate associated with event retrieval of the failed event type from the replica event store108. The management system126may compare the success rates, and may select the retrieval option (e.g., the main event store116or the replica event store108) associated with the higher success rate.

As shown by reference number136, the management system136may transmit an instruction, to the identified event processor110(e.g., to an event retrieval component114of the identified event processor110), instructing the identified event processor110to process second events from one of the main event store116or the replica event store108. By transmitting this instruction, the management system136may cause retrieval of one or more second events from one of the replica event store108or the main event store116for processing by the first event processor.

For example, as shown by reference number138, based on the instruction, the identified event processor110(e.g., using the retrieval component114) may retrieve second events from one of the main event store116or the replica event store108. For example, if the instruction indicates to retrieve second events from the main event store116, then the event processor110may retrieve second events, having an event type associated with the event processor110(e.g., which may be indicated by a tag or another indication stored in the main event store116), from the main event store116. As another example, if the instruction indicates to retrieve second events from the replica event store108, then the event processor110may retrieve second events, having an event type associated with the event processor110(e.g., which may be indicated by a tag or another indication stored in the replica event store108), from the replica event store108.

In some implementations, the management system126may determine that the identified event processor110is to attempt to retrieve one or more second events from the main event store116and is then to retrieve the one or more second events from the replica event store108only if attempted retrieval of the one or more second events from the main event store116fails. In some implementations, this may be a default configuration to be followed if there is no stored configuration that specifically indicates that second events are to be retrieved from one of the main event store116or the replica event store108. In this case, the management system126may instruct the identified event processor110to first attempt to access the main event store116and then access the replica event store108if access to the main event store116fails. The event processor110(e.g., using the event retrieval component) may attempt to retrieve one or more second events from the main event store116based on this instruction. If the event processor110determines that this attempt has failed, then the event processor110may retrieve the one or more second events from the replica event store108. This may enable more robust failover than attempting to retrieve events from only one event store.

As shown by reference number140, the identified event processor110may process the one or more second events that have been retrieved. After processing a second event, the event processor110may update a record of the second event stored in the event store from which the second event was retrieved (e.g., one of the replica event store108or the main event store116), which may cause the other event store to be updated due to database synchronization or mirroring. In this way, duplicate processing of events may be avoided (e.g., when the failure of the failed event processor120is resolved).

In some implementations, the system100(e.g., the event processor110that is processing second events or the management system126) may determine that the failure, associated with the corresponding second event processor120, has been resolved. For example, the event processor110and/or the management system126may receive a heartbeat message from the event processor120, may receive a heartbeat message that indicates that the event processor120is in a normal status, and/or may determine that one or more metrics indicate that the failure has been resolved for the event processor120. Based on the determination that the failure has been resolved, the event processor110may refrain from processing any additional second events (e.g., that have not yet been processed) that otherwise would have been processed by the event processor110. In some implementations, if the management system126determines that the failure has been resolved, then the management system126may instruct the identified event processor110to refrain from processing additional second events. In this case, the event processor110may continue to process first events from the main event store106.

The techniques described above may be used for failover in the system100(e.g., an event processing system having an event-driven architecture) to improve service availability, reduce delays associated with event processing, and/or prevent data loss.

As indicated above,FIG.1is provided as an example. Other examples may differ from what is described with regard toFIG.1. The number and arrangement of devices shown inFIG.1are provided as an example. In practice, there may be additional devices, fewer devices, different devices, or differently arranged devices than those shown inFIG.1. Furthermore, two or more devices shown inFIG.1may be implemented within a single device, or a single device shown inFIG.1may be implemented as multiple, distributed devices. Additionally, or alternatively, a set of devices (e.g., one or more devices) shown inFIG.1may perform one or more functions described as being performed by another set of devices shown inFIG.1. For example, a computing instance may perform operations described herein as being performed by the management system126, or vice versa.

FIG.2is a diagram of an example system200associated with event-driven system failover and failback. The system200shown inFIG.2is the same as the system100shown inFIG.1.FIG.2shows an example of the management system126booting up an event processor110to process second events from the second data center104after detecting a failure associated with a corresponding event processor120in the second data center104.FIG.2further shows event retrieval from the replica event store108.

For example, the system200(e.g., the management system126and/or one or more other components of the first data center102) may detect a failure of an event processor120and may identify a corresponding event processor110to process second events that would have otherwise been processed by the failed event processor120(e.g., events having an event type that the failed event processor120is configured to process), as described above in connection withFIG.1.

As shown by reference number202, the management system126may boot up an identified event processor110to process one or more second events (e.g., after identifying the event processor110, as described above in connection withFIG.1). For example, the management system126may boot up the identified event processor110if a configuration associated with the identified event processor110and/or the failed event processor120indicates that the identified event processor110is to be booted up (e.g., rather than selected from a group of event processors110that are already booted up), as described above in connection withFIG.1. Additionally, or alternatively, the management system126may boot up the identified event processor110based on a quantity of events stored in the main event store116that have an event type to be processed by the identified event processor110, as described above in connection withFIG.1. Additionally, or alternatively, the management system126may boot up the identified event processor110based on a sum of quantities of first events and second events having the event type to be processed by the identified event processor110, as described above in connection withFIG.1.

In some implementations, the management system126may determine whether an event processor110, corresponding to the failed event processor120, is already booted up. If the corresponding event processor110is not booted up in the first data center102, then the management system126may boot up that event processor110. The management system126may then instruct the event processor110to process the second events, such as based on a configuration and/or by transmitting a retrieval instruction, as described above in connection withFIG.1.

InFIG.2, the management system126instructs the booted up event processor110(e.g., an event retrieval component114of the booted up event processor110) to retrieve one or more second events from the replica event store108, as shown by reference number204. Based on this instruction, the booted up event processor110retrieves the one or more second events from the replica event store108, as described above in connection withFIG.1. The booted up event processor110may process the one or more second events as described above in connection withFIG.1, and may stop processing events when a failure is resolved, as also described above in connection withFIG.1.

When the management system126boots up an event processor110rather than selecting from an active event processor110that is already booted up and/or is already processing first events, the booted up event processor110and the failed event processor120may be configured for different processing of events based on execution of different code (e.g., rather than identical processing of events based on execution of identical code, as described above in connection withFIG.1). In this case, the booted up event processor110may include additional code that flags one or more second events, processed by the booted up event processor110, in the event store from which the one or more second events are retrieved (e.g., the replica event store108and/or the main event store116). For example, the one or more second events may be flagged in the event store as having been processed by the identified event processor110or in the first data center102rather than by the failed event processor120or in the second data center104. This flagging or marking enables these events to be reviewed.

As indicated above,FIG.2is provided as an example. Other examples may differ from what is described with regard toFIG.2. The number and arrangement of devices shown inFIG.2are provided as an example. In practice, there may be additional devices, fewer devices, different devices, or differently arranged devices than those shown inFIG.2. Furthermore, two or more devices shown inFIG.2may be implemented within a single device, or a single device shown inFIG.2may be implemented as multiple, distributed devices. Additionally, or alternatively, a set of devices (e.g., one or more devices) shown inFIG.2may perform one or more functions described as being performed by another set of devices shown inFIG.2.

FIG.3is a diagram of an example system300associated with event-driven system failover and failback. The system300shown inFIG.3is the same as the system100shown inFIG.1and the system200shown inFIG.2.FIG.3shows an example of the management system126booting up an event processor110to process second events from the second data center104after detecting a failure associated with a corresponding event processor120in the second data center104.FIG.3further shows event retrieval from the main event store116.

For example, the system300(e.g., the management system126and/or one or more other components of the first data center102) may detect a failure of an event processor120and may identify a corresponding event processor110to process second events that would have otherwise been processed by the failed event processor120(e.g., events having an event type that the failed event processor120is configured to process), as described above in connection withFIG.1.

As shown by reference number302, the management system126may boot up an identified event processor110to process one or more second events (e.g., after identifying the event processor110, as described above in connection withFIG.1), as described above in connection withFIG.1andFIG.2. InFIG.3, the management system126instructs the booted up event processor110(e.g., an event retrieval component114of the booted up event processor110) to retrieve one or more second events from the main event store116, as shown by reference number304. Based on this instruction, the booted up event processor110retrieves the one or more second events from the main event store116, as described above in connection withFIG.1. The booted up event processor110may process the one or more second events as described above in connection withFIG.1, and may stop processing events when a failure is resolved, as also described above in connection withFIG.1.

As indicated above,FIG.3is provided as an example. Other examples may differ from what is described with regard toFIG.3. The number and arrangement of devices shown inFIG.3are provided as an example. In practice, there may be additional devices, fewer devices, different devices, or differently arranged devices than those shown inFIG.3. Furthermore, two or more devices shown inFIG.3may be implemented within a single device, or a single device shown inFIG.3may be implemented as multiple, distributed devices. Additionally, or alternatively, a set of devices (e.g., one or more devices) shown inFIG.3may perform one or more functions described as being performed by another set of devices shown inFIG.3.

FIG.4is a diagram of an example environment400in which systems and/or methods described herein may be implemented. As shown inFIG.4, environment400may include an event processing system401, which may include one or more elements of and/or may execute within a cloud computing system402. The cloud computing system402may include one or more elements403-413, as described in more detail below. As further shown inFIG.4, environment400may include a network420, an input device430, and/or an output device440. Devices and/or elements of environment400may interconnect via wired connections and/or wireless connections.

The cloud computing system402includes computing hardware403, a resource management component404, a host operating system (OS)405, and/or one or more virtual computing systems406. The cloud computing system402may execute on, for example, an Amazon Web Services platform, a Microsoft Azure platform, or a Snowflake platform. The resource management component404may perform virtualization (e.g., abstraction) of computing hardware403to create the one or more virtual computing systems406. Using virtualization, the resource management component404enables a single computing device (e.g., a computer or a server) to operate like multiple computing devices, such as by creating multiple isolated virtual computing systems406from computing hardware403of the single computing device. In this way, computing hardware403can operate more efficiently, with lower power consumption, higher reliability, higher availability, higher utilization, greater flexibility, and lower cost than using separate computing devices.

Computing hardware403includes hardware and corresponding resources from one or more computing devices. For example, computing hardware403may include hardware from a single computing device (e.g., a single server) or from multiple computing devices (e.g., multiple servers), such as multiple computing devices in one or more data centers. As shown, computing hardware403may include one or more processors407, one or more memories408, one or more storage components409(e.g., one or more hard disk drives), and/or one or more networking components410. Examples of a processor, a memory, and a networking component (e.g., a communication component) are described elsewhere herein.

The resource management component404includes a virtualization application (e.g., executing on hardware, such as computing hardware403) capable of virtualizing computing hardware403to start, stop, and/or manage one or more virtual computing systems406. For example, the resource management component404may include a hypervisor (e.g., a bare-metal or Type 1 hypervisor, a hosted or Type 2 hypervisor, or another type of hypervisor) or a virtual machine monitor, such as when the virtual computing systems406are virtual machines411. Additionally, or alternatively, the resource management component404may include a container manager, such as when the virtual computing systems406are containers412. In some implementations, the resource management component404executes within and/or in coordination with a host operating system405.

A virtual computing system406includes a virtual environment that enables cloud-based execution of operations and/or processes described herein using computing hardware403. As shown, a virtual computing system406may include a virtual machine411, a container412, or a hybrid environment413that includes a virtual machine and a container, among other examples. A virtual computing system406may execute one or more applications using a file system that includes binary files, software libraries, and/or other resources required to execute applications on a guest operating system (e.g., within the virtual computing system406) or the host operating system405.

Although the event processing system401may include one or more elements403-413of the cloud computing system402, may execute within the cloud computing system402, and/or may be hosted within the cloud computing system402, in some implementations, the event processing system401may not be cloud-based (e.g., may be implemented outside of a cloud computing system) or may be partially cloud-based. For example, the event processing system401may include one or more devices that are not part of the cloud computing system402, such as device500ofFIG.5, which may include a standalone server or another type of computing device. The event processing system401may perform one or more operations and/or processes described in more detail elsewhere herein.

In some implementations, the system100ofFIG.1, the system200ofFIG.2, and/or the system300ofFIG.3may be or may include one or more elements of the event processing system401ofFIG.4. In some implementations, the first data center102may be or may include a first event processing system401, and the second data center104may be or may include a second event processing system401. As another example, the event processors110,120may include computing hardware403, a resource management component404, a host operating system405, and/or a virtual computing system406. Additionally, or alternatively, the main event stores106,116and/or the replica event stores108,118may include memory408and/or storage components409. Additionally, or alternatively, the management system126may include computing hardware403, a resource management component404, and/or a host operating system405.

Network420includes one or more wired and/or wireless networks. For example, network420may include a cellular network, a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a private network, the Internet, and/or a combination of these or other types of networks. The network420enables communication among the devices of environment400.

The input device430includes one or more devices that are an event source, an event producer, or a data source for events to be processed by the event processing system401, the system100, the system200, and/or the system300, and may transmit events (or event notifications) to one or more of these systems. The input device430may include a communication device and/or a computing device. For example, the input device430may include a database, a server, a database server, an application server, a client server, a web server, a host server, a proxy server, a virtual server (e.g., executing on computing hardware), a server in a cloud computing system, a device that includes computing hardware used in a cloud computing environment, or a similar type of device.

The output device440includes one or more devices that are event consumers for events processed by the event processing system401, the system100, the system200, and/or the system300. The output device440may receive instructions to perform one or more actions based on the processing of events by one or more of these systems (e.g., by one or more event processors). The output device440may perform such actions, such as triggering alerts, triggering workflows, or performing some other automated action or processing. The output device440may include a communication device and/or a computing device. For example, the output device440may include a server, such as an application server, a client server, a web server, a database server, a host server, a proxy server, a virtual server (e.g., executing on computing hardware), or a server in a cloud computing system.

The number and arrangement of devices and networks shown inFIG.4are provided as an example. In practice, there may be additional devices and/or networks, fewer devices and/or networks, different devices and/or networks, or differently arranged devices and/or networks than those shown inFIG.4. Furthermore, two or more devices shown inFIG.4may be implemented within a single device, or a single device shown inFIG.4may be implemented as multiple, distributed devices. Additionally, or alternatively, a set of devices (e.g., one or more devices) of environment400may perform one or more functions described as being performed by another set of devices of environment400.

FIG.5is a diagram of example components of a device500, which may correspond to main event store106, replica event store108, event processor110, main event store116, replica event store118, event processor120, management system126, event processing system401, input device430, and/or output device440. In some implementations, main event store106, replica event store108, event processor110, main event store116, replica event store118, event processor120, management system126, event processing system401, input device430, and/or output device440may include one or more devices500and/or one or more components of device500. As shown inFIG.5, device500may include a bus510, a processor520, a memory530, an input component540, an output component550, and a communication component560.

Bus510includes one or more components that enable wired and/or wireless communication among the components of device500. Bus510may couple together two or more components ofFIG.5, such as via operative coupling, communicative coupling, electronic coupling, and/or electric coupling. Processor520includes a central processing unit, a graphics processing unit, a microprocessor, a controller, a microcontroller, a digital signal processor, a field-programmable gate array, an application-specific integrated circuit, and/or another type of processing component. Processor520is implemented in hardware, firmware, or a combination of hardware and software. In some implementations, processor520includes one or more processors capable of being programmed to perform one or more operations or processes described elsewhere herein.

Memory530includes volatile and/or nonvolatile memory. For example, memory530may include random access memory (RAM), read only memory (ROM), a hard disk drive, and/or another type of memory (e.g., a flash memory, a magnetic memory, and/or an optical memory). Memory530may include internal memory (e.g., RAM, ROM, or a hard disk drive) and/or removable memory (e.g., removable via a universal serial bus connection). Memory530may be a non-transitory computer-readable medium. Memory530stores information, instructions, and/or software (e.g., one or more software applications) related to the operation of device500. In some implementations, memory530includes one or more memories that are coupled to one or more processors (e.g., processor520), such as via bus510.

Input component540enables device500to receive input, such as user input and/or sensed input. For example, input component540may include a touch screen, a keyboard, a keypad, a mouse, a button, a microphone, a switch, a sensor, a global positioning system sensor, an accelerometer, a gyroscope, and/or an actuator. Output component550enables device500to provide output, such as via a display, a speaker, and/or a light-emitting diode. Communication component560enables device500to communicate with other devices via a wired connection and/or a wireless connection. For example, communication component560may include a receiver, a transmitter, a transceiver, a modem, a network interface card, and/or an antenna.

Device500may perform one or more operations or processes described herein. For example, a non-transitory computer-readable medium (e.g., memory530) may store a set of instructions (e.g., one or more instructions or code) for execution by processor520. Processor520may execute the set of instructions to perform one or more operations or processes described herein. In some implementations, execution of the set of instructions, by one or more processors520, causes the one or more processors520and/or the device500to perform one or more operations or processes described herein. In some implementations, hardwired circuitry may be used instead of or in combination with the instructions to perform one or more operations or processes described herein. Additionally, or alternatively, processor520may be configured to perform one or more operations or processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software.

The number and arrangement of components shown inFIG.5are provided as an example. Device500may include additional components, fewer components, different components, or differently arranged components than those shown inFIG.5. Additionally, or alternatively, a set of components (e.g., one or more components) of device500may perform one or more functions described as being performed by another set of components of device500.

FIG.6is a flowchart of an example process600associated with event-driven system failover and failback. In some implementations, one or more process blocks ofFIG.6may be performed by a system (e.g., system100, system200, system300, and/or event processing system401). In some implementations, one or more process blocks ofFIG.6may be performed by another device or a group of devices separate from or including the system. Additionally, or alternatively, one or more process blocks ofFIG.6may be performed by one or more components of device500, such as processor520, memory530, input component540, output component550, and/or communication component560.

As shown inFIG.6, process600may include determining that a primary event processor, included in the primary data center, is associated with a failure, wherein the primary event processor is one of a plurality of primary event processors included in the primary data center and configured to process a plurality of first events stored in a main event store of the primary data center (block610). As further shown inFIG.6, process600may include identifying a secondary event processor, in the secondary data center, that is to process one or more first events of the plurality of first events based on the failure, wherein the primary event processor and the secondary event processor are configured to process a same type of event (block620). As further shown inFIG.6, process600may include causing, based on a configuration associated with the primary event processor or the secondary event processor, the one or more first events to be retrieved from one of the main event store or a replica event store for processing by the secondary event processor, wherein the replica event store is included in the secondary data center and is configured to mirror the main event store included in the primary data center (block630).

AlthoughFIG.6shows example blocks of process600, in some implementations, process600may include additional blocks, fewer blocks, different blocks, or differently arranged blocks than those depicted inFIG.6. Additionally, or alternatively, two or more of the blocks of process600may be performed in parallel.

The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications may be made in light of the above disclosure or may be acquired from practice of the implementations.

As used herein, the term “component” is intended to be broadly construed as hardware, firmware, or a combination of hardware and software. It will be apparent that systems and/or methods described herein may be implemented in different forms of hardware, firmware, and/or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and/or methods is not limiting of the implementations. Thus, the operation and behavior of the systems and/or methods are described herein without reference to specific software code—it being understood that software and hardware can be used to implement the systems and/or methods based on the description herein.

As used herein, satisfying a threshold may, depending on the context, refer to a value being greater than the threshold, greater than or equal to the threshold, less than the threshold, less than or equal to the threshold, equal to the threshold, not equal to the threshold, or the like.

Although particular combinations of features are recited in the claims and/or disclosed in the specification, these combinations are not intended to limit the disclosure of various implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and/or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of various implementations includes each dependent claim in combination with every other claim in the claim set. As used herein, a phrase referring to “at least one of” a list of items refers to any combination of those items, including single members. As an example, “at least one of: a, b, or c” is intended to cover a, b, c, a-b, a-c, b-c, and a-b-c, as well as any combination with multiple of the same item.

No element, act, or instruction used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items, and may be used interchangeably with “one or more.” Further, as used herein, the article “the” is intended to include one or more items referenced in connection with the article “the” and may be used interchangeably with “the one or more.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, or a combination of related and unrelated items), and may be used interchangeably with “one or more.” Where only one item is intended, the phrase “only one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based, at least in part, on” unless explicitly stated otherwise. Also, as used herein, the term “or” is intended to be inclusive when used in a series and may be used interchangeably with “and/or,” unless explicitly stated otherwise (e.g., if used in combination with “either” or “only one of”).