Switch event ordering

Examples disclosed herein relate to a method comprising detecting a plurality of changes in a database, wherein the database is used to configure a switch operating traffic on a network. The method may include determining that a subset of the plurality of changes are to be deferred before being used to configure the switch, wherein each change in the subset has a potential dependency with at least one other change in the subset. The method may also include iterating through each change in the subset. The iteration may include confirming that a target change has a dependency with another change in the subset, resolving the dependency and transmitting the target change to an object manager for configuration of the switch.

BACKGROUND

A network device, such as a switch, may be configured to operate network traffic in a desired way. Traditionally, a network device, such as a switch, may be configured by a text file or the like. For example, many switches may be configured via Command-Line Interface (CLI).

DETAILED DESCRIPTION

The present disclosure describes aspects of systems and methods for switch event ordering that includes a switch that utilizes a corresponding database to configure and/or operate the switch. The database may store configuration, status, states and statistics of the system. The systems and methods may leverage the database, which may include a specialized time series database, to correlate and analyze a variety of system data, triggering troubleshooting routines when an anomaly is detected and also providing a history of events leading to a root cause. However, because a large number of configuration changes could be made in the database during any given interval, it may be beneficial to manage the order in which the changes made in the database are used to configure the switch.

Aspects of the system and method described herein can properly order changes made to the database through the use of a Sequencer. Sequencer is a stream processing engine (SPE) that takes a stream of network traffic routing programing events with inherent interdependencies, applies speculative and concrete logic to re-sequence the events so that they can be effectively programed into hardware while preserving the interdependencies.

To arrive at a programmable sequence, the sequencer may examine each event and decide whether to submit the event for programing or defer because of possible outstanding dependencies that may be satisfied by other events in the stream. Sequencer may make multiple passes over the events arriving in a stream. The sequencer may start off processing events inline i.e. as they arrive and use speculative logic to arrive at a decision. Once the sequencer has processed each of the events in the stream, it may use concrete logic to make one or more passes analyzing the events until each of the events get submitted for programming.

A method for switch event ordering may include detecting a plurality of changes in a database, wherein the database is used to configure a switch operating traffic on a network. The method may also include determining that a subset of the plurality of changes are to be deferred before being used to configure the switch, wherein each change in the subset has a potential dependency with at least one other change in the subset and iterating through each change in the subset. The iteration may include confirming that a target change has a dependency with another change in the subset, resolving the dependency and transmitting the target change to an object manager for configuration of the switch.

FIG.1is a block diagram of an example system100where switch configuration troubleshooting may be useful. System100may implement a database driven network switch architecture. The system100may include a database140. The database may be internal or external to the switch150. For example, the database may reside in another switch connected to switch150, in a separate device, etc. In some aspects, the database may be stored on a device in a physically different location than the switch150, such as a cloud data center. In aspects where the database140is not part of the switch150, the database140and switch150may be connected by a link, which may be a physical link, such as an Ethernet connection or other physical connection, a wireless connection, a virtual connection, etc.

A network switch, such as switch150, may have routing capabilities such that the switch is responsible for routing data along a route (or equivalently, path) in a network. The switch may perform routing of data based on routing information accessible by the switch. For example, the routing information can be stored on a storage medium of the router, or on a storage medium separate from but accessible by the switch.

A database140, may store some or all of the configuration, status, states and statistics of the network, the switches150and/or other devices on the network at any given point at time. The different state data may be accessed from the databases150either individually (data for a given time, a given device, etc.) or in aggregate (data aggregated for particular items over a given time period, etc.). In other words, the state of the switch may be retrieved from any arbitrary state to any other arbitrary state by manipulating the database. The switch150may respond to these manipulations by reconfiguring itself to and perform any functionality required to achieve that state.

In this manner, the switch150has access to numerous data points of information for devices on the network (including the switch150itself) and the network itself at different points in time. This also provides the switch150with a wide variety of information for monitoring, analysis, troubleshooting, etc.

In addition to discovering and detecting events, anomalies, conditions, etc., errors may be dealt with programmatically. For example, if certain parameters meet certain conditions, certain actions to correct these parameters or other network conditions may be performed and/or a user may be notified to take manual intervention steps. Similarly, the parameters can be monitored over a certain period of time and the parameters can be compared over the period of time and/or compared to historical data. Additionally, a set of repeatable actions can be programmed to occur in certain circumstances based on the monitored parameters and/or other parameters. These automated responses may be programmed via REST, Ansible, as well as other scripting languages that can access data stored in a database.

Switch150may include a processor152and a memory154that may be coupled to each other through a communication link (e.g., a bus). Processor152may include a single or multiple Central Processing Units (CPU) or another suitable hardware processor(s). In some examples, memory154stores machine readable instructions executed by processor152for system150. Memory154may include any suitable combination of volatile and/or non-volatile memory, such as combinations of Random Access Memory (RAM), Read-Only Memory (ROM), flash memory, and/or other suitable memory.

Memory154stores instructions to be executed by processor152including instructions for change detector156, event manager158, sequencer160, event manager162, object manager164and/or other components. According to various implementations, system150may be implemented in hardware and/or a combination of hardware and programming that configures hardware. Furthermore, inFIG.1and other Figures described herein, different numbers of components or entities than depicted may be used.

Processor152may execute change detector156to detect a plurality of changes in a database, wherein the database is used to configure a switch operating traffic on a network. The plurality of changes may occur during a specific temporal period. Accordingly, the change detector156may also create a snapshot of the database at the end of the temporal period which is transmitted to the sequencer160for use in determining deferrals and potential dependencies.

Processor152may execute change manager158to convert each change in the plurality of changes into a data structure that can be used to configure the switch. Processor152may execute sequencer160to perform a first phase of processing through each change in the plurality of changes to determine that a subset of the plurality of changes are to be deferred before being used to configure the switch. Each change in the subset may have a potential dependency with at least one other change in the subset. Each change in the subset of the plurality of changes may be deferred because of an action associated with that change.

The potential dependencies may be based on the snapshot of the database. Sequencer160may transfer each change in the subset into a deferred queue. Sequencer160may transmit each change in the plurality of changes that is not to be deferred to the event manager to be further transmitted to a corresponding object manager for configuration of the switch. The first pass may be a single pass through and thus the sequencer160may determine that a subset of the plurality of changes are to be deferred without knowing each potential dependency in the plurality of changes.

The sequencer160may further iterate through each change in the subset via a second phase of processing through each change in the deferred queue. The iteration may include confirming that a target change has a dependency with another change in the subset, resolving the dependency and transmitting the target change to an object manager for configuration of the switch. The second phase of processing may also be based on the snapshot of the database as well as each other change in the deferred queue. Accordingly, the sequencer may iterate through each change in the subset with knowledge of each potential dependency in the plurality of changes. The sequencer160may determine if there are any deadlocks in the subset. Any change not resolved during the second phase may be considered a deadlock.

The sequencer160may further iterate through the deadlocks during a third phase. The sequencer160may resolve the deadlocks after each other change in the subset has been resolved.

Processor152may execute event manager162to determine, for each resolved change, a corresponding object manager164. Processor152may execute object manager164to program the change into hardware.

As described above, the database140may be used for the configuration of a network switch150. Accordingly, a change may be made to the database140concerning a network setting such as, for example, a route, a destination, a group, etc. These changes may be made automatically by one or more devices on a network, manually by a system administrator, may be caused by a network event (such as a new device connecting to a switch150), etc. Specifically, each change to the database140may be made in the form of an entry in the database140. The database140may be a time series database140, an Open vSwitch Database (OVSDB), a relational database, etc. as well as some combination of the above examples. A change detector156may monitor the database140for any changes. The change detector156may also periodically report changes made during certain time intervals. The change detector156may then transmit a notification of these changes to a change manager158. The change detector156may also take a snapshot of the database140at the end of the time interval and make the snapshot available to other modules in the switch150.

The change manager158may receive the changes from the change detector156and process the changes. During the processing, the change manager158may take changes that are formatted as database140entries and restructure the changes into a data structure that can be used to configure a network switch150. The change manager158may transmit the changes in the updated data structure to the sequencer160.

The sequencer160may receive the changes from the change manager158and enter into a first phase of processing the events. The first phase of the processing may be performed inline i.e. as the changes are received, and determine if any changes are to be deferred. Importantly, the first phase of processing may be a single pass phase, meaning that each change is looked at once. This also means that the sequencer160is making a decision to defer based on incomplete information, as the sequencer160has not seen each change that is received. During the first phase, the sequencer160may determine whether to defer a change based on a variety of factors. One factor may include if there are any dependencies in the change. A dependency is a reference in the change to another object that is not the subject of the change.

For example, a change may include a change to delete a group used by the network switch150. However, there is a potential dependency with that change because the group being deleted may still be used by another object in the configuration of the switch150, such as being in use by a route. Deleting the group while the route is still using the group may create errors and/or other consequences. Accordingly, the sequencer160may decide to defer this change. If there is a change to add/modify to a group, there is a potential dependency that the members of that group may not be ready yet and/or the ID of the member is not available yet. If there is a change to delete a destination, there is a potential dependency that destination may still be in use. If there is a change to add/modify a route, a potential dependency may be that the group or destination is not ready. If there is a change to modify/add a neighbor, a potential dependency maybe that the destination of the neighbor may not be ready. Although a number of examples have been provided, these are for exemplarily purposes only and the above list is not exhausted. A deferral may be made by the sequencer160for any number of reasons including certain actions, references, etc.

Each deferred change identified by the sequencer160may go to a defer queue. After the first phase is completely, the sequencer160may process each change in the deferred queue to resolve any dependencies and/or reorder the changes so that the dependencies are resolved. The second phase may include multiple passthroughs until each change in the deferred queue is resolved or identified as a deadlock. Deadlocks are resolved during a third phase of the sequencer160. Resolved changes are then sent to the event manager162.

An example environment may include a route 1 that leads to a group 1, the group 1 consisting of a destination A and a destination B. A change may be desired to modify route 1 to point to a destination C. This change may be performed using a plurality of configuration changes in the database140including delete group 1, delete destination A, delete destination B, add destination C and modifying route 1. However, if these changes are performed in the order that they are received, it may lead to a variety of errors, as the destination of route 1 is deleted before the new destination is changed. Accordingly, these five actions may be identified during phase 1 of the sequencer160processing as having potential dependencies. During the second phase of the processing, the sequencer160may reorder these five steps to avoid any errors. Accordingly, the sequencer160may put the changes in the following order—(1) add destination C, (2) modifying route 1, (3) delete group 1, (4) delete destination A, (5) delete destination B.

A deadlock may occur when one or more changes cannot be resolved without creating some sort of network disturbance. However, a deadlock may be able to be mitigated to some extent by the sequencer160. For example, a first route may have a destination A with ID1and a second route may have a destination B with ID2. A change may be desired to change route 1 to a destination C with ID2and change route 2 to a destination D with ID1. This change may be made in the database140with the following commands: Delete destination A, delete destination B, add destination C, add destination D, modify route 1 and modify route 2. During the first phase, the sequencer160may flag these changes as potential dependencies. During the second phase, the sequencer160may flag these changes as a deadlock because at some point during the resolution of the change, there will be no destination for route 1. However, the sequencer160may reorder the changes in order to reduce downtime, errors. Etc. Accordingly, the sequencer160may put the changes in the following order—(1) delete route 1, (2) delete route 2, (3) delete destination A, (4) delete destination B, (5) add destination C, (6) add destination D, (7) add route 1 and (8) add route 2.

After each change is resolved, during any of phases 1-3, each change may be sent to the event manager162. The event manager162may process each change and determine what object is involved in the change in order to send the change to the proper object manager164. Objects may include routes, groups, destinations, etc. Once the change is received by the corresponding object manager164, the object manager164determines whether the change is new or it is an update and pushes the change to the switch150hardware.

Referring now toFIGS.2-3, flow diagrams are illustrated in accordance with various examples of the present disclosure. The flow diagrams represent processes that may be utilized in conjunction with various systems and devices as discussed with reference to the preceding figures, such as, for example, system100described in reference toFIG.1, system400described in reference toFIG.4and/or system500described in reference toFIG.5. While illustrated in a particular order, the flow diagrams are not intended to be so limited. Rather, it is expressly contemplated that various processes may occur in different orders and/or simultaneously with other processes than those illustrated. As such, the sequence of operations described in connection withFIGS.2-3are examples and are not intended to be limiting. Additional or fewer operations or combinations of operations may be used or may vary without departing from the scope of the disclosed examples. Thus, the present disclosure merely sets forth possible examples of implementations, and many variations and modifications may be made to the described examples.

FIG.2is a flowchart of an example method200for switch event ordering. For example, method200may be used in an environment similar to system100, described above in reference toFIG.1. A sample database used in conjunction with the method and system ofFIG.1may be an Open vSwitch Database (OVSDB). The databases may have a plurality of rows and unique identifiers for the rows.

Method200may start at block202and continue to block204, where the method200may include detecting a plurality of changes in a database. The database may be used to configure a switch operating traffic on a network. The plurality of changes may occur during a specific temporal period. In some aspects, the method may also include creating a snapshot of the database at the end of the temporal period which is transmitted to the sequencer for use in determining deferrals and potential dependencies

The method may proceed to block206, where the method may include converting each change in the plurality of changes into a data structure that can be used to configure the switch. The method may proceed to block208, where the method may include determining that a subset of the plurality of changes are to be deferred before being used to configure the switch. Each change in the subset may have a potential dependency with at least one other change in the subset. Each change in the subset of the plurality of changes may be deferred because of an action associated with that change. The potential dependencies may be based on the snapshot of the database. Block208may be part of a first phase of processing, may be a single pass through and thus the sequencer160may determine that a subset of the plurality of changes are to be deferred without knowing each potential dependency in the plurality of changes.

At block210, the method may include transferring each change in the subset into a deferred queue and at block212, the method may include transmitting each change in the plurality of changes that is not to be deferred to the event manager to be further transmitted to a corresponding object manager for configuration of the switch. At block214, the method may include iterating through each change in the subset via a second phase of processing through each change in the deferred queue. The iteration may include confirming that a target change has a dependency with another change in the subset, resolving the dependency and transmitting the target change to an object manager for configuration of the switch. The second phase of processing may also be based on the snapshot of the database as well as each other change in the deferred queue. Accordingly, the method may include iterating through each change in the subset with knowledge of each potential dependency in the plurality of changes. Any change not resolved during the second phase may be considered a deadlock. At block216, the method may include determining if there are any deadlocks in the subset and at block218, the method may include resolving the deadlocks. The method may proceed to block220, where the method may end.

FIG.3is a flowchart of an example method300for switch event ordering. For example, method300may be used in an environment similar to system100, described above in reference toFIG.1. A sample database used in conjunction with the method and system ofFIG.1may be an Open vSwitch Database (OVSDB). The databases may have a plurality of rows and unique identifiers for the rows.

Method300may start at block302and continue to block304, where the method300may include detecting a plurality of changes in a database, wherein the database is used to configure a switch operating traffic on a network. At block306, the method may include determining that a subset of the plurality of changes are to be deferred before being used to configure the switch, wherein each change in the subset has a potential dependency with at least one other change in the subset. At block308, the method may include iterating through each change in the subset. At block310, the method may include confirming that a target change has a dependency with another change in the subset and at block312, the method may include resolving the dependency. At block314, the method may include transmitting the target change to an object manager for configuration of the switch. The method may proceed to block316, where the method may end.

FIG.4is a block diagram of an example system400for switch event ordering. In the example illustrated inFIG.5. System400may be similar to system100described above in reference toFIG.1. System400may include a processor402and a memory404that may be coupled to each other through a communication link (e.g., a bus). Processor402may include a single or multiple Central Processing Units (CPU) or another suitable hardware processor(s). In some examples, memory404stores machine readable instructions executed by processor402for system400. Memory404may include any suitable combination of volatile and/or non-volatile memory, such as combinations of Random Access Memory (RAM), Read-Only Memory (ROM), flash memory, and/or other suitable memory.

Memory404stores instructions to be executed by processor402including instructions for change detector406, sequencer408, sequence iterator410and/or other components. According to various implementations, system400may be implemented in hardware and/or a combination of hardware and programming that configures hardware.

Processor402may execute change detector406to detect a plurality of changes in a database, wherein the database is used to configure a switch operating traffic on a network. Processor402may execute sequencer408to determine that a subset of the plurality of changes are to be deferred before being used to configure the switch. Each change in the subset may have a potential dependency with at least one other change in the subset. Processor402may execute a sequence iterator410to iterate through each change in the subset, including: confirming that a target change has a dependency with another change in the subset; resolving the dependency and transmitting the target change to an object manager for configuration of the switch.

FIG.5is a block diagram of an example system500for switch event ordering. In the example illustrated inFIG.5, system500includes a processor502and a machine-readable storage medium504. In some aspects, processor502and machine-readable storage medium504may be part of an Application-specific integrated circuit (ASIC). Although the following descriptions refer to a single processor and a single machine-readable storage medium, the descriptions may also apply to a system with multiple processors and multiple machine-readable storage mediums. In such examples, the instructions may be distributed (e.g., stored) across multiple machine-readable storage mediums and the instructions may be distributed (e.g., executed by) across multiple processors.

Processor502may be at least one central processing unit (CPU), microprocessor, and/or other hardware devices suitable for retrieval and execution of instructions stored in machine-readable storage medium504. In the example illustrated inFIG.5, processor502may fetch, decode, and execute instructions506,508and510for switch event ordering. Processor502may include at least one electronic circuit comprising a number of electronic components for performing the functionality of at least one of the instructions in machine-readable storage medium504. With respect to the executable instruction representations (e.g., boxes) described and shown herein, it should be understood that part or all of the executable instructions and/or electronic circuits included within one box may be included in a different box shown in the figures or in a different box not shown.

Machine-readable storage medium504may be any electronic, magnetic, optical, or other physical storage device that stores executable instructions. Thus, machine-readable storage medium504may be, for example, Random Access Memory (RAM), an Electrically-Erasable Programmable Read-Only Memory (EEPROM), a storage drive, an optical disc, and the like. Machine-readable storage medium504may be disposed within system500, as shown inFIG.5. In this situation, the executable instructions may be “installed” on the system500. Machine-readable storage medium504may be a portable, external or remote storage medium, for example, that allows system500to download the instructions from the portable/external/remote storage medium. In this situation, the executable instructions may be part of an “installation package”. As described herein, machine-readable storage medium504may be encoded with executable instructions for context aware data backup. The machine-readable storage medium may be non-transitory.

Referring toFIG.5, detect instructions506, when executed by a processor (e.g.,502), may cause system500to detect a plurality of changes in a database, wherein the database is used to configure a switch operating traffic on a network. Determine instructions508, when executed by a processor (e.g.,502), may cause system500determine that a subset of the plurality of changes are to be deferred before being used to configure the switch. Each change in the subset may have a potential dependency with at least one other change in the subset. Iterate instructions510, when executed by a processor (e.g.,502), may cause system500to iterate through each change in the subset, wherein the iteration includes confirming that a target change has a dependency with another change in the subset, resolving the dependency and transmitting the target change to an object manager for configuration of the switch.

The foregoing disclosure describes a number of examples for switch event ordering. The disclosed examples may include systems, devices, computer-readable storage media, and methods for switch event ordering. For purposes of explanation, certain examples are described with reference to the components illustrated inFIGS.1-5. The content type of the illustrated components may overlap, however, and may be present in a fewer or greater number of elements and components. Further, all or part of the content type of illustrated elements may co-exist or be distributed among several geographically dispersed locations. Further, the disclosed examples may be implemented in various environments and are not limited to the illustrated examples.

Further, the sequence of operations described in connection withFIGS.1-5are examples and are not intended to be limiting. Additional or fewer operations or combinations of operations may be used or may vary without departing from the scope of the disclosed examples. Furthermore, implementations consistent with the disclosed examples need not perform the sequence of operations in any particular order. Thus, the present disclosure merely sets forth possible examples of implementations, and many variations and modifications may be made to the described examples.