Patent Description:
In both industrial and automotive domains, there is greater desire to use Ethernet technology for the network backbone to support Internet of Things (IoT)/Industrial Internet of Things (IIoT) connectivity to benefit from remote monitoring and management, centralized analytics, and the like. Ethernet standard, however, cannot guarantee that critical data will be delivered across a network from device A to device B in time - a key requirement for both industrial and automotive applications. Time-sensitive networking (TSN) which offers a fully deterministic real time delivery of Ethernet traffic is a solution that can meet the requirements of the industrial and automotive applications.

Document <CIT> relates to bandwidth on demand in SDN networks.

Document <CIT> pertains to enhanced flow entry table cache replacement in a software-defined networking switch.

The article "<NPL>, proposes a new routing algorithm, which is said to calculate bandwidth-delay constrained routes in a fast and efficient manner.

The present invention is embodied by the methods as defined in independent claims <NUM> and <NUM>. Preferred implementations are detailed in the dependent claims.

The present disclosure describes a system and method for processing messages with time-sensitive, time-aware and best-effort characteristic in a software-defined network (SDN) incorporating time management or time domain. Such time-sensitive SDN (TsSDN) can comprise time-sensitive as well as non-time-sensitive network devices. A TsSDN controller, in some embodiments, can manage the network by determining whether a data packet originating from a transmitting device and to be delivered to a receiving device is associated with one of: time-sensitive, time-aware or best effort characteristic. The TsSDN controller can then determine a network path for transport of the data packet from the transmitting device to the receiving device with a guaranteed end to end delay to satisfy the characteristic. The data packet can then be transmitted from the transmitting device via the network path to the receiving device. In some embodiments, the end to end delay for the data packet takes into account latency through each layer the data packet transitions through after it is conjured at an application layer of the transmitting device. Specifically, end to end delay can be determined based device delays encountered at application and stack layers in the transmitting and receiving devices as well as delays in control plane where the TsSDN controller resides and data planes where the network devices that forward data packets reside.

These and other embodiments and aspects of the TsSDN network as well as its advantages will now be described in detail.

Industrial control system network architectures are complex and hierarchical in nature. A typical industrial control system uses a hierarchy of networking levels to separate concerns into a tiered architecture. For example, an ICS network architecture can include a device network, a control network, an enterprise network and an office network. At the lowest level, the architecture is segmented into functional units where each industrial device such as a PLC has a well-defined sphere of influence. This is complex and hard to manage as connectivity between devices and across levels is strictly controlled. This solution requires multiple subnets implemented through separate physical networks or virtual local area networks (VLANs) and gateways which require difficult manual configuration. Implementing and maintaining these networks requires a high degree of networking skill that is often not available at installations.

There are two realms of concern when considering Ethernet switching technology - the control plane and the data plane. The data plane is the traffic flowing through the switch. The control plane is concerned with the management and configuration of the switch to control how the data traffic is directed, shaped, or segmented into various portions of the network. In the traditional switching technologies (described in reference to <FIG> and <FIG>) that are common in industrial automation, each switch manages its own control plane. Therefore, each switch is configured individually and acts somewhat autonomously from its neighboring switches. This is difficult to manage and system wide parameters such as VLANs and quality of service (QoS) must be carefully applied to each switch individually.

SDN presents an alternative to this architecture. SDN is a network architecture in which the system that makes decisions about where traffic is sent (i.e., the control plane) is decoupled from the underlying systems that forward traffic to the selected destination (i.e., the data plane). In simple terms, SDN makes network programmable. With SDN, network administrators can manage network services through abstraction of higher-level functionality.

SDN architecture is a layered architecture based on three basic principles:.

<FIG> is a block diagram illustrating traditional networking compared to SDN. <FIG> is a block diagram illustrating a traditional network device implementation compared to a SDN device implementation.

Traditional networking architecture comprises of dedicated networking devices <NUM> such as, but not limited to: routers, switches, firewalls, and the like, provided by various vendors, with both the control plane <NUM> and the data plane <NUM> on the same device. Moreover, the multi-vendor networking infrastructure warrants that each device is managed individually using vendor proprietary interface, thus making provisioning, maintenance and de-commissioning extremely time consuming and costly. Use of specialized hardware, and at times, customized protocols, warrants that networking features implementation and availability is dictated by vendors. It also follows vendor's business model and product life cycle rather than network deployment needs. For example, in <FIG>, switches 102a, 102b and 102c can each require its own proprietary application and configuration interface 104a, 104b and 104c respectively.

In contrast to traditional networking, SDN is characterized by decoupling of network control and forwarding functions. Network control or intelligence is logically centralized in a SDN controller <NUM>, which enables network administrators to dynamically adjust network-wide traffic flow to meet changing needs. Moreover, even as the software-based SDN controllers maintain a global view of the network, it appears to applications <NUM>, policy engines and/or other entities as a single, logical entity. When implemented through open standards (e.g., OpenFlow), SDN simplifies network design and operation because instructions are provided by an SDN controller <NUM> instead of multiple, vendor-specific devices and protocols. The SDN controller <NUM> directly controls the physical network devices 108a as well as software network devices 108b (e.g., Open vSwitch) hosted on a server or other hardware.

Referring to <FIG>, in a typical networking device <NUM> such as a router or a switch all intelligence is in the device itself. The device is usually implemented in three planes: data plane 110a, control plane 105a and management plane 115a. Data plane 110a is the layer responsible for moving packets, and is usually implemented in provider proprietary hardware with fixed forwarding methodology. Control plane 105a is the layer responsible for configuring forwarding decisions and exchanges of those with other devices. It may be implemented in hardware and/or firmware with vendor specific protocols and features. This type of implementation leads to existence of complex and dedicated networking devices. Management plane <NUM> is the layer that provides management interface, and is usually implemented as software in form of command-line interface (CLI). CLI implementation is vendor specific, and hence difficult to automate in multi-vendor environment. Thus, a traditional networking device <NUM> requires proprietary application and configuration <NUM> for operation.

Contrary to a traditional networking device <NUM>, the main approach in SDN is separation of control plane and data plane and connecting them, usually, with open protocol. This approach allows control plane protocols and requirements to be deployed separately from data plane thus creating an opening for generalization.

In an SDN device implementation <NUM>, control plane can be implemented on general-purpose CPU, thus reducing complexity of networking hardware and removing complex implementation of protocols in firmware. Furthermore, control plane is no longer tied to specific networking device hence consolidation of all devices' control planes is possible. This consolidation is what is known as the SDN controller <NUM>. It is the SDN controller <NUM> that provides centralized network intelligence and enables holistic view of the network. Management plane 115b in an SDN device <NUM> is the SDN application itself. This is the programmable part of SDN and it is targeted to provide freedom of network management and designs specific to users' network needs.

One aspect of "OpenFlow" based SDN is that the data plane 110b operates on flows rather than static lookup tables like MAC table in switches or routing tables in routers. Flows in SDN are best described as pattern matching rules used for packet switching. OpenFlow is vendor-neutral standard and one of the most common protocols used by SDN Controllers to program underlying data plane hardware. This principle of reducing complexity of control protocols to one protocol and enabling flow-based lookups using highspeed memory such as ternary content-addressable memory (TCAM) is what can lead to simplification of infrastructure devices and commoditized hardware.

The purpose of the SDN controller <NUM> is to separate network control from data path and provide an abstraction of network services. As illustrated <FIG>, SDN Controller is a mediator between network application or SDN application <NUM> and underlying network infrastructure comprising network elements <NUM>.

In the control plane 105a, the SDN controller <NUM> implements application-controller plane interface (A-CPI) and exposes north-bound interfaces (NBIs) or APIs <NUM> to users of SDN controller to develop network centric applications without being concerned about network implementation details.

The control plane 105a is the SDN controller <NUM> itself. This plane represents central intelligence of the SDN network. While the SDN controller <NUM> can be physically distributed, it is logically centralized in this plane. Detailed architecture of the controller is implementation specific but in general, SDN controller <NUM> implements Data-controller plane interface (D-CPI) and exposes south-bound APIs <NUM> to communicate with and directly control network elements or devices <NUM> residing in the data plane nob.

The current SDN controller landscape can be divided into open source and proprietary controllers. Open source is targeted mainly at general feature rich SDN controller while proprietary controllers are geared toward specific applications. For interoperability, it is preferable for the SDN controller to be vendor independent. Nonlimiting examples of SDN controllers include OpenDaylight (ODL) which offers many network services and south bound protocols and ONOS which offers high performance.

The control plane is centrally managed by the SDN Controller that doles out configurations to the individual switches or network devices. But for SDN capable switches this configuration can be much more versatile and provides a measure of control over the flows never before possible with conventional switches. This control allows communication flows between two devices to be singled out and directed at will throughout the network. For example, consider two different simultaneous communication flows between two devices, for example HTTP and Modbus TCP. With SDN, the two flows can be directed through very different paths through the network. The Modbus TCP may take a direct path between the two devices while the HTTP may be directed into a cloud platform through a virtual firewall, and then on to the other device. This is not possible with conventional switches as these switches have no visibility into the content of the messages and are relegated to an all or nothing control over the communications between the two devices.

This level of control over the flows obviates the need for the tiered and segmented architecture used today in industrial networks. Accordingly, enabled by the SDN controller, each flow can be individually controlled to implement a simple flat network for use by all devices connected at the same topological level.

One advantage of the granular level of control of the flows is an increased level of security because only sanctioned flows are allowed into the segment. This capability to control the flows can also be exploited for network and device redundancy. For network redundancy, the SDN controller can redirect flows around links that are down or are experiencing some degradation of performance. This same capability could be used to load balance the network links.

In this manner, SDN can provide unprecedented control over flows within a network and thereby enable flat network architectures for use in the ICS. The SDN controller can manage the flows within the network to provide the necessary flow separation that was previously accomplished with complex hierarchical topologies, enabling implementation of simpler and easier to manage flat architectures in the ICS.

Many different SDN controllers have been proposed or are available in the market, and as previously noted, the architecture of an SDN controller is implementation-specific, with different controllers having different sets of components and properties. An SDN controller <NUM> with an example set of components is illustrated in <FIG>. Some SDN controllers include a service layer abstraction (SLA) <NUM> which supports coexistence of several different southbound APIs (e.g., OpenFlow, OVSDB) and protocols (e.g., SNMP, NETCONF). All SDN controllers include some form of SDN control logic <NUM> that utilize network control functionalities provided by components such as a device manager <NUM>, a topology manager <NUM>, a stats manager <NUM>, a notification module <NUM>, a security module <NUM>, and/or the like. The device manager <NUM> can manage devices in the network. For example, the device manager <NUM> can detect when a device is added to or removed from the network. The topology manager <NUM> can detect topology changes in the network and maintain an up-to-date topology of the network. The stats manager <NUM> can collect statistics relating to the monitoring of network functions and devices (e.g., flow statistics generated by the forwarding devices). The notification module <NUM>, for example, can receive, process and take an action with respect to events (e.g., alarm notifications, state changes). The security module can provide security enforcement.

<FIG> is a block diagram illustrating industrial SDN control domain in accordance with some embodiments. SDN can be deployed in an industrial environment to manage the industrial network and devices. In addition to the real physical network devices and real physical industrial devices, the infrastructure layer 110b can also have virtual network devices and virtual industrial devices. Devices depicted in the virtual environment <NUM> are virtualized instances of physical devices including network devices <NUM> such as switches, routers, firewalls, and the like, and other devices such as PCs <NUM>, PLCs <NUM>, and the like. They are connected via virtual links. The network of virtual devices with virtual connections is referred to as virtualized network. Virtual devices and networks reside on physical server nodes in a cloud computing environment <NUM> which provides compute, storage, identity and networking resources. The cloud computing environment <NUM> is part of the real network. This means that real network and virtualized network are interconnected to form the control domain of the SDN controller.

Some infrastructure devices in the industrial domain integrate infrastructure and end device into one device. For example, a PLC that acts as a PLC and as a switch. As such, these types of industrial devices are suitable for direct management by SDN controller. To be managed by the SDN controller, the device should implement at least one south bound interface of a selected SDN controller. One example solution is to have such a device implement the OpenFlow or similar protocol. Implementation of OpenFlow at an industrial device level would bring SDN control to the device level.

Time-sensitive network (TSN) is one of the major set of technologies driving fully deterministic real time communication. TSN achieves deterministic real time communication by using centralized control, time synchronization, and scheduling. In a TSN network, messages or data packets are transmitted based on a time schedule and priority to guarantee a maximum bounded latency and reliability. As such, TSN technology can enable control applications, such as those in industrial and automotive domains to transmit time-critical data over a standard Ethernet infrastructure, obviating the need for specialized protocols to handle real time data.

Infrastructure devices adhere to classical network device architecture in sense that they have:.

Infrastructure can then be characterized based on device network capabilities and purpose. From the network perspective, there are following types of devices:.

Based on the above classification of devices, the entire conversation about infrastructure can actually be transformed to a conversation about data in which infrastructure transports and policies that govern data distribution.

Data (e.g., in industrial domain, in automotive domain) can be described as:.

The above data types can be associated with the following networking Standards:.

While SDN allows unprecedented and granular control of network resources and flows, it has no notion of time. SDN controllers can implement conventional time stamping mechanisms for stamping each message with the time it was received or sent. However, such mechanisms do not have a notion of time in the sense that they cannot interpret or understand the implications of time. For example, an SDN controller/device is not capable of rejecting packets received after midnight, or allowing packets to transition through a network device between <NUM> am and <NUM> am. Similarly, an SDN controller can determine a path through the network infrastructure, but such a path does not consider whether the data to be transmitted is time-sensitive, time-aware or best effort.

In accordance with the present disclosure, this limitation of the SDN technology can be overcome by integrating time domain management into the SDN controller to create a time-sensitive SDN (TsSDN) controller. Example components of a TsSDN controller in accordance with a first embodiment is depicted in FIG. The TsSDN controller 325A includes a TsSDN control logic 322A having a device manager <NUM>, a topology manager <NUM>, a stats manager <NUM>, a notification module <NUM> and a security module <NUM>. These components have similar/same functions as the components of the SDN controller <NUM> of <FIG>. The TsSDN control logic 322A also includes a time management module <NUM> responsible for bringing time awareness. The time management module <NUM> can have a north-bound interface for interfacing with applications and a south-bound interface for interfacing with network devices.

Bringing time awareness to the SDN controller does not necessarily mean full implementation of TSN requirements (i.e., the entire suite of TSN protocols). In some embodiments, time awareness can include the ability to differentiate flows (for Open Flow implementation) or data that are time-sensitive from those that are not. TsSDN controller would then be able to separate time-sensitive paths from general traffic paths and therefore create appropriate overlays to accommodate differentiation of device capabilities with respect to time awareness. In this manner, the TsSDN controller can provide end to end management of ART by using the principles of the TSN technologies including centralized management and global time as well as time awareness. TsSDN would thus be able to guarantee transport of messages across the network comprising TSN capable devices (or TSN devices) with bounded low latency, low packet delay variation and low packet loss. TSN capable devices, as used herein, are devices including network devices supporting or implementing some or all of the TSN protocols. In some embodiments, example methods implemented by the time management module <NUM> are described in reference to <FIG>6D.

A TSN includes TSN devices or entities implementing TSN protocols relating to control and governing of time-sensitive data. Although, the TSN technology is fast evolving in a direction where a special TSN configurator can configure the entire TSN, the technology still relies on specific hardware adhering to specific standard definitions and information exchange among TSN entities using predefined protocols and a sort of exclusive network. The TsSDN controller, on the other hand, takes over the responsibilities of devices' control plane. For example, the TsSDN controller can implement SRP on behalf of all devices. Similarly, OpenFlow as an SDN protocol can be extended to meet the TSN arsenal of protocols. With proper merger of functionality, a TsOpenFlow can be created to configure all aspects of the TSN devices directly from the TsSDN controller.

<FIG> illustrates example components of a TsSDN controller 325B in accordance with a second embodiment. In this embodiment, the SDN controller can be integrated with a TSN application <NUM> by extending the controller (e.g., Open Daylight) to implement a TSN application <NUM> which uses a southbound interface to directly sense and control capabilities of TSN devices. In this case, Open Flow protocol may not be used to implement time domain but rather to configure TSN devices in such way that it preserves TSN established time domain.

<FIG> illustrates example components of a TsSDN controller 325C in accordance with a third embodiment. In this embodiment, a time-sensitive multi-path routing module <NUM> can be provided for centralized calculation of the most efficient paths for routing, thereby providing redundancy and increasing reliability of the overall Ts-SDN controlled network. The module <NUM> can implement any of the link state protocols such as the shortest path bridging protocol (SPB), transparent interconnection of lots of links (TRILL), and the like, and use techniques such as equal cost multi-tree (ECMT), Dijkstra algorithm and/or traffic engineering to determine or calculate the most efficient paths based on time-sensitivity and/or network load. One specific implementation of the time-sensitive multi-path routing module <NUM> can be a time-sensitive SPB module (Ts-SPB module) that implements the shortest path bridging (SPB) protocol. SPB protocol is a standardized protocol specified in the IEEE <NUM>. 1aq standard which creates the shortest path of communication between network entities. In standard networks, it is implemented by each network device in its firmware. In this example, the SPB protocol implementation can be moved to the TsSDN controller 325C (e.g., in Ts-SPB module), and in conjunction with OpenFlow or other control protocols, the TsSDN controller 325C can generate a centralized view of all the shortest paths. In this example, the SPB protocol is augmented to include time and network load as attributes in equal cost multi-tree (ECMT) routing. In accordance with this embodiment, the Ts-SPB module can, using the augmented SPB protocol and the ECMT technique create or determine alternate shortest paths based on time sensitivity or load of the network, or a combination thereof. Congestion information that is indicative of network load can be collected by a SDN load balancer <NUM>. This behavior is illustrated in <FIG> which depicts a number of shortest paths through TSN devices <NUM> and non-TSN devices <NUM> that the Ts-SPB module can determine. In the diagram, the shortest path for transporting packet flows from device A to device B is shown to be through network devices 312A, 308A, 308B, 312F and an alternate shortest path from device A to device B is through network devices 312A, 312B, 308D, 312F. For both paths, no guarantees are provided with respect to time, and as such these paths may not be suitable for time-sensitive packet flows. Nevertheless, packet flows can be distributed between the two paths for load balancing. Also depicted in the same diagram is the shortest TSN path from device A to device B through network devices 312A, 312B, 312C, 312E, 312F. An alternate shortest TSN path can be through network devices 312A, 312B, 312C, 312D, 312E and 312F. Both TSN paths provide guarantee with respect to time, and as such packet flows can be distributed between the two TSN paths for load balancing. When network congestion is taken into account, the shortest TSN path is no longer the same. In the diagram, the shortest path for heavyTSN traffic from device M to device N is through network devices 312C and 312E. This network path provides the shortest path that is time-sensitive. Finally, the shortest path for heavy traffic that is non-time-sensitive from device X to device Y can be through network devices 308A and 308B. This path is the shortest path that is not time-sensitive. In this manner, the TsSDN controller 325C, via the Ts-SPB module and load balancer <NUM> can determine shortest paths bridging that is also time-sensitive and load balanced, bringing all aspects of traffic engineering under the realm of the TsSDN controller 325C.

In some embodiments, a Dijkstra algorithm can be utilized by the time-sensitive multi-path routing module <NUM> to identify the shortest path for TSN traffic and/or non-TSN traffic through the network. The Dijkstra algorithm continuously eliminates longer paths between the source or starting node and all possible destination nodes. Longer paths can have higher cost which is typically attributed to link speed (i.e., cost is inverse of link speed). The algorithm starts out by having confirmed or visited and unconfirmed/unvisited sets. Nodes in the unconfirmed sets have a path to the initial node, but are pending evaluation. Nodes in the confirmed sets have been evaluated. The following steps may be taken:.

The result of this algorithm is the determination of the shortest path from the initial node to all possible destinations. To have the network capable of determining the shortest path, the Dijkstra algorithm has to be implemented on each network device, which adds to the complexity of the network device. Moreover, the algorithm does not take into account time and load based costs. For example, a highspeed path through a network device A may have <NUM>% of its bandwidth reserved, causing the data plane to be fully loaded. This load information can be used to avoid the highspeed path, for example for time-sensitive traffic. A Dijkstra algorithm that has been enhanced to include load and time based costs that overcomes these and other problems of the Dijkstra algorithm will now be discussed.

An example of the enhanced Dijkstra algorithm that can be implemented by the TsSDN controller 325C (e.g., via the time-sensitive multi-path routing module <NUM>) is illustrated in <FIG>. In the diagram, some of the paths are associated with cost values. All unmarked paths are assumed to have a cost of <NUM>, and so these unmarked paths will not be sorted and will not be selected by the modified Dijkstra algorithm. In this example, network paths are evaluated based on a load based cost. In other words, the cost calculation can take into account network load conditions. Other additional factors such as link speed may be included in the cost calculation in some embodiments.

The Dijkstra algorithm can determine that shortest path between Device A and Device B is through network devices 312A, 308A, 308B, 312F because the path cost is <NUM>+<NUM>+<NUM> or <NUM> as opposed to <NUM>+<NUM>+<NUM> or <NUM> through network devices 312A, 312B, 308D, 312F. Now suppose heavy traffic is running between device X and device Y through network devices 308A and 308B. When the TsSDN controller detects such traffic levels (e.g., using OpenFlow diagnostic features), it can dynamically change cost of link between network devices 308A and 308B to a load based number. For example, if the load is <NUM>%, adding <NUM> to the existing cost would result in a new cost of <NUM>. Now when the Dijkstra algorithm is run again, it will yield <NUM>+<NUM>+<NUM> or <NUM> for the path through 312A, 308A, 308B, 312F while the path through 312A, 312B, 308D, 312F will have the cost of <NUM>+<NUM>+<NUM> or <NUM>. In this example, the second path will be the shortest path and hence will be chosen for packet transport.

Time-sensitive data may be treated in a similar manner. In this case, in addition to load based cost, there would be latency based cost. That means the slower the network device, the bigger the latency and thus the cost. For example, referring to <FIG>, let's take only TSN path in consideration. The Dijkstra algorithm can determine the cost of the fast unloaded path through network devices 312A, 312B, 312C, 312E, 312F is <NUM>+<NUM>+<NUM>+<NUM>=<NUM>. The slow unloaded path through network devices 312A, 312B, 312C, 312D, 312E, 312F is <NUM>+<NUM>+<NUM>+<NUM>+<NUM>=<NUM>. Now assuming some additional load on 312C and 312E, the cost of the first path would be updated to <NUM>+<NUM>+<NUM>+<NUM>=<NUM>. In this instance, the second path would be selected for transporting TSN traffic. In this manner, the TsSDN controller which is aware of how many TSN streams go across a single point in the network can adjust cost of links to that device and recalculate the shortest path accordingly.

Moreover, as shown in <FIG>, TSN devices can be easily isolated from non-TSN devices. This is another advantage of the centralized system. This centralization enables Dijkstra algorithms enhanced by load and latency based cost considerations to run on subsets of the network or the entire network.

<FIG> is a block diagram illustrating the network infrastructure under the management of a TsSDN controller. As depicted, the TsSDN controller can manage TSN devices <NUM> which implement the TSN protocols to provide guaranteed transport of messages from one device to another across the network. Existing network infrastructure is likely to have non-TSN devices <NUM> in addition to TSN devices <NUM>, as depicted. In such instances, the SDN devices <NUM> are also managed by the Ts-SDN controllers. The Ts-SDN controller <NUM> can directly program each of these TSN devices <NUM> and SDN devices <NUM> to implement various network policies including how to manage or process (e.g., prioritize, forward/reject) different classes of data.

<FIG> depicts communication between a talker device <NUM> and a listener device <NUM> across a network infrastructure that comprises a mix of TSN devices <NUM> and non-TSN devices <NUM> that exist in the same time domain. In "SDN as underlay" deployment model entire control of networking infrastructure (e.g., <NUM>, <NUM>) from all networking perspectives is managed by the TsSDN controller <NUM>. This means that underlying protocols such as Open Flow and TSN suite of protocols are directly controlled by SDN. Moreover, the TsSDN controller <NUM>, the talker device <NUM>, the listener device <NUM> as well as the network infrastructure implement time synchronization <NUM>. Time synchronization <NUM> is achieved by having the real-time clocks of all devices synchronized to a master clock with great accuracy (e.g., in nanosecond range). This provides a common time base for messages conjured at the talker device and presented at the listener device to have the same relative timing.

Time-sensitive SDN can be viewed as comprising a control plane, a data plane and time domain. The control plane implements the TsSDN controller, while the data plane realizes communication by providing data transport in timely manner. Time Domain enforces time awareness in SDN control domain. The planes are connected with two notions of responsibility:.

In some embodiments, SDN as the ultimate controller of entire network has potentially ultimate control of time domain, either through isolated path selection or thorough deployment and orchestration of TSN. This ultimate control makes it suitable for assurance of truly deterministic data delivery on network level.

Consider an example of a brownfield industrial network where there is mixture of TSN aware devices <NUM> and unaware devices <NUM>. TsSDN controller <NUM> can discover and ensure time a network path for time-sensitive messages through the existing infrastructure. To discover a network path that meets the time sensitivity requirements of messages to be transported, end to end delay for the messages needs to be determined. Typically, network delay calculations account for delay only in the data plane of the network, i.e., from one network device to another. However, in reality, starting from the time a data packet is conjured at the application layer <NUM>, the data packet transitions through several layers before reaching the data plane <NUM>. For example, at a talker device <NUM> (i.e., the source of the data packet or transmitting device), the data packet transitions from an application layer <NUM> through a stack <NUM> which includes the operating system providing a means of communication (e.g., TCP/IP stack, UDP stack, Linux, Windows, etc.) before transitioning to the control plane <NUM> and finally to the data plane <NUM> of the talker device. From thereon, the data packet transitions to the data plane of the network which includes forwarding elements <NUM>, <NUM>, i.e., network devices responsible for transport across the network. The data packet then transitions through the control plane <NUM>, the stack <NUM> and application layer <NUM> of the listener device <NUM> (i.e., receiving device). The delay the data packet encounters at each of these transitions through the layers are accounted for in calculating end to end delay. The TsSDN controller <NUM> can calculate the end to end delay as time synchronization <NUM> enables delay at the talker device level <NUM>, delay at the listener device level <NUM> and delay at the network data plane level to be propagated to the TsSDN controller.

In <FIG>, the TsSDN controller has discovered a network path for transporting a time-sensitive message from the talker device <NUM> to the listener device <NUM> using only the TSN devices <NUM>. The network path <NUM> guarantees that the end to end delay for the time-sensitive message meets the timing requirement of the time-sensitive message or application that conjured the time-sensitive message, and thus achieves end to end deterministic communication.

In SDN/OpenFlow, a flow is a sequence of packets between a source and a destination. <FIG> depicts handling of time-sensitive and non-time-sensitive flows by the Ts-SDN controller (e.g., 325A, 325B, 325C) in an industrial domain. An example PLC device <NUM> can produce two flows including data packets that are to be transported to a diagnostic service <NUM> and an industrial equipment <NUM> respectively. When a data packet associated with a flow to be delivered to the industrial equipment <NUM> is received by a network device, it is first forwarded or reported to the Ts-SDN controller <NUM> if the network device has no rule for forwarding the data packet. The TsSDN controller <NUM> then determines whether the data packet includes time-sensitive or time-aware data. As previously described, time-sensitive data (TSD) is mission-critical data that requires instant and reliable delivery, with ART often less than <NUM>, while time-aware data (TAD) has an ART that is typically in the millisecond range. In some embodiments, whether a data packet has TSD or TAD can be determined based on information received from the source device. For example, when a user qualifies or specifies a traffic as TSD or TAD (e.g., via an industrial control application executing on the PLC <NUM> or via a network application that interfacing with the TsSDN controller <NUM>), each such qualification can be translated to network configuration information which may be in the form of VLAN ID groups, QoS parameters, and the like. In some implementations, the network configuration information indicative of time-sensitivity characteristic may be included in a stream reservation protocol (SRP) talker advertise message. In other words, the TsSDN controller <NUM> can based on any SRP activity (which is a TSN specific protocol) detected from the source device determine that the data packet includes TSD. The TsSDN controller <NUM> can further detect that a data packet from the source device is TAD based on QoS parameters associated with the data packet. All other data packets can be considered best effort. If the data packet is time-sensitive, the Ts-SDN controller <NUM> determines a network path with a maximum latency such that the end to end delay for the data packet is less than the time-sensitivity requirement associated with the data packet. In some implementations, the TsSDN controller <NUM> can obtain time-sensitivity requirement from the industrial control application executing on the PLC <NUM> or via a network application that interfacing with the TsSDN controller <NUM>, e.g., via RESTFULL interface on the northbound side of the controller. The end to end delay, as previously described, can be calculated based on information associated with the source device (i.e., PLC <NUM>) and/or the destination device (i.e., industrial equipment <NUM>) and the network devices. In some embodiments, the intent of network usage including associated delay or latency contributing to the end to end delay, time-sensitivity characteristic of data packet, and the like can be reported by the source and/or destination devices and network devices to the TsSDN controller. The capability of the network devices (e.g., associated latencies, available bandwidth, type of data packets in queue) can be determined by the Ts-SDN controller by direct interrogation. Referring to <FIG>, the network path discovered for transporting time-sensitive flow from PLC <NUM> to the industrial equipment <NUM> is through TSN devices 512a and 512b. The TsSDN controller <NUM> selects this path comprising only time-sensitive devices to guarantee end to end delay in the transport of the data packets associated with the flow meets the strict timing requirement associated with the time-sensitive data.

As for the second flow from PLC <NUM> to the diagnostic service <NUM>, the TsSDN controller <NUM> can similarly determine, based on the content of a data packet associated with the flow, whether the flow is time-sensitive or time-aware. Suppose the flow is determined by the TsSDN controller <NUM> to be neither (i.e., best effort data), then the TsSDN controller <NUM> can determine a network path through only non-TSN devices. As depicted in <FIG>, the network path for the best effort flow is through SDN device 508a and 508c.

<FIG> is a logic flow diagram illustrating an example method of transmitting a time-sensitive data packet through a network infrastructure comprising time-sensitive and non-time-sensitive network devices. The example method comprises at block <NUM>, receiving a data packet originating from a first device and directed to a second device across a time-sensitive SDN. The TsSDN controller based on network configuration information associated with the data packet (e.g., SRP activity, QoS, VLAN ID groups) can determine that the data packet is a time-sensitive data packet that includes time-sensitive data at block <NUM>. At block <NUM>, the TsSDN controller can receive or determine a timing requirement associated with the time-sensitive data packet. At block <NUM>, the TsSDN controller can determine a network path through the network infrastructure using one or more of only the time-sensitive network devices. The network path has a predictable latency that causes end to end delay for the time-sensitive data packet to be less than the timing requirement. In some embodiments, end to end delay for a data packet takes into account latency through each layer the data packet transitions through after it is conjured at an application layer of the first entity. For example, the time-sensitive data packet can transition through an application layer and an operating system stack at each of the first and second devices and through a control plane and a data plane of the TsSDN network associated with the TsSDN controller. In some embodiments, determining the network path can include programming the network devices in the network path so that other data packets that are a part of the same flow can be transported using the same network path. At block <NUM>, the time-sensitive data packet is transmitted from the first device to the second device via the network path.

<FIG> is a logic flow diagram illustrating an example method of transmitting a time-aware data packet through the first device and network infrastructure in <FIG>. In this method, the TsSDN controller receives a data packet to be delivered from the first device to a third device across the network infrastructure at block <NUM>. At block <NUM>, the TsSDN controller based on network configuration information associated with the data packet (e.g., QoS) can determine that the data packet is a time-aware data packet that includes time-aware data. At block <NUM>, the TsSDN controller determines a timing requirement for delivering the time-aware data packet. At block <NUM>, the TsSDN controller determines a second network path through the network infrastructure using one or more network devices selected from the time-sensitive network devices and the non-time-sensitive network devices that comprise the network infrastructure. The second network path has a predictable latency that causes end to end delay for the time-aware data packet to be less than a timing requirement associated with the time-aware data packet. The time-aware data packet is then transmitted from the first device to the third device through the second network path.

<FIG> is a logic flow diagram illustrating an example method of transmitting a best-effort data packet through the first device and network infrastructure in <FIG> and <FIG>. At block <NUM>, the TsSDN controller receives a data packet to be delivered from the first device to a fourth device across the network infrastructure. The TsSDN controller based on network configuration information associated with the data packet determines that the data packet is a best-effort data packet that has no time-sensitivity or time-awareness at block <NUM>. At block <NUM>, the TsSDN controller determines a third network path through the network infrastructure using one or more of only the non-time-sensitive network devices to deliver the best-effort data packet. The best-effort data packet is then transmitted from the first device through the third network path to the fourth device.

<FIG> is a logic flow diagram illustrating an example method of transmitting packet flows through a network path reserved to guarantee a specified timing requirement. In the example method, at block <NUM>, a TsSDN controller (e.g., TsSDN controller 325A via time management module <NUM>) can receive a request specifying a timing requirement for communication between a talker device and a listener device. The request, in some embodiments, can be in the form of a standard network packet sent by the talker device to the TsSDN controller. At block <NUM>, the TsSDN controller can receive device delay associated with the talker device and the listener device. These device delays may be associated with the delays encountered at the device level as data packets transition from an application layer through a stack to a control plane and/or data plane. At block <NUM>, the TsSDN controller receives network load information from network devices in the network. The network load information may be indicative of the available bandwidth of the network device to handle additional packet flows. At block <NUM>, the TsSDN controller determines, based on the specified timing requirement and the network load information a network path that the talker device can use to communicate with the listener device. The network path has a predictable latency that causes end to end delay for the communication from the talker device to the listener device to be less than or equal to the specified timing requirement. At block <NUM>, the TsSDN controller programs one or more network devices in the network path to reserve bandwidth to guarantee the timing requirement for the communication between the talker device and the listener device. At block <NUM>, the TsSDN controller transmits data packets from the talker device to the listener device in a deterministic manner via the network path.

The above described method can be also be used for reverse, i.e., when the listener device wants to send data packets to the talker device with a time guarantee. It should be noted that depending on the timing requirement, the communication may be time-sensitive or time-aware. Similarly, depending on the timing requirement, the network path may include only TSN devices, only non-TSN devices or both TSN and non-TSN devices.

Thus, the role of the TsSDN controller is that of a mediator for TSN services in the infrastructure. A talker device can acquire a stream from the TsSDN controller which in turn secures a path to a listener device by configuring all devices on the path to the listener device. This enables the TsSDN controller to have a broader view of all streams, current and available, to which it can intelligently assign resources. One of the key differences between TsSDN and TSN is that TSN has a peer to peer to governance mandating that all participants understand all TSN protocols in order to guarantee the TSN level of service. In TsSDN, time management is logically centralized and it can eliminate a number of TSN protocols form TSN devices leaving them only with data plane standards. This approach has the advantage of significantly simplifying the management of time as well as the TSN devices themselves.

<FIG> is a block diagram of an exemplary machine/computer/apparatus that may perform various operations, and store various information generated and/or used by such operations in accordance with some embodiments. The computer <NUM> is intended to illustrate a hardware device on which any of the entities, components or services depicted in the examples of <FIG>, (and any other components described in this specification) and methodologies described in the examples of <FIG> can be implemented, such as a server, client devices, compute nodes, controller nodes such as the SDN controller (e.g., <NUM>), TsSDN controller (e.g., <NUM>, 325A, 325B, 325C, <NUM>, <NUM>), storage devices/nodes, databases, industrial devices (e.g., PLCs, PACs), network devices, and the like. The computer <NUM> includes one or more processors <NUM> and memory <NUM> coupled to an interconnect. The interconnect can represent any one or more separate physical buses, point to point connections, or both connected by appropriate bridges, adapters, or controllers.

The processor(s) <NUM> is/are the central processing unit(s) (CPU(s)) of the computer and, thus, control the overall operation of the computer. In certain embodiments, the processor(s) accomplish this by executing software or firmware stored in memory. The processor(s) maybe, or may include, one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application specific integrated circuits (ASICs), programmable logic devices (PLDs), trusted platform modules (TPMs), or the like, or a combination of such devices.

The memory <NUM> is or includes the main memory of the computer. The memory represents any form of random access memory (RAM), read-only memory (ROM), ternary content addressable memory (TCAM), flash memory, or the like, or a combination of such devices. In use, the memory may contain a code. In one embodiment, the code includes a general programming module configured to recognize the general-purpose program received via the computer bus interface, and prepare the general-purpose program for execution at the processor. In another embodiment, the general programming module may be implemented using hardware circuitry such as ASICs, PLDs, or field-programmable gate arrays (FPGAs).

Also connected to the processor(s) through the interconnect are a network adapter <NUM>, a storage device(s) <NUM> and I/O device(s) <NUM>. The network adapter provides the computer with the ability to communicate with remote devices, over a network and may be, for example, an Ethernet adapter or Fibre Channel adapter or wireless radio. The network adapter may also provide the computer with the ability to communicate with other computers within the cluster. In some embodiments, the computer may use more than one network adapter to deal with the communications within and outside of the cluster separately.

The I/O device(s) can include, for example, a keyboard, a mouse or other pointing device, disk drives, printers, a scanner, and other input and/or output devices, including a display device. The display device can include, for example, a cathode ray tube (CRT), liquid crystal display (LCD), or some other applicable known or convenient display device.

The code stored in memory can be implemented as software and/or firmware to program the processor(s) to carry out actions described above. In certain embodiments, such software or firmware may be initially provided to the computer by downloading it from a remote system through the computer (e.g., via network adapter). In some embodiments, memory <NUM> and the storage device(s) <NUM> can be a single entity.

The components introduced herein can be implemented by, for example, programmable circuitry (e.g., one or more microprocessors) programmed with software and/or firmware, or entirely in special-purpose hardwired (non-programmable) circuitry, or in a combination of such forms. Special-purpose hardwired circuitry may be in the form of, for example, one or more ASICs, PLDs, FPGAs, etc..

Software or firmware for use in the SDN/TsSDN system introduced here may be stored on a machine-readable storage medium and may be executed by one or more general-purpose or special-purpose programmable microprocessors. A "machine-readable storage medium", as the term is used herein, includes any mechanism that can store information in a form accessible by a machine.

A computer can also be a server computer, a client computer, a personal computer (PC), a tablet PC, a laptop computer, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a smart phone, a tablet, a phablet, a processor, a telephone, a web appliance, a network router, switch or bridge, a controller (e.g., PLC, PAC), or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.

A machine-accessible storage medium or a storage device(s) includes, for example, recordable/non-recordable media (e.g., ROM; RAM; magnetic disk storage media; optical storage media; flash memory devices; etc.), etc., or any combination thereof. The storage medium typically may be non-transitory or include a non-transitory device. In this context, a non-transitory storage medium may include a device that is tangible, meaning that the device has a concrete physical form, although the device may change its physical state. Thus, for example, non-transitory refers to a device remaining tangible despite this change in state.

The term "logic", as used herein, can include, for example, programmable circuitry programmed with specific software and/or firmware, special-purpose hardwired circuitry, or a combination thereof.

Unless the context clearly requires otherwise, throughout the description and the claims, the words "comprise," "comprising," and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of "including, but not limited to. " As used herein, the terms "connected," "coupled," or any variant thereof, means any connection or coupling, either direct or indirect, between two or more elements; the coupling of connection between the elements can be physical, logical, or a combination thereof. Additionally, the words "herein," "above," "below," and words of similar import, when used in this application, shall refer to this application as a whole and not to any particular portions of this application. Where the context permits, words in the above Detailed Description using the singular or plural number may also include the plural or singular number respectively. The word "or," in reference to a list of two or more items, covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list.

The above detailed description of embodiments of the disclosure is not intended to be exhaustive or to limit the teachings to the precise form disclosed above. For example, while processes or blocks are presented in a given order, alternative embodiments may perform routines having steps, or employ systems having blocks in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and/or modified to provide alternative or sub-combinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed in parallel, or may be performed at different times. Further any specific numbers noted herein are only examples: alternative implementations may employ differing values or ranges.

The teachings of the disclosure provided herein can be applied to other systems, not necessarily the system described above. The elements and acts of the various embodiments described above can be combined to provide further embodiments.

Aspects of the disclosure can be modified, if necessary, to employ the systems, functions, and concepts of the various references described above to provide yet further embodiments of the disclosure.

These and other changes can be made to the disclosure in light of the above Detailed Description. While the above description describes certain embodiments of the disclosure, and describes the best mode contemplated, no matter how detailed the above appears in text, the teachings can be practiced in many ways. Details of the system may vary considerably in its implementation details, while still being encompassed by the subject matter disclosed herein. As noted above, particular terminology used when describing certain features or aspects of the disclosure should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the disclosure with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the disclosure to the specific embodiments disclosed in the specification, unless the above Detailed Description section explicitly defines such terms.

Claim 1:
A method for transmitting a time-sensitive data packet in a time-sensitive software defined network, TsSDN, of an industrial control system, the TsSDN comprising time-sensitive (<NUM>) and non-time-sensitive (<NUM>) network devices, comprising:
receiving (<NUM>) by a TsSDN controller (<NUM>) in the TsSDN a timing requirement for delivering the time-sensitive data packet originating from a first entity (<NUM>) to a second entity (<NUM>) through the TsSDN, the TsSDN controller managing both the time-sensitive (<NUM>) and non-time-sensitive (<NUM>) network devices in the TsSDN;
determining (<NUM>) a first network path (<NUM>) through the TsSDN using one or more of only the time-sensitive network devices (<NUM>), the first network path (<NUM>) having an end to end delay that meets the timing requirement for delivering associated with the time-sensitive data packet; wherein the end to end delay takes into account latency associated with each layer (<NUM>, <NUM>, <NUM>, <NUM>) the data packet transitions through beginning from the application layer (<NUM>) of the first entity (<NUM>); and
transmitting (<NUM>) the time-sensitive data packet from the first entity (<NUM>) through the first network path (<NUM>) to the second entity (<NUM>).