QoS - Application of TSN to EtherNet/IP Networks
Jordon Woods
Director, Deterministic Ethernet Technology Group
Analog Devices, Inc.
Steve Zuponcic
Technology Manager
Rockwell Automation
February 22, 2017

AgendaDeterministic Ethernet in Industrial
What is TSN?
An Industrial Use CaseTSN in Control Applications
Effect of TSN features on Control Applications
Network Convergence and Centralized Configuration
Conclusions
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

2

Deterministic Ethernet in Industrial•

For over a decade, Industrial
Automation has used:––––––••

Cut through switching
Time synchronization
Traffic shaping techniques
Scheduling algorithms
Dynamic frame packing
Frame fragmentation

The results are excellent!
But, no single standard exists
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

3

Opportunity•

In general, these protocols are proprietary at layer 2– Specific to 10/100 Mbit (Fast Ethernet)– EtherNet/IP is an exception•The IIoT represents a huge inflection point in the market– Demands for network convergence and more data will force thesestandards bodies to adopt Gigabit Ethernet– These proprietary solutions do not scale easily to Gigabit– The emerging Time-Sensitive Networking (TSN) standards provide amigration path•

ODVA is in an excellent position to leverage this transition into
a competitive advantage

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

4What is TSN?
►A set of 802.1 sub-standards,addressing different needs►Not all sub-standard have to be
implemented►The important sub-standards forIndustrial Automation are:
 802.1AS (REV) Time Synchronization
 802.1Qbv Time Aware Traffic Shaper 802.1Qbu/802.3br Preemption
 802.1CB Seamless Redundancy

 802.1Qci Ingress Policing
 802.1Qcc Network Management
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

5802.1AS (REV) Time Synchronization•

In the context of TSN:
–––––

•

Time sync refers to IEEE 802.1AS and .1AS-REV:
uses a master-slave protocol for time synchronization
does not use “transparent clocks” to compensate for bridge latency
Peer-to-Peer mechanism requires every node be time-aware
Not practical for brownfield

Good news! Other TSN features are independent of time-sync profile– Need to manage various time profiles.

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

6

802.1Qbv Time Traffic Aware Shaper•••

IEEE802.1Qbv introduces time-aware “transmission gates”.
These gates are used to enable separate transmission queues.
The Qbv shaper provides a time-based circular schedule which opens andcloses the transmission gate at specific times

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

7

802.1Qbu/802.3br Preemption•Preemption (also called Interspersing ExpressTraffic):– Allows the switch to stop a transmission in midstream to allow a higher priority packet totransmit.
– Note that only one level of traffic is defined as
preemptive

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

8802.1CB Seamless Redundancy
••
••

•

Purpose: Provide lightweight redundancy forreliable delivery of traffic streams
How? Frame replication and eliminationSend two copies of a message along maximallydisjoint path to ensure delivery
Use of redundant paths minimize packet loss
due to– Link or device failures– Congestion
Discard duplicate frames upon reception

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

9802.1Qci Ingress Policing••••

Purpose: Prevent traffic overload conditions (DDoS, erroneous delivery)
from affecting the receiving node
How? Filtering traffic on a per stream basis by providing an input gate foreach stream
Input gate serves to enforce a "contract" between the talker and listener
Contract functions could be:–––––––

Pass/no-pass"Leaky bucket" policing
Time/bandwidth-based
Threshold counter
Burst sizes
Packet sizes
Misuse of labels, etc.

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

10802.1Qcc - Centralized Configuration•Adds a User Network Interface (UNI) which allows for a centralized network
configuration (CNC) entity.
– CNC performs network calculus, scheduling and other configuration tasks•uses a remote management protocol such as NETCONF or RESTCONF.

– A Centralized User Configuration (CUC) communicates to the CNC via a standard API.
• The CUC may be used to discover end stations, retrieve end station capabilities and user
requirements, and configure TSN features in end stations.

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

11

Industrial Use Case
• A single machine consisting of four
different sections of machinery•Section delivered via different OEMs•

Each section is a subnet with a unique VLAN

• 15 machines per site•

Common IP addressing scheme• 7 manufacturing sites worldwide.
• All sections synchronized andcoordinated to produce final product•Relevant events timestamped•

Data on the manufacturing floor can be correlated
against data in the MES system and data from thesupply chain.

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

12

Industrial Use Case
• The entire manufacturing facility uses
the same understanding of absolute
time and the common notion of “wallclock time.”• But, different 1588 profiles may bepresent:•

Component A may communicate with A’•

Component B may communicate with B’•

Component C produces data consumed by C’components in sections B, C, and D• Time gateway/translation is required forcomponents A and A’, and B and B’.
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

13

Time Gateways• Time bridging, or time gateway, must be
provided in the layer 2 switches• Provides the mechanism for migrating
legacy technologies into a TSN system.
• This functionality has NOT been
identified as a required work item forany standards communities. Individualsuppliers could develop these bridgefunctions as solutions for the market.

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

14

Challenges•

This use case illustrates the need for solutions at layer 2(switching), layer 3 (routing) and for time bridging functions••

How does centralized management deal with such a use case?
••••

Typical for a very wide range of industrial applications

Machine segments are configured and certified by manufacturer
Multiple CNC/CUCs involved
How are the configuration and traffic specification of these segments
integrated at the manufacturing site?

A plant-wide understanding of time becomes problematic••

Technical Track© 2017 ODVA, Inc.

Integration of different time profiles will be necessary (brownfield)
IEEE802.1AS could become the defacto standard for infrastructure devices2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

15

A Simple Control Model• Input data must arrive at Controller before theend of the input interval• Planned data outputs should be transmitted
before the end of the planner interval• Output data must arrive at the drive before theend of the output interval• Of course, this all assumes the drives andcontroller have a common understanding oftime.
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

16

Models vs. Reality
You will never strike oil by drilling through the map!
BUT: this does not, in any way, diminish the value of a map!

(Solomon Golomb: Mathematical models – Uses and limitations. AeronauticalJournal 1968)– Source: Dr Edward Lee (UC Berkley) TSNA’15 – The Internet of Important Things

What’s shown is a simple model intended to illustrate the effectsof TSN on an Ethernet-based control solution
There are many potential points of optimization in a complex, realworld system. Assumptions made herein are for the purpose ofdiscussion, not to suggest design approaches or solutions.

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

17

A Simple Motion Control Model• We’ll focus on a part of theproblem associated with network
performance• Ideally, we’d like all of the drives
to transmit their output data
simultaneously• In this way the link between thecontroller and bridge is optimallyutilizedTechnical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

18

A Simple Motion Control Model• A simple model for control on a “best-effort” network• Assumes all network elements are time-aware• Assumes standard QoS/priority throughout.
• Assume cut-through switch (cut-through latency ~2usec @ 100
Mbs; ~1usec @ 1 Gbs)
– Important for upcoming line topology discussion• Assumes some control of traffic volume and the size ofinterfering traffic on the network

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

19

A Simple Motion Control Model– Max Axis = 1 + {1/3 * Connection Update Period – (Drive Transmission Delay + (m + 1)
* Ethernet Transmission Time + m * Switch Latency + NIC Packet Processing Delay +Bus Interface Delay)}/NIC Packet Processing Delay– (Where m = # of hops)– Drive Transmission Delay: We’ll assume all drives have outputs queued prior to transmission,
so this is contribution is small with respect to other operands, effectively 0 usec

– Assume update packets are fairly small(124 bytes), so Ethernet Transmission Time is(124+20)*80ns/byte = 11.52 usec (at 100 Mbs)
– Switch Latency = (interfering packet size+20)*80ns/byte– NIC Packet Processing Delay – There are techniques to ensure the network is the bottleneck(e.g. 2 cycle processing): 11.5 usec for 100 Mbs, 1.15 for Gigabit..
– Bus Interface Delay: has a lot to do with the overall system architecture. could go effectively to0 (given good bus structure, DMA/ etc.). We’ll assume 0 for this analysis.

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

20

100 Mbs Baseline• Performance strongly influenced byinterfering traffic and thus, thenumber of hops

8

• In practice, control systems will4
engineer the network to limit the sizeof interfering packets (this example
assumes 500 bytes)
2Limit Interfering Traffic
Standard Traffic

10Technical Track© 2017 ODVA, Inc.

5

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

1015
www.odva.org

202521

100 Mbs w/ Preemption••

Preemption offers a standard, unified means tolimit the effects of interfering traffic
8

With a maximum fragment size of 64, themaximum interfering frame size is:•(2*64 bytes)-1 =127 bytes

4
w/ Preemption••

Simplifies the problem of isolating the control
network from interfering traffic
Still need to ensure that other traffic of thesame priority is not present on the wire or thatbandwidth is sufficient to deal with all suchtraffic
Technical Track© 2017 ODVA, Inc.

Limit Interfering Traffic
Standard Traffic

210

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

10
www.odva.org

2030
22

Controller

Scheduled Traffic (Qbv)…

Drives• Utilization of a line topology and scheduled traffic can further minimize effect ofinterfering traffic• Schedule of drives can be individually adjusted to compensate for drive transmission
delay, transmission time and switch latency.
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

23

100 Mbs w/ Scheduled Traffic• Effects due to switch latency areminimized
8• This still assumes a somewhat
isolated network (i.e. there is
not other traffic of the sameclass which might interfere withcontrol packets)
Technical Track© 2017 ODVA, Inc.

4

scheduled trafficHops• The effects of interfering traffic
are of less consideration than
the ability of the controller toprocess incoming packets

w/ Preemption
Limit Interfering Traffic
2Standard Traffic

105

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

1015

20

Axis/mS

www.odva.org

2530

24

1 Gbs Results• Gigabit transmission speeds
further reduce the effects ofinterfering traffic• Note that the benefit fromscheduled traffic across hops ismuch less significant

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

25

1 Gbs Preemption vs Scheduled Traffic• Properly engineered, line
topology limits the effects ofinterfering traffic to a single hop(i.e. control traffic is transmittedin a burst)

1 Gbs

scheduled traffic• With preemption, the effects ofinterfering traffic are minimal
with respect to a 1 mS update 100 Mbs
cyclew/ Preemption

0Technical Track© 2017 ODVA, Inc.

5

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

1015

Max Axis
www.odva.org

2025

3026

1 Gbs Preemption vs Scheduled Traffic• However, as the update
frequency is increased, network
effects again become significant.
250 uS• Note: this example assumes thenetwork, rather than controller
packet processing, is thebottleneck in the system.

scheduled traffic
w/ Preemption
125 uS

020

4060

80

Max Axis
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

27

Wire Speed vs. TSN Functionality:
Distributed Linear Segment Topology
440

Conclusions:(430) (432)420(411)400
380(375)360(365)

Common Configuration• i7 Processor• 2-cycle timing• 1 millisecond motion planner• 1500 byte non-motion packet• 1 Switch• Up to 50 nodes per linear segment

Number of Axis

340
320

300
280
240260(252)

240

220
200180160
140

120
100

Control
Link: 100
Mbit/sDevice Link:
100 Mbit/s

80

Control
Link: 1Gbit/s
Device Link:
100 Mbit/s

Control
Link: 1Gbit/s
Device Link:
1 Gbit/s

Legend:No TSN functions enabled
Preemption enabled
Scheduling enabled• Wire speed contributes to themajority of throughput andperformance on the wire.
––

Number or axes
Jitter• TSN functions allow for better“packing” of data at any given
wire speed• There is little difference inperformance benefit between
preemption and scheduling–

However, both can be used
together.

60
40(35) (37)20(17)

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

28

Network Convergence and Centralized Configuration•Given these results, do we needscheduled traffic?

•

As deployment of Ethernet inIndustrial Automation grows moredata is desired. Solutions must scale.

•As the Enterprise and Automation
networks become more integrated,data flows from various applications
must converge.

•IEEE802.1Qcc provides a practical
approach for achieving this vision
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

29

Summary••••

TSN technologies offer a scalable, predictable approach todeterministic networking.
Because Ethernet/IP products have always relied upon
standardized technologies, ODVA is in an excellent position
to leverage these emerging standards.
However, significant challenges remain.
The integration of various PTP profile and the convergence of
EtherNet/IP traffic with a scheduled TSN network are chief
among these challenges.

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

30

THANK YOU