High Availability - Guidelines for Use of DLR in EtherNet/IP Networks
Jordon Woods
Director, Deterministic Ethernet Technology Group
Analog Devices, Inc.
February 22, 2017

Abstract
With the addition of the Device Level Ring (DLR) protocol to the EtherNet/IP specification, there wasconcern that end users might not fully understand the ramifications of the protocol on network
infrastructure, especially with respect to other resiliency protocols. While all of the information
contained herein is contained also in the specification, it is hoped this overview will clarify theseimplications.
This whitepaper is intended to provide an overview of DLR and to provide guidelines for implementinga DLR network comprised of DLR and non-Non-DLR devices. It is intended for those manufacturing orIT professionals responsible for specification, implementation or deployment of networks utilizing
EtherNet/IP technologies.
Making this paper available to the masses of ODVA CIP users, will ease the pains of integrating thistechnology at the end user site.

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org2

Why a DLR Whitepaper?
••••In 2009, ODVA added the DLR Protocol to CIP Toolbox.
Device Level Ring (DLR) protocol provides a means for detecting, managingand recovering from faults in a ring-based network.
Implementation of DLR imposes certain requirements upon the supporting
network infrastructure.
The Infrastructure SIG has been tasked with developing a set of guidelines for
implementing a DLR network comprised of DLR and Non-DLR devices.
– It is intended for those manufacturing or IT professionals responsible for specification,
implementation or deployment of networks utilizing EtherNet/IP technologies.
– Available to all users (i.e. in the public area of the ODVA Website)– published as PUB00316R0(https://www.odva.org/Portals/0/Library/Publications_Numbered/PUB00316R0_Guideli
nes_for_Using_Device_Level_Ring_(DLR)_with_EtherNetIP.pdf).

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

3

DLR Whitepaper TOC

Contents
1.
2.

Introduction ....................................................................................................................................... 5
Overview of DLR ............................................................................................................................... 5
2.1. Ring Supervisor .......................................................................................................................... 5
2.2. Beacon-based Ring Node ............................................................................................................ 5
2.3. Announce-based Ring Node ........................................................................................................ 6
2.4. DLR Operation ........................................................................................................................... 6
2.5. Devices which do not support the DLR protocol. ......................................................................... 62.5.1.
Non-DLR Device .................................................................................................................... 62.5.2.
Non-compliant Device ............................................................................................................ 63. General Considerations....................................................................................................................... 73.1. Simple Stand-Alone DLR Ring ................................................................................................... 7
3.2. DLR Attributes, status indicators and diagnostics ........................................................................ 8
3.3. DLR, High-Performance applications and IEEE-1588 ............................................................... 11
3.4. DLR and QuickConnect. ........................................................................................................... 123.5. Inserting Non-DLR Devices in a DLR Ring .............................................................................. 12
3.5.1.
Requirements for Devices not supporting the DLR Protocol .................................................. 16
3.5.2.
General Configuration Requirements for devices not supporting the DLR protocol ................ 17
3.5.3.
Configuration Requirements for a Ring Network ................................................................... 17
3.6. DLR and Resiliency Protocols .................................................................................................. 19
3.6.1.
A non-DLR loop passing through a DLR ring ........................................................................ 19
3.6.2.
DLR Redundant Gateways .................................................................................................... 20

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

4DLR Overview•

Ring Supervisor––––––•

Ring Node, Beacon-based–––•At least one device must be configured to act as the Ring Supervisor
Sends beacon and announce messages
Prevents messages from circulating
Reconfigures on ring break and restoration
Multiple supervisors supported (algorithm to determine who is active supervisor)
It is strongly recommended that at least one additional device be capable of acting as a
back-up ring-supervisor
Required to process beacon frames within a specified beacon interval (100-400 uS)
Participates in fault detection (Check Neighbor Process, Fault Detection Process)Upon fault detection, reconfigures & relearns the network topology (Ring Recovery Process)Ring Node, Announce-based–––Required to process announce frames (1 second interval)
Participates in fault detection (Check Neighbor Process, Fault Detection Process)Upon fault detection, reconfigures & relearns the network topology (Ring Recovery Process)Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

5DLR Operation• A DLR network consists of anactive Ring Supervisor and any
number of Ring Nodes• Sends beacon and announce
frames on both ports• Ring Supervisor must be
capable of blocking traffic on
one port to avoid loops• Faults are detected when
beacon traffic is interrupted
and/or link/node failure isdetected by adjacent nodes.
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

6

Devices which do not support the DLR protocol• Non-DLR Device– Any device not supporting the DLR protocol but which complies withall of the guidelines outlined in the whitepaper.

• Non-compliant Device– A Non-DLR device that fails to comply with one or more of theguidelines outlined in the whitepaper.

Note: The use of Non-DLR devices is strongly discouraged
throughout the whitepaper. Non-compliant devices areconsidered unsupported.
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

7Devices which do not support the DLR protocol
Required Features
 Disable unicast MAC learning on those ports connected to the DLR ringIEEE 802.3 operation:Auto-negotiation, with 10/100Mbps, full/half duplex
Forced setting of speed/duplex
Auto MDIX (medium dependent interface crossover), in both auto-negotiate and forced speed/duplex
modes. Note: This is a PHY and transformer issue, not an embeddedswitch issue.

QoS:



2 queues
High priority queue for DLR frames, with strict priority scheduling for the high priority queue
Prioritization via 802.1Q/D. Usage shall be consistent with the EtherNet/IP QoS scheme shown inTable 3. For non-IP frames the priority in the 802.1Q header should be used.

Recommended Features
 QoS:


4 queues
Prioritization via DSCP Usage shall be consistent with the EtherNet/IP QoS scheme. For IP frames theswitch should use the DSCP value.

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

8

Devices which do not support the DLR protocol• Configuration Requirements– Quality of Service (QoS)
based upon either the
Differentiated Services CodePoint (DSCP) or the 3-bit
priority field in the VLAN ID

802.1D Priority1

Traffic TypeCIP Priority

PTP Event(IEEE 1588)

n/a59(‘111011’)

7

CIP Traffic Usage(recommended)n/a

PTP General(IEEE 1588)

n/a47(‘101111’)

5n/a

CIP class 0 /1

Urgent (3)55(‘110111’)

6

CIP Motion

Scheduled (2) 47(‘101111’)

5

Safety I/O
I/O

High (1)43(‘101011’)

4

I/O

Low (0)39(‘100111’)

3

No
recommendation
at presentAll35(‘100011’)

3

CIP messaging

CIP UCMMCIP class 3
All otherEtherNet/IPencapsulation messages

DSCP1 Sending 802.1Q tagged frames is disabled by default

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

9Devices which do not support the DLR protocol• Configuration Requirements– Ports connected directly to the ring must be configured to preserve IEEE 802.1Q
tag priority of ring protocol frames.
– Disable IP multicast filtering on the two ports of the switch connected to ring. This
step will assure uninterrupted delivery of EtherNet/IP multicast connection data
after a ring reconfiguration.
– Statically configure the three multicast addresses used by ring protocol to beforwarded only on two ports of switch connected to ring. This step must be done
to prevent multicast ring protocol frames from being forwarded on other ports ofswitch
Message Type
BeaconNeighbor_Check_Request/Response/Sign_on
Announce/Locate_Fault messages
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

MAC Address
01:21:6C:00:00:01
01:21:6C:00:00:02
01:21:6C:00:00:03
www.odva.org

10

DLR and 1588•••

The DLR protocol does not require
support for IEEE-1588.
High performance applications canbenefit from both the timesynchronization of IEEE-1588 and thefast fault detection and ring recovery ofDLR .
It is recommended that all nodes
participating in such a high-performancering support IEEE-1588 and the CIP
time sync object.
– Note: the device itself need not be timeaware (i.e. it need not process IEEE1588 messages locally), it should include
support for end-to-end (E2E) transparent
clock..
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

11

DLR and QuickConnect•••••

QuickConnect is an ODVA technology intended
to address specific applications in Automotive
Manufacturing.
Applications such as robots, tool changers andframers are required to quickly exchange tooling
fixtures which contain a section or segment of anindustrial network.
This requires the network and nodes to becapable of quickly connecting and disconnecting,both mechanically, and logically.
Consequently, when in QuickConnect mode, aport configured for forced speed and duplex
mode, QuickConnect devices do not use AutoMDIX (detection of the required cable connectiontype).
Because Auto-MDIX is disabled for
QuickConnect applications, extra care must be
taken to ensure cabling connections are correct.
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

12General Considerations• DLR is generally intended for a simple, single-ring topology
requiring fast recovery from network failures– Does not support the concept of multiple or overlapping rings• While a DLR ring can contain an arbitrary number of nodes,
recommended ring size is less than 50 nodes– As the number of nodes grows, the time required for DLR frames totraverse the ring increases, leading to increased fault detection andrecovery time– in a larger ring the probability of a fault increases, including double
faults in which a segment may be lost from the rest of the network– Ultimately the number of nodes a user selects for a given ringdepends upon the performance requirements the user has set forthat ringTechnical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

13

Example Ring Configuration Parameters and Performance

Number ofRing Nodes

2550(nominalnetworksize)
100
150
200
250

1.

BeaconInterval(usecs)

400
400

Round Trip
Time1 (usecs)

905
1810

BeaconTimeout(usecs)

1380
1960

400
3620
3120
400
5430
4280
400
7240
5440
400
9050
6600
Same for Beacon and Announce frames based nodes.

Technical Track© 2017 ODVA, Inc.

Physical Layer
Faults Recovery
Delay 1 (usecs)

Non-physicalLayer Faults
Recovery Delayfor Beacon
Frame BasedNodes (usecs)

Non-physicalLayer Faults
Recovery Delayfor AnnounceFrame BasedNodes (usecs)

Ring Restore
Delay for
Beacon frame
Based Nodes(usecs)Ring Restore
Delay forAnnounce frame
Based Nodes(usecs)

980
1885

1858
2890

2335
3820

1808
3165

2260
4070

3695
5505
7315
9125

4955
7020
9085
11150

6790
9760
12730
15700

5880
8595
11310
14025

7690
11310
14930
18550

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

14

DLR Attributes, status indicators and diagnostics• DLR object attributes can provide helpful information when afault occurs– Most attributes are available only at the Ring Supervisor– Many of the attributes depend upon a beacon frame traversing thenetwork to gather the attribute’s data (e.g. participant’s list)• While attribute 1, Network Topology, refers to both linear andring topologies, this does not imply that DLR supports a linetopology– Attribute 1 is a status bit indicating:• Supervisor has blocked traffic in one direction on the ring and the ring isin its RING_NORMAL_STATE (linear topology) or,• the ring is either in startup or a faulted state (ring topology)
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

15

DLR Attributes

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

16

DLR Attributes

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

17

Inserting Non-DLR Devices in a DLR Ring
•Non-DLR Devices have no adverse
impact on worst case fault detection
time.
– Supervisor will still detect a lack ofbeacon frames and initiate ring
recovery•

Including multiple Non-DLR devices ina ring can make isolating anddiagnosing a fault problematic if thefault occurs between two non-DLRnodes.

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

18

Inserting Non-DLR Devices in a DLR Ring•

Consider a simple ring consisting of:–––•Assume that a link failure occurs between N1
and D4–•

identify D2 as the “Last Active node on port 1” andD4 as the “Last Active node on port 2”.

Querying the link status will show:–•Supervisor will still detect a lack of beacon frames
and initiate ring recovery

The fault detection process will:––•

A supervisor (S1),four DLR nodes (D1 through D4) anda single Non-DLR device (N1).

from D4’s perspective, the link with D2 is down,from D2’s perspective the link is up

One can reasonably deduce that the fault lies
either with D4’s lower port, N1’s right port or thecabling between these nodes.

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

19Inserting Non-DLR Devices in a DLR Ring••Now consider that Non-DLR devices N2 andN3 have been added to our simple ring
Assume that a link failure occurs between N2and N3–•

The fault detection process will again:––•

identify D2 as the “Last Active node on port1” andD4 as the “Last Active node on port 2”.

Querying the link status will show:–•Supervisor will still detect a lack of beacon
frames and initiate ring recovery

from D4’s perspective, the link with D2 isdown, from D2’s perspective the link is upHowever, the fault might exist anywhere
between D2 and D4.
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

20

Inserting Non-DLR Devices in a DLR Ring••

Now consider that Non-DLR devices N2 andN3 have been added to our simple ring
Assume that a link failure occurs between N2and N3–•The fault detection process will again:––•

identify D2 as the “Last Active node on port1” andD4 as the “Last Active node on port 2”.

Querying the link status will show:–•Supervisor will still detect a lack of beacon
frames and initiate ring recovery

from D4’s perspective, the link with D2 isdown, from D2’s perspective the link is up

However, the fault might exist anywhere
between D2 and D4.
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

21

Inserting Non-DLR Devices in a DLR Ring
•If multiple Non-DLR devices are to beinserted in the ring, it is preferable toinsert DLR nodes between thesedevices for better isolation of faults .

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

22

DLR and Resiliency Protocols•••••

DLR rings may be connected to networks
supporting IEEE Spanning Tree protocols(RSTP, MSTP).
Care must be taken to ensure that theseprotocols do not interfere with DLR .
Spanning tree protocols use special control
frames called Bridge Protocol Data Units(BPDUs) to exchange information about
network connections and their cost.
The active Ring Supervisor will not forward
multicast messages with the address
01:80:C2:00:00:00 (BPDU frames) from one
ring port to the other, regardless of ring state.
This feature ensures that the spanning tree
protocol does not recognize and attempt tomanage the DLR ring.
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

23

A non-DLR loop passing through a DLR ring•

Avoid multiple connections into a DLR
ring:– switches S2 and S3 each connect tothe DLR ring through a DLR tap– the Ring Supervisor will block BPDUs
packets sent by each switch– the RSTP algorithm will be unable toresolve the network structure
potentially resulting in unaddressed
nodes, unmanaged rings and network
storms•DO NOT IMPLEMENT!
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

24

DLR Redundant Gateways•DLR supports multiple connections to a DLR
network through redundant gateway devices:–––––––•

Two DLR ports to connect to the DLR network
One or more uplink ports to connect to the network
Implements DLR protocol on its two DLR ports
Implements RSTP or MSTP on its uplink ports(other options possible)
Only one gateway is active (selection and switch
over is automatic)
The backup gateway only forwards DLR traffic
between its two DLR ports and forwards uplink port
traffic only between its uplink ports
This prevents the creation of an unmanaged ring

It is strongly recommended that multiple
connections to the network infrastructure
outside of the DLR network be made via thesegateway devices
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

25Conclusions••

Some of the requirements which DLR imposes upon the network may not be
obvious to those unfamiliar with .
The DLR Whitepaper “Guidelines for Using Device Level Ring (DLR) with
EtherNet/IP” is:– a resource for end-users and those responsible for specification, implementation or
deployment of networks utilizing EtherNet/IP technologies and– an educational tool for manufacturers of EtherNet/IP equipment– available to all users (i.e. in the public area of the ODVA Website)– published as PUB00316R0(https://www.odva.org/Portals/0/Library/Publications_Numbered/PUB00316R0_Guideli
nes_for_Using_Device_Level_Ring_(DLR)_with_EtherNetIP.pdf).

•My thanks to Adrienne Meyer, the TRB and especially the membersof the Infrastructure SIG for their support of this effort!

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

26

References
1.

2.
3.
4.

5.

6.

7.

IEEE Std 802.3 ¨C 2005, Part 3: Carrier sense multiple access withcollision detection (CSMA/CD) access method and physical layer
specifications.
IEEE Std 802.1D ¨C 2004, Media Access Control (MAC) Bridges.
IEEE Std 802.1Q ¨C 2003, Virtual Bridged Local Area Networks.
IEEE Std 802.1s ¨C 2002, Virtual Bridged Local Area Networks ¨CAmendment 3: Multiple Spanning Trees.
The CIP Networks Library, Volume 1, Common Industrial Protocol (CIP),
Edition 3.5, December 2008.
The CIP Networks Library, Volume 2, EtherNet/IP Adaptation of CIP,
Edition 1.6, December 2008.
Moldovansky, A. et. al., Introduction to Device Level Ring. Proc. of ODVA
2009, Howey in the Hills, FL.

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org27THANK YOU