High Availability in EtherNet/IP Systems
Using Frame Replication and Elimination for
Reliability (FRER)
as defined in the
TSN Standard IEEE 802.1CB-2017
George A. Ditzel III
Ethernet Architect
Schneider ElectricPresented at the ODVA2018 Industry Conference & 18th Annual Meeting
October 10, 2018
Stone Mountain, Georgia, USA
Abstract
This paper presents an overview of providing high availability in EtherNet/IP™ systems using the Time
Sensitive Networks (TSN) feature Frame Replication and Elimination for Reliability (FRER) defined in theIEEE 802.1CB-2017 standard.
Critical system applications are often required to maintain high availability for all transmission, generation,
distribution, and communication network components. For critical infrastructures and time sensitive
processes, downtime is never allowed.
These systems may incorporate redundancy to cope with points of failure in their infrastructure. The key
performance factor of redundancy is the recovery time. Recovery time is the time needed to restore
normal operation after a disruption. A key characteristic of recovery time is its determinism, i.e., thecertainty that the desired recovery time is met.
The time required for a dynamic network control protocol to recover from an equipment failure may beunacceptable in certain applications. Stringent system applications require zero recovery (switchover)time.
FRER provides zero recovery time using frame replication with multiple paths called compound streams.
This paper will present an overview of using FRER in industrial automation, utility infrastructures, andtunnel applications that demand uninterrupted operation.

2018 ODVA Industry Conference

1©2018 ODVA, Inc.

Introduction
This paper is an introduction to developing high availability for networks in EtherNet/IP systems using the
Frame Replication and Elimination for Reliability (FRER) protocol for demanding high availability
applications in industrial scenarios. The FRER protocol is defined in the IEEE 802.1CB-2017 standard.
This paper will provide:






A discussion of high availability
Introduction to the FRER High Availability solution
Applications of the FRER protocol
The fault tolerance features of a FRER network
The installation and operation of a FRER protocol in an EtherNet/IP system

High Availability
High availability is based on the concept of availability. The availability of a network is the probability (inpercent) that the network is in service and available for use at any instant in time.
High availability is represented as a percentage, usually referred to as the 9s. If the availability metric is
specified as five nines, it is understood to mean that the network should be functional for 99.999% of thedesired duty cycle (24-hours/day).
Availability is expressed using the following measures of reliability.
Availability = MTTF / (MTTF+MTTR)

(1)

where
MTTF is the mean time to failure; a measure of the reliability of a network, otherwise known is its
failure rate. The MTTF is the interval in which the network or element can provide service without
failure.
MTTR is the mean time to repair; a measure of reliability that represents the time it takes toresume service after a failure has been experienced.
As equation (1) shows, the availability of a network can be increased by designing network elements thatare highly reliable (high MTTF), and/or by reducing the time required to repair the network and return it toservice (low MTTR).
Since it is impossible to create networks that never fail, the key to high availability is to make recovery
time as brief as possible. Availability is increased in networks by introducing redundancy.

2018 ODVA Industry Conference

2©2018 ODVA, Inc.

Redundancy
High availability can be achieved economically by using techniques that detect points of failure and avoid
service interruptions through redundancy in the system. There are two forms of redundancy, dynamic andstatic.
In dynamic (standby) redundancy the replicated components activate after a failure has been detected.
Dynamic redundancy does not actively participate in the control. Switchover logic determines when to
insert and activate redundancy. This requires detection and/or recovery as shown in Figure 1.

Figure 1

In static (parallel) redundancy the replicated components are active concurrently. Static redundancy
usually participates in the control. No special processing is needed on errors. This provides bumpless (0ms) switchover, with continuously exercised redundancy and increased point-of-failure detection with failsafe behavior. Static redundancy is provided at the cost of duplication, as shown in Figure 2.

Figure 22018 ODVA Industry Conference

3©2018 ODVA, Inc.

The FRER Solution
The Frame Replication and Elimination for Reliability (FRER) is a static (parallel) redundancy highavailability capability as defined the the IEEE 802.1CB-2017 standard. This solution will be illustrated bytransforming the following EtherNet/IP Network (Figure 3) into an FRER supported network.

Figure 3: Example EtherNet/IP Network

FRER provides increased reliability (reduced packet loss rates) for a Stream by using a sequencenumbering scheme, and by replicating every stream packet in the source. FRER also eliminates those
replicated stream packets in the destination. FRER provides the following:
1. Packet replication: sending replicated frames on separate paths, and then using inserted
identification information to eliminate replicates, reducing the probability of frame loss.
2. Multicast or unicast: A path on which a Stream is sent can be a point-to-point path or a point-tomultipoint tree.
3. Latent error detection: some means of detecting a failure to deliver copies of each packet isprovided at the point that the replicated packets are discarded.
4. Interoperability: a small number of controls are provided that that make interoperation with otherstandards possible.
5. Backward compatibility: To provide the ability to be connected to a network that is not aware of
FRER, and for a network of conformant relay systems to offer these benefits to unaware end
systems.
6. Zero congestion loss: provide a Stream with zero (or very low) packet loss due to congestion.

2018 ODVA Industry Conference

4©2018 ODVA, Inc.

The FRER supported network consists of several device types through the combination of defined FRER
component and equipment elements (illustrated in Figure 4).

Figure 4: FRER Described Device Types

The FRER protocol provides increased reliability (reduced packet loss rates) for a Stream by using asequence numbering scheme, and by replicating every stream packet in the source end system and/or inrelay systems in the network. FRER also eliminates those replicates in the destination end system and/or
in other relay systems. The devices types described in the standard are:End Systems (ES): End Systems may contain a Talker component, a Listener component, or both.
The Talker component is the originator of single or redundant streams. When generating
redundant streams, the Talker component performs the following functions: Supplies sequencing information in frames; Replicates each frame, assigning each replicate a different stream handle, at most one ofwhich can be the same as the original passes unchanged;
 Encodes the sequencing information into the frame in a manner such that it can be
decoded by its peer.
The Listener component consume single or redundant streams. When receiving redundant
streams, the Listener component performs the following functions: Extracts and decodes the sequencing information from a received frame.
 Examines this sequencing information in received frames packets and discards frames
indicated to be a duplicate of a frame previously received; Also monitors the variables todetect latent errors of Streams;
End Systems are represented in Figure 4 by: The Sngl-Port Controller; Sngl-Port Drive(s); one
Multi-Port Sensor; IP Camera; Camera Recorder; and Monitor.

2018 ODVA Industry Conference

5©2018 ODVA, Inc.

Relay Systems (RS): Relay Systems will either transfer packets belonging to redundant streams, or
act as a proxy Talker or Listener for End Systems not capable of handling redundant streams. As a
Proxy the Relay system performs the following functions:
The Talker proxy will generate redundant streams from single streams:
 Supplies sequencing information in frames; Replicates each frame passed to it, assigning each replicate a different stream handle, at
most one of which can be the same as the original passes unchanged;
 Encodes the sequencing information into the frame in a manner such that it can be
decoded by its peer.
When the Listener proxy receives redundant streams:
 Extracts and decodes the sequencing information from a received frame.
 Examines this sequencing information in received frames packets and discards frames
indicated to be a duplicate of a frame previously received and forwarded; Also monitors
the variables to detect latent errors of Streams;
Relay Systems are represented in Figure 4 by the Ethernet Switch(es).
Relay-End Systems (RES): Relay-End Systems are not defined in the IEEE Standard but areelements within the EtherNet/IP Network. The Rely-End System is created by combining the FRER
End system and Relay System capabilities. Relay-End Systems are represented in Figure 4 by theMulti-Port IO Device(s) and one Multi-Port Sensor.
Additional network interconnections needed within the network are representing in figure 4 by thedashed lines. These interconnections change an existing network into a Mesh network, which will be
explained later in this paper.

2018 ODVA Industry Conference

6©2018 ODVA, Inc.

FRER in a Ring Topology
This section describes the use of FRER in the Ring network topology (illustrated in Figure 5) portion of
the EtherNet/IP Network example. The FRER protocol is defined to interoperate with any of the Network
Control Protocols defined in the IEEE 802.1Q-2018 Standard (i.e. RSTP, MSTP, SBB, etc.). The FRER
protocol is described to also work with the IEC 62439-3 (PRP and HSR) network control protocols,
illustrated later in this paper.

Figure 5: Single Talker/Listener Ring Network Example

In this scenario the Multi-Port IO Devices is a Relay-End System (RES) acting as a Listener/Talker thatreceives/transmits redundant streams to the Sngl-Port Controller, which is and End System (ES).
The Multi-Port IO Device (RES), acting as a Talker generating a redundant stream to the One-Port
Controller (ES), performs the following FRER functions: Supplies sequencing information in frames; Replicates each frame, assigning each replicate a different stream handle, at most one ofwhich can be the same as the original passes unchanged;
 Encodes the sequencing information into the frame in a manner such that it can be decodedby its peer.
The Multi-Port IO Device (RES), acting as a Listener receiving a redundant stream from the One-Port
Controller (ES), performs the following FRER functions: Extracts and decodes the sequencing information from a received frame.
 Examines this sequencing information in received frames packets and discards frames
indicated to be a duplicate of a frame previously received and forwarded; Also monitors
the variables to detect latent errors of Streams;

2018 ODVA Industry Conference

7©2018 ODVA, Inc.

The other Multi-Port IO Devices (Relay-End Systems) and Ethernet switch (Relay System) are acting as
relay systems which transfer frames belonging to redundant streams.
One Ethernet Switch (RS), acting as a relay system for the Sngl-Port Controller, performs the following
FRER functions:
The Talker proxy will generate redundant streams from single streams:
 Supplies sequencing information in frames; Replicates each frame passed to it, assigning each replicate a different stream handle, at
most one of which can be the same as the original passes unchanged;
 Encodes the sequencing information into the frame in a manner such that it can be
decoded by its peer.
When the Listener proxy receives redundant streams:
 Extracts and decodes the sequencing information from a received frame.
 Examines this sequencing information in received frames packets and discards frames
indicated to be a duplicate of a frame previously received and forwarded; Also monitors
the variables to detect latent errors of Streams;

2018 ODVA Industry Conference

8©2018 ODVA, Inc.

FRER in a Mesh Topology
This section describes the use of FRER in the Mesh network topology portion of the EtherNet/IP Network
example. The Mesh network is supported any of the Network Control Protocols defined in the IEEE
802.1Q-2018 Standard (i.e. RSTP, MSTP, SBB, etc.). The network recovery time of these control
protocols is irrelevant due to the seamless redundancy nature of the FRER protocol.

Figure 6: Single Talker/Multiple Listeners Mesh Network Example

In this scenario the IP Camera is an End System (RES) acting as a Talker that transmits a stream to both
the Camera Recorder and the Monitor, both are End Systems (ES).
The Ethernet Switch is a Relay System (RS) connected to the Monitor is acting as a translator from thesingle stream to a redundant stream for the IP Camera End System (ES):The Talker proxy will generate redundant streams from single stream: Supplies sequencing information in frames; Replicates each frame passed to it, assigning each replicate a different stream handle, at
most one of which can be the same as the original passes unchanged;
 Encodes the sequencing information into the frame in a manner such that it can be
decoded by its peer.
The Multi-Port IO Devices (Relay-End Systems) and Ethernet switch (Relay System) are acting as relay
systems which transfer frames belonging to redundant streams.
Two Ethernet Switches (ES) are connected to the Monitor and Camera Recorder are acting as Listeners
for the Redundant stream from the IP Camera:
When the Listener proxy receives redundant streams:
 Extracts and decodes the sequencing information from a received frame.
 Examines this sequencing information in received frames packets and discards frames
indicated to be a duplicate of a frame previously received and forwarded; Also monitors
the variables to detect latent errors of Streams;

2018 ODVA Industry Conference

9©2018 ODVA, Inc.

In this scenario the Multi-Port Sensor is an End System (ES) acting as a Listener/Talker thatreceives/transmits redundant streams to the Sngl-Port Controller, which is and End System (ES).

Figure 7: Single Talker/Listener Mesh Network Example

The Multi-Port Sensor (ES), acting as a Talker generating a redundant stream to the One-Port
Controller (ES), performs the following FRER functions: Supplies sequencing information in frames; Replicates each frame, assigning each replicate a different stream handle, at most one ofwhich can be the same as the original passes unchanged;
 Encodes the sequencing information into the frame in a manner such that it can be decodedby its peer.
The Multi-Port Sensor (ES), acting as a Listener receiving a redundant stream from the One-Port
Controller (ES), performs the following FRER functions: Extracts and decodes the sequencing information from a received frame.
 Examines this sequencing information in received frames packets and discards frames
indicated to be a duplicate of a frame previously received and forwarded; Also monitors
the variables to detect latent errors of Streams;

2018 ODVA Industry Conference

10©2018 ODVA, Inc.

The Multi-Port Sensor (Relay-End System) and Ethernet switches (Relay System) are acting as relay
systems which transfer frames belonging to redundant streams.
One Ethernet Switch (RS), acting as a relay system for the Sngl-Port Controller, performs the following
FRER functions:
The Talker proxy will generate redundant streams from single streams:
 Supplies sequencing information in frames; Replicates each frame passed to it, assigning each replicate a different stream handle, at
most one of which can be the same as the original passes unchanged;
 Encodes the sequencing information into the frame in a manner such that it can be
decoded by its peer.
When the Listener proxy receives redundant streams:
 Extracts and decodes the sequencing information from a received frame.
 Examines this sequencing information in received frames packets and discards frames
indicated to be a duplicate of a frame previously received and forwarded; Also monitors
the variables to detect latent errors of Streams;

2018 ODVA Industry Conference

11©2018 ODVA, Inc.

FRER in a PRP and HSR Topologies
This section describes the use of FRER in a Parallel Redundancy Protocol (PRP) and a High-availability
Seamless Redundancy (HSR) network. This will be illustrated with an update to the EtherNet/IP Network
example in this paper (Figure 8). The PRP and HSR network control protocols are defined in the IEC
62439-3 standard.

Figure 8: PRP and HSR networks

There are additional devices in this network that need to support the FRER protocol. These devices are:
PRP and HSR Redundancy Boxes [Redbox] (RS): These are Relay Systems will need perform thestandard FRER Relay System functions as well as their respective PRP or HSR function. The
redboxes will substitute the FRER protocol tag with either the PRP or HSR protocol tag.

2018 ODVA Industry Conference

12©2018 ODVA, Inc.

In this scenario the Multi-Port IO Devices is a Relay-End System (RES) acting as a Listener/Talker thatreceives/transmits redundant streams to the Sngl-Port Controller, which is and End System (ES).

Figure 9: Single Talker/Listener Ring Network Example

The Multi-Port IO Device (RES), acting as a Talker generating a redundant stream to the One-Port
Controller (ES), performs the following FRER functions: Supplies HSR sequencing information in frames; Replicates each frame, assigning each replicate a different stream handle, at most one ofwhich can be the same as the original passes unchanged;
 Encodes the sequencing information into the frame in a manner such that it can be decodedby its peer.
The Multi-Port IO Device (RES), acting as a Listener receiving a redundant stream from the One-Port
Controller (ES), performs the following FRER functions: Extracts and decodes the HSR sequencing information from a received frame.
 Examines this sequencing information in received frames packets and discards frames
indicated to be a duplicate of a frame previously received and forwarded; Also monitors
the variables to detect latent errors of Streams;
The other Multi-Port IO Devices (Relay-End Systems) within the HSR network and one HSR Redbox (RS)are acting as relay systems which transfer frames belonging to redundant streams.
Two HSR Redboxes (RS) are acting as relay systems which transfer frames belonging to redundant
streams to the PRP Network, converting the sequencing information to the PRP Trailer tag. One HSR
Redbox will generate a stream for the red network, the other will generate the stream for the blue
network.
The Ethernet switches (Relay System) in the PRP network are acting as relay systems which transfer
frames belonging to redundant streams.

2018 ODVA Industry Conference

13©2018 ODVA, Inc.

One PRP Redbox (RS), acting as a relay system for the Sngl-Port Controller, performs the following
FRER functions:
The Talker proxy will generate redundant streams from single streams:
 Supplies PRP sequencing information in frames; Replicates each frame passed to it, assigning each replicate a different stream handle, at
most one of which can be the same as the original passes unchanged;
 Encodes the sequencing information into the frame in a manner such that it can be
decoded by its peer.
When the Listener proxy receives redundant streams:
 Extracts and decodes the PRP sequencing information from a received frame.
 Examines this sequencing information in received frames packets and discards frames
indicated to be a duplicate of a frame previously received and forwarded; Also monitors
the variables to detect latent errors of Streams;

Conclusion
The addition of FRER, an existing network can be converted to a seamless redundant network withchanging the core network equipment and adding some additional interconnections. Most on theinterconnection wiring and Edge Equipment can be reused. This is less expensive than having the
duplicate the network or replacing the Edge Equipment as well. This will allow the convergence ofexisting/legacy network with new TSN deterministic (Guarantee-of-Service vs Quality-of-Service)
capabilities.
******************************************************************************************************************************************************The ideas, opinions, and recommendations expressed herein are intended to describe concepts of the author(s) for the possible useof ODVA technologies and do not reflect the ideas, opinions, and recommendation of ODVA per se. Because ODVA technologies
may be applied in many diverse situations and in conjunction with products and systems from multiple vendors, the reader andthose responsible for specifying ODVA networks must determine for themselves the suitability and the suitability of ideas, opinions,and recommendations expressed herein for intended use. Copyright ©2018 ODVA, Inc. All rights reserved. For permission to
reproduce excerpts of this material, with appropriate attribution to the author(s), please contact ODVA on: TEL +1 734-975-8840
FAX +1 734-922-0027 EMAIL odva@odva.org WEB www.odva.org. CIP, Common Industrial Protocol, CIP Energy, CIP Motion, CIP
Safety, CIP Sync, CIP Security, CompoNet, ControlNet, DeviceNet, and EtherNet/IP are trademarks of ODVA, Inc. All othertrademarks are property of their respective owners.

2018 ODVA Industry Conference

14©2018 ODVA, Inc.