High availability process safety applications enabled
by Concurrent Connections

Filip Zembok, Rockwell Automation
Gregory Majcher, Rockwell Automation
Darren Klug, Rockwell Automation

Abstract
In the Spring 2023 publication of the CIP family of specifications, ODVA
announced the addition of an important new technology, ConcurrentConnections. This technology enables flexible, zero switchover time, endto-end redundancy solutions.
This presentation briefly introduces the basic terms and concepts related withhigh availability.
This presentation discusses characteristics of process industry systems andpoints out some of the deficiencies of current high availability solutions.
Finally, the presentation explains the Concurrent Connections technology anddescribes how Concurrent Connections improve high availability andsafety systems.
Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

2

Overview of High Availability•Availability– Mean Time To Failure– Mean Time To Restore/Repair••High Availability
Redundancy– Hot, Warm, or Cold
– Active or Passive synchronization– Switchover or Concurrent•Fault Tolerance

Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

3

Process Industry•••

Large-scale and complex installations
Processes cannot be easily stopped
Consequences of failure can be catastrophic
––––

••

Bhopal gas tragedy
Piper Alpha oil rig explosion
Deepwater Horizon oil spill
Tianjin explosions

High Availability is critical
Highly regulated

Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

4

High Availability and Redundancy in the CIP Specification•CIP Networks Library, Volume 1 Common Industrial Protocol–•CIP Networks Library, Volume 2 EtherNet/IP Adaptation of CIP––•Redundant Owner connection type, Ownership of Outputs (ROO) and Claim Output Ownership (COO) bits
PRP
DLR

CIP Networks Library, Volume 4 ControlNet Adaptation of CIP–

Media redundancy with ring topologies

Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

5

Existing Redundancy solutions and their problems••Vendor specificSolution 1: switchover on IP address
Device CIP address: 192.168.1.4

Switchover
In an event ofDevice A failure

EtherNet/IP network

Device C
IP address: 192.168.1.4

EtherNet/IP network

Redundant devices

Redundant devices

Keep up to date

Device A (Active)
IP address: 192.168.1.2•••

Device B (Backup)
IP address:

Device A (failed)
IP address:

Device B (Active)
IP address: 192.168.1.2

Solution 2: Delay applying connection fault action
Solution 3: Redundancy realized at the application level…

Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

6

Concurrent Connections••••“PRP on the CIP connection level”
Redundant participantsMultiple paths for transferring CIP data
One logical CIP connection from anapplication perspective

Concurrent Connection branch(Originator: Participant A –Router: Participant A)

Concurrent Connection branch(Router: Participant A –Target: Participant B)

Originator

RouterTargetParticipant A

Participant A

Participant A

Participant B

Participant B

Participant B

Concurrent Connection hop Concurrent Connection hop(Originator - Router)(Router - Target)

Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

7

Concurrent Connections – connection management•New Connection Manager services:–

Concurrent_Forward_Open and Large_Concurrent_Forward_Open•–

Concurrent_Forward_Close••“Forward_Open + Concurrent Connections Protocol Version field”
Same format as Forward_Close

New Extended Network Segment–Concurrent Connection Path (list of Concurrent Connection Hops)

Controller

Participant A

Hop 1

Participant A

Egress port: Port number on Controller
Number of Link Addresses: 2
Link Address 1: Router Participant A
Link Address 2: Router Participant B
Participant B

Technical Track© 2023 ODVA, Inc.

I/O Device

RouterHop 2

Participant AEgress port: Port number on Router
Number of Link Addresses: 2
Link Address 1: I/O Device Participant A
Link Address 2: I/O Device Participant B
Participant B

Participant B

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

8

Concurrent Connections - Connection Opening ProcessController

RouterI/O Device

Participant A

Participant A

Participant A

Participant B

Participant B

Synchronize
connection
parameters

Participant B
Hop 1

Hop 2

Egress port: Port number on Controller
Number of Link Addresses: 2
Link Address 1: Router Participant A
Link Address 2: Router Participant B

Egress port: Port number on Router
Number of Link Addresses: 2
Link Address 1: I/O Device Participant A
Link Address 2: I/O Device Participant B

Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

Legend:• Concurrent
Forward Openrequest• Concurrent
Forward Openresponse• Open branch ofConcurrent
Connection

9

Concurrent Connections - Connection Closing Process
Controller

RouterI/O Device

Participant A

Participant A

Participant A

Participant B

Participant B

Participant B

Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

Legend:• Open branch ofConcurrent
Connection• Concurrent
Forward Close
request• Concurrent
Forward Close
response10Concurrent Connections – Data flow
Controller

RouterI/O Device

Participant A

Participant A

Participant A
Synchronize
connection data
before setting thefield device

Synchronize
connection
data

Participant B

Participant B

Participant B

Legend:• Open branch ofConcurrent
Connection• Last Concurrent
Connection
Packet (CCSC=1)• Fresh Concurrent
Connection
Packet (CCSC=2)• Discarding theredundant packet

The same process happens in I/O Device to Controller direction.
Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

11

Concurrent Connections - Fault tolerance••Controller

RouterI/O Device

Participant A

Participant A

Participant A

Participant B

Participant B

Participant B

The connection remains open as long as there is at least
one available path between connection endpoints
Zero switchover time
Technical Track© 2023 ODVA, Inc.

••

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

Legend:• Open branch ofConcurrentConnection• Failure

Redundant devices are functionally equivalentand participate in the process all the timeHot, active, and concurrent redundancy
www.odva.org

12

Concurrent Connections – Branch recovery•

Controller

RouterI/O Device

Participant A

Participant A

Participant A

Participant B

Participant B

Participant B

Legend:• Open branch ofConcurrent
Connection• Failure• Concurrent
Forward Openrequest• Concurrent
Forward Openresponse

Local, periodic attempts to reopen failed Concurrent Connection branches
Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

13

Concurrent Connections - Flexibility
Controller

Router 1

Router 2

Participant A

Participant A

Participant A

Router 1

Router 2

I/O device

Controller

I/O device
Participant B

Participant B

Participant B

I/O device

I/O Device
Participant A

Participant A

Controller

Router 1

Router 2

Participant B

Controller

Router 1

Router 2
Participant B

Participant C

Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

14

Concurrent Connections - real example
Concurrent Connections will be supported by Wireshark 4.2 (November 2023 release)
Controller

Router 1

Participant A

Participant A

Participant B

Technical Track© 2023 ODVA, Inc.

Router 2
Wireshark

IP: 10.192.71.101

Participant B
IP: 10.192.71.1022023 Industry Conference & 22nd Annual Meeting
All rights reserved.

Participant A
IP: 10.192.71.103
Slot: 0Participant B
IP: 10.192.71.104
Slot: 1www.odva.org

I/O Device
Participant A
Slot: 2

Participant B
Slot: 315

Real example – Concurrent_Forward_Open

Controller

Router 1

Participant A

Participant A

Participant B

Technical Track© 2023 ODVA, Inc.

Router 2
Wireshark

IP: 10.192.71.101

Participant B
IP: 10.192.71.1022023 Industry Conference & 22nd Annual Meeting
All rights reserved.

Participant A
IP: 10.192.71.103
Slot: 0Participant B
IP: 10.192.71.104
Slot: 1I/O Device
Participant A
Slot: 2

Participant B

www.odva.org

Slot: 316

Real example – Concurrent_Forward_Open format
Legend:• Originators shalluse the samevalues• Routers andTargets shallcheck if values
are the same fora given
Connection Triad• Difference from
Forward_Open

Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

17

Real example – Concurrent Connection Path

Concurrent Connection Path

Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

18

Real example – Concurrent_Forward_Close

Concurrent Connection Path

Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

19Real example – Real-Time data O->T

Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

20

Concurrent Connections implementation hints
Concurrent Connections extend existing CIP connections
Concurrent Connections functionality

ODVA CIP Specification Vol 1

Concurrent Connection Path Extended Network Segment

C-1.4.3.6.2, 3-7.1

Concurrent_Forward_Open Connection Manager service

3-5.6.5, 3-7.2

Concurrent_Forward_Close Connection Manager service

3-5.6.6, 3-7.3

Concurrent Connection Packet production and consumption

3-7.4.1.1, 3-7.4.3, 3-7.4.4

Branch recovery

3-7.5

Diagnostics

3-5.3.1.10, 3-5.3.1.11, 3-5.3.1.12

Originators synchronization of Concurrent Connection parameters

3-7.2

Endpoints synchronization of produced connection data

3-7.4.2

Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

www.odva.org

21Concurrent Connections SummaryPros•••••ConsStandardized end-to-end solution for •redundant device communication•
across a system with devices from •multiple vendors
Enable high availability systems,
maximize system MTTF
Flexible
Eliminate switchover solution
deficiencies
Relatively easy to implement (extensionof existing CIP connections)Technical Track© 2023 ODVA, Inc.

2023 Industry Conference & 22nd Annual Meeting
All rights reserved.

Higher use of network bandwidthAdditional packet processing steps
Require active synchronization of theredundant endpoints (vendor-specific)

www.odva.org

22