IT and OT Convergence - Recommendations for Building
an Industrial IoT-Ready Manufacturing Network
Arun Siddeswaran, Cisco Systems, Inc.
Gregory Wilcox, Rockwell Automation, Inc.
February 22, 2017

Agenda•Secure Connectivity between Manufacturing and Business Systems– Business Outcomes– Bridging OT-IT– Key Requirements / Key Tenets••

Key Takeaways
Recommended Resources

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

3

Industrial IoT - Business OutcomesIndustrial IoT - Business Outcomes

ReducedDowntime

Reduction in Defects49%

48%
Unplanned downtime
down from 11% to5.8%

Defect rate
down from
4.9% to 2.5%

New ProductIntroduction

OEE
Improvement

Improved
Inventory

Reduction inEnergy Use

23%

16%

35%

18%

New productintroduction cycle
time reduced from 15
to 11 months

Average OEE
improvedfrom 74% to 86%

Inventoryturns increased
from 14 to 19

Annual energy
cost down from
$8.4M to $6.9M

Source: SCM World/Cisco “Smart Manufacturing and the Internet of Things 2015” survey of 418 Manufacturing Business Line Executives
and Plant Managers across 17 vertical industries.
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

5

Industrial IoT - Business Outcomes••••Smart Devices, Smart Machines, Smart Manufacturing
Customer choice of best-in-class products through Industrial IoT device
coexistence and interoperability
Standard Network Services; Standard Network Tools
Pervasive Asset Optimization and Utilization– Common infrastructure devices and tools– Human assets: knowledge, experience, training•

Better Analytics– Device/Machine, System/Plant, Enterprise•

Enables Innovative Technologies– Mobility – Personnel and Equipment– Cloud –On Premise and Off PremiseTechnical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

6

Industrial OT vs Enterprise IT Networks

Industrial OT vs. Enterprise IT Networks
Criteria

Industrial OT Network

Enterprise IT Network

Network

Standard IEEE 802.3 Ethernet and proprietary (non-Standard IEEE 802.3 Ethernet

Technologystandard) versions

Standard IETF Internet Protocol (IPv4 and IPv6)

Standard IETF Internet Protocol (IPv4) and proprietary(non-standard) alternatives

Network

Switch-Level and Device-Level Topologies

Switch-Level topologies

Availability

Ring Topology is predominant for both, Redundant Star for

Redundant Star Topology is predominantStandard IEEE, IETF, and vendor specific Layer 2 and Layer 3
resiliency protocols

switch topologies is emerging
Standard IEEE, IEC and vendor specific Layer 2 resiliency
protocolsService Level

Mean time to recovery (MTTR) - Minutes, Hours

Mean time to recovery (MTTR) - Hours, Days

Mostly Static

Mostly Dynamic

Agreement (SLA)
IP Addressing
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

9Industrial OT vs. Enterprise IT Networks
Criteria

Industrial OT Network

Enterprise IT Network

Traffic TypePrimarily local – traffic between local assets
Information, control, safety, motion, time synchronization,energy management
Smaller frames for control traffic
Industrial application layer protocols: CIP, PROFINET, IEC
61850, Modbus TCP, etc.

Primarily non-local – traffic to remote assets
Voice, Video, Data
Larger packets and framesStandard application layer protocols: HTTP, SNMP, DNS,
RTP, SSH, etc.

Performance

Low Latency, Low Jitter
Data Prioritization – QoS – Layer 2 & 3

Low Latency, Low Jitter
Data Prioritization – QoS – Layer 3

Security

Open by default, must close by configuration andarchitecture
Industrial security standards – e.g. IEC, NIST
Inconsistent deployment of security policies
No line-of-sight to the Enterprise or to the InternetPervasive
Enterprise security standards
Strong security policies
Line-of-sight across the Enterprise and to the Internet

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

10What are best practices

Structured and Hardened Architectures
Key Requirements• Scalable• Reliable• Safe• Secure• Future-readyTechnical Track© 2017 ODVA, Inc.

Key Tenets• Smart Endpoints• Segmentation (Zoning)• Managed Infrastructure• Resiliency• Time-critical Data
• Wireless - Mobility• Holistic Defense-in-Depth
Security• Convergence-ready2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

13

Zoning Through Segmentation
Level 5
Level 4Enterprise Network
Site Business Planning and Logistics Network

E-Mail, Intranet, etc.
Remote Desktop
Gateway Services

Patch Management

Application Mirror

Enterprise Security Zone
Levels 4-5

FirewallAVServerWeb Services Operations

Web
E-MailCIP

Application
ServerIndustrial DMZ
Level 3.5

FirewallLevel 3

Level 2

Level 1

Level 0Technical Track© 2017 ODVA, Inc.

FactoryTalk
Application
Server

FactoryTalk
Directory

Engineering
Workstation

FactoryTalk
ClientRemote Access
ServerFactoryTalk
ClientOperator
Interface

Batch
ControlEngineering
Workstation

Discrete Control

Sensors

Site Operations
and Control

Drive Control

DrivesContinuous
Process Control

Actuators

Robots2017 Industry Conference & 18th Annual Meeting
All rights reserved.

Operator
Interface

Industrial Security Zone(s)Levels 0-3

Area
Supervisory
Control
Cell/Area Zone(s)Basic Control Levels 0-2

SafetyControl

Process
www.odva.org

14

Zoning Through Segmentation

Plant-wide Zoning• Functional / Security Areas• Smaller Connected LANs
• Smaller Broadcast Domains• Smaller Fault Domains• Smaller Domains of Trust• Industrial IoT Technology• Building Block Approach forScalability

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

15

Zoning Through Segmentation
Wide Area Network (WAN)
Data Center - Virtualized Servers
•••••

Cloud

Enterprise

ERP - Business Systems
Email, Web Services
Security Services - Active Directory (AD), Identity Services (AAA)
Network Services – DNS, DHCP
Call Manager

External DMZ/Firewall

Enterprise Zone
Levels 4-5

Identity Services

Plant Firewalls

Physical or Virtualized Servers•••••••••Patch Management
AV Server
Application Mirror
Remote Desktop Gateway Server

Physical or Virtualized Servers
•••

Core
SwitchesIdentity Services

FactoryTalk® Application Servers andServices Platform
Network & Security Services – DNS,AD, DHCP, Identity Services (AAA)
Storage Array

AccessSwitches

ActiveWireless
LAN Controller(WLC)

Level 3 - Site Operations
Standby(Control Room)

Industrial
Demilitarized Zone
(IDMZ)

Active/Standby
Inter-zone traffic segmentation
ACLs, IPS and IDS
VPN Services
Portal and Remote Desktop Services proxy

Cell/Area Zone
Levels 0–2

RemoteAccess
ServerDistribution
Switch Stack

Industrial ZoneLevels 0–3(Plant-wide Network)

AccessSwitches

Distribution
Switch Stack

Cell/Area Zone
Levels 0–2

IFW

LWAP
Camera

LWAP

LWAP

HMI

SSID
2.4 GHzAP
SafetyController

Phone
SSID
5 GHz

WGB

Safety
I/O

SSID
5 GHz
WGB

Soft
Starter

IFW

WGB

Instrumentation
Drive
Controller

Controller

Controller

I/O

Servo
Drive

HMI

Robot

Cell/Area Zone - Levels 0–2

Cell/Area Zone - Levels 0–2

Cell/Area Zone - Levels 0–2

Redundant Star Topology - Flex Links Resiliency
Unified Wireless LAN(Lines, Machines, Skids, Equipment)

Ring Topology - Device Level Ring (DLR) Protocol
Unified Wireless LAN(Lines, Machines, Skids, Equipment)

Linear/Bus/Star Topology
Autonomous Wireless LAN(Lines, Machines, Skids, Equipment)

SafetyI/OKey Tenets• Smart Endpoints• Segmentation(Zoning)• ManagedInfrastructure• Resiliency• Time-critical Data
• Wireless - Mobility• Holistic Defensein-Depth Security• Convergenceready16

Wired Access Overview

Typical Zone Traffic Flows
IDMZ
Industrial Zone
Levels 0-3

Engineering Laptop

NetworkManagement

CIP Implicit Traffic- Producers &Consumer>80% local
Cyclical I/O traffic, UDP unicast and multicast

Distribution Switch<500 Bytes, Frequent 0.5 to 10’s of ms, typically 20ms
HMIHMICIP Explicit Traffic - Informational
control and administration

Controller

Intra- and inter-cell/area zone traffic flow
Drive

Technical Track© 2017 ODVA, Inc.

Cell/Area Zone
Levels 0-2

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

Cell/Area Zone
Levels 0-2

Non-critical administrative or data traffic using TCP~1500 Bytes, infrequent

www.odva.org

19

Industrial Network Topologies

Switch-level Topologies
RedundantStarStar/Bus
LinearRing

Flex Links

Resilient Ethernet
Protocol (REP)

Switch Stack

Switch Stack

Cisco
Catalyst 2955
HMIHMIControllerControllers

HMIControllers

HMI

Controllers,Drives, and Distributed I/O

Cell/Area Zone
Cell/Area Zone

Controllers, Drives, and Distributed I/OControllers, Drives, and Distributed I/O

Cell/Area Zone

Cell/Area Zone

Device-level Topologies
Switch-level andDevice-level Topologies
I/O
VFD
Drive

I/O

I/O

VFD
DriveController
Servo
Drive
Controllers,Drives, and Distributed I/O

VFD
Drive

HMI

HMI

Cell/Area Zone

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

I/O
Controller

Instrumentationwww.odva.org

20Performance Requirements
Process AutomationAutomation
Process

Discrete Automation
Automation
DiscreteLoss Critical
Critical
LossFunction

Information Integration,
Slower Process Automation

Time-critical
Discrete Automation

Multi-axis Motion Control

Comm. Technology.Net, DCOM, TCP/IP

Industrial Protocols, CIP, Profinet

Hardware and Software solutions, e.g.
CIP Motion, PTP

Period1 second or longer

1 ms to 100 ms

100 µs to 10 ms

Industries

Oil & Gas, chemicals,
energy, water

Auto, food and beverage, electricalassembly, semiconductor, metals,
pharmaceuticalUtilities
Subset of Discrete automation

Applications

Pumps, compressors, mixers;
monitoring of temperature,pressure, flow

Material handling, filling, labeling,
palletizing, packaging; welding,stamping, cutting, metal forming,soldering, sorting

Life/equipment safety, Synchronization ofmultiple axes: printing presses, wiredrawing, web making, picking and placing

Source: ARC Advisory Group
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

21

Network Resiliency Protocols
Resiliency
ProtocolMixed
VendorRing

RedundantStarNet Conv
>250 ms

Net Conv
50-100 ms

Net Conv< 0~10 ms

Layer 3Layer 2

STP (802.1D)

Process and Information

RSTP (802.1w)
MSTP (802.1s)

PVST+Time CriticalREP
EtherChannel(LACP 802.3ad)
MRP (IEC 62439-2)*
Flex Links
PRP/HSR (IEC 62439)*

Loss Critical

DLR(IEC & ODVA)
StackWise
HSRP
VRRP(IETF RFC 3768)
Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

22

Convergence-ReadyChallenge - Ethernet Growing Pains•

Ethernet networks continue to grow:– Each skid/machine adds another 5 - 50 EtherNet/IP enabled devices– Every line adds another 250 - 1,000 EtherNet/IP enabled devices

How do I connect all these skids/machines into a plant network
to gain the advantages?

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

24

Layer 2 Network Address Translation (NAT)
Outside Subnet(ex. 10.0.0.x)

One to One (1:1) NAT

Many Outside IP addresses(One per device wishing to be accessible from the Outside Subnet

NAT Enabled Device
Many Inside IP addresses(One per connected device)

Inside Subnet(ex. 192.168.1.x)

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

25

Layer 2 NAT Design Scenario #1
Single-Cell, Single VLAN per Switch
Aggregation
switchVLAN10Line Controller
10.10.10.30

Trunk

OUTSIDE

Inside to Outside
NAT Table
Outside to inside
NAT Table

Inside192.168.1.10
Outside10.10.10.30

Outside10.10.10.10
Inside192.168.1.30

INSIDEVLAN10Inside Address
192.168.1.10

IES

MachineTechnical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

28

Wireless Access
Overview

Wireless Technology Overview - Benefits of Industrial WLAN
•Lower installation and operational costs– Cabling and hardware reduction– Minimizing cable failures••

Connection to hard-to-reach and restricted areas
Equipment mobility– New and more efficient applications•

Workforce mobility– Higher productivity and less downtime– Operators, engineering and maintenance, Industrial IT•

Asset Tracking– Track assets of people to optimize cost and for safety

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

32

Challenges of wireless communication•

Half-duplex shared medium:– Only one radio can transmit on a particular wirelesschannel– A radio cannot transmit and receive at the same time onthe same channel•

Higher latency, jitter and packet loss compared towired Ethernet– Media contention, collisions and interference– Can be minimized but not eliminated•Signal quality may change over time

Technical Track© 2017 ODVA, Inc.

Wireless advantages > challenges when
• WLAN is designed and maintained properly• Used for appropriate applications

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

33

Autonomous WLAN Architecture

AutonomousAP

WGB

AutonomousAP

SSID1
5 GHz…

Technical Track© 2017 ODVA, Inc.

WGB

SSID25 GHz
WGB

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

WGB…

www.odva.org

40Unified WLAN Architecture
WLC

SSID3
2.4 GHzLWAP

WGB

Technical Track© 2017 ODVA, Inc.

LWAP

SSID1
5 GHz

…

LWAP

LWAP

WGB(Roaming)2017 Industry Conference & 18th Annual Meeting
All rights reserved.

SSID2
5 GHz
WGB

…

WGB

www.odva.org

41

Holistic Defense-in-Depth Security

Industrial Network Security Framework
CloudEnterprise
External DMZ/Firewall

IEC 62443- Zones & Conduits- Confidentiality, Integrity,
AvailabilityNIST 800-82- Cybersecurity Framework
ICS-CERT- Recommended Practices

Identity Services
Enterprise Zone: Levels 4-5
Industrial Demilitarized Zone (IDMZ)

Physical or Virtualized Servers• Patch Management• AV Server• Application Mirror• Remote Desktop Gateway ServerIndustrial Zone: Levels 0-3
Core
Switches

Wireless LAN
Controller (WLC)
ActiveControl System
Engineers (OT)
Control System Engineers
in Collaboration with IT
Network Engineers(Industrial IT)
IT Security Architects inCollaboration with Control
Systems Engineers

StandbyDistribution
Switch Stack

Level 3 – Site Operations

FactoryTalkClientSSID
2.4 GHz

Level 2 – Area Supervisory ControlLWAP

IFW
SSID
5 GHz

I/O
Controller

Controller

Level 1 - Controller

SoftStarter

WGB

MCC

Level 0 - Process

47
I/O

Drive

Secure Remote Connectivity
Enterprise Zone: Levels 4-5
Enterprise
WAN

Cloud
External
DMZ / Firewall

WLC (Enterprise)
ISE PAN/PSN
ISE MnT

Industrial Demilitarized Zone (IDMZ)

Firewalls(Active/Standby)

Industrial Zone
Levels 0-3

Core
switches

ISE PSN
WLC(Active)(Plant-wide Network)

WLC(Standby)

Distribution
switchLevel 3Site Operations
IES

LWAP

IES
IES

IES
IES

Cell/Area Zones - Levels 0-2

WGB

Controller(Lines, Machines, Skids, Equipment)
FactoryTalk Client

I/O48
DriveController

Controller

Key Takeaways•••••

Plant-wide reference architectures - Simplified design, quicker deployment,
reduced risk in deploying new technology
Wired access topology and protocols based on plant layout, convergence
and application requirements
Layer 2 NAT helps end users to easily integrate skids/machines into theirlarger plant network without extensive coordination with OEMs
Wireless access offers multiple advantages, enables secure personnel
access, equipment to equipment communication and asset tracking
Defense-in-depth security offers multiple layers of threat detection andprevention

Technical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

49

Recommended Resources••

ODVA– The Common Industrial Protocol(CIP) and the Family of CIP
Networks– Network Infrastructure for
EtherNet/IP: Introduction andConsiderations– Media Planning and Installation
Manual– Guidelines for Using Device Level
Ring (DLR) with EtherNet/IP– Securing EtherNet/IP Networks

Converged Plantwide Ethernet(CPwE) Architectures– Cisco– Rockwell Automation•Education / Awareness– Industrial IP Advantage (IIPA)eLearning industrial–ip.org•Training / Certification– Industrial Networking Specialist• IMINS Training, 200-401 Exam

– CCNA Industrial• IMINS2 Training, 200-601 ExamTechnical Track© 2017 ODVA, Inc.

2017 Industry Conference & 18th Annual Meeting
All rights reserved.

www.odva.org

50

Thank You