Troubleshooting EtherNet/IPNetworks
By Merrill Harriman
Schneider ElectricTechnical Trackwww.odva.org

Introduction
Overview

Mysterious counters
Diagnostic tools
Access mechanisms
Ethernet counters
Troubleshoot with counters
Profile counter behavior
QuestionsTechnical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 2www.odva.org

Introduction
Overview







Mysterious counters
Diagnostic Tools
Access mechanisms
Ethernet counters
Troubleshoot with counters
Profile counter behavior
QuestionsTechnical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 3www.odva.org

Mysterious Counters
Ethernet Statistics = MysteryAccessed• Via Ethernet Link Object (0xF6) or SNMPFew people understand them
• Vague or terse definitions

Fewer people know how to apply them
• Relationships between counters• What a value does not mean
• Rate VS valueMisused or misinterpreted• Wrong conclusions derails troubleshooting• Wastes time chasing non-existent problems

Technical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 4www.odva.org

Overview





Mysterious counters
Diagnostic Tools
Access mechanisms
Ethernet counters
Troubleshoot with counters
Profile counter behavior
QuestionsTechnical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 5www.odva.org

Diagnostic Tools
What tools are used instead?
Ping (ICMP Echo)• Most often used tool
– “Reach-ability”– Connection establishment ? Application : Network• Most often misused tool
– No measure of congestion / delay» Propagation delay inconsequential» Processed at lower stack layer - Faulty comparison– No indication of disturbance / load related• Failed Ping
– Numerous causes– Provides no useful information

Technical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 6www.odva.org

Diagnostic Tools
What tools are used instead?
Packet Capture• See “every” packet on the wire
– Second most common diagnostic tool
– Packet filtering, parsing, decomposition– Intricate detail• Complex – Micro view
– Second most common misused diagnostic tool
– Easy to get lost / mislead by detail– Only shows data for single link
– Location of capture critical for good analysis– Use of hubs may disturb network of interest– Corrupted packets not visible
Technical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 7www.odva.org

Diagnostic Tools
What tools are used instead?
Web Pages• Often supported– Easy access– Usually has diagnostic page
» Often same data as Link Object (0xF6)» De facto standard• Often different– No standard level of support– Vendor / device specific• This discussion – not assumed to exist– Concepts still relevant– General discussion favors general solutions

Technical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 8www.odva.org

Overview







Mysterious counters
Alternative Diagnostic Tools
Access mechanisms
Ethernet counters
Troubleshoot with counters
Profile counter behavior
QuestionsTechnical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 9www.odva.org

Diagnostics Access
EtherNet/IP AccessTCP/IP Ojbect (0xF5)– Mostly control parameters– IP params– Multicast params– ACD Status



Ethernet Link Object (0xF6)• Useful Ethernet statistics– Interface Flags– Interface Counters– Media Counters

Technical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 10www.odva.org

Diagnostics Access
SNMP Access

Simple Network Management Protocol• MIB Browser – SNMP client software• MIB (Management Information Base) => Objects• OID (Object ID) ~= Object AddressSupported by some industrial equipment
Supported by most infrastructure• All “Managed” devices support SNMP• Diagnostics & configurationMIBs• MIB II• etherLikeMIB

Technical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 11www.odva.org

Overview








Mysterious counters
Alternative Diagnostic Tools
Access mechanisms
Ethernet counters
Troubleshoot with counters
Profile counter behavior
QuestionsTechnical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 12www.odva.org

Ethernet Counters
EtherNet/IP Link Object (0xF6)
Group 1 – Interface Flags• Link status – Up / Down
• Duplex status – Operational value – HD / FD• Auto-negotiation status – Results of Auto-Neg

Group 2 – Collision Centric Counters• Half-Duplex OnlyCounterDeferred Transmissions

Carrier Sense detected line busy on TX event – defers TX

Single Collisions

Frame is successfully TX after 1 collision

Multiple Collisions
Excessive Collisions
Late Collisions

Frame is successfully TX after >1 collisions
Frame discarded after >16 collisions
Collision detected too late - discarded

Technical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 13www.odva.org

Ethernet Counters
EtherNet/IP Link Object (0xF6)
Group 3 – Other Important Counters
CounterDefinition

FCS Errors

A count of frames received that do not end on a byteboundary and do not pass the FCS check.
Frame Check Sequence – detects packet corruption.

In / Out Discards

Number of good packets discarded; input / output queue full.

In / Out Errors

Aggregation of a multiple specific errors.

In / Out Ucast Packets

Number of unicast packets rcvd/sent – including point-topoint & explicit messaging connections.
Number of non-unicast packets rcvd/sent including
broadcast (ARP, DHCP, …) & multicast (class 0 & 1).
Carrier signal not detected
For legacy equipment – not used today.
Implementation specific “catch-all”

Alignment Errors

In / Out NUcast Packets
Carrier Sense Errors
SQE Test Error
MAC Transmit / Receive Errors
Technical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 14www.odva.org

Overview

Mysterious counters
Alternative Diagnostic Tools
Access mechanisms
Ethernet counters
Troubleshoot with counters• Duplex Mismatch

Profile counter behavior
QuestionsTechnical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 15www.odva.org

Duplex MismatchCause

2 Devices• Dev1 => Static Duplex Config = FD
• Dev2 => Auto-Negotiation

Link Negotiation• Speed from link pulse– OK = 100 MB• Duplex(Auto)
HalfDuplex
DeviceCollisions

TX
ORRX(Static)
Full
Duplex
DeviceTX & RXFCS & Alignment
Errors– Dev1 – no negotiation – always FD– Dev2 – IEEE802.3 – MUST HD

Infrastructure Devices• Default = Auto-Negotiation

Technical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 16www.odva.org

Duplex Mismatch
Effect

CSMA/CD Circuitry• Carrier Sense Multiple Access / Collision Detection• Used in HD – Not used in FDCollision Detection• Dev2 (Auto-HD)– Uses CSMA/CD - Detects collision– Re-TX after collision detection• Dev1 (FD)– No CSMA/CD – No collision detection– Never Re-TX after collision– Frame lost foreverTechnical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 17www.odva.org

Duplex Mismatch
Anatomy of a CollisionCollision• HD end starts TX• Detects collision – increments counter• Stops TX & Sends Jam signal (101010101)• TXed frame = Few bytes good frame + Jam• Discards inbound frame from FD end – never Re-TXFCS / Alignment Errors• FD end detects FCS & Alignment Errors• Discards bad frame

Duplex Mismatch Performance• Much worse than HD-HD Link

Technical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual MeetingAll rights reserved.

page 18
www.odva.org

Duplex Mismatch
Detection
Explicitly Check for Dup. Mismatch• Implies that I suspect the problem• Check device configs• Interface flags of 0xF6

Use Link Object from end devices• Interface flags & counters

Switch to Device Links• Few switches support EtherNet/IP• Switches support SNMP
– Same counters available via SNMP

• Duplex status illusive in SNMPTechnical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 19
www.odva.org

Duplex Mismatch
Take awayEffects• Packet loss very high• Performance < halfduplex link
• Slow / DroppedConections

Half Duplex End• Many collisions• Maybe some FCS &Alignment Errors• Few Late & Excessive
Collisions

Technical Track© 2014 ODVA, Inc.

(Auto)
Half
Duplex
DeviceTX
ORRX

TX & RXFCS & Alignment
ErrorsCollisions

(Static)
FullDuplex
DeviceFull Duplex End• Zero collisions• Many FCS & Alignment
Errors• Zero Late & Excessive
collisions

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 20www.odva.org

Duplex Mismatch
EtherNet/IP VS. SNMP
EtherNet/IP

SNMP

Ethernet Link Object (0xF6)
Attribute
ID:

Name:

RFC 3635 - etherLike MIB(1.3.6.1.2.1.10.7)

RFC 1213 – MIB II(1.3.6.1.2.1)

None

2

Interface Flags(Duplex configurationand operational status)

Dot3StatsDuplexStatus(1.3.6.1.2.1.10.7.2.1.19)

5

Alignment Errors

dot3StatsAlignmentErrors(1.3.6.1.2.1.10.7.2.1.2)
dot3StatsFCSErrors(1.3.6.1.2.1.10.7.2.1.3)
dot3StatsLateCollisions(1.3.6.1.2.1.10.7.2.1.8)
dot3StatsExcessiveCollisions(1.3.6.1.2.1.10.7.2.1.9)

FCS Errors
Late Collisions
Excessive Collisions

Technical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

ifInErrors(1.3.6.1.2.1.2.2.1.14)

ifOutErrors(1.3.6.1.2.1.2.2.1.20)

page 21www.odva.org

Overview






Mysterious counters
Alternative Diagnostic Tools
Access mechanisms
Ethernet counters
Troubleshoot with counters• Electrical NoiseProfile counter behavior
QuestionsTechnical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 22www.odva.org

Electrical NoiseCausesElectrical Interference• Causes packet corruption• FCS / Alignment Errors

Sources of Interference• Cabling– Bends, crushed, corrosion– Routing – motors, power feeds, fluorescent lights– Shielding & grounding– Connectors – to much untwist– X-talk – TX & RX channels

Technical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 23www.odva.org

Electrical NoiseDetection
Ping• Stochastic – low probability ping gets clobberedPacket Capture• Corrupted frames discarded at receiving interface• Never shown in packet captureTechnical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 24www.odva.org

Electrical NoiseDetection

Full Duplex Link
• FSC & Alignment Errors• Noise or Duplex Mismatch?
– No collisions – both ends at FDHalf Duplex Link
• FCS & Alignment Errors• Both ends – few Single / Multi Collisions – expectedFCS / Alignment on 1 end only – No Collisions• Suspect bad single pair

Understand failure modes• Relative position of count values matters

Technical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 25www.odva.org

Overview




Mysterious counters
Alternative Diagnostic Tools
Access mechanisms
Ethernet counters
Troubleshoot with counters
Profile counter behavior
QuestionsTechnical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 26
www.odva.org

Profiling Types of Errors
Relative Counter Values

Not absolute – General GuideProblem Type

Normal Collisions
Rate

Error Collisions
RateFCS / Alignment
Errors Rate0HighDuplex
Mismatch

Static Config(FD) End0HighMedLow

Overloaded
Devices

AutoNeg (HD)EndHalf Duplex
Full Duplex

Med to High0

Med to High0Low

Half Duplex
Full Duplex
Half Duplex
Full Duplex

Med0Med0Med0High0High

Noise / Cable
Fault
Cable too long
Technical Track© 2014 ODVA, Inc.

Discards
Rate

Zero to few

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

HighHigh

Rate >> 0
indicates
potential
problemZero to few
Zero to few

page 27www.odva.org

Overview

Mysterious counters
Alternative Diagnostic Tools
Access mechanisms
Ethernet counters
Troubleshoot with counters
Profile counter behavior
QuestionsTechnical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 28
www.odva.org

Questions

Technical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 29
www.odva.org

Overview

Mysterious counters
Alternative Diagnostic Tools
Access mechanisms
Ethernet counters
Troubleshoot with counters• Overloaded DeviceProfile counter behavior
QuestionsTechnical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual MeetingAll rights reserved.

page 30
www.odva.org

Overloaded Device
CauseLine rate > processing rate
• “Bursty” traffic - overloads buffers• Maximum sustainable rate
– Packet size
– Type of packets– Unicast / multicast / broadcast• Multicast flooding– IGMP Snooping – pruning delay– Topology change

Technical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 31www.odva.org

Overloaded Device
Effect & Detection
Full Buffers• No room for new packets => dropped• Increments Discard counter• Sluggish, retries, connection timeoutsIn / Out Discards• Link Object – Interface Counters• SNMP – MIB II – interface table– ifInDiscards (ifTable.13)– ifOutDiscards (ifTable.19)

High CPU utilization• Optional attribute – Connection Manager Object (0x06)

Technical Track© 2014 ODVA, Inc.

2014 Industry Conference & 16th Annual Meeting
All rights reserved.

page 32
www.odva.org