# Presentation: 832942

## Ethernet/IP Performance Metrics

- James Gilsinn
- NIST, Intelligent Systems Division
- james.gilsinn@nist.gov

- 6th RIA/NIST Workshop on Open Architecture Control in Robotics

- June 4, 2002

## Overview

- Introduction
- Conformance vs. Performance
- Overview of ODVA Ethernet/IP Implementors Workshop
- Suggestions
- Comments?

- 6th RIA/NIST Workshop on Open Architecture Control in Robotics

- June 4, 2002

## Introduction

- Most industrial machines will be networked
- Most commercial networked devices are both conformance and performance tested.
- Most industrial networked systems conformance tested but NOT performance tested

- 6th RIA/NIST Workshop on Open Architecture Control in Robotics

- June 4, 2002

## Conformance vs. Performance

- Conformance testing proves a device meets the basic specification and doesn’t break
- Performance testing proves how well a device works under different conditions
- Not all devices break when something goes wrong
- Devices can be tailored to tests, but performance tests give basic comparison of different devices (vehicle city/highway gas mileage)

- 6th RIA/NIST Workshop on Open Architecture Control in Robotics

- June 4, 2002

## Conformance vs. Performance

- Conformance Tests
- Does the device...
- Send/receive properly formatted messages?
- Shutdown when it receives improper messages?
- Perform the desired task(s)?
- Use the correct physical characteristics?
- Etc.

- Performance Tests
- Does the device...
- Handle message prioritization correctly?
- Continue to work at low/high network loads?
- Suffer from OS overhead?
- Block on communications?
- Handle multiple connections?
- Etc.

- 6th RIA/NIST Workshop on Open Architecture Control in Robotics

- June 4, 2002

## Overview of ODVA Ethernet/IP Implementors Workshop

- May 22, 2002 – Ann Arbor, MI
- Major Topics Discussed
- Terms & Definitions
- IP Address Configuration Methods
- Performance Requirements & Benchmarking
- Workshop Interoperability Demonstration
- IP Address Configuration Methods
- Rockwell IP Tool (BootP)
- BootP vs. DHCP (DHCP Server, either client)

- 6th RIA/NIST Workshop on Open Architecture Control in Robotics

- June 4, 2002

## Overview of ODVA Ethernet/IP Implementors Workshop

- Performance Requirements & Benchmarking
- Rockwell Presentation, *Brian Batkey*
- Ran set of experiments to test Packet Rate, Connection Size, and Number of Connections vs. throughput
- Propose performance specification with:
- Packet Size vs. Rate
- Connection Overhead vs. Number of Connections
- Max Connections
- Tests did not include:
- Other class 3 CIP traffic
- Non-CIP traffic
- Rejected Multicast Packets

- 6th RIA/NIST Workshop on Open Architecture Control in Robotics

- June 4, 2002

## Overview of ODVA Ethernet/IP Implementors Workshop

- Performance Requirements & Benchmarking
- BCIT Presentation, *Eric Byres*
- Most commercial networking equipment uses RFCs for performance tests (RFC 1242, 2285, 2544, 2889)
- RFC 1242 – Benchmarking Terminology for Networking Interconnection Devices
- RFC 2285 – Benchmarking Terminology for LAN Switching Devices
- RFC 2544 – Benchmarking Methodology for Networking Interconnection Devices
- RFC 2889 – Benchmarking Methodology for LAN Switching Devices
- Need to define something similar for industrial networked devices and equipment
- Testing should be done at all ISO levels, not just TCP/UDP and below
- Closed loop tests for both command/response and command/action

- 6th RIA/NIST Workshop on Open Architecture Control in Robotics

- June 4, 2002

## Overview of ODVA Ethernet/IP Implementors Workshop

- Performance Requirements & Benchmarking
- Participants from GM, BCIT, Rockwell, and NIST would develop a proposed set of metrics and tests for next meeting (July 10-11, 2002)
- Workshop Interoperability Demonstration
- GM sponsored workshop to demonstrate interoperability of Ethernet/IP equipment
- Many different companies & industries represented
- GM would define demonstration purpose and scope, then planning could be done for miscellaneous devices

- 6th RIA/NIST Workshop on Open Architecture Control in Robotics

- June 4, 2002

## Suggestions	

- Metrics
- Throughput *(Quan.)*
- Frame rate
- Number of connections
- Stability *(Qual.)*
- Robustness of system
- Reaction to bad info
- Response Latency *(Quan.)*
- Closed-loop time for command/response

- Tests
- Throughput
- Packets/sec
- Connections before fail
- Packets/connection
- Stability
- Bad packets
- Bad parameters
- Recovery
- Response Latency
- Time for response from device (status)

- 6th RIA/NIST Workshop on Open Architecture Control in Robotics

- June 4, 2002

## Suggestions

- Metrics
- Action Latency *(Quan.)*
- Closed-loop time for command/action
- Prioritization *(Quan.)*
- Latency & throughput due to low priority CIP or other network traffic

- Tests
- Action Latency
- Time for action by device (digital and/or analog)
- Prioritization
- Load network with low priority CIP and other traffic, check latency & throughput (multiple loadings)

- 6th RIA/NIST Workshop on Open Architecture Control in Robotics

- June 4, 2002

## Suggestions

- Tests should be performed with dedicated device
- Software-based tests will have same OS related latencies that devices will
- Devices WILL be targeted to work well on tests, but never perform up to specs
- This can’t be stopped, but performance tests will give some basis for comparison between devices

- 6th RIA/NIST Workshop on Open Architecture Control in Robotics

- June 4, 2002