Performance Test Terminology for
EtherNet/IP Devices
Version 1.1
March 14, 2005
PUB00080R11

Published by
EtherNet/IP Implementors Workshop
ODVAPerformance Test Terminology for EtherNet/IP Devices
Document Revision Log
Revision
0.7

Sections

0.8

2.7, 4.x

Date
7/15/04(2.7) Removed text referring tothroughput being tested with nobackground traffic.
(4.x) Split out definitions of
Loaded, Unloaded, Within-Spec,and Outside-Spec.
Updates made per Workshop #13
comments
First release after Workshop #13
approvalUpdated document per
recommendations from
EtherNet/IP Performance
Workgroup.

9/14/04

Author(s)EtherNet/IP
Performance
Workgroup
James Gilsinn

9/16/04

James Gilsinn

03/14/05

James Gilsinn

Initial Draft

1.0
1.1

Remarks

Various

March 14, 2005

ii

PUB00080R11

Performance Test Terminology for EtherNet/IP Devices

Abstract
This document discusses and defines a number of terms that are used in describing performance
tests and the results for EtherNet/IP devices. The terms and definitions presented in thisdocument will be used in additional documents to describe performance tests and the suggested
format in reporting the results for these tests. This document is a product of the EtherNet/IP
Performance Workgroup of the Open DeviceNet Vendor Association (ODVA) EtherNet/IP
Implementors Workshop.

Table of Contents
1 Introduction .......................................................................................................1
2 Metrics................................................................................................................2
2.1
2.2
2.3
2.4
2.5
2.6
2.7

3

Types of Equipment ..........................................................................................6
3.1
3.2
3.3
3.4
3.5
3.6
3.7

4

Term (Definition Format) ............................................................................................... 2
Action Latency................................................................................................................ 2
Jitter/Variability .............................................................................................................. 3
Latency............................................................................................................................ 3
Overload Behavior .......................................................................................................... 4
Response Latency (a.k.a. Response Time) ..................................................................... 4
Throughput...................................................................................................................... 5
Term (Definition Format) ............................................................................................... 6
Device Under Test (DUT) .............................................................................................. 6
Network Analyzer........................................................................................................... 6
Network Infrastructure Equipment ................................................................................. 7
Protocol Analyzer ........................................................................................................... 7
Test Equipment ............................................................................................................... 8
Traffic Generator ............................................................................................................ 8

Other Terms ......................................................................................................9
4.1
4.2
4.3
4.4

Unloaded ......................................................................................................................... 9
Loaded............................................................................................................................. 9Within-Spec .................................................................................................................... 9
Outside-Spec ................................................................................................................... 9567

EtherNet/IP Performance Workgroup Members ........................................10
Acknowledgments ...........................................................................................10
References ........................................................................................................10

1

Introduction

The specifications used to define the performance characteristics of network devices are often
written in different ways from different vendors. This makes it difficult for users to compare
similar devices from without doing painstaking searching through manuals or hours contacting
engineers from the vendors to determine how the different performance characteristics relate.
This document and follow-up documents attempt to define a specific set of terminology and tests
that vendors can use to measure and report the performance characteristics of network devices.

March 14, 2005

1

PUB00080R11

Performance Test Terminology for EtherNet/IP Devices
This will provide the user comparable data from different vendors with which to evaluate thesedevices. [1]2Metrics

2.1 Term (Definition Format)
Definition:
The specific definition for the term.
Discussion:
A brief discussion of the term, its application, and any restrictions on themeasurement procedure.
Measurement Units:
The units used to report measurements for this term, if applicable. When possible,
the International System of Units (SI) will used.
Issues:A discussion of issues or conditions that relate to the term.
See Also:
A list of other terms or references that relate to this term or its discussion.
2.2 Action Latency
Definition:
The closed-loop latency of a device to process a command and return a desired
physical output or vice versa.
Discussion:
Action Latency measures the latency for a device to either process a network
command and produce a physical output or process a physical input and produce a
network packet. An example might be for a device to change a digital output fromoff (logical 0) to on (logical 1). The Action Latency would be calculated as the timedifference between the last bit of the command packet received by the DUT (or thelast bit of the last command packet if the command requires more than one packet)
and the physical output being turned on. A similar example might be to measure thetime difference for a physical input to a device to be changed from off to on and thefirst bit of the network packet received by the test equipment.
Measurement Units:
Nano-Seconds (ns)Issues:Action Latency for EtherNet/IP devices may be affected by the Requested Packet
Interval (RPI) or Actual Packet Interval (API). Since data is only sent over the wireat specific rates, the Action Latency for an input device would be affected by the APIof the device. Output devices may be less affected by the API value, since they are

March 14, 2005

2

PUB00080R11

Performance Test Terminology for EtherNet/IP Devices
likely to send the output as quickly as they can after receiving the command. (Tests
will have to be conducted to determine the validity of this for individual devices.)
See Also:
Latency
Response Latency
2.3 Jitter/Variability
Definition:
The amount of change in the measured times for a series of events.
Discussion:
This term is a modifier for other terms, i.e. Throughput Variability or Latency Jitter.
(When used separately, the terms Jitter or Variability usually refer to Throughput
Jitter/Variability.) This measure includes the difference between the minimum andmaximum values as well as the standard deviation of the measured times.
Measurement Units:
Nano-Seconds (ns)Issues:None
See Also:
Reference [1]2.4 Latency
Definition:
The time interval between a message being sent to a device and a corresponding
event occurring.
Discussion:
Latency shows the time delay for a device to actually process data. A device may be
asked to process a command and return some data or may be commanded to perform
some action. The type of action will be determined by the test data required. When
the DUT is receiving a network packet, latency is calculated as the time difference
between the last bit of the message packet received by the DUT (or the last bit of thelast command packet if the command requires more than one packet) and thecorresponding event occurring. When the DUT is sending a network packet, latency
is calculated as the time difference between the event occurring and the first bit of
the message packet sent by the DUT.
Measurement Units:
Nano-Seconds (ns)Issues:None

March 14, 2005

3

PUB00080R11

Performance Test Terminology for EtherNet/IP Devices
See Also:
Action Latency
Response Latency
Reference [1]2.5 Overload Behavior
Definition:
A qualitative description of the behavior of a device in an overload state.
Discussion:
Overload states exist when the device’s internal resources either receive too muchinformation to process or receive enough bad information to cause the device to go
into a state other than its normal run mode.
Data recorded for overload states should:- Describe the device behavior when its resources are exhausted.
- Describe the system management response in an overload state.
- Describe the device recovery from an overload state.
Measurement Units:
None
Issues:None
See Also:
Reference [1]2.6 Response Latency (a.k.a. Response Time)Definition:
The closed-loop latency of a device to process a command and respond to it.
Discussion:
Response Latency measures the latency for a device to process a network command
and return a value from memory. This should represent the minimum latency for adevice to process a command through its network stack. An example might be for adevice to return a value from the EtherNet/IP identity object. Since the identity
object is a specific EtherNet/IP and CIP concept, the command would have to beprocessed through the entire communications stack to the application layer. The
primary performance concern is for real-time I/O response time, so the actual object
and values requested from the device will have to be determined before the test is
conducted. The Response Latency is calculated as the time difference between thelast bit of the command packet leaving the test equipment (or the last bit of the lastcommand packet if the command requires more than one packet) and the first bit of
the response packet received by the test equipment.
Measurement Units:
Nano-Seconds (ns)March 14, 2005

4

PUB00080R11

Performance Test Terminology for EtherNet/IP Devices
Issues:Response Latency for EtherNet/IP devices may be affected by the Requested Packet
Interval (RPI) or Actual Packet Interval (API). Since data is only sent over the wireat specific rates, the Response Latency for an input device would be affected by the
API of the device. (Tests will have to be conducted to determine the validity of this.)
See Also:
Action Latency
Latency
2.7 Throughput
Definition:
The maximum continuous traffic rate that a device can send/receive without
dropping a single packet.
Discussion:
This metric determines the maximum amount of traffic a device is capable ofproducing at one time. The metric should produce a 2D matrix of results for the datacollected during the series of tests conducted.
This metric gives the user an understanding of how much data the device is capableof actually producing or consuming. It shows the difference in the data rates fromthe device in contrast to the data rates of the network interface (10/100 Mbits/s).
Measurement Units:
Frames per second (s) @ a given frame size (# of bits/bytes) or Bits/Bytes per second(s)Issues:
The amount of data produced or consumed by the device will be dependant on thenumber of connections the device is capable of handling and the RPI values for thoseconnections. An example would be a device that is capable of handling 4
connections at a RPI rate of 10 ms and 8 connections at 50 ms all at a frame size of
128 bytes.

[(

) (

)]

Throughput = 4 frames × 1+ 8 frames × 1@(128 bytes frame )
10ms
50ms
Throughput = 560 frames s @ 128 bytes frame = 71680 bytes s = 573440 bits s

See Also:
Reference [1]March 14, 2005

5

PUB00080R11

Performance Test Terminology for EtherNet/IP Devices

3

Types of Equipment

3.1 Term (Definition Format) *Definition
The specific definition for the term.
Discussion:
A brief discussion of the term, its application, and any other pertinent information.
Examples:Examples of the equipment type. These examples may give manufacturer or productnames.
See Also:
List of other terms or references that relate to this term or its discussion.
3.2 Device Under Test (DUT)
Definition:
The device to be tested.
Discussion:
The device should be conformance tested to the EtherNet/IP specification before
being subjected to performance tests.
Examples:Programmable Logic Controller (PLC)
Distributed/Remote Input/Output Block
See Also:
Reference [1]3.3 Network Analyzer
Definition:
A device capable of recording network packets and analyzing the timing of thosepackets.
Discussion:
These devices may be software or hardware based. They may be combined with
Protocol Analyzers to form a more generic solution.
If the network analyzer is software based, the timing of the packets is subject towhatever limits exist for the hardware platform. On a Microsoft Windows platform,
timing resolutions may be as high as 150 ms depending on how the software is*Commercial equipment and software referred to in this document are identified for informational purposes only,and does not imply recommendation of or endorsement by ODVA, the National Institute of Standards andTechnology, HRL, or the other organizations represented in this document, nor does it imply that the products so
identified are the best available for the purpose.

March 14, 2005

6

PUB00080R11

Performance Test Terminology for EtherNet/IP Devices
configured to use the system’s clock. However, most of the software packages are
configured to get near millisecond resolution of their timing signals.
Hardware network analyzers are usually dedicated pieces of hardware specificallydesigned to facilitate performance testing and network troubleshooting. They havegreater capabilities, higher resolution timing, and more flexibility. Many times,
these devices have removable network interface cards that allow them to be used for
networks other than Ethernet. In addition, they usually have the capability to analyze
network packets at higher layers in the ISO/OSI 7-layer Reference Model above
layer 2 (Data Link layer). Some of the devices are capable of full 7 layer operation.
Examples:Frontline Test Equipment’s NetDecoder
Spirent’s SmartBits
Ixia’s 400T System
See Also:
Protocol Analyzer
Traffic Generators
Test Equipment
3.4 Network Infrastructure Equipment
Definition:
Any device associated with connecting the Test Equipment to the DUT over the
network.
Discussion:
Network infrastructure equipment is a necessary piece of the test but should not
impede the test in any way. The network infrastructure equipment should be
performance tested prior to testing any other devices. Any jitter or latencies
associated with the infrastructure equipment can be taken into account whenanalyzing the results from the DUT.
Examples:Ethernet Switch
Ethernet Hub
Network Router
See Also:
Device Under Test (DUT)
Test Equipment
3.5 Protocol Analyzer
Definition:
A device capable of decoding and analyzing network packets.

March 14, 2005

7

PUB00080R11

Performance Test Terminology for EtherNet/IP Devices
Discussion:
Protocol Analyzers are usually software packages capable of decoding all the
protocols in a network packet. They allow the individual portions of the packet to beanalyzed separately and in protocol specific terms instead of trying to interpret thebits or hex-code equivalent. They are often bundled with network analyzer
functionality so they can capture packets and analyze the timing of the individual
packets.
Examples:Frontline Test Equipment’s NetDecoder
Ethereal
See Also:
Network Analyzer
3.6 Test Equipment
Definition:
Any equipment used during the performance tests to stimulate, record, or control thetest.
Discussion:
Any device used during the test that is not specifically the DUT or Network
Infrastructure Equipment is considered Test Equipment. Test equipment may consistof more than one device if a complex test is to be performed.
Examples:Network Analyzer
Traffic Generator
Protocol Analyzer
Programmable Logic Controller (PLC)
Distributed/Remote Input/Output Block
Laptop Computer
See Also:
Traffic Generator
Network Analyzer
Protocol Analyzer
3.7 Traffic Generator
Definition:
A device capable of generating network packets during the test.
Discussion:
Traffic generators will control the flow of traffic during the test. They provide thestimulus that other devices react to. These devices may produce industrial specificnetwork traffic or background traffic used to determine the performance of the DUT.
Before any test is conducted, the performance of the traffic generator must be
understood. If it is a dedicated traffic generating device, the performance is probably
March 14, 2005

8

PUB00080R11

Performance Test Terminology for EtherNet/IP Devices
orders of magnitude better than the DUT, which should not affect the results of thetests. Other devices that do not include such dedicated hardware or are themselves
industrial devices may have performance characteristics that are similar to the DUT.
These performance characteristics need to be accounted for when analyzing theresults of any test conducted.
Examples:Spirent’s SmartBits
Ixia’s 400T System
Programmable Logic Controller (PLC)
Distributed/Remote Input/Output Block
Laptop Computer
See Also:
Test Equipment

4

Other Terms

4.1 Unloaded
The test is conducted with no background traffic on the network.
4.2 Loaded
The test is conducted with background applied to the network.
4.3 Within-Spec
The test is conducted within the manufacturer’s specifications.
4.4 Outside-Spec
The test is conducted outside the manufacturer’s specifications.

March 14, 2005

9

PUB00080R11

Performance Test Terminology for EtherNet/IP Devices

5

EtherNet/IP Performance Workgroup Members
•
•
•
•
•
•

6

Brian Batke, Rockwell Automation, babatke@ra.rockwell.com
James Gilsinn, National Institute of Standards & Technology (NIST),
james.gilsinn@nist.gov
Kevin Martin, HRL Laboratories, martin@hrl.com
Anatoly Moldovansky, Rockwell Automation, amoldovansky@ra.rockwell.com
Mark Weisenborn, Frontline Test Equipment, mweisenborn@fte.com
Gary Workman, General Motors, gary.c.workman@gm.com

Acknowledgments

The EtherNet/IP Performance Workgroup would like to thank the following additional people
who provided input to this document:• Scott Bradner, Harvard University

7

References[1] Bradner, S. ed., “Benchmarking Terminology for Network Interconnection Devices,” RFC
1242, July 1991, Internet Engineering Task Force (IETF),
http://www.ietf.org/rfc/rfc1242.txt .
[2] Bradner, S., McQuaid, J., ed., “Benchmarking Methodology for Network Interconnection
Devices,” RFC 2544, March 1999, Internet Engineering Task Force (IETF),
http://www.ietf.org/rfc/rfc2544.txt .
[3] EtherNet/IP Specification, version 1.0, June 5, 2001, Open DeviceNet Vendor Association(ODVA), http://www.odva.org/ .

March 14, 2005

10

PUB00080R11