Whitepaper

Resource Efficiency Testing
Step by Step Instructions
OPC Foundation
May 16, 2014www.opcfoundation.org

Contents
Executive Summary .......................................................................................................................................................................... 3
Intended Audience ....................................................................................................................................................................... 3
OPC Foundation Membership................................................................................................................................................... 3Introduction ....................................................................................................................................................................................... 4
OPC Server Testing ..................................................................................................................................................................... 4
Multiple data-Sources .............................................................................................................................................................. 5
Multiple OPC Clients .............................................................................................................................................................. 6
Intermittent Connection Simulation ..................................................................................................................................... 7
OPC Client Testing ...................................................................................................................................................................... 8
Multiple OPC Servers ............................................................................................................................................................. 8Intermittent connection simulation ...................................................................................................................................... 9
Performance Monitoring ........................................................................................................................................................... 10
Windows Performance Monitor .......................................................................................................................................... 10Linux Performance monitor ................................................................................................................................................ 13
Windows Batch File ................................................................................................................................................................... 14

2|Pagewww.opcfoundation.org
www.opcfoundation.org

Executive Summary1
All OPC products can be certified (Clients, Servers, or both) for Unified Architecture, OPC Classic, or both.
Most products that fail certification testing do so because of resource leaks or crashes found in the 36-hour resource
efficiency testing. This document provides step-by-step instructions for you to replicate the resource efficiency testing
conducted in the OPC Foundation Certification Test Lab to help you avoid this problem.
This paper covers the following:Configuring your OPC Client/ServerConfiguring products to connect to your Client/ServerConfiguring the environment for simulating network communications outagesObtaining and analyzing performance metrics

Vendors need to perform this test on their Client/Server products before submitting to the test lab.

Intended Audience
This document contains technical information and is intended for software developers and QA personnel.

OPC Foundation MembershipThis document is primarily intended for members of the OPC Foundation.

Copyright © 2014 OPC Foundation. All rights reserved.
No part of this publication may be reproduced, transmitted, transcribed, stored in a retrieval system or translated into any language, in any form or by anymeans, electronic, mechanical, photocopying, recording, or otherwise, without prior written permission from the OPC Foundation.
All copyright, confidential information, patents, design rights, and all other intellectual property rights of whatsoever nature contained herein are and shallremain the sole and exclusive property of the OPC Foundation.
The information furnished herein is believed to be accurate and reliable. However, no responsibility is assumed by the OPC Foundation for its use or for
any infringements of patents or other rights of third parties resulting from its use. All other trademarks are the property of their respective owners.

3|Pagewww.opcfoundation.org
www.opcfoundation.org

Introduction
A 36-hour resource-efficiency test will combine stress-testing with robustness and recovery. The goal is to ensure
products are reliable, able to automatically recover from communications loss, and are efficient with system resources(no memory-leaks etc.)
The general testing principle is the same for OPC Clients and Servers. However, some subtle differences require us totreat them differently. Please review the applicable sub-section as applicable to your product.

OPC Server Testing
You will need the following:
Multiple data-sources (PLCs, Modbus Simulators etc. if supported)Multiple OPC Clients (5, by minimum)

Windows Performance Monitor (See section Performance Monitor on page 9)

Intermittent network connection (simulation)

4|Pagewww.opcfoundation.org
www.opcfoundation.org

Multiple data-Sources
Note: you can skip this section if your server does not support Device connections


The Test lab uses “PeakHMI TCP Slave” / MODBUS Ethernet PLC simulator.

Multiple data-sources are needed (at least 2 remote data sources) as shown here:
Data Source A

PLC/Modbus
Simulator

Data Source B

PLC/ModbusSimulator

OPC SERVER

Data Source C

PLC/Modbus
Simulator

5|Pagewww.opcfoundation.org
www.opcfoundation.org

Multiple OPC Clients

A minimum of 5 OPC clients’ are needed:
OPC Client 1

OPC Client 2

OPC Client 5

OPC SERVEROPC Client 3

OPC Client 4

Each client must consume a minimum of 1000 items/nodes from each Server covering all devices.

If the server does not support device connections, configure the client with all the available items (internal/
simulation tags) in the serverA list of typical Classic/UA Clients used in the test lab is shown here:

Classic Iconics

UAIndusoft-

Softing Demo

OPCF DA

Kepware-OPC Quick

Genesis64

WebStudio

ClientSample Client

ClientV 10.81+FH2

V 7.1

V 4.30.1 Build 2692

V 334

V 5.11.250.0

Iconics

Indusoft-

HB-Soft Comet

OPCF UA

Unified Automation-

Genesis64

WebStudio

UA Browser

Sample Client

UA Expert

V 10.81+FH2

V 7.1

V 1.0.3

V 1.1.334.0

V 1.2.2

Please contact the vendors directly to obtain the demo version of their software.

6|Pagewww.opcfoundation.org
www.opcfoundation.org

Intermittent Connection Simulation
Configure the performance monitor tool to record the data as mentioned in Performance Monitor section on page 9.

Let the system run for an hour without any interruption to get a base-line of resource consumption.

Run the batch file (see section Batch File on page 13) to start simulating intermittent network connectivity.

There are two different sets of configuration for the intermittent connection. Please use the configuration
applicable for your application:
Configuration 1: When the server supports device connections:
Data Source A

PLC/Modbus

Intermittent connection toData Source A

OPC Client 1

OPC Client 2
Data Source B

OPC SERVER

PLC/
ModbusSimulator

OPC Client 3
Intermittent Connection to Client 4

Data Source C

OPC Client 4

PLC/
ModbusSimulator

OPC Client 5

Configuration 2: When the server is not connected to an external data source:
OPC Client 2

OPC Client 1

OPC Client 3

OPC SERVER

Intermittent Connection to Client 4
Data Source C

OPC Client 4

OPC Client 5

7|Pagewww.opcfoundation.org
www.opcfoundation.org

OPC Client Testing
You need to have the following in order to perform the resource efficiency test on a Client.
Multiple OPC Servers (5, by minimum)

Windows Performance Monitor (See section Performance Monitor on page 9)

Intermittent connection (simulation)

Multiple OPC Servers
A minimum of 5 Servers need to be used for the efficiency test:OPC Server
1

OPC Server
2

OPC Server 5

OPC Client

OPC Server 3

OPC Server 4

Connect the client to each server as shown and configure 1000 items per OPC server connection.

A list of typical Classic/ UA servers used in the test lab are shown in the below table

Classic

UAKepware

Matrikon

Softing-OPC Toolbox Demo

OPCF Sample

Cyberlogics- MBX OPC Server

KepServerEXUCSServerServerV 7.1

V 5.11

V 2.1.0.1602

V 4.30.1 Build 2692

V 1.0.01

KepwareMatrikon

Softing – OPC UA C++ Test

OPCF UA Reference

UnifiedAutomation- UA Demo

KepServerEX

UCSServerServerServer

V 5.11

V 2.1.0.1602

V 5.20.0.8536

V 1.1.334.0

V 1.3.2.200

8|Pagewww.opcfoundation.org
www.opcfoundation.org

Intermittent connection simulation
Configure the Client to connect to multiple Servers, as shown here:
Data Source A

OPC Server
2

OPC Server
1

OPC Server
3

OPC Client
Intermittent Connection to Server 5

Data Source COPC Server
4

OPC Server5

Configure the performance monitor tool to record the data as mentioned in Performance Monitor section on page 9.

Let the system run for an hour without any interruption to get a baseline of resource consumption.

Run the batch file (See section Batch File on page 13) to enable the intermittent connection simulation

9|Pagewww.opcfoundation.org
www.opcfoundation.org

Performance Monitoring
Windows Performance Monitor
Windows performance monitor is used to record performance data and is then converted to charts for reporting.
1. Open Performance monitor by typing “Perfmon.msc” in windows Start menu search.
2. Create a new collector set in the “Data Collector Sets/ User Defined” section of the performance monitor:

3. Select “Create Manually” option on the screen and select Performance counter under “Create data logs” option
on the next screen:

10 | P a g e
www.opcfoundation.org
www.opcfoundation.org

4. Select the performance counters in the next screen. Click “ Add” to include the counters that needs to bemonitored:

5. Here is the list of counters needed for the test:

Processor
Processooooooooo

Counters

Instances% Processor Time
% User Time
% Processor Time% User Time
Elapsed Time
Handle CountPriority BaseThread Count
Working Set_Total
All related Instances of yourOpc Server/ OPC Client

6. Add the counters listed above, as shown here:

11 | P a g e
www.opcfoundation.org
www.opcfoundation.org

7. Set the Sampling Interval, e.g. 15-seconds, and then follow the steps to create the collector set:

8. Right click on the Data collector set and choose properties to set the Stop Condition:

9. To record the data for 36 hours specify the “Overall duration” as 36 hours in the “Stop Condition” and click “OK”:

12 | P a g e
www.opcfoundation.org
www.opcfoundation.org

10. Right click on the DataCollector01 to see the DataCollector01 properties.

11. Select the log format and click "ok”.
12. Start the collector once the OPC Server/ OPC Client is ready to start the 36-hour test.

Linux Performance monitor
Linux provides the top command to monitor the performance during the efficiency test.

Below is the screenshot of how the top command looks in the terminal:Use the following command in the terminal to view the output:./top_pname.sh <OPC Server/ OPC Client>Use the following command to redirect the output to a csv file:./top_pname.sh <OPC Server/ OPC Client> &> <Filename.csv>

Use Ctrl+c to terminate the process.

13 | P a g e
www.opcfoundation.org
www.opcfoundation.org

Linux TOP Shell Script
The following script will remove unwanted information from the TOP output:#!/bin/bash
if [ $# = 0 ]thenecho "arg error : ?"echo "top_pname.sh <exename>"
exit
else
echo ""
fi
h1="Time,Mem total,Mem used,Mem free,Mem buffers,PID,USER,PR,NI,VIRT,RES,SHR,S,%CPU,%MEM,TIME+,COMMAND"
pname=$1
echo -e "$h1"
print_vals(){l1=`/usr/bin/top -bn1p $(pidof $pname)|sed -n -e 1p|awk '{print $3,","}'`
l2=`/usr/bin/top -bn1p $(pidof $pname)|sed -n -e 4p|awk '{print $2,",",$4,",",$6,",",$8,","}'`
l3=`/usr/bin/top -bn1p $(pidof $pname)|sed -n -e 8p|awk '{print$1,",",$2,",",$3,",",$4,",",$5,",",$6,",",$7,",",$8,",",$9,",",$10,",",$11,",",$12}'`echo -e "$l1$l2$l3"}
while true
do print_vals
sleep 4doneWindows Batch File
Test lab uses a batch file to enable/disable the “Local Area Connection” network adapter in a windows environment.
Copy the below text into a “.bat” file and run it using Administrator/elevated permission to enable the intermittent
connection.
@echo off
SET DTIME = 120
CLS
ECHO Delay time is: %DTIME%ECHO Network Connection Enabler/Disabler started at: %date% %time%:start
ECHO Disabling network adapter at %time%
netsh interface set interface "Local Area Connection" DISABLED
timeout /t %DTIME%ECHO Enabling network adapter at %date% %time%
netsh interface set interface "Local Area Connection" ENABLED
choice /c:yn /m "Repeat network adapter recycle (y/n)?" /t %DTIME% /d y
ECHO %ERRORLEVEL%IF ERRORLEVEL 2 GOTO end
IF ERRORLEVEL 1 GOTO start:end
ECHO.
ECHO Network adapter recycle complete at %date% %time%

You can change the time period of the intermittent connection by varying DTIME in the above code.

14 | P a g e
www.opcfoundation.org
www.opcfoundation.org

Contact Information:
OPC Foundation
16101 N. 82nd Street, Suite 3B
Scottsdale, AZ 85260
USATel: 480-483-6644
Fax: 480-483-1830
Email: compliance@opcfoundation.org

15 | P a g e
www.opcfoundation.org
www.opcfoundation.org

Page Intentionally Left Blank

16 | P a g e
www.opcfoundation.org
www.opcfoundation.org

Revision History
VersionDateAuthor1

May 16, 2014

DT

ChangeInitial version.

17 | P a g e
www.opcfoundation.org
www.opcfoundation.org