OPC UA versus ROS, DDS, and MQTT:
Performance Evaluation of Industry 4.0 Protocols
Stefan Profanter, Ayhun Tekat, Kirill Dorofeev, Markus Rickert, Alois Knoll

Abstract—Ethernet-based protocols are getting more and more
important for Industry 4.0 and the Internet of Things. In
this paper, we compare the features, package overhead, andperformance of some of the most important protocols in thisarea. First, we present a general feature comparison of OPC UA,
ROS, DDS, and MQTT, followed by a more detailed wire
protocol evaluation, which gives an overview over the protocol
overhead for establishing a connection and sending data. In theperformance tests we evaluate open-source implementations ofthese protocols by measuring the round trip time of messages indifferent system states: idle, high CPU load, and high network
load. The performance analysis concludes with a test measuring
the round trip time for 500 nodes on the same host.

I. I NTRODUCTION
Industry 4.0 is a current trend in industrial automation
which is often referred to as the fourth industrial revolution.
Its main goal is to modernize the way manufacturing andproduction work by easing the integration of devices and byimproving the communication between all the devices on the
shop floor. Components from different manufacturers need tocommunicate in a common language by using standardized
communication protocols. Current manufacturing systems use
are a lot of vendor-dependent field bus protocols that handle
this task. As a result, an integrated system can suffer frommany different communication paradigms which reduce thesystem’s overall flexibility and adaptability.
In this paper, we evaluate the performance and resource
usage of some of the most common protocols in the area ofindustrial automation and Internet of Things (IoT): OPC UA,
DDS, ROS, MQTT. We chose these protocols since they
are commonly used in our research projects. Other possible
candidates for further evaluation could be CoAP, XMPP, and
AMQP.
According to [1], these protocols are among the most important ones to be considered in IoT applications. All presented
protocols try to deal with the problem described above by
providing a standardized, open, and manufacturer-independent
protocol. With new and improved real-time features added
to Ethernet-based protocols over the last few years, they areslowly replacing conventional and proprietary field bus communication [2]. One of the major advantages of Ethernet-based
protocols is the well known standard which is implemented inmany different operating systems and microcontrollers, thus
resulting in lower hardware costs. Field bus protocols on the
S. Profanter, K. Dorofeev, M. Rickert, A. Knoll are with fortiss, An-Institut
Technische Universität München, Munich, Germany.
A. Tekat is with Technische Universität München, Munich, Germany.

other hand are either implemented as closed source or only on
specific hardware in combination with licensing fees.
Today, there is a growing demand in the industry for
distributed applications, where multiple devices from different
vendors control different types of objects [3]. Selecting anappropriate middleware, a software layer located between
applications and operating system, becomes one of the most
crucial tasks to simplify and speed up a development of
distributed systems [4]. As these industrial systems usuallyrequire hard real-time constraints, the performance of a middleware layer becomes a critical issue when designing such asystem.
This paper is split into multiple sections: Section II gives
a short overview on related work. The main concepts andapplication domains of the different middlewares are describedin Section III followed by a more detailed evaluation of the
package overhead of the underlying protocol in Section IV.
Section V introduces our architecture for performance testing
and lists some of the tests we used for comparison. Thissection also describes some of the tools used to test the
protocols under heavy system and network load. In Section VI
the most important results of the comparison are shown anddiscussed. In Section VII we give a summary of the results
and an outlook on future work.
II. R ELATED W ORK
Numerous performance tests have already been conductedin various domains. In the IoT domain, where a usual scenario
includes resource-constrained devices communicating witheach other over low-bandwith or probably unreliable wirelessnetworks, the question of the communication protocol’s performance evaluation is seen as a crucial one. [5] compares
bandwidth consumption and latency of most common IoTprotocols, including MQTT, CoAP, and DDS. [6] evaluates
CoAP, MQTT, MQTT-SN, TCP, and Websockets, also comparing energy performance and CPU power consumption foreach of the protocols. A performance comparison of data usage
and the time spent to send and receive messages for MQTT
and OPC UA can be found in [7].
A survey of supported communication paradigms between
OPC UA and DDS is presented in [8], with a focus on how
both protocols can be run in hybrid deployments.
Some papers present performance evaluations for a specificprotocol: [9] shows how an OPC UA server performs in afield device, measuring response times, memory, and CPU
utilization of an OPC UA Server running on an Altera Cyclone I FPGA. [10] evaluates the performance of different

OPC UA features, such as security, binary transport, and SOAP
transport, while [11] gives an overview over the features ofdifferent OPC UA implementations, including open62541, an
open source C (C99) implementation of OPC UA used in our
tests. A more detailed comparison between ROS and ROS2,
especially considering different DDS implementations such as
Connext, OpenSplice, and FastRTPS is evaluated in [12]. Theyshow that using DDS for ROS2 gives a significant performance
improvement compared to ROS1.
As of now previous related work mainly focuses on thecomparison of one specific middleware, or on the performance
evaluation between a subset of the protocols presented inthis paper. For our evaluation we focus on some of themost used protocols in Industry 4.0 and present a side-byside comparison for OPC UA, DDS, MQTT and ROS. We
also evaluate the performance of multiple nodes on the same
host, showing significant differences between the presented
protocols.

itself. In combination with, e.g., Time Sensitive Networking (TSN) on layer 2, OPC UA Publish/Subscribe or MQTT
can also support additional QoS principles.
DDS (Data Distribution Service) is a data-centric publishsubscribe middleware for highly dynamic distributed systems.
It is standardized by the Object Management Group (OMG)2 .
Compared to OPC UA, DDS is more data centric: data is published into the DDS domain and subscribers can subscribe todata from that domain without knowing where the information
came from or how it is structured, as the information package
already describes itself. In OPC UA every node is describedin the address space of the server. A client can query thisinformation and use it together with the received data. DDS
provides an extensive set of QoS parameters, e.g., durability,
lifespan, presentation, reliability, and deadlines3 . Similar to
OPC UA, DDS also supports dynamic discovery without acentral instance. According to the OMG’s website, DDS is oneof many protocols used in industry sectors such as railway
networks, air traffic control, smart energy, medical services,military and aerospace, and industrial automation4 .
ROS (Robot Operating System) is an open-source software framework originally developed by Willow Garage andsupported by the Open Source Robotics Foundation (OSRF)
together with a large community5 . Its main target are research
institutes in various areas with a focus on encouraging collaborative robotics software development within a large ecosystem.
ROS industrial is trying to extend ROS by capabilities for theindustrial manufacturing area. The successor of ROS, ROS2
is currently under heavy development. Instead of using the
proprietary message formats from ROS, it is built on top of
DDS. As we also include DDS in our evaluation, specificallyeProsima Fast RTPS as the default DDS implementation for
ROS2, the results from this evaluation can also be transferred
to ROS2. A more detailed comparison between DDS and
ROS2 and the different DDS implementations will be conducted at a later stage.
MQTT (Message Queuing Telemetry Transport) declares
itself as an extremely lightweight publish/subscribe machineto-machine and Internet of Things connectivity protocol6 .
It is an open message protocol which mainly focuses ona small code footprint and low network bandwidth usage,
while handling high latency or bad network connections.
Communication between sensors via satellite link is therefore
one of its use cases. Since 2013, MQTT is standardized by theOrganization for the Advancement of Structured Information
Standards (OASIS) as the protocol for the Internet of Things7 .
MQTT uses the concept of an MQTT-Server, also known
as a broker, which holds the complete data of all of its

III. M IDDLEWARE C OMPARISON
A middleware provides a communication abstraction layer
between different components in a distributed system. It isable to bridge the gap between individual subsystems runningon different hardware platforms, operating systems, and programming languages. The development of a distributed systemcomposed of heterogeneous devices from various vendors
becomes a complicated task that, among others, must ensure communication between numerous devices. Middlewares
enable the developer to manage the complexity of this task
by introducing an intermediate software layer that provides
a high-level API with an abstraction of the low-level details
related to communication and application distribution.
OPC UA (Open Platform Communications Unified Architecture) is a service-oriented machine-to-machine communication protocol mainly used in industrial automation anddefined in the IEC 62541 specification. Its main goals are toprovide a cross-platform communication protocol while using
an information model to describe the transferred data. The
various features and components of OPC UA are describedin different specification parts released and publicly available
by the OPC Foundation1 . OPC UA is mainly driven by theEuropean manufacturing industry and thus gains more andmore importance in that area while also becoming one of
the more important protocols worldwide. The major strengthof OPC UA is the semantic description of the address spacemodel together with various companion specifications which
extend the basic semantic descriptions for various domains
like PLCopen, robotics, or computer vision. OPC UA recently
released a Publish/Subscribe specification. It enriches OPC UA
with a Publish/Subscribe concept similar to DDS, whereservers can publish data and clients can subscribe to this data,
independent of the data origin. OPC UA Publish/Subscribe
does not include any quality of service (QoS) mechanisms

2 https://portals.omg.org/dds/
3 http://download.prismtech.com/docs/Vortex/html/ospl/DDSTutorial/qos.
html
4 http://portals.omg.org/dds/who-is-using-dds-2/
5 http://www.ros.org/
6 http://mqtt.org/
7 https://www.oasis-open.org/news/announcements/
mqtt-version-3-1-1-becomes-an-oasis-standard

1 https://opcfoundation.org/about/opc-technologies/opc-ua/

2

TABLE I: Comparison of the protocols used in the evaluationand their main features.

communication partners. With this concept, small devices
simply report data to the Broker and do not need to store thedata themselves. The devices can also be controlled throughthe broker. The published data is grouped hierarchically similar
to DDS and multiple devices can publish to the same topic.
MQTT supports some basic QoS to define if and how often
a message should be re-sent until it is acknowledged by thebroker and if the server should cache topic data.
OPC UA’s device-centric approach focuses on device interoperability, where devices may be used in different systems,
while DDS focuses on software integration mainly in a single
type of system. This may be one of the reasons why OPC UA
is more likely to be adopted in the manufacturing industry:a typical production line is built up using many different
types of systems, whereas systems like air traffic control comefrom one vendor and all system components are fine-tuned
to each other. The focus of ROS is hardware abstraction,
the collaboration between different research groups, and reusability of software components. MQTT’s focus is on small
code footprint and low network bandwidth, which is one of
the reasons MQTT does not offer as many extra features such
as Remote Procedure Call (RPC) or Discovery.
Table I gives an overview of the basic features foreach of the presented middleware protocols. Every protocol can be used on top of TCP, whereas MQTT does notsupport UDP communication. In ROS, all clients have tosupport TCP (TCPROS), optionally they can also support
UDP (UDPROS) which is currently only implemented byroscpp. Some DDS implementations also support shared memory (SHM): if multiple nodes are running on the same host, thedata is exchanged via shared memory instead of the loopback
network adapter. Additionally, DDS standardizes the user API,
which means that in theory it should be possible to exchange
different DDS implementations without changing the sourcecode. Practically this is hard to achieve since every DDS
implementation adds additional specific methods to configure
the stack.
All presented middlewares support the Publish/Subscribe(Pub/Sub) pattern. Remote Procedure Calls (RPC) are natively
supported by OPC UA and ROS. RPC for DDS is standardized
but not implemented in many implementations. MQTT does
not support RPC at all.
OPC UA uses the address model to provide data to theclients. This address model includes a semantic annotationof the data which allows clients to automatically infer themeaning of specific data values without previous knowledge.
In ROS, DDS, and MQTT, data values are published on
specific topics, which means that clients have to know the topic
they need to subscribe to in advance. DDS additionally uses
domains and partitions for the information scope. The only
way to infer some additional meaning is to use the topic name,which is not as powerful as it is in OPC UA. In OPC UA,
every node can have multiple typed references to other nodes,
which is similar to a triple storage database as it is commonlyused for semantic modeling.
Security is an important feature for every middleware which

OPC UAROS

TCP, UDP

TCP, UDP

QoS

RPC,
Pub/SubNo

RPC,
Pub/Sub
No

TCP, UDP,
SHM(RPC),
Pub/Sub
Yes

Authentication
Encryption
Std. API
Semantic Data

User, PKI
Yes
No
Yes

(Mac)
No
No
No

PKI
Yes
Yes
No

Communication
Patterns

DDS

MQTT
TCP
Pub/Sub
Yes
User, PKI
Yes
No
No

is used in the domain of industrial automation and IoT.
Table I shows that OPC UA and MQTT support authentication via username and password or by using a private key
infrastructure (PKI). DDS only supports PKI authentication,
ROS only supports authentication via MAC Address using
third-party packages. ROS is also the only protocol which
does not support application layer encryption. The additional
efforts (network overhead, CPU load) for encrypting ROS
communication are evaluated in [13].
In our tests, we used the following implementations of each
protocol, all of which are available as open source software.
Commercial stacks exist for OPC UA, DDS, and MQTT.
open62541: OPC UA implementation. License MPL2.0.
Version 0.3-rc2 (for Pub/Sub: current master df58cf8)• ROS C++: ROS implementation. License BSD. Version
Kinetic Kame
• eProsima Fast RTPS: DDS implementation. License
Apache 2, Version 1.6.0• Eclipse Paho MQTT C: MQTT implementation. License
EPL1.0. Version 1.2.1•

IV. PACKAGE OVERHEAD
As a first step, we compared the different protocols froma theoretical point of view by looking at the overhead foreach transmitted package. Every protocol needs some kind of
package header for each data it transmits as a payload of the
TCP package so the remote side knows which kind of data
is sent. This package header adds additional overhead to eachdata message transmitted on the network and thus limits thepossible maximal bandwidth which can be reached.
To compare the protocol headers, we had a look at every
protocol specification and compared the protocol header fordifferent payload sizes of 0 byte, 100 bytes, 1000 bytes and
10 000 bytes as shown in Table II. The table shows the protocol
payload size that is passed from the middleware (OSI layer 5,
Session) to the UDP/TCP transport (OSI layer 4). Note, thatthese values do not include TCP or UDP header sizes andare therefore not influenced by the Ethernet frame size, as the
middleware payload size is independent from this if the frames

3

TABLE II: Package size in bytes transmitted as TCP/UDP
payload for each protocol with given payload in bytes. The
difference between the final package size and the protocol payload shows the protocol overhead. The last column summarizes
the total bytes for the connection setup in the evaluation.

are split at a lower layer. Additionally, we verified the values
using Wireshark8 .
For OPC UA, we evaluated the overhead for a variable
write request from the client to the server without encryption, which would add additional overhead. In OPC UA,
the client first needs to open a secure channel (OpenSecure
ChannelRequest, 132 bytes), get the available endpoints (Get
EndpointsRequest, 93 bytes), create a session (CreateSession
Request, 138 bytes, depending on the hostname, here
localhost:4840), and then activate the session (Activate
SessionRequest, 137 bytes, depending on the identity token
length, here open62541-anonymous-policy), before awrite request can be sent. At the end, the client should close
the session (CloseSessionRequest, 75 bytes) and the securechannel (CloseSecureChannelRequest, 57 bytes). The totalsum of these TCP payload bytes for opening and closing theconnection in OPC UA without the WriteRequest sums up to
132 + 93 + 138 + 137 + 75 + 57 = 632 bytes.
MQTT is using its own lightweight TCP-based binary protocol. The payload of the connect command to the MQTT broker
depends on the publisher name. In our case, we only used one
character for this, which resulted in a payload of 15 bytes
for the connect command and 2 bytes for the disconnect
request. The size of the publish message also depends on
the topic name. With one character as the topic name, theoverall overhead for connection handling in MQTT sums up
to 15 + 2 = 17 bytes.
In ROS, the first step is to start the ROS core on one
machine. Every started ROS node requires the ROS core’s
IP address. On startup, the node sends XMLRPC requests tothe core to exchange information about the current system
state and any subscribers and publishers for the node’s topics.
Only if there is a matching subscriber/publisher pair, thetwo nodes are connected via a separate TCP connection andthe TCPROS protocol specifically for this subscription. On
shutdown, the ROS node unregisters itself from the core via
XMLRPC9 . In our test, the overall sum of the TCP package’s
payload size for outgoing XMLRPC requests—connecting
and exchanging information with the core (5693 bytes) anddisconnecting at the end (3046 bytes)—was 8739 bytes. This
does not include the TCPROS messages that are exchanged
between the nodes. On the separate TCPROS channel, thesubscriber node connects to the publisher node with additional
information about the desired topic. The publisher node thenreturns its own publishing info of size 176 bytes for setting up
the connection. If there is no subscriber for a specific topic,
the data published by the node will not be sent through the
network at all. The final sum of TCP payload bytes for opening
and closing the connection in ROS without published data is8739 + 176 = 8915 bytes.
The discovery process in DDS depends on the used implementation stack. For our tests, we used the eProsima Fast
RTPS implementation. It uses two phases to discover other

Payload

0

100

1000

10 000Connection

OPC UA C/S
MQTT
ROS
DDS

96
5
8
88

196
105
108
188

1096
1006
1008
1088

10 09610 00610 00810 088

632
17
8915
8348

nodes in the network: in the participant discovery phase,the node sends out multicast messages to discover otherparticipants in the network and periodically sends heartbeat
packages. When two nodes have found each other, they exchange information about published and provided topics withthe other endpoint in the endpoint discovery phase. During theendpoint discovery phase, the subscriber and publisher haveto acknowledge the other side. Only then can they exchange
information. During this whole setup process in our test, theserver sent a total amount of 8348 bytes, encapsulated in
RTPS (Real-Time Publish Subscribe) packages and delivered
as UDP payload. In OpenDDS, another open source DDS
implementation used in our tests, the discovery process is
implemented by using a central discovery repository. Upon
start, the publisher uses IIOP (Internet Inter-ORB Protocol) to
exchange information with the directory service (add domain
participant, assert topic, add subscription, add publication) on
its own publish data and on the current subscribers. This procedure sends a total amount of 13 TCP messages with a sumof 4522 bytes. Unregistering (remove publication/subscription,
remove topic, remove domain participant) results in another1580 outbound bytes. The total sum of outbound bytes without
exchanging any messages in OpenDDS is 6102. When amessage is published, DDS uses the RTPS protocol, which issent through UDP unicast in our setting. The RTPS protocol
can also be configured to use UDP multicast or TCP.
Overall, the evaluation shows that MQTT not only adds
the smallest amount of additional data during the connection
initialization, it also has the smallest overhead when sendingout data messages, as can be seen in Table II. OPC UA with
Client/Server is in second place when looking at the connection setup overhead, but comes last regarding the message
overhead. ROS requires the largest amount of extra bytes forsetting up the connection between two nodes. This is dueto the fact that it uses XMLRPC, which is sending noncompressed XML text to the ROS core. On the other hand,
ROS is very efficient when looking at the payload overhead.
DDS has a slightly smaller payload overhead compared to
OPC UA and requires a few bytes less than ROS for setting
up the connection between publisher and subscriber. It can also
be seen, that the protocol overhead is almost constant for all
investigated protocols, independent of the payload size. MQTT
requires an additional byte for messages larger than 127 bytes:
the packet length header field in MQTT is a variable between 1

8 https://www.wireshark.org/
9 http://wiki.ros.org/ROS/Technical%20Overview

4and 4 bytes (7 bits plus continuation bit) and is adapted to thecorresponding payload size.
One important reason for the significant differences between
MQTT and ROS compared to OPC UA and DDS lies in thefact, that MQTT and ROS create a dedicated TCP connection
for every subscriber and publisher pair. They do not need toinclude additional information about the published data in thetransmitted package. In OPC UA and DDS different kind ofdata can be sent in the same connection, therefore the payload
needs additional identification data.
DDS and OPC UA include the possibility to add additional
diagnostics information for every transmitted data package,
while for the lightweight protocols (ROS, MQTT) this dataneeds to be collected separately.

requests are made continuously without waiting in between
until 5000 requests are sent. In the next step, the payload size isdoubled, starting from 2 bytes and going up to 32 768 bytes in
one request. This process is repeated for both test modes (ACKand echo).
In addition to the ACK and echo tests, we also tested theresting RTT. This is the time it takes the server to respond to arequest when it is in a resting state. Within 30 seconds, a client
waits for a random amount of time between 0 and 3 seconds,
sends a single request in ACK mode, and then measures the
RTT.
To evaluate the performance in non-ideal scenarios, whereother processes on the end-device cause high CPU load or the
network is working at full capacity, we used third-party tools togenerate extensive network traffic and CPU usage. For network
traffic, we used the Ostinato11 traffic generator, which cancreate sequential and interleaved streams of different protocols
at different rates to generate artificial network load [14]. This
results in a full capacity utilization of the network stack, but
the test should still succeed. The stress12 tool was used tocreate artificial CPU load. It is designed to apply configurable
CPU, memory, I/O, and disk stress on the system.
This setup results in 7 tests per middleware: echo and ACK
when the system is in idle state, echo and ACK for CPU load,
echo and ACK for network load, and the resting RTT test.
As a final step, we also evaluated the performance of the
middleware when 500 server instances of the same protocol
are started at the same time on the server machine. The clientmachine will then simultaneously send 10 packages with apayload of 10 240 bytes to 10 nodes running on the servermachine. These 10 nodes will then immediately forward thereceived package to the next node on the same machine. The
last step is repeated 50 times, which results in 10 sending
streams running in parallel. In the last node, the messages
are sent back to the client, which will then measure thecomplete RTT of all 10 messages. The whole process isrepeated 100 times to average the results. This test shows thesuitability of the protocol for the use case, where a lot of nodes
are running on the same host and data is exchanged primarily
between these nodes. This setup is often used with ROS, e.g.,where a ROS node is reading a camera image, another nodesubscribes to this image, applies preprocessing, and outputs
the result in a new topic.
In the next section, we evaluate some of the most interesting
test results and present the performance of each middleware
implementation.

V. T ESTING S ETUP
Our testing environment is composed of the following
components: a client machine is connected to the tested Linuxserver machine acting as a remote host for the latency andthroughput tests via a gigabit network switch (TP-Link TLSG1024DE). Both Linux machines have the following specification: Intel i7-8700K CPU with 3.70 GHz and 64 GB RAM
running on Ubuntu 16.04.4 with Preempt-RT Kernel 4.14.59rt37. We used a real-time kernel to ensure high performanceand reproducibility on the tests. A basic evaluation of the setup
results in an average round-trip time (RTT) ping of 0.35 ms
between the two Linux machines. A measurement of the
bandwidth using the Linux iperf command resulted in a valueof up to 724 Mbit/s.
The evaluated middlewares have different feature sets as
described in Section III. To compare the performances of
OPC UA, DDS, ROS, and MQTT, we used the request-reply
pattern as it can be implemented in all protocols and is an
important and often-used feature. In this setup, the server is
responsible for providing the methods a client can call. The
client is responsible for calling the methods on the server andmeasuring the server’s performance. For MQTT, DDS, andOPC UA Pub/Sub, we implemented a request-reply patternusing two topics: one topic where the data is sent and anothertopic where the response is returned. All the tests are designed
to be reproducible with a single command, which also logs
the timing values in a file. Our test suite developed for thisevaluation is open source and available on GitHub10 .
To compare the performance and system load during thetests, we employed the following metrics: CPU usage, RAM
usage, messages per second, and round-trip time (RTT).
For the tests, we used two distinct modes: the acknowledge (ACK) and echo mode. In the ACK mode, a client
sends a request and waits for a simple acknowledgment
message (1 byte) from a server. This mode can be used tomeasure the response time for a single message. In the echomode, a server replies with the same data bytes, which isuseful to measure the throughput of a middleware. Echo andACK modes are executed one after the other. For each mode,

VI. P ERFORMANCE E VALUATIONThis section presents the results of our tests as described in
Section V. Note, that for conducting the tests we are using themost common C/C++ open-source implementations of every
protocol. Therefore, the results may also be influenced by thecorresponding performance of the implementation itself andmay not apply to other implementations of the same protocol.
11 https://ostinato.org/

10 https://github.com/Pro/middleware evaluation

12 https://people.seas.harvard.edu/∼apw/stress/5(a) Echo RTT for different data message size using OPC UA
Client/Server via TCP on Idle, CPU Load, and Network Load.

(b) Echo RTT for different data message size using OPC UA
Publish/Subscribe via UDP on Idle, CPU Load, and Network Load.

(c) Echo RTT for different data message size using MQTT on Idle,
CPU Load, and Network Load.

(d) RTT for sending data with a simple ACK message using different
protocols.

(e) Echo RTT for different data message size using ROS on Idle,
CPU Load, and Network Load.

(f) RTT for sending and receiving data messages using different
protocols.

(g) Echo RTT for different data message size using DDS on Idle,
CPU Load, and Network Load.

(h) Resting RTT for the evaluated protocols sending a payloadof 4 bytes waiting randomly between 0 and 3 seconds.

(i) RTT for sending 10 packages
with a payload of 10240 bytes
through 500 nodes on a different
host.

Fig. 1: Plots showing the measurement results for an excerpt of the collected data.
6

TABLE III: Average RTT in microseconds for echo and ACK mode with various payloads (in bytes) as visualized in Fig. 1.
Middlware

Load

2

4

8

16

32

64

128

256

512

1024

2048

4096

8192

16 38432 768OPC UA C/S

Idle
CPU
Net

520
279
875

283
258
869

288
233
876

280
260
858

270
276
827

270
265
864

360
284
866

313
293
871

456
310
875

546
346
860

606
461
1056

978
752
1158

1003
830
1248

1171
955
1397

1770
1467
1894

OPC UA P/S

Idle
CPU
Net

590
224
919

589
177
930

510
178
880

338
182
902

162
183
911

270
265
864

169
188
921

174
206
901

305
242
924

469
373
936

521
529
1003

446
620
1019

395
618
1110

1142
1083
1286

1924
1676
1641

MQTT

Idle
CPU
Net

584
912
1017

629
905
1042

645
912
1038

641
917
1033

581
911
1040

601
934
1027

622
945
1054

699
944
1039

694
964
1085

725
1015
1135

716
1094
1239

920
1540
1347

1186
1582
1456

1225
1504
1622

1511
1843
2029

ROS

Idle
CPU
Net

618
256
1017

628
320
1014

446
295
1009

396
252
1012

398
190
982

182
305
1021

296
332
1011

319
317
963

495
359
1040

814
421
1063

698
563
1124

1007
1077
1263

1273
1114
1343

1486
952
1488

1823
1427
1884

DDS

Idle
CPU
Net

726
624
1091

234
631
1086

236
626
1070

243
625
1111

247
624
1063

252
628
1088

251
622
1102

279
644
1099

270
659
1123

314
442
1114

343
351
1184

454
464
1190

495
489
1270

802
782
1432

1144
1050
1806

TCP

Idle

250

253

284

303

254

255

307

270

289

315

273

823

811

892

1261

UDP

Idle

241

231

208

213

208

162

168

198

238

193

263

263

268

893

1398

OPC UA C/SOPC UA P/S
MQTT
ROS
DDS
TCP
UDP

Idle
Idle
Idle
Idle
Idle
Idle
Idle

383
185
647
642
320
235
259

354
180
602
537
228
243
223

301
199
556
321
234
283
220

294
207
518
168
236
187
190

306
196
606
157
241
239
155

291
190
626
248
243
279
153

293
180
636
623
244
196
164

321
195
646
608
242
208
169

310
215
650
437
252
189
174

305
279
643
197
272
237
215

481
222
629
226
275
58
359

710
277
575
871
355
71
317

681
298
556
992
343
81
255

969
785
740
1192
514
167
502

1033
1265
800
1377
784
338
725

ACK

Echo

Mode

Figure 1 shows an excerpt of some of the most interesting
results from our tests. The corresponding numeric values forthe average RTT are also shown in Table III. In addition to
the protocols described in previous sections, we implemented
two simple echo/ACK servers in C that listen for TCP and
UDP connections and return the received data to the sender or
acknowledge the data by sending one single byte as a response.
The results of these raw TCP and UDP implementation are
included in the figures and show the best reachable RTT forthe corresponding tests without any overhead from a specificmiddleware implementation.
Figures 1a to 1c, 1e, and 1g show the box plots for the
RTT of the message being sent to the server side, which
immediately returns the same message for all four protocols.
Every diagram shows the values for the system idle state withno specific load on the system or the network. The second
value row shows the RTT with 100 percent CPU load on theserver and the last row shows the RTT under high network
load.
Comparing the values for the server in idle state versus
the one with high CPU load, the test results show that theRTT of OPC UA and ROS does not correlate to the CPU
load of the host and is nearly the same as in the idle
state. Note, that we are using a Preempt-RT patched Linuxkernel and the processes are set to the highest priority for all
protocols. MQTT and DDS show a significant slowdown ofapproximately 300 µs compared to the idle state. This leads
to the conclusion, that the Paho MQTT and eProsima Fast
RTPS implementations require more CPU power to process themessages and are thus slowed down when the CPU utilization
is at 100 percent.
The results of the RTT during high network load show that

protocols using TCP (OPC UA Client/Server, MQTT) wereless influenced by a network interface running at maximum
capacity than protocols using UDP (OPC UA Pub/Sub, DDS,
ROS). All protocols show a slowdown of more than 400 µs.
The combined view in Fig. 1d and 1f shows the directcomparison of the protocols, including a simple TCP and UDP
echo/ACK server implemented in C. It can be seen, that the
raw UDP implementation is the fastest way for exchanging
messages of small sizes, which are either acknowledged orechoed back, closely followed by TCP, which has a better
performance for bigger message sizes. Excluding the resultsof TCP and UDP, the open62541 OPC UA Pub/Sub implementation described in [15] is the fastest middleware for almost all
package sizes independent of the mode. eProsima Fast RTPS
is in second place, followed by OPC UA Client/Server, ROS,
and then MQTT. The diagrams also show that MQTT has thehighest number of outliers.
To investigate further which components of a middleware
have the highest impact on the RTT, we also ran tests using
the OpenDDS middleware. It features a zero-copy or memory
allocation mode for sending back the echo message. This has
shown that the way messages are read from the socket and thenforwarded to the user code has a high impact on the RTT, sincememory allocations are an expensive operation. Additionally,
OpenDDS was significantly slower than FastRTPS, whichshows that the serialization of messages may also lead tohigher RTT. The eProsima Fast RTPS and open62541 OPC UA
implementations pass constant data pointers to the user code,
whereas MQTT and ROS use multiple copy operations to
duplicate the data.
Figure 1h shows the RTT for sequential requests, where
the client waited a random amount of seconds (between zero7and three) to send one single request with 4 bytes of payload
and waited for the acknowledgment messages. This test was
repeated 100 times. These results indicate, that DDS needs
more time to reactivate from the resting state compared toother middleware protocols. For OPC UA, MQTT, and ROS,
the values are similar to the ones in Fig. 1d, which shows the
ACK RTT.
As a last test, we also evaluated the case where 500 instances are running on the same host and the instances need to
exchange data between each other, as described in Section V.
For OPC UA Client/Server, we simply started 500 OPC UA
servers on the same host, using different endpoint ports. The
client from the remote host sends 10 simultaneous write
requests with a payload of 10 240 bytes and every server
forwards this request to the next server in line, resulting in
50 write requests per package, multiplied by 10 simultaneous
streams. The last 10 servers then return the value to theclient, which measures the overall time. For OPC UA Pub/Sub,
MQTT, ROS, and DDS, the same procedure is achievedby using topics with different names and starting multiple
processes. One of the biggest issues for this test was theresource usage of ROS and DDS. Starting 500 ROS forces
the CPU to work at full capacity and filled 60 GB of thetotal 64 GB of installed RAM on that host. Using Fast RTPS,
we were not even able to start more than 100 nodes, as the
CPU was already working at full capacity and the DDS nodes
were not able to discover any topics after some time. We
then used the OpenDDS implementation, which also supports
shared memory, to conduct the tests. The high load is theresult of the discovery process in ROS and DDS: as explainedin Section IV, these two protocols produce a higher number ofpackages with more data compared to OPC UA and MQTT.
This test shows that the open62541 OPC UA Client/Server
implementation is still the fastest protocol and is even fasterthan the DDS shared memory implementation. The reason forthis huge performance gap to MQTT and ROS is the direct
TCP connection between nodes in OPC UA, whereas MQTT
uses stateless UDP connections. In [12], the authors also statethat for small data, shared memory does not improve thelatency compared to local loopback. ROS is slowed down by
the high CPU load it produces on the host.

Future experiments will evaluate the performance of the protocols on bad network connections, for example on a wirelessnetwork. Additionally, we will evaluate the performance ofdifferent implementations of the same protocol.
R EFERENCES[1] I. Ungurean, N. C. Gaitan, and V. G. Gaitan, “A middleware basedarchitecture for the industrial internet of things,” KSII Transactions on
Internet and Information Systems, vol. 10, no. 7, pp. 2874–2891, 2016.
[2] J.-D. Decotignie, “Ethernet-based real-time and industrial communications,” Proceedings of the IEEE, vol. 93, no. 6, pp. 1102–1117, 2005.
[3] K. Bauer, B. Diegner, J. Diemer, W. Dorst, S. Ferber, R. Glatz,
A. Hellinger, W. Herfs, M. Horstmann, T. Kaufmann, C. Kobsda,
C. Kurz, U. Löwen, V. Stumpf et al., “Umsetzungsempfehlungen für das
Zukunftsprojekt Industrie 4.0,” Promotorengruppe Kommunikation derForschungsunion Wirtschaft - Wissenschaft, acatech, Abschlussbericht
des Arbeitskreises Industrie 4.0, 2013.
[4] M. Saghian and R. Ravanmehr, “A survey on middleware approaches
for distributed real-time systems,” Journal of Mobile, Embedded andDistributed Systems, vol. 6, no. 4, 2014.
[5] Y. Chen and T. Kunz, “Performance evaluation of IoT protocols under aconstrained wireless access network,” in Proceedings of the International
Conference on Selected Topics in Mobile & Wireless Networking, 2016.
[6] D.-H. Mun, M. L. Dinh, and Y.-W. Kwon, “An assessment of Internet ofThings protocols for resource-constrained applications,” in Proceedingsof the IEEE Annual Computer Software and Applications Conference,
jun 2016, pp. 555–560.
[7] M. S. Rocha, G. S. Sestito, A. L. Dias, A. C. Turcato, and D. Brandao, “Performance comparison between OPC UA and MQTT for dataexchange,” in Proceedings of the IEEE International Workshop on
Metrology for Industry 4.0 and IoT, 2018, pp. 175–179.
[8] J. Pfrommer, S. Grüner, and F. Palm, “Hybrid OPC UA and DDS:
Combining architectural styles for the industrial internet,” in Proceedingsof the IEEE World Conference on Factory Communication Systems,
2016.
[9] A. Veichtlbauer, M. Ortmayer, and T. Heistracher, “OPC UA integration
for field devices,” in Proceedings of the IEEE International Conferenceon Industrial Informatics, 2015, pp. 419–424.
[10] S. Cavalieri and G. Cutuli, “Performance evaluation of OPC UA,”in Proceedings of the IEEE Conference on Emerging Technologies &Factory Automation, 2010.
[11] H. Haskamp, M. Meyer, R. Mollmann, F. Orth, and A. W. Colombo,“Benchmarking of existing OPC UA implementations for Industrie
4.0-compliant digitalization solutions,” in Proceedings of the IEEE
International Conference on Industrial Informatics, 2017, pp. 589–594.
[12] Y. Maruyama, S. Kato, and T. Azumi, “Exploring the performance of
ROS2,” in Proceedings of the International Conference on Embedded
Software, 2016.
[13] F. J. Rodrı́guez-Lera, V. Matellán-Olivera, J. Balsa-Comerón, Á. M.
Guerrero-Higueras, and C. Fernández-Llamas, “Message encryption in
Robot Operating System: Collateral effects of hardening mobile robots,”
Frontiers in ICT, vol. 5, no. 2, pp. 1–12, 2018.
[14] S. Srivastava, S. Anmulwar, A. M. Sapkal, T. Batra, A. K. Gupta, andV. Kumar, “Comparative study of various traffic generator tools,” inProceedings of the International Conference on Recent Advances inEngineering and Computational Sciences, 2014.
[15] J. Pfrommer, A. Ebner, S. Ravikumar, and B. Karunakaran, “Open source
OPC UA PubSub over TSN for realtime industrial communication,” inProceedings of the International Conference on Emerging Technologies
and Factory Automation, 2018.

VII. C ONCLUSION
In this paper, we gave an overview on the different features of OPC UA, ROS, DDS, and MQTT and compared
their performance in several benchmarks. OPC UA has its
strength in the semantic modeling of information. ROS ismainly used on robots for research purposes and provides
many different pre-implemented feature packages. DDS has anextensive set of Quality-of-Service settings, whereas MQTT
mainly focuses on a lightweight publish/subscribe protocol.
The performance comparison of the protocol implementations
shows that open62541 for OPC UA and eProsima FastRTPS
for DDS deliver high performance, whereas the MQTT and
ROS implementations show a significant slowdown in the RTTof packages sent to the server.

8