NEW SERVICE DELIVERYIoT InterOp-WARE
A Heck of a Challenge
Latif Ladid, IPv6 Forum President
in IoT today. This won’t move IoT to where it should go.
In an Industry Forum session at Globecom with Vint Cerf
and Geoff Mulligan, IPSO Chair and inventor of 6LoWPAN
and we came to the conclusion that IoT, M2M and oneM2M
are digging themselves into the same hole.
The Future Interop (F-Interop) project funded by theEuropean Commission has just started its research workback in November 2015 for three years to research theissues of the new IoT interop paradigm.
Current applications often follow the “one-networkone-application” paradigm. We are shifting away from thisparadigm as IoT applications often require merging different
solutions from a vertical into a horizontal market. This
results in an increased need for interoperable solutions,where “umbrella” networks are used to carry a large numberof independent traffic.
Latif Ladid, IPv6 Forum President

32InterComms www.intercomms.net

Figure 1. F-Interop addresses the standards valuecreation process
In order to be widely adopted, new technologies, products
and solutions go through the steps illustrated in Figure 1:
1.	Standardization: stakeholders discuss and align theirviews on a common standard.
2.	Conformance: test and validate that an implementation
conforms to the standard.
3. Optimization: in terms of Quality of Service, scalability,
energy consumption, etc.
4.	Market Launch: the solution is ready for initial roll-out
into the market.
Each phase requires extensive testing. Verifying
conformance to a standard, and verifying interoperability
with other vendors typically requires interoperability testing
events and interaction with third-party certification labs.
The traditional approach is to organize interoperability
events8, where different vendors meet face-to-face totBy 2020, the Internet of Things (IoT) is expected
to connect 50 to 100 Billion smart things andobjects12, paving the way to great economic
opportunities and challenges3. Forrester4 expects
the digital universe to grow by a factor of 32 by 2020
compared to 2012. According to Bell labs5, in 2013, ICT
consumed about 6% of global energy at about 108.4
gigawatts (or 109 nuclear plants, or 19 times the energy
consumption of New York City). According to Cisco6, by
2018, over half of all IP traffic will originate with non-PC
devices and the machine-to-machine (M2M) traffic willgrow at an annual rate of 84%. According to the IERC
and the ITU, the largest barriers hindering the Internet ofThings market development is the lack of interoperability 7.
Internet of Things, M2M and oneM2M are major
emerging trends at the present and for the coming years of
the communication society. Several challenges have alreadybeen identified in terms of governance, security/privacy andconvergence.
The current approach to IoT is obsolete as it still uses oldInternet models like gateways or NATed/Intranet or more
IntraNAT. We are simply making old mistakes of the Internet

test interoperability by going through an exhaustive
list of “interoperability tests”. Since this approach relies
on face-to-face meetings, it has a direct impact on the
speed of development of a standard, the cost of products
implementing such standards, and their time-to-market. Thisis all the more problematic as it is commonplace for sometests to fail for trivial reasons (packet formats, addressing
length, etc.). When this happens, vendors have to return
home, change their implementation, and wait until the next
event to re-test, usually months later.
The consequence is that:• The current process is extremely labor-intensive, as
engineers travel across the globe only to find out theyneed to make a minor fix;• The cost associated with engineering time and travel
expenses is often too high for SMEs;• Time-to-market is unnecessarily stretched, giving
vendors who want to adopt emerging standards a
disadvantage compared to vendors who come tomarket with entirely proprietary solutions.
This process often scares vendors away from standardsbased solutions. The end-user is therefore often andunnecessarily locked into proprietary solutions, asstandards-based products haven’t hit the market when theybought their first product.
The F-Interop project proposes a ground-up redesignof the way conformance, interoperability and performance
testing of protocols and standards is done. A novel
architecture centered on the cloud will enable the vast
majority of these tests to be done remotely.
The outcome is that typically only one interop event
will be required for a company to develop standardsbased interoperable products, cutting time-to-market by
6-12 months, and significantly lowering the engineering/financial overhead.
Interoperability tests require an implementation tointeract with other devices running another implementationof the same standard. Scalability tests require a large
number of such devices. It is impractical for an SME to buy alarge number of devices “just” for testing.
The European FIRE+ community provides several largescale open testbeds, largely used by the academic researchcommunity. We propose to position them at the center ofthe cloud-based testing architecture. We are targeting the
testbeds below, covering the full spectrum of networked
devices. Partners of the consortium play the lead role ineach of the following testbeds.
• Fed4FIRE (http://www.fed4fire.eu/testbeds/) is afederation of (at the moment of writing) 24 FIRE+
testbeds spread over Europe, bringing together
technologies such as cloud, IoT/wireless/wireless mobile,
LTE, openflow, network emulation, all accessible throughthe same toolset and account. All testbeds speak thesame API (Aggregate Manager API, XMLRPC based)
and are monitored continuously on their availability.

•

The total number of nodes (physical servers, physicalwireless devices, physical switches) is around 1000.
OneLab (https://onelab.eu/) is an experimental facility
made up of a federation of future Internet testbeds,
which together offer large-scale experimentation acrossheterogeneous resources. The four platforms federated
under OneLab include:o FIT IoT-Lab (https://www.iot-lab.info/), a 2728-node
testbed of wireless devices deployed in France,
representative of tomorrow’s low-power wireless meshnetworks.
o FIT CorteXlab (http://www.cortexlab.fr/), a cognitive
radio testbed with 80 wireless nodes deployed in Lyon,
Franceo FIT NITOS-Lab (http://fit-nitos.fr/), a wireless testbed
with over 100 nodes divided between sites in Volos(Greece), Paris, Sophia-Antipolis, and Evry (France)o PlanetLab Europe (PLE, http://planet-lab.eu/) an
Internet overlay testbed with 300 servers located in
150 different sites across Europe.

The FIT IoT-Lab testbeds will primarily be used in
F-Interop; however, all of the testbeds federated under
OneLab will be accessible through the F-Interop platform
and will be available to users for experiment deployment.
This will enable those users who wish to extend their testing
beyong the IoT domain to so do.
• The European IoT lab project9 (http://www.iotlab.eu/)
federates several IoT-related testbeds together withcrowd sourcing and crowd sensing capabilities. It aims at
researching the potential of crowdsourcing to extend IoT
testbed infrastructure for multidisciplinary experimentswith more end-user interactions. On the IoT testbed side,in includes several testbed, including smart campuses
and smart buildings, as well as a smart office testbed.
The platform gathers a large heterogeneity of IoT devices
enabling F-Interop to expose all sorts of devices using
distinct communication protocols. On the user side, theplatform enables interaction with end-users distributed
across Europe.
The F-Interop project will develop a set of tools
enabling new forms of online testing on top of the existing
FIRE+ testbeds. This includes tests that require interactions
between a large number of devices, to verify scalability,
end-to-end Quality of Service (QoS) and energy efficiency.
Each of the federated platforms provides a different
type of hardware, depending on the testbed, allowing
for expansive testing scenarios to suit the needs of all
experimenters.
This places the FIRE+ testbeds at the heart of the
F-Interop testing infrastructure, thereby significantly
increasing the usage of the FIRE+ testbeds by theEuropean industry.
F-Interop will research, develop and enable innovative
tools to support the development of new technologies andstandards, from their genesis to their maturity. It will provide

www.intercomms.net InterCommstt

NEW SERVICE DELIVERY33

cloud-based remote interoperability and conformance testing
tools, as well as tools for testing and measuring the scalability,
end-to-end Quality of Service (QoS) and energy efficiency of
any tested solution. This will extend the capability of existing
FIRE+ testbeds (Fed4Fire, OneLab, IoT lab), which will play acentral role the proposed testing architecture.
Objectives of F-Interop
The goal of F-Interop is to design and develop a FIRE+based online conformance, interoperability andperformance test and validation platform to support
researchers, product development by SME, andstandardization processes. More specifically, F-Interop will:
1. E
 xtend FIRE+ through research on an online testingtools- Conformance testing tools: an Implementation Under
Test (IUT) located at the vendor remotely connects to atest system that verifies its conformance to a standard.
- Interoperability testing tools: remote IUTs run
interoperability tests through a variety of real-lifescenarios, by interacting with (or directly on) FIRE+
testbeds.
- Scalability tests, by leveraging the FIRE+
infrastructure, test and analyses the capacity of agiven implementation to handle large numbers ofconnections and interactions.
- End-to-end Quality of Service (QoS) and Quality ofExperience (QoE) tests: through interactions between
FIRE+ testbed(s), verify QoS and QoE performance
metrics such as end-to-end reliability, latency and jitter.
This is particularly applicable to SDN/NFV standards.
- Energy efficiency tests, including when the IUT is
deployed at large scale.
2. I ntegrate several FIRE+ testbeds into a sharedplatform “Testbed as a Service”
- Mutualize 3 FIRE+ federating testbeds: Fed4FIRE,
OneLab and IoT lab, bringing together over 32 testbeds
and 4755 nodes.
- Design a common reference architecture modelfor on-line test and standardization support in close
collaboration with ETSI, IETF, W3C, ITU and IEEE.
- Security by design: the architecture will consider the
security and privacy of the test results, gathered data
and any test code/scripts passed over shared networks.
- Develop a “Testbed as a Service” (TBaaS) model:
based on virtualization and Software as a Service(SaaS), simplify remote access and interaction with theexperimental platform.
- Adopt a future-proof and flexible design, allowing
seamless integration of future testbeds.
3. S
 upport standardization and enable closer
cooperation with the industry
- Close collaboration with standardization bodies,directly contributing to three global emerging standards:

34

InterComms www.intercomms.net

oneM2M, IETF 6TiSCH and Web of Things (W3C).
- Supporting and enabling new online certificationand labelling mechanisms with a direct use by the
IPv6 Ready logo.
- Enabling easier participation of researchers andindustry in the standardization process.
- Open call for SMEs and developers to use and enrich
the developed testing tools.
The clear set of objectives of F-Interop and the new tools
developed during the project will have a significant impact
on standardization activities and the availability of compliant
products:
- Standardization support with emerging standards at
ETSI, IETF and W3C. The flexible F-Interop architecture
will enable new standards to be added to the available
test plans. Standards and test plans will be developedside-by-side resulting in faster publication of better
standards. This will also allow certification andlabeling for those standards.
- Researchers and SMEs can remotely test
implementations, products and applications.
By streamlining conformance, interoperability andperformance test, we evaluate the time-to-market ofstandards-based will be reduced by 6-12 months.
Faster availability of standards-based products resultsin wider adopt standards and a larger portion ofinteroperable products.
- Improving European leadership and influence on
global standardization processes.
The consortium includes the leading figures of the
3 targeted FIRE+ testbeds (Fed4Fire, OneLab, IoT lab). Its
members bring expertise from over research projects on
IoT, with strong links with standardization bodies (ETSI,IETF, IEEE, ITU, etc.) and international fora (IoT Forum, IPv6
Forum, etc.). This will ensure the alignment of the researchwith the industry needs, and an effective exploitation andtransfer of F-Interop results into standards, products andservices.
Overview: Best Practice in Conformance andInteroperability Testing
Testing is arguably the most important activity during thedevelopment of an implementation. When implementing
an existing standard, two types of tests are typically done:
conformance tests and interoperability. Today’s bestpractices on both types of tests are highlighted in Sections
1.3.1.1 and 1.3.1.2, resp.
A typical test (including conformance andinteroperability test) involve an Implementation Under Test
(IUT) and a “test harness”. The test harness (also referred
to as “test execution engine” or “test script repository”)
interfaces to various devices, and monitors their activity
while executing a set of tests. The ISO/IEC 9646 series andthe ETSI document EG202 237 describe a generic approach
for interoperability testing. These foundational documentstt

NEW SERVICE DELIVERY

will be followed by the F-Interop project.

Figure 2. Relationship between Standards, Validation &Testing.
Conformance testing
According to ISO/IEC 9646, the components of conformance
testing specifications are the Abstract Test Method/
Architecture, Test Purposes, the Abstract Test Suite (ATS) and
its Test Suite Structure, and Implementation Conformance
Statements (ICS).
The Abstract Test Architecture describes how an
Implementation Under Test (IUT) is tested independentlyof any Means of Testing. It includes the IUT, the testing
interfaces called PCO (Point of Control and Observation)and the chosen category of testing from the four
possibilities: Remote, Local, Distributed and Coordinated.
The architecture is a description of the interfaces where
test system and IUT are attached, the relationship between
the IUT, the type of test system, cabling/connection
requirements, and test and IUT operator locations. In somecases, the IUT cannot be directly connected to the test
system; it is then tested through a SUT (System Under Test)in which the IUT resides.
A Test Purpose is a plain-English description of atesting objective focusing on a single requirement or a setof requirements to be tested on the IUT. An abstract test
case is a complete and independent specification (at the
Abstract Test Method’s level of abstraction) of the actions
required to achieve a specific test purpose. It is abstract

Figure 3. From standard to
executable test suite.

insofar that it does not depend on the specific hardware
and/or interfaces used for test execution. Run-time
interfaces, software libraries and drivers are required forinterfacing the executable tests to the test equipment. The
Abstract Test Suite (ATS) is a collection of abstract test casesorganized into a Test Suite Structure, allowing multiple test
cases to be grouped into logical test groups; a test groupverifying the conformance of a particular part of a standard.
The ICS (Implementation Conformance Statements) is aquestionnaire filled in by the supplier (the vendor) thatprovides a statement about which capabilities and optionsof the standard have been implemented. It is used forselecting and parameterizing the tests to be run at the endon the IUT.
Figure 3 shows how a standard is translated into an
executable test suite. Figure 4 then shows how this test
suite, when executed against an IUT, is used for labeling andcertification.
Interoperability Testing
A framework for interoperability testing follows the sameprocess as conformance testing, except that there may
be several standards for abstract interoperability test
generation, and that the interoperability test suites are
executed on two or more devices (not only one as inconformance testing).
The purpose of interoperability testing is to demonstratethat an IUT works with other products, and to prove that
end-to-end functionality between (at least) two devices per
the implemented standard(s).
Interoperability test specifications include:- A test architecture, an abstract framework within
which any SUT scenario or configuration (a SUT being
composed of a number of IUTs coming from different
suppliers) will fit. This test architecture must clearly
identify the different IUTs and their functional role, thecommunication paths between them and protocols, APIs
and/or data models used for communication.
- An Interoperable Functions Statement (IFS) which
identifies standardized functions that an IUT may
support. An IFS is used as a pro-forma by suppliers

Figure 4. General overview ofconformance testing execution.

www.intercomms.net InterCommstt

NEW SERVICE DELIVERY35

to state which functions their IUT supports wheninteroperating with other IUTs.
- A Test Description (TD) which specifies detailed
steps to be followed to achieve stated test objectives,
usually written in a structured and tabulated natural
language which allows a manual execution. Automatedinteroperability Test Descriptions are also possible.
A Test Environment can be required: a combination ofother equipment and procedures enabling interoperability
testing. The Test Environment ensures the selection,interpretation and execution of the Test Descriptions,coordination and synchronization of the actions on the test
interfaces, and provides mechanisms for logging, monitoringand observing the interactions among IUTs.
A taxonomy of interoperability tests
Figure 5 illustrates the four categories of interoperability
tests, following ETSI’s taxonomy:• Technical Interoperability is associated with hardware/
software components, systems and platforms thatenable machine-to-machine communication. Technicalinteroperability is centered on (communication)
protocols and the infrastructure needed for those
protocols to operate.
• Syntactical Interoperability is associated with data
formats. Messages exchanged through a communication
protocol need to conform to the agreed-upon syntax andencoding. High-level transfer syntaxes such as HTML,XML or ASN.1 are used to represent the exchanged data.
• Semantic interoperability is associated with the meaningof content, and concerns human rather than machineinterpretation of the content. There needs to be acommon understanding of the meaning of the content(information) being exchanged.
• Organizational interoperability is the ability oforganizations to effectively communicate over different
information systems, infrastructures, geographic regions
and cultures. Organizational interoperability depends
on successful technical, syntactical and semanticinteroperability.

Figure 5. Different levels of interoperability

36InterComms www.intercomms.net

F-Interop focuses on conformance, technicalinteroperability testing (including syntacticalinteroperability checking and conformance testing) andsemantic interoperability testing.
The project will develop complementary tools that help
testing scalability, Quality of Service (QoS) and Quality ofExperience (QoE), as well as energy consumption, using the
FIRE+ testbeds.
The complexity of constrained network testingF-interop aims to develop several tools for testing different
technologies. Among the tools proposed, particularly
challengeable will be the design and implementation of
those testing constrained protocols for low-power wirelessmesh networks, (e.g., IETF 6TiSCH, 6LoWPAN and CoAP).
The added complexity of testing those standards (incomparison for example to testing IPv6 or HTTP) is due tothe constrained nature of the wireless devices and (lossy)
environments, where they run. Therefore, the testing tools
will be developed according to the strict requirements interm of time synchronization, and energy consumption.

Interop tests, plugfests and plugtests
Partners in the consortium have a good experience inorganizing interop tests, plugfests and plugtests. The mostimportant such events are:
- TODO ETSI: CoAP, 6LoWPAN plugtests. Dates, #participants, input ot WGs (3-4 lines)- ETSI is organizing the first IETF 6TiSCH plugtest, to beheld at IETF93 Prague in July 2015. The purpose of theevent is to test interoperability of implementations ofdraft-ietf-6tisch-minimal. 10 participants are expected.
- Inria co-organized an IPSO plugtest even in 2011
around IEEE802.15.4e and CoAP, held in Santa Claraover the course of two days, with 5 implementations
from industrial/academic partners. With one company
participating remotely from France, this event resulted
in the first trans-Atlantic communication between CoAPenabled low-power devices.
- The EANTC and Upperside Conferences invitedinterested vendors to a public multi-vendor
interoperability test and showcase at the MPLS SDNWorld Congress 2015 (http://www.uppersideconferences.
com/mpls-sdn/index.html). Together with the NFV &SDN Summit and the V6 World Congress 2015 taking
place at the same time, EANTC presented the results
live to the combined audience of all three conferences.
By presenting a realistic example of a next generationtt

NEW SERVICE DELIVERYt

NEW SERVICE DELIVERYinteroperable service provider network, participants
demonstrated the maturity and applicability of theirdevices and solutions.
- Inria participated in several specialist task forces(STF) at ETSI related to Test Specifications for IPv6
Interoperability. Since 2001, Inria brings expertise to the
ETSI Plugtest services for the organization of the several
sessions of IPv6 related interoperability events. Morerecently, in 2012, Inria contributed to the one 6LoWPAN
and the two first CoAP interoperability Plugtests events
organized by ETSI.
- Since 1995, the EANTC organizes interoperability test
events, usually 1-3 times a year, executed in the EANTC
lab. Results are demonstrated at shows (including theCarrier Ethernet World Congress).
- Inria is a founding and active member of the worldwide
IPv6 Ready Logo certification program (supported by
the IPv6 Forum). It is the representative of Europe in thisprogram, both as technical expert for test specification
development, and technical advisor for test results
analysis in the certification process. In 2006-2008, Inria
participated in the Go4IT Project (the FP7 European
project on TTCN-3 based Test Tools and Services forIPv6 protocols) and more recently in the PROBE-IT(Pursuing Roadmap and Benchmark in IoT) dedicated to
deployment, testing and interoperability issues of IoT.
Inria co-organized 2 plugfests around IETF 6TiSCH
technology (IETF89 London, March 2014, IETF90 Toronto,
July 2014), each attended by 8 academic/industrial teams.
The purpose was to bring together implementations
around 6TiSCH technology, including demonstrations of afederated backbone by Cisco and Linear Technology, and an
IEEE802.15.4e stack by the OpenWSN team.
The main objective of these interoperability events is thevalidation of the specification which gives a higher chancefor interoperable implementations, and allowed vendors
to detect and correct errors in a controlled manner. At thesame time, they are an opportunity for vendors to test theirimplementations and prototypes, to reduce time to market,
to promote technology and to originate new ideas.

References:
1 “Vision and Challenges for Realising the Internet of Things”,
Cluster of European Research Projects on the Internetof Things (CERP-IoT), page 13, March 2010. [available
online: http://www.internet-of-things-research.eu/pdf/
IoT_Clusterbook_March_2010.pdf, consulted 17-Feb-2015].
2 “Ericsson CEO Predicts 50 Billion Internet Connected
Devices by 2020”, GigaOm, April 2010 [available online:
http://gigaom.com/2010/04/14/ericsson-sees-the-internetof-things-by-2020/, consulted 17-Feb-2015].
3 “Disruptive technologies: Advances that will transform
life, business, and the global economy”, McKinsey Global
Institute, May 2013. [available online: http://www.
mckinsey.com/insights/business_technology/disruptive_
technologies, consulted 17-Feb-2015]
4 Jittender Miglani’s blog on Forrester.com. [available online:
http://blogs.forrester.com/jitender_miglani, consulted 17Feb-2015]
5 Global “What If” Analyzer of neTwork energy consumpTion(G.W.A.T.T.), Alcatel-Lucent. [available online: http://www.
gwatt.net/, consulted 17-Feb-2015]
6 “The Zettabyte Era—Trends and Analysis”, Cisco Systems,
June 2014. [available online: http://www.cisco.com/c/en/
us/solutions/collateral/service-provider/visual-networkingindex-vni/VNI_Hyperconnectivity_WP.html, consulted 17Feb-2015]
7 “The Internet of Things”, Internet Report, International
Telecommunication Union (ITU), 2005. [available online:
http://www.itu.int/osg/spu/publications/internetofthings/,
consulted 17-Feb-2015]
8 Interoperability events are branded as “PlugtestsTM” whenorganized by ETSI.

www.f-interop.eu
Come to the European Commission annual event on April
20 http://netfutures2016.eu/programme/interoperabilitybenchmaking-experimentation/

F-Interop remote interoperability tools will allow
validating of specifications and implementations in aquicker and more responsive manner which, will contributeto the technology success.
F-Interop will organize Plugtest event in which F-Interopinteroperability tools will be used and validated.

www.intercomms.net InterComms37