H2020-EUJ-02-2016
H2020 Grant Agreement Number 723076
NICT Management Number 18302

Profiling Specifications for
Cloud-Edge Orchestration
January 31st, 2018

ABSTRACT
Edge computing is a new paradigm which can move data processing close todata sources in order to reduce bandwidth consumption and latency. The
infrastructure of a smart city is a typical cloud-edge environment with a large
number of geo-distributed and heterogeneous edge nodes, which is a perfectenvironment to apply edge computing. However, to utilize edge computing,smart city service providers are still facing high complexity and low efficiency
when developing and managing their IoT services over such a cloud-edge
environment.
Our research activities in WP4 aim to reduce such complexities and enableefficient “Cloud-edge programming” by defining advanced edge
programming model and providing scalable edge computing framework.
This deliverable documents our achievements from these two aspects, focusing
more on the concrete specification of our cloud-edge programming model andframework, including the detailed profiling of IoT devices, data processing
modules, processing flows, and upper layer services and management
interfaces, which are the core of our cloud-edge orchestration.

This work is licensed under the Creative Commons Attribution 4.0 International License.
To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

H2020 EUJ-02-2016

CPaaS.io

Page 1 of 40

Disclaimer
This document has been produced in the context of the CPaaS.io project which is jointly funded by theEuropean Commission (grant agreement n° 723076) and NICT from Japan (management number 18302).
All information provided in this document is provided "as is" and no guarantee or warranty is given thatthe information is fit for any particular purpose. The user thereof uses the information at its sole risk andliability. For the avoidance of all doubts, the European Commission and NICT have no liability in respectof this document, which is merely representing the view of the project consortium. This document issubject to change without notice.

Document Information
EditorAuthors

Bin Cheng (NEC);Bin Cheng (NEC);
Toshihiko Yamakami (ACCESS)

Reviewers

François Carrez (UoS)

Delivery Type
Dissemination
Level
Contractual
Delivery Date
ActualDelivery Date
Keywords

R
Publicst

31 January, 2018st

31 January 2018
Cloud-Edge computing, programming model, task orchestration, task
migration, heavy edges, lightweight edges

H2020 EUJ-02-2016

CPaaS.io

Page 2 of 40

Revision History
Rev.
v0.1
v0.5
v0.6
v0.8
v1.0

Date
09/12/2017
24/01/2018
27/01/2018
28/01/2018
29/01/2018

H2020 EUJ-02-2016

Description
Initial structure of the deliverable
Inputs from the Japan side
First draft
Review
Final version

Contributors
Bin Cheng (NEC)Toshihiko Yamakami (ACCESS)
Bin Cheng (NEC)François Carrez (UoS)
Bin Cheng (NEC)CPaaS.io

Page 3 of 40

Table of Contents
1

2

3

4

5

6

7

Introduction............................................................................................................................................................................... 7
1.1

Cloud-Edge Computing............................................................................................................................................... 7

1.2

Major Challenges ........................................................................................................................................................... 8

1.3

Our Approach .................................................................................................................................................................. 9

1.4

Design Goal .................................................................................................................................................................... 10

CPaaS.io Cloud-Edge Computing Framework............................................................................................................ 102.1High Level View ............................................................................................................................................................ 10

2.2

Relation with Other CPaaS.io Activities ............................................................................................................... 11

Cloud-Edge Orchestration with Heavy Edge .............................................................................................................. 12
3.1

FogFlow Edge Computing Framework ................................................................................................................ 12

3.1

Major Workflow ............................................................................................................................................................ 14

3.2

Distributed Context Management ......................................................................................................................... 15

3.3

Declarative Programming Model ........................................................................................................................... 17

3.4

Service Orchestration ................................................................................................................................................. 19

3.5

Task Instantiation ......................................................................................................................................................... 20

3.6

Deployment View......................................................................................................................................................... 22

Cloud-Edge Orchestration with Lightweight Edge ................................................................................................... 23
4.1

Execution Environment for Binary Tasks ............................................................................................................. 23

4.2

Standby Task Instantiation ....................................................................................................................................... 24

4.3

Adapters for non-NGSI .............................................................................................................................................. 25

Development Process .......................................................................................................................................................... 26
5.1

Design Phase ................................................................................................................................................................. 27

5.2

Implementation Phase ............................................................................................................................................... 27

5.3

Usage Phase ................................................................................................................................................................... 28

Programmable Application Interfaces........................................................................................................................... 29
6.1

Implement a docker-ized operator ....................................................................................................................... 29

6.2

Register an implemented operator programmatically .................................................................................. 33

6.3

Define and register a service topology................................................................................................................ 34

6.4

Trigger/Terminate a service topology.................................................................................................................. 35

Graphical Interfaces .............................................................................................................................................................. 36
7.1

Registering a docker-ized operator ...................................................................................................................... 36

7.2

Editing and submitting a service topology; ....................................................................................................... 37

7.3Managing the registered service............................................................................................................................ 38

8Conclusions and Next Steps ............................................................................................................................................. 39

9References................................................................................................................................................................................ 40

H2020 EUJ-02-2016

CPaaS.io

Page 4 of 40

List of Figures
Figure 1: cloud-edge computing with multiple edge layers ............................................................................................... 7
Figure 2: major challenges being addressed in WP4 ........................................................................................................... 8
Figure 3: overall approach for our research activities in WP4 ......................................................................................... 9
Figure 4: separation and collaboration between EU and Japan in WP4 .................................................................... 10
Figure 5: CPaaS.io cloud-edge computing framework...................................................................................................... 11
Figure 6: relation to the CPaaS.io activities .......................................................................................................................... 12
Figure 7: system architecture of FogFlow ............................................................................................................................. 14
Figure 8: major workflow ............................................................................................................................................................ 15
Figure 9: single view of distributed context management system in FogFlow ........................................................... 17
Figure 10: an example of task based on the operator “anomaly” ................................................................................. 18
Figure 11: steps to orchestrate a service ............................................................................................................................... 20
Figure 12: steps to instantiate a task ...................................................................................................................................... 22
Figure 13: deployment view of the FogFlow system ............................................................................................................ 23
Figure 14: framework for binary migration with lightweight edges .............................................................................. 24
Figure 15: a flow of orchestration at Light-weight edge.................................................................................................... 25
Figure 16: framework of adapters to transform Non-NGSI to NGSI ............................................................................ 26
Figure 17: development process with FogFlow ..................................................................................................................... 26
Figure 18: list of all submitted operators ............................................................................................................................... 36
Figure 19: registration of a docker-ized operator ................................................................................................................ 37
Figure 20: graphical editor of service topology ..................................................................................................................... 38
Figure 21: defining a requirement with a specified geo-scope ........................................................................................ 39

List of Tables
List of Acronyms
Acronym
AMQP
API
COAP
DAG
ID
NGSI
SLA
TM
URL
IoT

Definition
Advanced Message Queuing Protocol
Application Programming Interface
COnstrained Application Protocol
Directed Acyclic Graph
IDentifier
Next Generation Service Interfaces
Service Level Agreement
Topology Master
Uniform Resource LocatorInternet of Things

List of Editing Shortcuts
Shortcut
e.g.
etc
feat.
h/w

Definition
exempli gratia – for example
et ceterafeaturinghardwareH2020 EUJ-02-2016

CPaaS.io

Page 5 of 40

i.e.
incl.
p/f
resp.
s/w
tbd
vs.
w.r.t.

id est – that isincludingplatform
respectively
softwareto be discussed/decided
versuswith respect to

H2020 EUJ-02-2016

CPaaS.io

Page 6 of 40

1 Introduction
Towards the goal of providing users and application developers with more control upon the City
platform in a seamless and easy way, the research activities in WP4 aims to support a cloud-edge based
programming paradigm, in which users and developers cannot only access and combine data, but also
implicitly determine aspects of system deployment and task execution. More specifically, what we havebeen doing in WP4 is to enable efficient “Cloud-edge programming” by defining advanced edge
programming model and providing scalable edge computing framework. This deliverable documents
our achievements from these two aspects, but focusing more on the concrete specification of our cloudedge programming model, including the detailed profiling of IoT devices, data processing modules,
processing flows, and also the upper layer services and management interfaces.
In this section we first start with the background and major challenges of cloud-edge computing andthen introduce the overall approach of how we structure our research activities within WP4 to tackle
those challenges and achieve our design goals, which are derived from the use case analysis in therelevant deliverable D4.1 [1].

1.1 Cloud-Edge Computing
Nowadays more and more stream data are constantly generated by a huge number of IoT devices thathave been widely deployed at different locations in our cities. By outsourcing more and more dataprocessing logics from the central cloud down to the network edges, such as Base Stations, IoT gateways,
even some endpoint devices, we are able to largely reduce the data transmission cost and also enablereal-time data analytics service with fast response time and better privacy-preserving capability. In such a
cloud-edge environment, we need to decompose our IoT services into smaller tasks so that the small
tasks could be dynamically migrated between cloud and edges.
As illustrated in Figure 1, a typical cloud-edge environment usually has several layers. At the top layer isthe cloud, which can be powerful compute nodes in a public cloud (such as AWS EC2, Microsoft Azure)or a private cloud (e.g. OpenStack based cluster). However, in this multi-layer environment what is called“edge” is somehow not very clear. To avoid the confusion, in CPaaS.io we separate edges into thefollowing two categories based on their capability and runtime environments.

Figure 1: cloud-edge computing with multiple edge layers

H2020 EUJ-02-2016

CPaaS.io

Page 7 of 40

Lightweight edge: meaning IoT devices located at the bottom layer, used as end points. Those devices
are very constrained in terms of storage and computation. They have limited size and/or battery lifetime.
In terms of execution environment, they do not support virtualization; they can run certain binary tasks to
pre-process the generated data. For example, any Arduino-based device is considered to be alightweight edge.
Heavy edge: any compute nodes that are deployed in between the cloud(s) at the top layer and IoT
devices at the bottom layer. In terms of storage and computation, those computer nodes are much more
powerful than constrained devices, but less powerful than cloud nodes in the cloud. In addition, heavyedges can support certain virtualization, such as docker container or virtual machine. They are considered
to have connected power supplier so that energy saving is not a big concern for heavy edges. For
example, an IoT Edge gateway deployed at streets or a small datacentre located at base station can be aheavy edge.

1.2 Major Challenges
The infrastructure of a smart city is a typical cloud-edge environment with a large number of geodistributed and heterogeneous edge nodes, including both heavy edges and lightweight edges. By
taking advantage of edge computing in a smart city platform, city service providers could offer timesensitive services with largely reduced bandwidth consumption. As compared to the cloud-centric
approach, using edge computing could not only improve QoS of their services, but also reduce theiroperation cost. However, as illustrated in Figure 2, in order to adopt edge computing, they have to face
the following challenges due to the heterogeneity, openness, and dynamics of such an environment.
High complexity to realize their services/applications during the development phase
High complexity to manage system resources in order to provide the best service QoS during the
deployment phase

Figure 2: major challenges being addressed in WP4
The CPaaS.io edge computing framework is designed and implemented to solve this problem. It provides
efficient programming model and tools for service providers to easily program decomposable IoT
services and help them automatically manage the services with optimized task orchestration during theruntime.

H2020 EUJ-02-2016

CPaaS.io

Page 8 of 40

1.3 Our Approach
As illustrated in Figure 3, we carry out our research activities in WP4 based on a bottom-up approach.
First, we go through a large set of smart city use cases and analyse their requirements based on the dataprocessing flows they need in order to produce their required results. After that, we try to abstract theirconcrete requirements into some common and generic patterns, which form the basis of our edge
programming model. To realize such a programming model, we design and implement the CPaaS.io
edge computing framework, which provides specific interfaces for service developers to design andoperator their IoT services. In parallel we can carry out some studies to optimize the performance of thecore algorithms in such a framework, such as task allocation algorithm or task migration algorithm. Oncethe initial framework is finished, we start to validate its programming model and interfaces with some
implemented use cases, such as smart parking use case, lost child finding use case.

Figure 3: overall approach for our research activities in WP4
As shown in Figure 4 below, the research focus for EU side and Japan side is separated as follows basedon their initial research interests: the EU side focuses more on the cloud-edge orchestration for heavyedges while the Japan side focuses more on the cloud-edge orchestration for lightweight edges.
However, the cloud-edge computing framework between cloud nodes and heavy edges can beconsidered as the “virtual cloud” at backend for lightweight edges. In the end, we share the same
programming model and upper layer interfaces for service developers at both sides. For the dedicatedorchestration algorithms for heavy edges or lightweight edges, we can choose and use them according
to the specific system setup environment.

H2020 EUJ-02-2016

CPaaS.io

Page 9 of 40

Figure 4: separation and collaboration between EU and Japan in WP4

1.4 Design Goal
According to the technical requirements derived from the use cases documented in D4.1, we aim toachieve the following design goals for the cloud-edge programming framework in the CPaaS.io platform.
From functionality perspective:






Cloud-edge programming model for service/application developers
Dynamic task orchestration between cloud and heavy edges
Binary migration between cloud and lightweight edges
Sharing intermediate results across tasks/services
Enabling secure data processing flows with access control policy

From performance perspective:High throughput and low latency to exchange context information across tasks
Fast response time to launch or terminate a service
Reduced bandwidth consumption between cloud and edges

2 CPaaS.io Cloud-Edge Computing Framework
2.1 High Level View
Figure 5 shows the high level view of our CPaaS.io Cloud-Edge Computing framework. The core part ofthis framework is the Service Orchestrator, which can dynamically generate, configure, and deploy dataprocessing flows over cloud and edges in an optimized manner. The Service Orchestrator is designed andimplemented to support a declarative edge programming model. For the easy use of our programming
model, an application interface layer is designed for service developers to easily program their IoT
services over cloud and edges, and also for service operators to easily manage their services on the fly.
The interface is provided as both defined APIs and web-based graphical interfaces.
Under the Service Orchestrator are the data processing part and the context information management
part. The data processing layer part consists of different types of execution engines in various
H2020 EUJ-02-2016

CPaaS.io

Page 10 of 40

environments (e.g. cloud nodes in public/private clouds, heavy edge nodes like IoT gateways, lightweightedges like IoT Engine). Each execution engine can perform some tasks assigned by the Service
Orchestrator. By our own definition, tasks can be categorized as data ingestion task or context analytics
task. For a data ingestion task, it can take any data in and then produce NGSI context; while for a context
analytics task, it will take NGSI data in and further produce the analytics results as NGSI data as well.
Different execution engines might accept different forms of data processing tasks. For example,
lightweight edges can only perform certain binary tasks since they have no virtualization support; while
heavy edges can accept any docker image-based tasks because they can support docker containers. In
this project we assume a data processing tasks defined by service developers can be transformed into
different images via an automated image builder tool like Unikraft [2].
The context information management part consists of a centralized Discovery component and a networkof many Brokers. Each Broker can manage part of context information for its responsible region, but they
can all provide a single view of all context entity data in the entire broker network. A broker component
can be deployed together with a heavy edge or on a separate and dedicated node. The Discovery
component will record which broker can provide which type of context entities, therefore supporting thediscovery and subscription of context availability information. Typically, the Discovery component isdeployed in the cloud as a centralized component.

Figure 5: CPaaS.io cloud-edge computing framework

2.2 Relation with Other CPaaS.io Activities
The CPaaS.io Cloud-Edge Computing framework is mainly responsible for realizing the data processing
logic and deploying it over clouds and edges when we need to provide time-sensitive IoT services oversmart city data. Figure 6 shows how our Cloud-Edge computing framework is related to the otheractivities in the project.

H2020 EUJ-02-2016

CPaaS.io

Page 11 of 40

First, using the programming model supported by our Cloud-Edge computing framework, we are
implementing some cloud-edge based smart city applications. One of the concrete examples is “smart
parking” use case.
Second, our Cloud-Edge computing framework can interact with the CPaaS.io platform via NGSI. On theother hand, it can fetch some open data from the CPaaS.io platform for doing further advanced data
analytics; on the other hand, it can forward any data analytics results to the CPaaS.io platform via NGSI
on demand so that we can share the generated real-time context information via the CPaaS.io platform
across services and applications, at both information layer and the semantic knowledge layer.
Third, we are utilizing the access control policy provided in WP5 to enhance the information control of
our processing flows.

Figure 6: relation to the CPaaS.io activities

3 Cloud-Edge Orchestration with Heavy Edge
In CPaaS.io, we have been working on the open source FIWARE Generic Enabler (GE) for cloud-edge
orchestration to realize the edge programming model to realize the cloud-edge orchestration the
CPaaS.io edge computing. In this section we report the major design of FogFlow framework, including its
programming model, core mechanisms for service orchestration, task instantiation, and also introduce
the deployment view of the FogFlow system.

3.1 FogFlow Edge Computing Framework
The system architecture of FogFlow is illustrated in Figure 7. The figure includes the FogFlow framework,
geo-distributed infrastructure resources, and FogFlow's connection with the users (system operator andservice developers) and external applications through its API and interfaces. Infrastructure resources arevertically divided as cloud, edge nodes, and devices. Computationally intensive tasks such as big data
H2020 EUJ-02-2016

CPaaS.io

Page 12 of 40

analytics can be performed on the cloud servers, while some tasks such as stream processing can beeffectively moved to the edge nodes (e.g. IoT gateways or endpoint devices with computation
capabilities). Devices may include both computation and communication capabilities (e.g. tablet
computers), or only one of them (e.g. beacon nodes advertising Bluetooth signals). The FogFlow
framework operates on these geo-distributed, hierarchical, and heterogeneous resources, with threelogically separated divisions: service management, data processing, and context management.
Service Management

The service management division includes task designer, Topology Master (TM), and docker image
repository, which are typically deployed in the cloud. Task designer provides the web-based interfaces
for the system operators to monitor and manage all deployed IoT services and for the developers todesign and submit their specific services. Docker image repository manages the docker images of all
dockerized operators submitted by the developers. TM is responsible for service orchestration, meaningthat it can translate a service requirement and the processing topology into a concrete task deployment
plan that determines which task to place at which worker.


Data Processing

The data processing division consists of a set of workers (w1, w2… wm) to perform data processing tasks
assigned by Topology Master. A worker is associated with a computation resource in the cloud or on anedge node. Each worker can launch multiple tasks based on the underlying docker engine and theoperator images fetched from the remote docker image repository. The number of supported tasks islimited by the computation capability of the compute node. The internal communication between
Topology Master and the workers is handled via an Advanced Message Queuing Protocol (AMQP)-based
message bus such as RabbitMQ to achieve high throughput and low latency.
Context Management

The context management division includes a set of IoT Brokers, a centralized IoT Discovery, and a
Federated Broker. These components establish the data flow across the tasks via NGSI and also manage
the system contextual data, such as the availability information of the workers, topologies, tasks, andgenerated data streams. The IoT Discovery handles registration of context entities and discovery of them.
This component is usually deployed in the cloud. IoT Brokers are responsible for caching the latest viewof all entity objects and also serving context updates, queries, and subscriptions. In terms of deployment,
IoT Brokers are distributed on the different nodes in the cloud and on the edges. They are also
connected to the other two divisions (Workers, Task Designer, Topology Master, and external
applications) via NGSI.
Federated Broker is an extended IoT Broker used as a bridge to exchange context information with allother Federated Brokers in different domains. For instance, Federation Broker enables communicationfrom one deployment in a European smart city (e.g. Domain A: Heidelberg) to another in a Japanesesmart city (e.g. Domain B: Tokyo). These deployments are considered as two different domains. Within
the same domain, all IoT Brokers and Federated Broker are connected to the same IoT Discovery.

H2020 EUJ-02-2016

CPaaS.io

Page 13 of 40

Figure 7: system architecture of FogFlow

3.1 Major Workflow
Figure 8 below illustrates the major workflow, including the following steps:
Step 1: developers define a topology (data representation is a JSON object) and submit it to the TM;
Step 2: TM receives the submitted new topology and keep it; meanwhile TM sends a
NGSI10/SUBSCRIBE(topology) message to IoT Broker for subscribing to the NGSI10/UPDATE(topology)
messages, issued by the system operator;
Step 3: to trigger stream processing tasks for a given topology, the system operator needs to issue a
NGSI10/UPDATE(topology) message to IoT Broker and then IoT Broker will notify TM about the update;
Step 4: Once receiving the message NGSI10/NOTIFY(topology), TM starts to generate and configure
stream processing tasks and then assign them to different workers for execution;
Step 5: Workers receive the commands from TM via AMQP and then launch each assigned stream
processing task in a docker container; Also the workers need to configure the container for proper
computation resources (e.g., number of CPUs, memory size) and some input parameters (the parameters
are passed to the docker-ized stream processing tasks as environment variables) in order to tell thelaunched stream processing task where and how to publish the generated results), meanwhile helping
the stream processing task to get their input streams from a nearby IoT Broker.
For each edge node, there is an associated nearby IoT Broker. During the initialization phase, the worker
daemon on an edge node will look for a nearby IoT Broker based on the hierarchy network topology,
starting from the local edge node and then check the upper layer parent site if no IoT broker cannot befound on the same edge node. All IoT brokers are registered to IoT Discovery. The searching of a nearby
IoT Broker is based on the query results from IoT Discovery.
H2020 EUJ-02-2016

CPaaS.io

Page 14 of 40

For consumers/producers, if they are IoT devices (sensors or actuators), they will be first redirected to anedge node based on network proximity in the hierarchy network topology; if they are processing task
instances running on some edge node, they will only talk to the IoT Broker that has been associated withthe edge node.
Step 6: producers (sensors) publish newly generated data by sending NGSI10/UPDATE(stream) messages
to IoT Broker(s);Step 7: stream processing tasks get their input streams by receiving NGSI10/NOTIFY (stream) from IoT
Broker(s) and publish their generated output data streams by sending NGSI10/UPDATE(stream)
messages to IoT Broker(s); all generated output streams are registered to IoT Discovery so that they canbe discovered by other tasks or consumers;
Step 8: To subscribe the final results or some intermediate results (they are all output streams generated
by some tasks), consumers (actuator) issue a NGSI10/SUBSCRIBE(stream) message to IoT Broker(s); TheNGSI10/SUBSCRIBE(stream) message defines, in which topology and which type of output streams are
expected under which scope; the scope defines the scope of producers;
Step 9: By relying on IoT Discovery, IoT Broker(s) can discover the matched streams for the received
NGSI10/SUBSCRIBE message and forward them to the consumers via NGSI10/NOTIFY(stream).

Figure 8: major workflow

3.2 Distributed Context Management
The context management system is designed to provide a global view for all system components andrunning task instances to query, subscribe, and update context entities via the unified NGSI. Our NGSIbased context management has the following additional features. These features are different from theones provided by traditional pub-sub message brokers such as MQTT-based Mosquitto or Apache Kafka,
but they play an important role to support FogFlow's NGSI-based programming model.

H2020 EUJ-02-2016

CPaaS.io

Page 15 of 40







It provides separate interfaces to manage both context data (via NGSI10) and context availability(via NGSI9). This feature enables flexible and efficient data management with standardized openAPIs.
It supports not only ID-based and topic-based query and subscription but also geoscope-based
query and subscription. This feature enables FogFlow to efficiently manage all geo-distributed
resources such as workers, tasks, and generated streams.
It allows a third-party to issue subscriptions on behalf of the subscribers. This feature provides the
chance to achieve the minimized complexity within the operators and the maximized flexibility ofthe operators.

Context availability represents the outline of context data. Usually context availability changes lessfrequently than context data over time. For example, the following availability information is used toregister context entities: context type, attribute list, and domain metadata (e.g., provider information).
The FogFlow programming model benefits from these separated interfaces because of two reasons. First,
FogFlow can automatically manage context availability information on behalf of tasks so that we reducethe complexity of operator implementation for developers. Second, context availability and context data
updates are forwarded to task instances via separate channels; therefore, we do not have to feed theunchanged context availability information to the tasks repeatedly. This can significantly reduce thebandwidth consumption of cross-task communication.
In FogFlow, whenever a service topology is triggered, a large number of geo-distributed task instances
are created, configured, and instantiated on the fly in a very short time. This introduces two challenges tothe context management system: 1) it must be fast enough to discover available resources in a specified
scope; 2) it must provide high throughput to forward context data from one task to another. In addition,
we assume that data processing tasks can only be instantiated from a service topology within a single
FogFlow-enabled smart city IoT platform. However, they should also be able to share and reuse context
data from other smart city IoT platforms as long as these platforms are compatible with NGSI. In terms ofterminology, each smart city IoT platform is represented by a domain and the FogFlow framework can be
duplicated to realize other smart city platforms for different domains. Different domains can be different
cities or business domains such as transportation and e-health in the same city.
Currently, the Orion Context Broker developed by Telefonica is the most popular message broker
supporting NGSI; however, it is not scalable due to the lack of distributed solutions and federation
support. To achieve a scalable context management system, we apply the following two approaches.
Scaling light-weight IoT Broker up with shared IoT Discovery

As illustrated in Figure 9, within each smart city platform domain a large number of IoT Brokers work
together in parallel with a shared IoT Discovery. The centralized IoT Discovery provides a global view ofcontext availability of context data and provides NGSI9 interfaces for registration, discovery, andsubscription of context availability. Each IoT Broker manages a portion of the context data and registers
data to the shared IoT Discovery. However, all IoT Brokers can equally provide any requested context
entity via NGSI10 since they can find out which IoT Broker provides the entity through the shared IoTDiscovery and then fetch the entity from that remote IoT Broker.
IoT Discovery
The centralized IoT Discovery provides a global view of context availability of context data and provides
NGSI9 interfaces for registration, discovery, and subscription of context availability.
IoT Broker
H2020 EUJ-02-2016

CPaaS.io

Page 16 of 40

The IoT Broker in FogFlow is very light-weight, because it keeps only the latest value of each context
entity and saves each entity data directly in the system memory. This brings high throughput and low
latency for the data transfer from context producers to context consumers.
Each IoT Broker manages a portion of the context data and registers data to the shared IoT Discovery.
However, all IoT Brokers can equally provide any requested context entity via NGSI10 because they canfind out which IoT Broker provides the entity through the shared IoT Discovery and then fetch the entity
from that remote IoT Broker.

Figure 9: single view of distributed context management system in FogFlow

Connecting different domains via Federated Broker:

In each domain there is one Federated Broker responsible for announcing what the current domain
provides and fetching any context data from the other domains via NGSI10. Within the domain,
Federated Broker informs IoT Discovery that it can provide any context data out of the current domain.
Federated Broker needs to coordinate with the other Federated Brokers in different domains. The
coordination can be done using different approaches, such as table-based, tree-based, or mesh-based.
In the table-based approach, all Federated Brokers can know which Federated Broker is
responsible for which domain via a shared and consistent table that is maintained and updated
by a bootstrap service.
In the tree-based (hierarchical) approach, a hierarchical relationship between different domains is
configured manually or maintained automatically by a root node.
In the mesh-based approach, each Federated Broker maintains a routing table based on its partial
and local view and relies on a next hop from the routing table to locate its targeted domain.

In practice, which approach to take is up to the actual scale of domains. Due to a limited number ofdomains in our current setup, FogFlow takes the table-based approach and looks up the Federated
Broker for a target domain directly from the shared table.

3.3 Declarative Programming Model
The FogFlow programming model defines the way of how to specify a service topology using declarative
hints and how to implement tasks based on NGSI. First, developers decompose an IoT service into
H2020 EUJ-02-2016

CPaaS.io

Page 17 of 40

multiple tasks and then define its service topology as a Directed Acyclic Graph (DAG) in JSON format toexpress the data dependencies between different tasks. On the other hand, the FogFlow programming
model provides declarative hints for developers to guide service orchestration without introducing muchcomplexity.
The following concepts are required to understand the FogFlow programming model.


Operator

In FogFlow an operator presents a type of data processing unit, which receives certain input streams as
NGSI10 notify messages via a listening port, processes the received data, generates certain results, andpublishes the generated results as NGSI10 updates.
The implementation of an operator is associated with at least one docker images. To support various
hardware architectures (e.g. X86 and ARM for 64bits or 32 bits), the same operator can be associatedwith multiple docker images.


Task

A task is a data structure to represent a logic data processing unit within a service topology. Each task is
associated with an operator. A task is defined with the following properties, like the example provided in
Figure 10:






name: a unique name to present this task
operator: the name of its associated operatorgroupBy: the granularity to control the unit of its task instances, which is used by service
orchestrator to determine how many task instances must be created
input_streams: the list of its selected input streams, each of which is identified by an entity typeoutput_streams: the list of its generated output streams, each of which is identified by an entity
type

Figure 10: an example of task based on the operator “anomaly”
In FogFlow, each input/output stream is represented as a type of NGSI context entities, which are usually
generated and updated by either an endpoint device or a data processing task.
During the runtime, multiple task instances can be created for the same task, according to its granularity
defined by the “groupBy” property. In order to determine which input stream goes to which task
instances, the following two properties are introduced to specify the input streams of tasks:

H2020 EUJ-02-2016

CPaaS.io

Page 18 of 40

Shuffling: associated with each type of input stream for a task; its value can beeither broadcaster unicast.
broadcast: the selected input streams should be repeatedly assigned to every task instance of thisoperator
unicast: each of the selected input streams should be assigned to a specific task instance only
once

Scoped: determines whether the geo-scope in the requirement should be applied to select the input
streams; its value can be either true or false.
Topology

A service topology presents the decomposed data processing logic of a service application.
Decomposing the entire service application logic into small data processing tasks allows FogFlow todynamically and seamlessly migrate data processing tasks between the cloud and edges. It also enables
better sharing of intermediate results.
Each service topology is represented as a DAG of multiple linked tasks, with the following properties:

description: a text to describe what this service topology is about
name: a unique name of this service topology
priority: the task priority of this service topology
tasks: the list of its tasks

3.4 Service Orchestration
Once developers submit a specified service topology and the implemented operator docker images, theservice data processing logic can be triggered on demand by a high level processing requirement.
The processing requirement is sent (as NGSI10 update) to the submitted service topology entity. It isissued either by the system operator via Task Designer or by a subscriber via an external application.
The following three inputs are necessary for Topology Master to carry out service orchestration.


Expected Output represents the output stream type expected by external subscribers. Based on
this input parameter, the Topology Master decides which part of service topology should betriggered. This allows FogFlow to launch only part of the data processing logic defined in theservice topology.
Scope is a defined geoscope for the area where input streams should be selected. This allows
FogFlow to carry out the selected data processing logic for the selected area such as a specificcity or a polygon area.
Scheduler decides which type of scheduling method should be chosen by the Topology Masterfor task assignment. Different task assignment methods lead to different Service Level Agreements(SLAs) because they aim for different optimization objectives. For instance, we provide two
methods in FogFlow: one for optimizing the latency of producing output results and the other for
optimizing the internal data traffic across tasks and workers.

For a given processing requirement, Topology Master performs the following steps to dynamicallyorchestrate tasks over cloud and edges, as illustrated in Figure 11.

H2020 EUJ-02-2016

CPaaS.io

Page 19 of 40

Topology Lookup: iterating over the requested service topology to find out the processing tree in orderto produce the expected output. This extracted processing tree represents the requested processing
topology which is further used for task generation.
Task Generation: first, querying IoT Discovery to discover all available input streams and then deriving
an execution plan based on this discovery and the declarative hints in the service topology. The
execution plan includes all generated tasks that are properly configured with right input and output
streams and also the parameters for the workers to instantiate the tasks.
Task Deployment: performing the specified scheduling method to assign the generated tasks to geodistributed workers according to their available computation capabilities. The derived assignment result
represents the deployment plan. To carry out the deployment plan, Topology Master sends each task to
the task's assigned worker and then monitors the status of the task. Each worker receives its assigned
tasks and then instantiates them in docker containers. Meanwhile, worker communicates with the nearby
IoT Broker to assist the launched task instances for establishing their input and output streams.

Figure 11: steps to orchestrate a service
Since the focus of this paper is on the NGSI-based programming model and its supporting systemframework, we skip the algorithms for task generation and task assignment. More details can be found in
our previous paper [3].

3.5 Task Instantiation
Developers need to implement each operator in a service topology as a docker-ized application. As
illustrated in Figure 12, once a worker instantiates a docker-ized operator application (a task instancerunning in a docker container), the task instance interacts with the FogFlow framework via the following
steps.
First, before starting its internal processing logic, the task instance receives a JSON-based configurationfrom the worker through environment variables. The initial configuration includes which IoT Broker thetask instance should talk to and also the metadata of its input and output streams. Later on, if there is
H2020 EUJ-02-2016

CPaaS.io

Page 20 of 40

any configuration change, those changes can be sent to the task instance via a listening port (input port).
In FogFlow, the important stream metadata required by the task instance include:The entity type and entity ID of the associated stream that can be used by the task instance toknow which entity to subscribe as inputs and which entity to update as outputs
The way of how the stream data is provided from the producer to consumers, which can be
PUSH-based or PULL-basedMore specifically, PUSH-based means that the stream entity will be updated by its context producer
actively and context consumers can receive the updates from IoT Broker via subscriptions, while PULLbased means that the stream entity only represents the information of the context producer and theactual stream data must be pulled by the task instance from a service URL, which is part of the stream
entity. For example, a temperature sensor that actively sends temperature observations periodically canbe represented as a PUSH-based stream entity; a webcam that sends captured images or video streams
on request can be represented as a PULL-based stream entity.
Second, after the task instance is launched and configured, it will start to get its input streams and thenprocess the received data. If the stream is PUSH-based, the task instance can receive all input stream
data as NGSI10 notification via the input port without sending any subscription, because the worker
issues NGSI10 subscriptions to the IoT Broker on behalf of the task; if the stream is PULL-based, e.g.
video streams from an IP camera, the task instance needs to fetch the input stream data from a provided
URL in the stream metadata.
Lastly, once some results are generated from the received stream data, the task instance publishes orannounces them as output streams. If the output stream is PUSH-based, the task instance sends thegenerated outputs to the IoT Broker as NGSI10 update under the specified entity type and ID; if theoutput stream is PULL-based, the worker can register the output stream on behalf of the task. With thisdesign, we allow the worker to handle higher management complexity in order to dynamically configureand establish the data flows across different task instances. Therefore, we can try to reduce thecomplexity of the implementation of docker-ized operators and reduce the required effort fromdevelopers; on the other hand, we can provide enough flexibility for various application use cases andalso comply with the NGSI standard.
The detailed process to instantiate a task at edges is described below, as illustrated in Figure 12.
1. the worker asks its local docker engine to create a new container for running the new task
instance with a specified listening port number
2. the task instance starts to run and to listen on the given port number in order to receive input
data streams
3. the worker sends a configuration object to the running task instance via its listening port
4. the worker issues NGSI10 subscriptions on behalf of the running task instance
5. the required input data streams flow into the running task instance for further processing via its
listening port
6. the running task instance processes the received input stream data and then generate its result
7. the running task instance publishes the generated result as its output by sending NGSI10 updates

H2020 EUJ-02-2016

CPaaS.io

Page 21 of 40

Figure 12: steps to instantiate a task

3.6 Deployment View
Figure 13 below shows the overall deployment view of the FogFlow system, including three parts:
Core Components running on the CloudAll core service components are deployed on a single cloud node or in a cloud cluster. As illustrated by
the deployment view Figure 13 below, the following components are started in the cloud:









IoT Discovery: the centralized IoT Discovery that manages the availability of context entities andtheir metadata
IoT Broker(cloud): the IoT Broker in the cloud
Topology Master: it is responsible for service orchestration and global task generation,
configuration, and management
Worker(cloud): it launches or terminates the tasks assigned by Topology Master
RabbitMQ: it is the AMQP-based message bus used for exchanging the control commands
between the TM and Workers
Task Designer: it is a nodejs-based web server to provide the web-based interfaces for systemmanagement and for showing use cases demonstrations

Edge node

An edge node must deploy a worker and install a docker engine to run task instances. Normally, it is also
required to deploy an IoT broker at each edge node so that the data flow exchanged between task
instances on the edge node can be done locally. However, the IoT Broker at the edge node remains
optional.
Registry service

H2020 EUJ-02-2016

CPaaS.io

Page 22 of 40

FogFlow relies on an external docker registry service to manage all registry docker images for each
operator. We can either use the public docker registry service or set up our own private docker registry
service.

Figure 13: deployment view of the FogFlow system

4 Cloud-Edge Orchestration with Lightweight Edge
The Japan approach in cloud-edge orchestration has been focused on fault-tolerant edge computing
with binary migration for standby activation. In this section, we report the initial design of such binary
task migration over cloud and lightweight edges. This design has been mainly explored by the Japan side
to carry out dynamic task migration between cloud and constrained devices. In addition, we also explain
the approach of how we can utilize the adapter concept in FogFlow to support the integration between
FogFlow and constrained devices, which might not support the NGSI standard due to their concerns onenergy consumption and their battery lifetime.

4.1 Execution Environment for Binary Tasks
The programming model and overall cloud-edge orchestration mechanism in FogFlow provides a solid
basis for the binary migration function on constrained edge devices. With constrained edge devices, local
orchestration at edges is optional. As illustrated by the deployment view in Figure 14 below, thefollowing components are started in the cloud:

IoT Discovery: the centralized IoT Discovery manages the availability of context entities and their
metadata
IoT Broker(cloud): is the IoT Broker in the cloudH2020 EUJ-02-2016

CPaaS.io

Page 23 of 40






Topology Master: it is responsible for service orchestration and global task generation,
configuration, and management
Migration Manager: it launches or terminates the tasks assigned by the Topology Master forbinary migration
Task Designer: is a cloud server to provide the cloud-based interfaces for system management
Management Repository: is a repository to store binary migration configuration data and binary
images

The execution environment for this orchestration is provided with coordination of IoT-Aggregator andIoT-Engine. Notice that the main logic of IoT-Aggregator can be realized on top of FogFlow based on its
programming model and framework for real centre cloud(s) and distributed heavy edges. For constrained
devices like IoT-Engine, the FogFlow system framework with heavy edges can provide them a virtual
cloud environment at backend.

IoT Broker(cloud)Cloud

Migration
Manager(Optional)

Edge

Topology
Master

Task
Designer

Management
Registry

IoT
Discovery

Migration
Manager

Alternative
DeviceManager

DeviceManager

Figure 14: framework for binary migration with lightweight edges

4.2 Standby Task Instantiation
A flow of orchestration at a lightweight edge is depicted in Figure 15. With this design, we can share thesame programming model for both heavy edges and lightweight edges.
Device Topology

Execution Plan

Deployment Plan
Cloud

Binary allocation
Management

B
A

D

Execution
ManagerC

NormalHot
Standby

Binary B1

Binary B2StatesDual

Cold
Standby

Binary B3

Task
specificationEdge-Device
Dependency Model

H2020 EUJ-02-2016

Binary migration
Plan generation

Binary B1

Device stand-by
deployment

Proactive
Management

CPaaS.ioLight-weight
Edge 1

Light-weight
Edge 2

Page 24 of 40

Figure 15: a flow of orchestration at Light-weight edge
The standby task is assigned by IoT-Aggregator using the proactive analysis of possible faults in devices.






IoT-Aggregator will assign the following task using the analysis
Normal task
Dual task
Hot-standby task
Cold-standby task

Task allocation is done by device-specific risk functions (fault possibility etc.) and cost functions(hardware costs, operation costs, etc.). Currently, the results for binary task migration between cloud andlightweight edges are still preliminary. More details will be reported in the following project period.

4.3 Adapters for non-NGSI
Within the project we have already considered the adoption of binary task migration to the upper layer
CPaaS.io Edge Computing framework, which is mainly provided by FogFlow. In this subsection, we
explain how the adapter concept in FogFlow can be utilized to integrate binary tasks that are executedon constrained lightweight edge nodes and only support more lightweight data models/communication
protocols (e.g. MQTT, u2) than NGSI.
In FogFlow we dynamically compose multiple data processing tasks to form the data processing flow ofeach IoT service based on the standardized NGSI data model and service interface. However, in order tointeract with sensors and actuators, we still need to deal with the diversity of various IoT devices. For
example, some devices (e.g. IoT Engine in the u2 architecture from Japan) might not be able to talk with
the FogFlow system via NGSI due to their limited upload bandwidth; some existing devices might only
support other protocols such as MQTT or COAP; or some devices need to be turned into sleep mode
from time to time in order to save their battery lifetime. In order to handle these issues, we introduce
virtual devices to unify the communication between FogFlow and IoT devices.
Figure 16 below illustrates this concept. Any physical device that already supports NGSI can be integrated
to the FogFlow system directly by interacting with a nearby IoT Broker. On the other hand, a physicaldevice which does not support NGSI must be integrated into the FogFlow system as a virtual device via a
device adapter. As shown in Figure 16, the device adapter is a proxy that is triggered and initialized froma device profile in order to mediate the communication between the device and the FogFlow system. The
proxy is a task instance instantiated from a specific mediation operator, which handles the detailed
mediation procedure for different types of devices. For conversion between different interfaces, different
mediation operators must be developed.
By adding a device profile, we can trigger an adapter task to integrate a non-NGSI device into the
FogFlow system. The device profile provides which operator should be used and the necessaryconfiguration information to interact with the device. In the end, the physical device is presented as a
NGSI context entity via its device adapter. All FogFlow services just need to interact with the NGSI
context entity associated with the physical device, such as sending context updates or subscribing tosome attribute changes.
Using this virtual device approach, we handle the availability and reliability issues of IoT devices within
the device adapters.

H2020 EUJ-02-2016

CPaaS.io

Page 25 of 40

Figure 16: framework of adapters to transform Non-NGSI to NGSI

5 Development Process
In FogFlow a service is represented by a service topology, which consists of a set of linked dataprocessing tasks. Therefore, we have to carefully consider how we can decompose the entire dataprocessing logic into multiple small tasks so that they can be dynamically created and placed either inthe cloud or at the edges. Here we outline what needs to be done from the design and implementation
phase to the service usage phase. Figure 17 visualizes the development process with three major phrases.

Figure 17: development process with FogFlow
H2020 EUJ-02-2016

CPaaS.io

Page 26 of 40

5.1 Design Phase
During the design phase we can take a top-down approach to think about how the entire service logic
can be divided into small tasks.
1. Analyze the major data processing part in the service
Basically we consider the entire data processing part as a black box and then start to define what the
inputs are and which type of outputs need to be produced.
2. Break down the major data processing part into small pieces (data processing tasks)
We can follow a data-driven approach to check for each output what are the required inputs and dataprocessing logics. Meanwhile, we can check if we can break the data processing logic into small tasks
mainly for two purposes: 1) the intermediate results can be shared across various tasks to produce
different outputs; 2) tasks can be smaller so that they can be dynamically migrated between edges andclouds.
3. Verify the entire data processing flows with inputs and outputs:
After having the separated tasks of the data processing logic in the service, the service provider can startto verify how they can be specified based on the FogFlow programming model, mainly by checking if
each task can be described with the FogFlow task specification, such as input or output data stream
types, task granularity.

5.2 Implementation Phase
During the implementation phase, the following steps are required to implement a service.
1. implement a docker-ized operator
For each task object in a defined service topology, its data processing logic is provided by an operator,
which is mapped to one or more than one docker-ized applications. The name of an operator must be
unique and the same operator can be mapped to multiple docker-ized applications, which provide theimplementation of the same operator for different operating systems (such as Ubuntu, Windows, MacOS)
and different pieces of hardware (such as x86-based or ARM-based, either 64bits or 32bits).
The detailed instruction on how to implement a docker-ized application is provided in Section 8.
2. push the docker image to the docker registry
Once one builds up the docker image for the operator, one needs to push the docker image to the
public or private docker registry, which is configured and used by FogFlow.
# if the default public docker registry is useddocker push task1# if you use your own private docker registry, which is using the domain name "myRegistry.com"
docker tag task1 myRegistry.com/task1
docker push myRegistry.com/task1

3. register the implemented operator
In order for FogFlow to use the operator, we must register it. This can be done in two different ways:
H2020 EUJ-02-2016

CPaaS.io

Page 27 of 40

register the docker-ized operator image via FogFlow Task Designer
register the docker-ized operator image by sending a NGSI update to the FogFlow Context
Management system

4. define and register a service topology
Once all required operators have been registered in FogFlow, we can start to define the service topology.
Currently, FogFlow provides two ways to define and register a service topology:

register a service topology via FogFlow Task Designer
registery a service topology by sending a NGSI update to the FogFlow Context Management
system

5.3 Usage Phase
Once the defined service topology is registered in FogFlow, we can trigger the service topology bysending processing requirements. The detailed steps are the following:
1. define a requirement object
The processing requirement is defined as an NGSI-based context entity, including the following
attributes:

topology: which topology to trigger
expected output: the output stream type expected by external subscribers
scope: a defined geoscope for the area where input streams should be selectedscheduler: which type of scheduling method should be chosen by Topology Master for task
assignment

Please referrer to the code example in Section 6.4.
2. send the defined requirement as a context update to trigger the service topology
The constructed requirement must be sent as a NGSI10 update. Since Topology Master alreadysubscribes to all requirement updates on any service topology, it will be immediately informed by thisnew requirement objects and then start to orchestrate a data processing flow accordingly. A unique ID
will be returned for this new requirement entity.
3. query or subscribe to the generated outputs
To query or subscribe to the generated outputs, developers just have to follow the standard NGSIinterfaces. For example, the developers need to specify the type of the subscribed entities, and/or a geoscope. More details can be found from the following document [4].
4. delete the requirement to terminate its data processing flows
To terminate the triggered data processing flows within the FogFlow system, we just need to delete thecreated requirement entity via the NGSI 10 interface. More details can be found from the following
document [4].

H2020 EUJ-02-2016

CPaaS.io

Page 28 of 40

6 Programmable Application Interfaces
As explained in the previous sections, FogFlow can automatically orchestrate dynamic data processing
flows and deploy them over cloud and edges in an optimized manner, given that a service topology has
been specified and the required building blocks (operators) are also provided. In this section, we describe
the application interfaces for operator developers to program a new operator and also for service
operators to manage their service topology in a programmable way.

6.1 Implement a docker-ized operator
Currently, FogFlow provides the templates to implement operators for JavaScript (nodejs-based) andPython. The operator developers can reuse them when developing their own operators. In the following,
we explain how to implement an operator based on the provided nodejs template.
After downloading the nodejs template, the operator developers can follow the provided framework to
implement their own application logic. Overall, they need to know the following important interfaces:
1. What is the main logic part of the provided template?
The following code block shows the main logic part of our provided nodejs template. It uses two
libraries: ngsiclient.js and ngsiagent.js.


ngsiclient.js is a nodejs-based based NGSI client to interact with FogFlow broker via NGSI10.
ngsiagent.js is a server to receive NGSI10 notify. With the provided libraries, the operator
application can perform NGSI10 update, subscribe, query.

In the main logic, three important functions are called. The responsibility of each function is explained as
below:
startApp: to do some initialization of an operator program; this function will be called right after
NGSIAgent starts listening in order to receive input data;
handleAdmin: to handle the configuration commands issued by FogFlow;handleNotify: to handle all received notify messages; please note that the subscription of the input data
is done by FogFlow on behalf of the operator application;
const NGSIClient = require('./ngsi/ngsiclient.js');
const NGSIAgent = require('./ngsi/ngsiagent.js');// get the listening port number from the environment variables given by the FogFlow edge worker
var myport = process.env.myport;// set up the NGSI agent to listen on
NGSIAgent.setNotifyHandler(handleNotify);
NGSIAgent.setAdminHandler(handleAdmin);NGSIAgent.start(myport, startApp);process.on('SIGINT', function() {NGSIAgent.stop();
stopApp();
process.exit(0);});

H2020 EUJ-02-2016

CPaaS.io

Page 29 of 40

function startApp(){console.log('your application starts listening on port ',myport);// to add the initialization part of your application logic here}
function stopApp(){console.log('clean up the app');}

2. How to initialize something when the operator program starts?
By overwriting the following function, the operator developers can implement the initialization logic of
their own operator programs. This function will be called only once when the operator program islaunched in a docker container.
function startApp(){console.log('your application starts listening on port ',myport);// to add the initialization part of your application logic here}

3. How is the operator configured by FogFlow?
For any generated results, the operator can publish them by sending NGSI updates to the given IoT
broker. The given IoT broker and the entity type to be used for publishing the generate results areautomatically configured by FogFlow.
Once FogFlow launches a task associated with the operator, FogFlow worker will send the configuration
to the task (a running instance of the operator) via the listening port. In the following code block, weshow how the operator should handle this provided configuration. The following global variables are
configured by FogFlow.







brokerURL: URL of the IoT Broker assigned by FogFlow;ngsi10client: the NGSI10 client created based on the given brokerURL
myReferenceURL: the IP address and port number that the operator application is listening;
Using this reference URL the operator application can subscribe to extra input data from theprovided IoT Broker
outputs: the array of the generated output entities
isConfigured: to indicate whether the configuration has been done// global variables to be configured
var ngsi10client;var brokerURL;
var myReferenceURL;
var outputs = [];var isConfigured = false;// handle the configuration commands issued by FogFlow
function handleAdmin(req, commands, res){handleCmds(commands);

H2020 EUJ-02-2016

CPaaS.io

Page 30 of 40

isConfigured = true;
res.status(200).json({});}// handle all configuration commands
function handleCmds(commands){for(var i = 0; i < commands.length; i++) {var cmd = commands[i];handleCmd(cmd);}}// handle each configuration command accordingly
function handleCmd(commandObj){
switch(commandObj.command) {case 'CONNECT_BROKER':
connectBroker(commandObj);
break;
case 'SET_OUTPUTS':
setOutputs(commandObj);
break;
case 'SET_YOUR_REFERENCE':
setReferenceURL(commandObj);
break;}}// connect to the IoT Broker
function connectBroker(cmd){brokerURL = cmd.brokerURL;
ngsi10client = new NGSIClient.NGSI10Client(brokerURL);
console.log('connected to broker', cmd.brokerURL);}
function setOutputs(cmd){var outputStream = {};outputStream.id = cmd.id;
outputStream.type = cmd.type;outputs.push(outputStream);
console.log('output has been set: ', cmd);}function setReferenceURL(cmd){myReferenceURL = cmd.referenceURL
console.log('your application can subscribe additional inputs under the reference URL: ',myReferenceURL);}

4. How to handle the received entity data?
All input data are received as NGSI notification messages. The following function must be overwritten to
implement the data processing logic of the operator program.
// handle all received NGSI notify messages
function handleNotify(req, ctxObjects, res){console.log('handle notify');

H2020 EUJ-02-2016

CPaaS.io

Page 31 of 40

for(var i = 0; i < ctxObjects.length; i++) {console.log(ctxObjects[i]);processInputStreamData(ctxObjects[i]);}// process the input data stream accordingly and generate output stream
function processInputStreamData(data){
var type = data.entityId.type;console.log('type ', type);// do the internal data processing
if (type == 'PowerPanel'){// to handle this type of input} else if (type == 'Rule') {// to handle this type of input}}

5. How to send an update within the operator application?
For any generated results, the operator application can publish them by sending NGSI updates to thegiven IoT broker The given IoT broker and the entity type to be used for publishing the generate results
are configured by FogFlow.
// update context for streams
function updateContext(anomaly){
if (isConfigured == false) {console.log('the task is not configured yet!!!');
return;}var ctxObj = {};ctxObj.entityId = {};var outputStream = outputs[0];ctxObj.entityId.id = outputStream.id;
ctxObj.entityId.type = outputStream.type;ctxObj.entityId.isPattern = false;
ctxObj.attributes = {};ctxObj.attributes.when = {type: 'string',
value: anomaly['when']};ctxObj.attributes.whichpanel = {type: 'string',
value: anomaly['whichpanel']};
ctxObj.attributes.shop = {type: 'string',
value: anomaly['whichshop']};
ctxObj.attributes.where = {type: 'object',
value: anomaly['where']};ctxObj.attributes.usage = {type: 'integer',value: anomaly['usage']

H2020 EUJ-02-2016

CPaaS.io

Page 32 of 40

};
ctxObj.metadata = {};ctxObj.metadata.shop = {type: 'string',
value: anomaly['whichshop']};ngsi10client.updateContext(ctxObj).then( function(data) {console.log('======send update======');
console.log(ctxObj);
console.log(data);}).catch(function(error) {console.log(error);console.log('failed to update context');});}

6.2 Register an implemented operator programmatically
The operator developer can also register an operator docker image by sending a constructed NGSI
update message to the IoT Broker deployed in the cloud.
Here is a Javascript-based code example to register an operator docker image. Within this code example,
we use the Javascript-based library to interact with FogFlow IoT Broker. The library can be found from thegithub code repository (designer/public/lib/ngsi). The ngsiclient.js module must be included into the
web page.
var image = {name: "counter",tag: "latest",
hwType: "X86",
osType: "Linux",
operatorName: "counter",prefetched: false};//register a new docker image
var newImageObject = {};newImageObject.entityId = {id : image.name + ':' + image.tag,
type: 'DockerImage',
isPattern: false};
newImageObject.attributes = {};newImageObject.attributes.image = {type: 'string', value: image.name};
newImageObject.attributes.tag = {type: 'string', value: image.tag};
newImageObject.attributes.hwType = {type: 'string', value: image.hwType};
newImageObject.attributes.osType = {type: 'string', value: image.osType};
newImageObject.attributes.operator = {type: 'string', value: image.operatorName};
newImageObject.attributes.prefetched = {type: 'boolean', value: image.prefetched};
newImageObject.metadata = {};newImageObject.metadata.operator = {type: 'string',
value: image.operatorName};// assume the config.brokerURL is the IP of cloud IoT Broker
H2020 EUJ-02-2016

CPaaS.io

Page 33 of 40

var client = new NGSI10Client(config.brokerURL);
client.updateContext(newImageObject).then( function(data) {console.log(data);}).catch( function(error) {console.log('failed to register the new device object');});

6.3 Define and register a service topology
Service developers can also register a service topology by sending a constructed NGSI update message
to the IoT Broker deployed in the cloud.
Here is a Javascript-based code example to register an operator docker image. Within this code example,
we use the Javascript-based library to interact with FogFlow IoT Broker. The library can be found from thegithub code repository (designer/public/lib/ngsi). Also, the ngsiclient.js library must be included into theheader part of the web page.
var topology = {"description":"detect anomaly events from time series data points","name":"anomaly-detection","priority": {"exclusive": false,"level": 100},"trigger": "on-demand","tasks":[{"name":"AnomalyDetector","operator":"anomaly","groupBy":"shop","input_streams":[{"type": "PowerPanel","shuffling": "unicast","scoped": true},{
"type": "Rule","shuffling": "broadcast","scoped": false}],"output_streams":[{"type":"Anomaly"}]},{"name":"Counter","operator":"counter","groupBy":"*","input_streams":[{"type":"Anomaly","shuffling": "unicast","scoped": true}],
"output_streams":[{"type":"Stat"}]

H2020 EUJ-02-2016

CPaaS.io

Page 34 of 40

}]}Given a service topology object defined as above, here is the Javascript-based code example to register aservice topology object.
var topologyCtxObj = {};topologyCtxObj.entityId = {id : 'Topology.' + topology.name,type: topology.name,isPattern: false};
topologyCtxObj.attributes = {};topologyCtxObj.attributes.status = {type: 'string', value: 'enabled'};topologyCtxObj.attributes.template = {type: 'object', value: topology};// assume the config.brokerURL is the IP of cloud IoT Broker
var client = new NGSI10Client(config.brokerURL);// send NGSI10 updateclient.updateContext(topologyCtxObj).then( function(data) {console.log(data);}).catch( function(error) {console.log('failed to submit the topology');});

6.4 Trigger/Terminate a service topology
Once a service topology has been submitted to FogFlow, we can use a NGSI application to trigger the fullor partial data processing of this service topology, by sending a specific NGSI entity update, which iscalled “requirement” in FogFlow.
Here is the Javascript-based code example to send a customized requirement object via NGSI10 update.
This requirement entity is identified by its entity ID. Later on, if the service operators would like to issue
any change to the same requirement, they can send a NGSI10 update to this requirement entity basedon its entity ID.

var rid = 'Requirement.' + uuid();var requirementCtxObj = {};requirementCtxObj.entityId = {id : rid,
type: 'Requirement',
isPattern: false};
var restriction = {scopes:[{scopeType: geoscope.type, scopeValue: geoscope.value}]};requirementCtxObj.attributes = {};requirementCtxObj.attributes.output = {type: 'string', value: 'Stat'};requirementCtxObj.attributes.scheduler = {type: 'string', value: 'closest_first'};requirementCtxObj.attributes.restriction = {type: 'object', value: restriction};requirementCtxObj.metadata = {};requirementCtxObj.metadata.topology = {type: 'string', value: curTopology.entityId.id};
console.log(requirementCtxObj);

H2020 EUJ-02-2016

CPaaS.io

Page 35 of 40

// assume the config.brokerURL is the IP of cloud IoT Broker
var client = new NGSI10Client(config.brokerURL);
client.updateContext(requirementCtxObj).then( function(data) {console.log(data);}).catch( function(error) {console.log('failed to send a requirement');});

A detailed application example can be found from the FogFlow code repository [5].

7 Graphical Interfaces
FogFlow Service Orchestration provides the following functionalities to service providers (behave as both
service developers and service operators) via the FogFlow Task Designer.

7.1 Registering a docker-ized operator
Figure 18 shows the list of all registered operator docker images and the key information of each image.

Figure 18: list of all submitted operators
After clicking the “register” button, the service provider can see a form as shown in Figure 19 below.
Please fill out the required information and click the “register” button to finish the registration. The form
is explained as the following.






Image: the name of the operator docker image
Tag: the tag that is used to publish the operator docker image; by default it is “latest”
Hardware Type: the hardware type that the docker image supports, including X86 or ARM (e.g.
Raspberry Pi)
OS Type: the operating system type that the docker image supports; currently this is only limited
to Linux
Operator: the operator name, which must be unique and will be used when defining a service
topology

H2020 EUJ-02-2016

CPaaS.io

Page 36 of 40Pre-fetched: if this is checked, that means all edge nodes will start to fetch this docker image inadvance; otherwise, the operator docker image is fetched on demand, only when edge nodes
need to run a scheduled task associated with this operator.

Figure 19: registration of a docker-ized operator

7.2 Editing and submitting a service topology;
Figure 20 shows the graphical user interface used to specify a service topology. As seen in the picture,
the following important information must be provided.
1. define a topology profile, including

topology name: the unique name of the service topology
service description: some text to describe what this service is aboutpriority: define the priority level of all tasks in the topology, which will be utilized by edge nodes
to decide how resource should be assigned to tasks
resource usage: define if the tasks in this topology can use the resources on edge nodes in an
exclusive way, meaning that not sharing resources with any task from the other topologies

2. draw the graph of data processing flows within the service topology
With a right click at some place of the design board, the service provider will see a memu pops up andthen the service provider can start to choose either task or input streams to define the data processing
flows according to the logical design that has been finished in the design phase.
3. define a task profile for each task in the data flow
As shown in the following picture, the service provider can start to specify the profile of each task in thedata processing flow by clicking the configuration button. The following information is required to
specify a task profile.


name: the name of the task
operator: the name of the operator that implements the data processing logic of this task; please
register the operator beforehand so that it can be shown from the list.
groupby: to determine how many instances of this task should be created on the fly; currently
including the following cases:

H2020 EUJ-02-2016

CPaaS.io

Page 37 of 40

if there is only one instance to be created for this task, please use “groupby” = “all”
if one instance is needed for each unique entity ID of the input streams, please user
“groupby” = “entityID”
 if one instance is needed for each unique value of a specific context metadata, please use
the name of this registered context metadata
shuffling of input streams: to indicate how the input stream should be assigned to the instance(s)of this task during the runtime, including the following two cases: “shuffling” = “broadcast”: the selected input streams should be repeatedly assigned toevery task instance of this operator
 “shuffling” = “unicast”: each of the selected input streams should be assigned to a specifictask instance only once
entity type of output streams: to specify the entity type of the produced output stream







Figure 20: graphical editor of service topology

7.3 Managing the registered service
Figure 21 shows the interface for service operators to start/stop a service topology with a defined geoscope on the map. By doing so, we can define a requirement with the selected geo-scope. Once we click
the start button, the defined requirement will be issued to the FogFlow system.

H2020 EUJ-02-2016

CPaaS.io

Page 38 of 40

Figure 21: defining a requirement with a specified geo-scope

8 Conclusions and Next Steps
This deliverable reports the overall design of the CPaaS.io Cloud-Edge computing framework, withregards to both heavy edges and lightweight edges. It also describes the specification of our edge
programming model and the framework introduced in order to realize such a programming model.
Below we summarize the latest achievements and next steps for WP4.


Our achievements in WP4

In terms of profile specification for Task 4.2, we designed and implemented a Cloud-edge programming
model for service/application developers;
In terms of cloud-edge orchestration mechanism for Task 4.1, we can already support dynamic service
orchestration between cloud and heavy edges and also explore the binary migration between cloud andlight-weight edges;
In terms of optimization of task deployment for Task 4.3, we implemented a locality aware task
deployment algorithm based on geo-location of IoT devices;
In terms of use case validation, we already finished the design of “smart parking” use case based on our
programming model supported by FogFlow and the use case implementation is now under further
development.
What to do next

We will continue the development of “smart parking” use case and also enhance our programming
model to provide better adoption to a wide range of smart city applications.
H2020 EUJ-02-2016

CPaaS.io

Page 39 of 40

Since the basic programming model and cloud-edge computing framework have established, we will
focus more on the optimization of task migration algorithms with regards to device mobility and secure
and privacy constraints. Also we will continue to improve the overall system performance and carry out
some performance evaluation. The Japan side will continue to enhance and realize their approach of task
migration over lightweight edges.

9 References[1] CpaaS.io deliverable D4.1, “Analysis and Selection of Task Migration Technologies”[2] Unikraft, http://sysml.neclab.eu/projects/unikraft[3] B. Cheng, A. Papageorgiou and M. Bauer, “Geelytics: Enabling On-Demand Edge Analytics over
Scoped Data Sources,” 2016 IEEE International Congress on Big Data (BigData Congress), San
Francisco, CA, 2016, pp. 101-108[4] NGSI10 APIs based on SWAGGER, https://app.swaggerhub.com/apis/fogflow/broker/1.0.0[5] FogFlow GE, https://github.com/Fiware/context.FogFlow

H2020 EUJ-02-2016

CPaaS.io

Page 40 of 40