モノのインターネットのためのグリーンインテリジェントエッジコンピューティング
メタデータ

言語: English
出版者:
公開日: 2022-12-19
キーワード (Ja):
キーワード (En):作成者: シュ, シャ
メールアドレス:所属:

URL

https://doi.org/10.15118/00010869

Green Intelligent Edge Computing forInternet of Things

SHA ZHU
Department of Sciences and Informatics
Muroran Institute of Technology

This dissertation is submitted for the degree ofDoctor of Philosophy of Engineering

June 2022

Declaration

I hereby declare that this thesis is my own work and effort and that it has not been submitted
anywhere for any award. Wherever contributions of others are involved, every effort ismade to indicate this clearly, with due reference to the literature, and acknowledgement ofcollaborative research and discussions.
SHA ZHU
June 2022

Acknowledgements

Firstly, I would like to express my sincere gratitude to my supervisor Prof. Kaoru Ota for thecontinuous support of my Ph.D. study and related research and for his patience, motivation,and immense knowledge. I have learned so many things from you, including the research
process, writing papers, giving talks, and many more.
Besides my supervisor, I would like to thank the rest of my thesis committee: Prof.
Kudo, Yasuo, and Prof. Mianxiong Dong, for their insightful comments and encouragement,
but also for the hard questions which encouraged me to widen my research from various
perspectives.
I want to thank Prof. Dong and Prof. Xu. Their guidance helped me in all the time ofresearch and writing of this thesis. I could not have imagined having a better advisor andmentor for my Ph.D. study.
I would like to acknowledge the financial, academic, and technical support of the MuroranInstitute of Technology and the financial support from the Japan Science and Technology
Agency, which has brought me many opportunities to publish our works in journals.
I owe a debt of gratitude to Xi, Ming, Bin, Weize, Hanxuan, Wenxin, Geigesen, Shuning,
Surina, and all my friends and colleagues in ENeS Lab at Muroran IT. Thanks for everything
you have done.
Last but not least, I would like to thank my family, my parents, and my husband forsupporting me spiritually throughout writing this thesis and my life in general.
SHA ZHU
June 2022

Abstract

Artificial intelligence (AI) driven IoT is an emerging opportunity for the future IoT since thedeep learning technology in the AI area efficiently absorbs and assesses the huge amount ofdata from billions of IoT devices. However, the energy consumption for processing AI tasks
is much higher than general tasks in the IoT environment. This dissertation presents a green
intelligent edge computing solution to improve the energy efficiency in processing AI tasks
from IoT devices.
There are three major challenges in work on the green intelligent edge computing
framework for IoT. The first challenge is to build the infrastructure of intelligent edge
computing for IoT. Moreover, the first challenge also introduces a scheduling strategy
to optimize the energy consumption of the proposed framework. The second challenge
mainly focuses on the energy efficiency of the intelligent edge computing framework in
the industry IoT (IIoT) environment. The second challenge shows the efficiency of a deep
reinforcement learning-based online scheduling strategy in reducing the energy consumption
and guaranteeing the processing latency for AI-driven IIoT. Finally, the last challenge
introduces an application of the proposed intelligent edge computing framework, which
improves the service capacity for early lung cancer diagnosis.

Table of contents
List of figuresxi

List of tables

xiii

1

Introduction
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 System Outline and Challenges . . . . . . . . . . . . . . . . . . . . . . . .
1.2.1 Building Green Intelligent Edge Computing for IoT . . . . . . . . .
1.2.2 Optimizing Energy Efficiency of Intelligent Edge Computing for
Industry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2.3 Applying Intelligent Edge Computing to Medicine . . . . . . . . .

1
1
3
3
4
4

2

Energy Efficient Artificial Intelligence of Things with Intelligent Edge
2.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.1 AIoT Applications . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2.2 Intelligent Edge Computing . . . . . . . . . . . . . . . . . . . . .
2.3 Intelligent Edge Computing Framework for AIoT . . . . . . . . . . . . . .
2.3.1 Multilevel Intelligent Edge Computing . . . . . . . . . . . . . . .
2.3.2 Energy Efficient Intelligent Edge Computing for AIoT . . . . . . .
2.4 Problem Formulation And Algorithm Design . . . . . . . . . . . . . . . .
2.5 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5
5
7
7
8
9
9
11
13
16

3

Green AI for IIoT: Energy Efficient Intelligent Edge Computing for Industrial
Internet of Things
25
3.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2.1 AI-driven IIoT . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2.2 Intelligent Edge Computing . . . . . . . . . . . . . . . . . . . . . 28

x

Table of contents
3.3

Scenario and Framework . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3.1 AI-driven IIoT Scenario . . . . . . . . . . . . . . . . . . . . . . .
3.3.2 Intelligent Edge Computing Framework . . . . . . . . . . . . . . .
Scheduling Problem Formulation and Solution . . . . . . . . . . . . . . . .
3.4.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.2 Scheduling Strategy . . . . . . . . . . . . . . . . . . . . . . . . .
Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5.1 Demonstration System . . . . . . . . . . . . . . . . . . . . . . . .
3.5.2 Experimental Result Analysis . . . . . . . . . . . . . . . . . . . .

29
29
30
33
33
35
39
39
44

4

Multiple-layer Intelligent Edge Computing in Early Cancer Diagnosis
4.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 DL for Early Cancer Diagnosis . . . . . . . . . . . . . . . . . . . . . . . .
4.3 Routes to Cancer Diagnosis . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4 MLIEC for Early Cancer Detection . . . . . . . . . . . . . . . . . . . . . .
4.5 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

47
47
48
49
51
52

5

Conclusions and Future Directions

57

3.4

3.5

References

59List of figures
1.1

Outline of the green intelligent edge computing for IoT . . . . . . . . . . .

3

2.1 Multilevel Intelligent Edge Scenario . . . . . . . . . . . . . . . . . . . . . 10
2.2 Structure of the intelligent edge computing framework for AIoT Applications 12
2.3 Power consumption of different devices . . . . . . . . . . . . . . . . . . . 17
2.4 Model Loading Time (YOLOv3) . . . . . . . . . . . . . . . . . . . . . . . 18
2.5 Average Processing Time (YOLOv3) . . . . . . . . . . . . . . . . . . . . . 18
2.6 Model Loading Time (VGG16) . . . . . . . . . . . . . . . . . . . . . . . . 19
2.7 Average Processing Time (VGG16) . . . . . . . . . . . . . . . . . . . . . 19
2.8 Frame Rate of the video processing task . . . . . . . . . . . . . . . . . . . 20
2.9 Increasing tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.10 Increasing edge nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.11 Increasing active ratio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.12 Increasing tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.1 A typical scenario for AI-driven IIoT applications . . . . . . . . . . . . . . 29
3.2 The intelligent edge computing framework for AI-Drive IIoT . . . . . . . . 31
3.3 An example of input states, including the edge node scheduling and three tasks 38
3.4 Power consumption of four edge devices in the testbed . . . . . . . . . . . 40
3.5 Model loading time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.6 Average processing time . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.7 Energy consumption with increasing tasks . . . . . . . . . . . . . . . . . . 42
3.8 Energy consumption with different number of edge nodes . . . . . . . . . . 43
3.9 Energy consumption with increasing active ratio . . . . . . . . . . . . . . . 43
3.10 Energy consumption with different latency requirments . . . . . . . . . . . 44
4.1
4.2

A route to cancer diagnosis in the healthcare hierarchy . . . . . . . . . . .
Structure of the multiple-layer intelligent edge computing for early cancer
diagnosis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

49
50

xii

List of figures
4.3
4.4
4.5

Power consumption and volume of different edge hardware . . . . . . . . .
Service capability of different DL models with different hardware . . . . .
Service capability and volumes in different edge layers . . . . . . . . . . .

53
54
55

List of tables
2.1

Hardware setting of the testbed . . . . . . . . . . . . . . . . . . . . . . . .

16

3.1
3.2

Hardware setting of the testbed . . . . . . . . . . . . . . . . . . . . . . . .
Parameter settings in all simulations . . . . . . . . . . . . . . . . . . . . .

39
45

4.1

Hardware settings of the testbed . . . . . . . . . . . . . . . . . . . . . . .

52

Chapter 1
Introduction
1.1

BackgroundBy 2025, there will be 42 billion Internet of things IoT devices globally that generate hugeamounts of data [19]. Since the deep learning technology in the artificial intelligence (AI)area efficiently absorbs and assesses large data, AI-driven IoT has become a most emerging
area in the future IoT. Typical AI-driven IoT applications, such as wearables, smart homes,
and smart cities, are changing human life.
However, the energy issue of AI-driven IoT will become more critical due to the highenergy consumption in processing AI tasks. A traditional way to build AI services in the IoT
environment is to employ high-performance cloud servers to process collected data from IoT
devices. Processing complicated AI tasks in the cloud usually needs high-end computing
hardware, including the graphics processing unit (GPU) and tensor processing unit, which
consume excessive energy. Moreover, transferring large data from IoT devices to the cloud isalso energy extensive.
Edge computing that brings computing from the cloud to the network edge plays anessential role in supporting IoT applications. Edge computing also improves the energy
efficiency of the entire IoT system with much fewer data transfers since the computing
is much near to IoT devices. Meanwhile, with the development of AI accelerators, newhardware platforms show incredible performance in processing AI tasks with low power
consumption. For example, NVIDIA Jetson AGX Orin, an embedded computer, has 275
TOPS of AI performance, similar to NVIDIA T4, an AI accelerator launched three years ago
for the datacenter [49]. The total power consumption of the new embedded device, including
CPU, GPU, and other modules, is only about 20
Another solution to design lightweight AI libraries or frameworks also reduces the energy
consumption of AI computing. Most AI-driven IoT applications focus on the inference phase2Introductionof deep learning, which has fixed weights in the learning model during these applications are
executed. Therefore, it is possible to reduce the complexity of the models and accelerate theprocessing. For instance, TensorRT, a machine learning framework provided by NVIDIA,
can accelerate deep learning inference in low spec hardware.
Therefore, intelligent edge computing will be an excellent opportunity for AI-drive IoT
applications by orchestrating state of the art on the new AI acceleration technologies intoedge computing:
1 It is possible to deploy new AI hardware to the edge environment, and some companies
have announced AI devices for edge computing.
2 Lightweight AI models can reduce the hardware requirements since the device spaceof the edge environment is much more strict than in the cloud.
3 Because edge computing is very near to IoT devices, the energy consumption of
network traffic will be significantly reduced.
However, there are still some challenges to improving the energy efficiency of intelligent
edge computing for IoT. The first challenge is from the heterogeneous hardware settings inmost edge environments. It is challenging to migrate tasks between different hardware forresource scheduling to improve the energy efficiency of the entire edge system. The second
challenge is unilinear relationships between task requirements and energy consumption inprocessing AI tasks. In most deep learning tasks, relationships between energy consumption
and task parameters, models, input data, and accuracy are usually unpredictable. Applying
intelligent edge computing is also a challenge since the development and deployment of AI
applications in the edge environment are very different from the cloud.

3

1.2 System Outline and Challenges

APPLICATION

Applying
Intelligent Edge
Computing to
Medicine

ALGORITHM
INFRASTRUCTURE

Optimizing Energy
Efficiency of
Building Intelligent
Intelligent Edge
Edge Computing
Computing forfor Internet ofIndustry
ThingsFig. 1.1 Outline of the green intelligent edge computing for IoT
As a result, as shown in Fig. 1.1, the research in this dissertation has three challenges,
building green intelligent edge computing for IoT, optimizing the energy efficiency ofintelligent edge computing for industry, and applying intelligent edge computing to medicine.

1.2

System Outline and Challenges

In this section, three major challenges in the dissertation are briefly introduced, and thefollowing chapters will discuss these challenges in detail.

1.2.1

Building Green Intelligent Edge Computing for IoT

Artificial Intelligence of Things (AIoT) is an emerging area of future IoT to support intelligent
IoT applications. In AIoT, intelligent edge computing technologies accelerate intelligent
services’ processing speed with much lower cost than simple cloud-aided IoT architecture.
However, it is still a lack of resource strategy to optimize the energy efficiency of AIoT withintelligent edge computing. Therefore, in the first major challenge of this dissertation, we
focus on the energy consumption of edge devices and cloud services in processing AIoT
tasks and formulate the optimization problem in scheduling tasks in the edge and the cloud.
Meanwhile, a novel online method is proposed to solve the optimization problem. We
investigate the energy consumption of several typical intelligent edge devices and the cloudservice in an intelligent edge computing testbed. Extensive simulation-based performance

4

Introduction

evaluation shows that the proposed method outperforms other strategies with lower energy
consumption.

1.2.2

Optimizing Energy Efficiency of Intelligent Edge Computing for
Industry

AI technology is a huge opportunity for the industrial Internet of Things (IIoT) in the fourthindustrial revolution (Industry 4.0). However, most AI-driven applications need high-end
servers to process complex AI tasks, bringing high energy consumption to IIoT environments.
In the second major challenge of this dissertation, we introduce intelligent edge computing,emerging technology to reduce energy consumption in processing AI tasks, to build green AI
computing for IIoT applications. We first propose an intelligent edge computing frameworkwith a heterogeneous architecture to offload most AI tasks from servers. To enhance theenergy efficiency of various computing resources, we propose a novel algorithm to optimize
the scheduling for different AI tasks. In the performance evaluation, we build a small testbed
to show the AI-driven IIoT applications’ energy efficiency with intelligent edge computing.
Meanwhile, extensive simulation results show that the proposed online scheduling strategy
consumes less than 80% energy of the static scheduling and 70% of the first-in, first-out(FIFO) strategy in most settings.

1.2.3

Applying Intelligent Edge Computing to Medicine

Intelligent edge computing is becoming a significant architecture for supporting AI applications in healthcare and medicine. It will be a great opportunity for early cancer diagnosis
from imaging tests. However, due to the high computing workload of existing computer
vision models, it is challenging to apply high precision models to the edge environment
with low spec hardware. In the last major challenge of this dissertation, the author analyzes
the performance of popular computer vision models for early cancer diagnosis in an edge
computing environment. According to the hierarchy of the healthcare system, the author also
proposes a multiple-layer intelligent edge computing (MLIEC) structure. The multiple layer
structure shows potential for large-scale early cancer diagnosis from the experimental results
in an edge computing testbed.

Chapter 2
Energy Efficient Artificial Intelligence ofThings with Intelligent Edge
2.1

Motivation

As the connection of the internet of things (IoT) and artificial intelligence (AI) technologies,
the artificial intelligence of things (AIoT) will deeply change human life by providing various
high intelligent services [10]. AIoT will play a critical role in traditional IoT areas, and othernew applications such as autonomous driving and health care [59, 46]. New AI technologies
and high-speed mobile networks will accelerate the deployment of AIoT in more potential
fields [55].
Due to the complex computing of AI tasks, cloud-aided AIoT that high-performancecloud servers process collected data from IoT devices with AI models is the general way tobuild AIoT service. However, the long distance between the cloud datacenter and IoT devices
seriously affects the quality of experience (QoE) of AIoT applications [72]. Intelligent edge
computing brings the AI process from the cloud near to AIoT devices by deploying AI
accelerators in the edge [36].
The performance and energy consumption of edge devices in processing AI tasks differ
from different hardware, and task types [47]. For instance, the NVIDIA EGX AI platform
has the highest performance and energy consumption while Jeston Nano has the lowest
performance energy consumption [51, 14]. Therefore, it is necessary to build an energyefficient intelligent edge computing framework for AIoT by orchestrating resources fromdevices, the edge to the cloud.
There are two challenges in building an energy-efficient edge computing framework for
AIoT, the limited scalability during edge devices processing massive AI tasks and the lack of6Energy Efficient Artificial Intelligence of Things with Intelligent Edge

cooperation between different computing hardware, including edge devices and the cloud[13, 3]. Depending on the newest edge devices, it is possible to process AI tasks with enoughperformance. However, a single device’s scalability is very limited due to the master–slave
architecture of most AI accelerators in computer systems [20]. Very few products support
concurrent processing with virtualization technologies designed for the cloud environment[28]. Therefore, the difficulty is to get good energy efficiency in both peak and valley loadtime for the edge computing framework.
Another challenge is that multiple edge devices are hard to parallel or distributed AI tasks
due to the low network performance between edge devices or between edge devices and thecloud. Meanwhile, only a few AI models support parallel or distributed processing [9]. As aresult, in the intelligent edge computing framework, it is necessary to consider the energy
consumption during processing AI tasks of different scales.
In this article, we try to address the above challenges and propose a multilevel intelligent
edge framework that orchestrates lightweight edge devices, high-performance edge devices,and the cloud to processing AI tasks in different load times and different scales with highenergy efficiency. Meanwhile, the QoE of AIoT tasks is also guaranteed with acceptablelatency and a transparent deployment interface.
Based on the proposed framework, we formulate the resource scheduling problem andintroduce the reinforcement learning model to support online task assignments. A small
testbed is applied to investigate the energy consumption in processing different AI tasks
in several edge devices and servers. Finally, we take extensive simulations to evaluate theproposed online scheduling method in a large-scale AIoT environment.
We list three major contributions of this paper as follows.
• We first design an intelligent edge computing framework to improve AI applications’energy efficiency, which is an original and creative work after a literature survey.
• We then state the proposed framework’s scheduling problem and give a reinforcement
learning-based solution for online scheduling.
• A demonstration framework is developed in a small testbed and the intelligent edge
devices show excellent performance-to-power ratio in processing AI tasks. Meanwhile, results from extensive simulations show that the proposed scheduling method
outperforms other solutions.

2.2 Related Work

2.2

7

Related WorkIn this section, we first present the background of the AIoT applications and then introduce
intelligent edge computing.

2.2.1

AIoT Applications

There are mainly four categories of AIoT applications, healthcare, smart home, smart
transportation, and industry. In smart healthcare, AIoT has been applied for health monitoringand disease analysis. It is easy to deploy wearable IoT devices for collecting data fromhuman life. To correctly analyze the collected raw data to health information, AI technology
plays a more critical role. A successful AI model, long short-term memory (LSTM), hasbeen applied to process various time series collected data such as accelerometer data andheart rate data [77]. Meanwhile, the convolutional neural network (CNN) is another efficientmodel for processing collected data from wearable devices [24].
Computer vision methods are also playing an essential role in smart healthcare foranalyzing medical image data. In the last decade, CNN has been used for recognizing orclassifying diseases in the professional medical area. Moreover, analyzing diseases fromdaily life images is also becoming possible with rapidly developed AI models. Lopez et al.
[56] introduce the VGGNet CNN model in recognizing skin diseases from a standard skinphoto, which shows acceptable accuracy on a given dataset.
The smart home is another important area of AIoT applications. AI technologies mainly
help smart home IoT devices in three ways, including indoor localization, intelligent control,and home robotics. AI models can find the correct position information of IoT devices
from the slightly changed Wi-Fi signals [35]. Intelligent control means IoT devices can becontrolled by human behaviors, such as gestures and human voice, rather than switches [33].
AIoT is mainly applied in autonomous navigating home robotics by processing sensing data
to understand home environments. Most works focus on the CNN-based object recognition
from robots’ cameras [16]. Meanwhile, light detection and ranging (LiDAR) has been used
for analyzing environments for home robotics [1].
AIoT applications in smart transportation mainly include three ways, traffic flow prediction, traffic monitoring, and autonomous driving. It is possible to predict traffic flow by
AI models through collected traffic data, and the results are usually better than traditional
machine learning methods. Traffic monitoring applies AI models to recognize vehicles andpedestrians from video data collected by surveillance cameras. Autonomous driving is themost challenging problem of AIoT applications in smart transportation, and many researchersand companies focus on new technologies or hardware for safe driving.

8

Energy Efficient Artificial Intelligence of Things with Intelligent Edge

In the smart industry, two significant areas apply AIoT technologies, data analysis
and automated manufacturing. Various data analysis-based predictions, such as demand
forecasting and predictive maintenance, are significant to industrial manufacturing [22].
Therefore, AI technologies, including traditional machine learning and deep learning methods,
have been applied in analyzing data collected from industrial environments. Automatedmanufacturing usually applies computer vision-based AI technologies to recognize theobjects recorded by sensors for future decision-making [15, 7].

2.2.2

Intelligent Edge Computing

Since AIoT has been applied in many critical areas, it is necessary to find an energy-efficient
way to build AIoT fundamental. Intelligent edge computing is the biggest opportunity
for building a stable and scalable infrastructure for future AIoT applications. After Edge
computing meets AI technologies, intelligent edge computing brings an evaluation to the
traditional edge and IoT areas [43].
The first stage of the intelligent edge is offloading AI tasks from the cloud to the edge.
However, the spec of earlier edge hardware is too low to execute traditional compute-boundAI tasks. There are several ways to reduce the computing requirement of AI tasks for
deployment in edge devices.
The simplest way is pruning the AI model and trimming weights to reduce the model
complexity. Tiny-YOLO [75] is the most successful object recognition model for lowspec devices, which maintains good enough performance after drastically reducing the
network complexity. However, since it is hard to fit different hardware with fixed pruning,
Bhattacharya et al. [5] propose an efficient pruning method named weight factorization to
optimize the complex AI models adaptively.
Segmentation of AI models is another solution that divides existing AI models into
different parts and pushes parts of models to edge devices. The benefit of AI model segmentation reduces the computing load of edge devices without model modification. Li et al.
[35] propose a vertically partitioning to support CNN in low-spec edge devices to support
IoT applications. However, the extensive data transferring in the network communications
between the edge and the cloud still affects the processing performance.
AI models with multiple outputs are also applied for adapting different edge devices. For
low-spec devices, AI tasks are processed through the outputs with less computing load. In
high-spec devices, AI tasks will be processed through the entire model for best precision.
Teerapittayanon et al. [63] first propose the AI model with different outputs for the local
device, edge devices, and cloud servers. The entire network will process input data for the
cloud output and process the input data with several neural layers for the local output.

2.3 Intelligent Edge Computing Framework for AIoT9

Due to the rapid development of edge devices, most AI models can be processed in new
published hardware without modification. NVIDIA has introduced the most powerful A100
GPU to the edge, which can process AI tasks and processors in data centers [52]. Meanwhile,
low-cost edge devices (e.g., NVIDIA Jetson Xavier NX [53]) can process multiple AI tasks
simultaneously.
Therefore, in existing AIoT environments, the edge devices are powerful enough toprocess AI tasks from AIoT devices. Due to the heterogeneous architectures and distributed
structure, efficient scheduling strategies are among the most challenging intelligent edge
computing issues. Q-learning-based scheduling is a promising methodology applied to
optimizing resource scheduling in edge environments [26]. However, it still needs an efficient
resource scheduling strategy for intelligent edge computing due to AI tasks’ particular
computing model. This paper focuses on the dedicated scheduling strategy for intelligent
edge computing in AIoT environments.

2.3Intelligent Edge Computing Framework for AIoT

This section introduces the concept of intelligent edge computing for AIoT and then presents
the proposed framework design.

2.3.1

Multilevel Intelligent Edge Computing

As shown in Fig. 2.1, we use a typical edge structure to illustrate multilevel edge computing
in a general network environment. We simply divide the network into several levels, including
the device lever, customer edge level, network edge level, access, and core network level. In
most common sense, edge computing refers to computing in the first four levels. We discuss
the limitations and potentials of intelligent edge computing in these four levels.
There are two limitations, network connections and hardware specifications, for processing AI tasks at the device level. Wireless connections connect most IoT devices with very
limited bandwidth. For example, the Zigbee protocol is still used for connecting IoT devices
due to its low energy consumption and a large number of connectable devices. The hardware
specifications of most IoT devices are also very low for low cost and energy consumption.
However, processing AI tasks is possible with specific hardware such as field-programmablegate array (FPGA) or application-specific integrated circuit (ASIC) chips in high-end IoT
devices. Since no data is transferring between IoT devices and upper levels, it is possible tosupport lightweight AIoT applications in IoT devices.

10

Energy Efficient Artificial Intelligence of Things with Intelligent Edge*+$#,-)+)'&(!""#$$ %&'()$

34-5(
6&'%7&8,"8$-.&2()*+&

9&$:-2;()*+&

377&88(
9&$:-2;.&/#,0#)1&/2.3&45,-#/6'"#!"#$%#&'&#()*+&(,-./"$%0+(1-2(34-5
Fig. 2.1 Multilevel Intelligent Edge Scenario
There is enough space to deploy powerful edge nodes for processing AI tasks at thecustomer edge level, such as a home, a vehicle, or a factory. From the NVIDIA company,edge computing has been applied in supper markets, manufacturing factories, city roads,
hospitals, and so on [50]. Moreover, these powerful edge devices can process multiple AI
tasks concurrently. For example, the Jeston AGX Xavier platform can process the four object
recognition tasks concurrently at the frame rate of 25 FPS. The network performance is also
better than the device level using high-speed wireless links or wired links. However, it ischallenging to afford high-end edge servers for most customers, which is hard to share withmultiple user groups.
The network edge refers to the access points of the public network services. At thislevel, space and power are still limited for deploying large edge servers. Unlike the edge
nodes in the customer edge level, users from different groups can share the edge nodes at
the network edge level. It is also easy to aggregate collected data in a small area for further
processing. At this level, both customers and carriers can deploy edge computing services
for AIoT applications, and some high-performance edge servers will be placed at this level.
Since only carriers can deploy edge servers at the access network level, the edge computing services will be more general to satisfy different requirements. Customers can rent
computing resources at this level to process some complex AI applications. Depending on
the access network fundamentals, it is possible to deploy small data centers with high-speed2.3 Intelligent Edge Computing Framework for AIoT

11

network connections. However, since the distance from the access network level to devices islonger than the bottom levels, the response latency will be affected by the data transferring.

2.3.2

Energy Efficient Intelligent Edge Computing for AIoT

According to the four levels of intelligent edge computing, we design an energy-efficient
framework to reduce energy consumption as well as to guarantee the QoE in processing AI
tasks. From Fig. 2.2, the edgy framework consists of four major modules, including the edge
node scheduler, the task manager, the AI processing interface, and the management interface.
The framework manages different edge nodes from the customer edge to the accessnetwork. We use the edge node scheduler to record all edge nodes’ status and information in
the network, including the workload, hardware specs, and software settings. Meanwhile, theresource scheduling strategy is executed to find the best edge node for processing the coming
AI tasks.
The task manager module monitors tasks in all edge nodes, including executing time, edge
node identifications (IDs), task identifications (IDs), and resource consumption. The edge
node scheduler can schedule AI tasks according to the information from the task manager
module. Meanwhile, the task manager module controls the states of tasks and edge nodes toswitch a task or an edge node from one state to another.
AI processing interface module provides a unified command set for the AI model storage.
Since different edge nodes and AI platforms have different commands for processing AI
tasks, the AI processing interface module can transfer the device-related or platform-related
interface to the unified interface for upon module.
Like the AI processing interface module, the management interface module also provides
the unified interface for the upon module to control edge nodes. After the task manager
module sends the edge node ID or task ID and the operations to the management interface
module, all operations will be transferred to the direct commands to the given edge node.
We also add an AI model management in the framework to improve the QoE of AIoT
application deployment. Because there are many types of edge hardware, customers needto develop hardware-specific models for different interfaces. In the proposed framework,
we add a module to store some pre-trained AI models for different hardware in the edge
environment. Meanwhile, a model converter is also deployed in the framework to transfer
user models to the hardware-specific models for better energy efficiency.

12Energy Efficient Artificial Intelligence of Things with Intelligent Edge

Intelligent Edge Framework for AIoT
Edge Node
Scheduler

Task Manager

AI Processing
InterfaceManagement
Interface

Access Network

Edge
Node

Edge
Node

Edge
Edge
Access Point Node Access Point Node
Customer Edge

Edge
Node

AIoT
DevicesCustomer Edge

AIoT
DevicesEdge
NodeFig. 2.2 Structure of the intelligent edge computing framework for AIoT Applications

13

2.4 Problem Formulation And Algorithm Design

2.4

Problem Formulation And Algorithm Design

In this section, the energy-efficient scheduling problem in the intelligent edge computing
framework for AIoT is formulated and then solved by an online scheduling method.
First, let set E denote edge nodes in four edge computing levels and ei denote an edge
node in set E. All AI tasks are denoted by set W , and one task is denoted by w j . The
total time of the scheduling has T timeslot, and a timeslot is denoted by t. Therefore, thescheduling for task w j and edge ei is presented by Xitj , given by(
1, task w j is processed in edge node ei ,Xitj =
0,elsewise.

(2.1)

Meanwhile, we use ci to denote the capacity of edge node ei and b j to denote the required
computing resource units of tasks w j . In the scheduling strategy, the required computing
resource units of assigned tasks are no more than the capacity of edge nodes, given by|W |∑ Xitj · b j ≤ ci.

(2.2)

j=1

Then, we formulate the energy consumption of the given scheduling. We have investigatedmost existing edge devices for processing AI tasks and found that all devices have stable
power supplies. Therefore, in this paper, we focus on optimizing the total power consumptionof the entire edge computing system and assuming that the energy consumption of different
devices has the same weight. Let P denote the total power consumption of the scheduling,
and Pt denote power consumption in timeslot t. For edge node ei , there are two types of power
consumption, idle power consumption denoted by psi and computing power consumption
denoted by pci , in the scheduling.
Since AI tasks have different power consumption, we use pwj to denote the power consumption of task w j . In the given scheduling in time t, the computing power consumption is
calculated by
|W |

pci =

∑ pwj · Xitj .

(2.3)

j=1

The power consumption Pt in time t is calculated by
|E|

|W |
|W |
s
t
Pt = (pi · (H( Xi j )) +pwj · Xitj )
i=1
j=1
j=1

∑

∑

∑

(2.4)

14

Energy Efficient Artificial Intelligence of Things with Intelligent Edge

where function H(x) is the heaviside step function. We use a calculable representation of
H(x), given by
1 1x−x
H(x) = + · (⌊
⌋−⌊
⌋).
(2.5)
2 2
|x| + 1| − x| + 1
We assume the total energy consumption P is the sum of power consumption in eachtimeslot, given byT

P = ∑ Pt .

(2.6)

t=1

Since the framework is designed to guarantee the QoE in processing AI tasks, we use thecustomer satisfaction ratio denoted by Q. Considering response latency is the major part that
affects customer experience, the satisfaction ratio Q is defined by|W |∑ j Yj
Q=
|W |

(2.7)

where Y j denotes whether task w j is finished in a required time.
To simply the formulation of the scheduling. We assume the response time satisfies
customer requirements if task w j is finished in an edge node. Therefore, Y can be defined by(|E|T1, ∑t=1∑i=1 Xitj ≥ 0
Yt =0,elsewise.

(2.8)

From above definitions, the scheduling problem can be summarized byminimize :sub jectto :P
Q ≥ Qr(2.9)|W |∑ j=1 Xitj · b j ≤ ci .

Scheduling problem in energy efficient intelligent edge framework for AIoT: given
a set of intelligent edge node and a set of AIoT tasks, the edge node scheduler attempts to findscheduling set {X1 , X2 , ..., XT } to minimize the energy consumption of the edge computing
framework.
Since the scheduling problem is a Markov decision process (MDP) problem [36], we
design a reinforcement learning-based algorithm to solve the MDP problem. We first
define the reinforcement learning reward, denoted by rn , as the negative value of the power
consumption, given by
rn = −Pt .
(2.10)

2.4 Problem Formulation And Algorithm Design

15

Let Rn denote the total future discounted reward that is defined as|T |Rn = ∑ γ t−n · rn

(2.11)

t=nwhere γ < 1 is the discounted factor.
We use set O = {O1 , O2 , ..., O|T | } and Z = {Z1 , Z2 , ..., Z|T | } to denote actions and statesof the scheduling strategy, respectively. We also use set O = {O1 , O2 , ..., O|T | } and Z ={Z1 , Z2 , ..., Z|T | } to denote entire admissible action and state space. Let Q∗ (·) denote theaction-reward function with maximum reward in state Zn and an optimal action Zn . From the
Bellman optimality equation [44], the value of Q∗ (Zn , On ) can be estimated byQ∗ (Zn , On ) ≈ Q(Zn , On ; θ ).

(2.12)

where θ is the sequence of approximation weights.
In our scheduling strategy, we adopt the deep reinforcement learning method and let
Li (θi ) denote the loss function, given by
Li (θi ) =Eρ(Zn ,On ) [(EZn+1 [rn+ γ · Q(Zn+1 , On+1 ; θi−1 )|Zn , On ](2.13)− Q(Zn , On ; θi ))2 ]where ρ(Z, O) denotes the behavior distribution of Z and O.
The stochastic gradient descent (SGD) is applied to optimize (2.13) with the gradient as∇θi Li (θi ) = Eρ(Zn ,On ) [(rn + γ · Q(Zn+1 , On+1 ; θi−1 )− Q(Zn , On ; θi )) · ∇θi · Q(Zn , On ; θi )].

(2.14)We use distinct images to represent the scheduling states, including the scheduled edge
nodes and tasks. The image of the scheduled edge nodes is a table of edge capacities, inwhich each column is one unit of the processing resource and each row means the timeslot.
Colored cells represent allocated resources, and different color means different task. The
image of a task is also a table in which the width is the needed number of resource units, andthe height represents the time series.
The action space is given by {ea1 , ea2 , ..., ea|A| } where eak means the edge node assigned totask ak and Xeak k = 1. If the task ak ends or does not begin, the value of eak is zero. If theaction for one task is valid, the value of eak is ∅. In each timeslot, the scheduler will find avalid action or an invalid action. After the scheduler finds an action, all tasks will put into the16

Energy Efficient Artificial Intelligence of Things with Intelligent Edge

table for edge node scheduling with different colors. If eak = ∅ exists, the scheduler will shift
up by a one timeslot and find another action space.
Algorithm 1 Training algorithm for the scheduler
Initial replay buffer
for m form 1 to M do
Z0 is set as an empty state;
for n from 1 to |T | dop ← random(0, 1)
if p ≤ 1 − ε thenOn ← arg maxOn Q(Zn , On ; θn );else
Randomly select On from On ;end ifCalculate rn from (2.10);
Assign edge nodes with On and get Zn+1 ;Store (Zn , On , rn , Zn+1 ) in the replay buffer;
Sample batch B from the replay buffer;
for n′ from 1 to |B| do
Update θn by performing SGD;
end forθn−1 ← θn ;end forend for

2.5

Performance Evaluation
Table 2.1 Hardware setting of the testbed

Abbreviation
AGX
NX
Nano
A6000

Device Name
Jetson AGX Xavier
Jetson Xavier NX
Jatson Nano
RTX A6000 Server

Operating System
Ubuntu 18.04
Ubuntu 18.04
Ubuntu 18.04
Debian 10CPU8-core ARM v8.2
6-core ARM v8.2
4-core ARM A57
10-Core Core i9

Memory32GB 256-Bit LPDDR4x
8 GB 128-bit LPDDR4x
4 GB 64-bit LPDDR448 GB 256-bit GDDR6GPU
512-core Volta
384-core Volta
128-core Maxwell
10752-core Ampere

We first introduce the experimental testbed for investigating the power consumption ofseveral NVIDIA edge devices. The specifications of all devices, including NVIDIA Jeston
AGX Xavier, Jetson Xavier NX, and Jetson Nano, are summarized in Table 2.1. In themultilevel intelligent edge framework, the Jeston AGX Xavier is deployed at the accessnetwork level, the Jetson Xavier NX is deployed at the network edge level, and Jetson Nano

17

2.5 Performance Evaluation

devices are deployed at the customer edge level. We also use a server with an NVIDIA RTX
A6000 Graphic Card to show the performance and energy consumption of a cloud server.
We test the power consumption and average processing time of two deep learning models,
YOLOv3 and VGG16. In these tests, the edge devices process selected 50 images from
COCO 2017 image dataset, and we record the model loading time and processing time withtwo models. We also test the video processing speed of all edge devices with YOLOv3.
The video data is a 100-second video with a 623 kbps bit rate and 1280x720 resolution.
We converted the models to TensorRT-enabled models in edge nodes. The intelligent edge
framework is developed in a Macbook Pro 15 2018 computer with Python 3.9.2. We applied
TensorRT 7.1.3 library and CUDA 10.2 in edge nodes and CUDA 11.1 in the server. However,
we also test TensorRT models in the A6000 server, which results in performance degradation.
Considering that TensorRT may not well support the newest ampere architecture, we choose
the results of the original model in the A6000 server.
As shown in Fig. 2.3, we test the power consumption in processing AI tasks and idle stateof four edge devices. Since the Jetson AGX Xavier has the highest performance, its power
consumption is more than 30 watts. Jetson Xavier NX and Jeston Nano’s power consumption
is similar, while the former is more powerful than the latter. The power consumption of theserver is much higher than all edge nodes.

Power Consumption (Watt)

350
300

Idle
Processing

250
200
150
100
50
0

Nano

NX

AGX

A6000

DevicesFig. 2.3 Power consumption of different devices

18

Model Loading Time (ms)Energy Efficient Artificial Intelligence of Things with Intelligent Edge

12000
10000
8000
6000
4000
2000
0

Nano

NX

AGX A6000

Devices

Average Processing Time (ms)Fig. 2.4 Model Loading Time (YOLOv3)

1200
1000
800
600
400
200
0

Nano

NX

AGX A6000

DevicesFig. 2.5 Average Processing Time (YOLOv3)

19

Model Loading Time (ms)

2.5 Performance Evaluation

12000
10000
8000
6000
4000
2000
0

Nano

NX

AGX A6000

Devices

Average Processing Time (ms)Fig. 2.6 Model Loading Time (VGG16)

5000
4000
3000
2000
1000
0

Nano

NX

AGX A6000

DevicesFig. 2.7 Average Processing Time (VGG16)

20

Energy Efficient Artificial Intelligence of Things with Intelligent Edge

The results of the processing latency in the demonstration system are given by Fig.
2.4,2.5,2.6 and 2.7. Before processing AI tasks in edge devices, it is necessary to load thepre-trained or converted models to the edge nodes. From the results shown in Fig. 2.4,
because of its highest specifications, the A6000 server loads the AI models much faster thanall edge devices.
As shown in Fig. 2.5, the average processing time in edge nodes is different from modelloading time. As the quickest device, the processing time with the YOLOv3 TensorRT modelin Jeston AGX Xavier is about 270 ms, while the VGG16 TensorRT model is about 850
ms. The performance difference between Jeston AGX Xavier and Xavier NX is slight with
TensorRT optimizations. The performance of the Jetson Nano is also good enough, whichis less than three times the processing time on the server with the YOLOv3 model and fivetimes with the VGG16 model. Considering the power consumption of the A6000 serveris more than 30 times the Jetson Nano device, the energy efficiency of the edge device ismuch better than the high-end hardware. The intelligent edge computing framework shows
good enough performance with much lower energy consumption from the testbed’s power
consumption and performance test than the cloud platform.
Fig. 2.8 shows the results of the video processing speed in the demonstration system.
The A6000 server outperforms other devices with nearly 150 frames per second (FPS), while
the lightweight Jeston AGX Xavier platform can process more than 100 video frames in one
second. The Jetson Nano is not an appropriate platform for object detection tasks based on
the several FPS performance.

Frame Rate (FPS)

150

100

50

0

Nano

NX

AGX A6000

DevicesFig. 2.8 Frame Rate of the video processing task

2.5 Performance Evaluation

21

We then evaluate the performance of the scheduling strategy in the intelligent edge
computing framework for AIoT in a large environment. We apply extensive simulations forthe performance evaluation. A simulator is developed with Python 3.9.2 and the training is
executed in the A6000 server. The operating system is Debian 10. We adopt Keras 2.4.0 as
the deep learning platform. In the experiments, we apply the RMSProp optimizer for Keras
library. We set the size of mini-batches to 64. For the ε-greedy in the scheduler training, we
set the value of ε to 0.1.
We set the parameters of all simulations according to the experimental results. The tasks
are randomly generated in the time series, and the power consumption for processing tasks
is evenly distributed in [5, 25] watts. The required processing latency is evenly distributedin [2, 600] ms. The capacity of edge nodes is evenly distributed in [1, 16], and the required
resource units of tasks are evenly distributed in [1, 2]. The number of edge nodes is set
from 100 to 200. The number of tasks is set from 200 to 500 and the time series has 3600
time-slots where one timeslot is 1 second long. The ratio that tasks are active in the total timeseries is evenly distributed in [0.8, 1]. The required satisfaction ratio is set in [0.95, 0.99].
We compare the online scheduling results with a static scheduling algorithm (Static),a first-in, first-out (FIFO) strategy, and DeepRM, a resource management method withdeep reinforcement learning [45], in all simulations. The static algorithm is based on thescheduling method applied in the article [35], while FIFO is a simple strategy that formulates
all edge nodes as a queue and schedules tasks according to arriving time. All simulations are
executed 20 times, and the average values are recorded.

22

Energy Efficient Artificial Intelligence of Things with Intelligent Edge

Energy Consumption (Wh)

5000
4000

OnlineFIFO
Static
DeepRM

3000
2000
1000
0
200

250

300

350

400

450

500

Number of Tasks
Fig. 2.9 Increasing tasks

Energy Consumption (Wh)

2800
2600
2400
2200
2000
1800
100

OnlineFIFO
Static
DeepRM

120

140

160

180

Number of Edge Nodes
Fig. 2.10 Increasing edge nodes

200

23

2.5 Performance Evaluation

Energy Consumption (Wh)

5000

4000

OnlineFIFO
Static
DeepRM

3000

2000

1000
0.5

0.6

0.7

0.8

0.9

1

0.99

1

Task Activity Ratio
Fig. 2.11 Increasing active ratio

Energy Consumption (Wh)

6000
5000

OnlineFIFO
Static
DeepRM

4000
3000
2000
1000
0.95

0.96

0.97

0.98

Satisfaction Ratio
Fig. 2.12 Increasing tasks

24Energy Efficient Artificial Intelligence of Things with Intelligent Edge

All results of the performance evaluation are represented in Fig. 2.9, 2.10, 2.11 and2.12. As shown in Fig. 2.9, we change the number of tasks from 200 to 500 and test the
total energy consumption in one hour. The online scheduling strategy shows the lowest
energy consumption with the different number of tasks. DeepRM is almost the same as theonline scheduling strategy when the number of tasks is less than 250. When the number oftasks is less than 300, the FIFO method performs better than the Static algorithm results.
When the number of tasks increases to more than 300, the Static algorithm brings less energy
consumption than the FIFO method.
We also test the energy consumption with different numbers of edge nodes. The numberof tasks is set to 400, and the number of edge nodes increases from 100 to 200. From Fig.
2.10, when the number of edge nodes is less than 140, four methods perform similarly, andthe total energy consumption is more than 2600 Wh. After more edge nodes begin to process
AI tasks, the total energy consumption decreases. The online scheduling method still has thelowest energy consumption in the four methods, while DeepRM has a similar performancewith the online scheduling method.
The active ratio of the task in the time series also affects the total energy consumption. In
Fig. 2.11, we adjust the active ratio from 0.5 to 1.0 and test the total energy consumption.
The number of edge nodes is set to 200, and the number of total tasks is set to 500. The
results show that a higher active ratio will bring more energy consumption, and the online
scheduling strategy performs better than the other three algorithms.
Finally, we test the energy consumption with different satisfaction ratio requirements.
We set the number of tasks to 400 with 200 edge nodes, and the required satisfaction ratio
increases from 0.95 to 0.99. As shown in Fig. 2.12, the total energy consumption increases
with an increased satisfaction ratio. When the satisfaction ratio is less than 0.96, fourscheduling methods perform similarly, and the FIFO has less energy consumption than theStatic algorithm. When the satisfaction ratio becomes more than 0.97, the online schedulingstrategy performs much better than the other three methods.

Chapter 3
Green AI for IIoT: Energy EfficientIntelligent Edge Computing for
Industrial Internet of Things
3.1

Motivation

Artificial intelligence (AI) driven industrial internet of things (IIoT) will play a prominentrole in supporting high automation manufacturing [64, 58]. For example, different object
recognition models have been deployed in automated product assembly, factory unattended
monitoring, automated quality inspections, and other core processes of industrial production[25, 11]. Since most AI tasks are compute-bound, it is necessary to deploy high-end
infrastructure to support AI-driven IIoT in finishing each task in a short time interval [27, 78].
However, high-performance computing hardware, including processors, graphics processing
units (GPUs), and tensor processing units (TPUs), usually has high energy consumption inprocessing AI tasks [68]. As one of the most efficient computing devices for AI tasks, atypical NVIDIA RTX GPU has 400 watts peak power consumption [51].
In general AI-driven applications, users apply cloud to process AI tasks to reduce theadditional cost from both infrastructure and energy consumption [38, 47]. Large cloudproviders also have good energy efficiency due to the high utilization of resources and maturescheduling strategies on homogeneous architectures [14]. However, in IIoT environments, thecloud-based structure for general AI applications loses its effectiveness due to inconsistency
between the long distance from the cloud and high real-time requirements of IIoT applications[72]. By bringing AI computing near to the location where it is needed, intelligent edge
computing becomes an excellent opportunity for AI-driven IIoT because the data transfer time

Green AI for IIoT: Energy Efficient Intelligent Edge Computing for Industrial Internet of
26
Thingsis reduced [36]. Some intelligent edge nodes have adequate capability to process complex AI
tasks, including computer vision and natural language processing to support AI-driven IIoT
applications [73]. Meanwhile, unlike homogeneous structure in most cloud platforms, it ispossible to customize intelligent edge computing with heterogeneous hardware, which has
good energy efficiency in processing specific AI tasks [54, 21].
This paper tries to introduce intelligent edge computing into the IIoT scenario to support energy-efficient AI-driven IIoT applications. The most challenging problem in IIoT
environments is to guarantee the processing latency, which is a matter of life and death tomost industrial applications [74, 37]. In our design, a framework organizes all heterogeneous
resources and then finds the most feasible hardware to handle different AI tasks in therequired processing time. Since existing AI tasks are developed in several major AI libraries
such as PyTorch or TensorFlow, the framework can connect various AI tasks and different
AI hardware without modification. Another problem is to schedule heterogeneous hardware
for better energy efficiency in processing different AI tasks in intelligent edge [39]. Because
special AI hardware has better energy efficiency in processing specific tasks, the proposed
framework’s scheduling strategy leverages the task type and latency requirement to minimize
energy consumption and guarantee processing latency.
We also implement the demonstration system in a small testbed with different edge
devices to test the process latency of several basic AI tasks in the proposed framework.
Meanwhile, the scheduling strategy in the intelligent edge is evaluated in large-scale scenarios
by extensive simulations. The simulation results show that the online scheduling strategy can
reduce more than 20% energy consumption compared to static scheduling strategy and morethan 40% compared to the first-in, first-out (FIFO) strategy in most settings. We list threemajor contributions of this paper as follows.
• We first design an intelligent edge computing framework to improve the energy efficiency of AI-driven IIoT applications. To the best of our knowledge, the proposed
framework is original and creative.
• We then state the scheduling problem in the proposed framework and give two solutions
to solve the problem in both static and online scenarios, respectively.
• A demonstration framework is developed in our testbed, and the intelligent edge devices
show excellent performance-to-power ratio in processing AI tasks. Meanwhile, resultsfrom extensive simulations show that two proposed scheduling methods outperform
other solutions.

3.2 Related Work

3.2

27

Related WorkIn this section, we introduce existing research related to AI-driven IIoT and the intelligent
edge.

3.2.1

AI-driven IIoT

Bisio et al. [6] first investigate the relationship between industry 4.0 and IoT concept andthen analyze several context-aware schemes of AI-driven IIoT applications. From their work,
AI-driven IIoT systems can understand the surrounding environment and react to different
conditions according to the sensed data and pre-trained models.
In IIoT environments, two major areas apply AI technologies, data analysis, and automated manufacturing. Various data analysis-based predictions, such as demand forecastingand predictive maintenance, are significant to industrial manufacturing. Therefore, AI technologies, including traditional machine learning and deep learning methods, have been
applied in analyzing data collected from IIoT environments. Demand forecasting is usuallyapplied for forecasting energy demand in the IIoT environment. Han et al. [22] propose adeep learning framework that adopts a Long short-term memory (LSTM) model to predict
the future energy consumption of IIoT devices. In their design, cloud computing shows
good performance to support data analysis and energy prediction in IIoT environments.
Kanawaday et al. [32] apply a traditional machine learning method to analyze sensing datafrom a slitting machine and predict possible failures for further maintenance. Nguyen et al.
[48] design a deep learning-based framework to support dynamic predictive maintenance. A
basic LSTM model is used in this framework, which shows good precision in predicting thefuture system failure time.
Unlike data analysis in IIoT, automated manufacturing usually applies computer visionbased AI technologies to recognize the objects recorded by sensors for future decision-making.
AI technologies have been applied for anomaly detection which is an important issue inautomated manufacturing. Supervised deep learning [15, 7], semi-supervised deep learning[69], and unsupervised deep learning methodologies [57] have been introduced to solve theanomaly detection problem in IIoT environments. Meanwhile, some researchers apply new
deep learning methods, such as federated learning [42] and explainable AI [2], to detect
anomalies in manufacturing.
Radio-frequency identification (RFID) is an efficient technology in sensing objects in
manufacturing, which is stabler than the vision-based way in many scenarios [70, 71]. Huang
et al. [30] show a deep learning-based approach to monitor and predict manufacturing
progress in factories. The proposed approach design a transfer learning-based AI model toGreen AI for IIoT: Energy Efficient Intelligent Edge Computing for Industrial Internet of28
Thingsimprove the prediction performance with insufficient labeled data. However, due to the gap
between AI technologies and industrial informatics, the AI-driven IIoT area still needs astable and scalable solution for more comprehensive applications.

3.2.2Intelligent Edge ComputingIntelligent edge computing is the opportunity for building a stable and scalable infrastructure for future AI-driven IIoT applications. After Edge computing meets AI technologies,
intelligent edge computing brings an evolution to the traditional edge and IoT areas.
The first stage of the intelligent edge is offloading AI tasks from the cloud to the edge.
However, the spec of earlier edge hardware is too low to execute traditional compute-boundAI tasks. There are several ways to reduce the computing requirement of AI tasks for
deployment in edge devices. The simplest way is pruning the AI model and trimming
weights to reduce the model complexity. Tiny-YOLO [75] is the most successful object
recognition model for low-spec devices, which maintains good enough performance after
drastically reducing the network complexity. However, since it is hard to fit different hardware
with fixed pruning, Bhattacharya et al. [5] propose an efficient pruning method named weightfactorization to optimize the complex AI models adaptively.
Segmentation of AI models is another solution that divides existing AI models into
different parts and pushes parts of models to edge devices. The benefit of AI model segmentation reduces the computing load of edge devices without model modification. Li etal. [35] propose a vertically partitioning to support convolutional neural networks (CNN) in
low-spec edge devices to support IoT applications. However, the extensive data transferringin the network communications between the edge and the cloud still affects the processing
performance.
AI models with multiple outputs are also applied for adapting different edge devices. For
low-spec devices, AI tasks are processed through the outputs with less computing load. In
high-spec devices, AI tasks will be processed through the entire model for best precision.
Teerapittayanon et al. [63] first propose the AI model with different outputs for the local
device, edge devices, and cloud servers. The entire network will process input data forthe cloud output and process the input data with several neural layers for the local output.
Due to the rapid development of edge devices, most AI models can be processed in new
published hardware without modification. NVIDIA has introduced the most powerful A100
GPU to the edge, which can process AI tasks and processes in data centers [52]. Meanwhile,
low-cost edge devices (e.g., NVIDIA Jetson Xavier NX [53]) can process multiple AI tasks
simultaneously. Therefore, in existing IIoT environments, the edge devices are powerfulenough to process AI tasks from IIoT devices. Due to the heterogeneous architectures29

3.3 Scenario and Framework

Cloud
Access Network

Access Point

Access Point
Access Point

AI ModelAI ModelAI ModelAI Task Processor

Industrial
Robot

AI Task Processor

AI Task Processor
Heavy
Equipment

Fig. 3.1 A typical scenario for AI-driven IIoT applications

Transport
Vehicleand distributed structure, efficient scheduling strategies are among the most challenging
intelligent edge computing issues. Q-learning-based scheduling is a promising methodology
applied to optimizing resource scheduling in edge environments [26]. However, it still needs
an efficient resource scheduling strategy for intelligent edge computing due to AI tasks’particular computing model. This paper focuses on the dedicated scheduling strategy for
intelligent edge computing in AI IIoT environments.

3.3

Scenario and Framework

We present the research motivation in this section, including the research scenario of AIdriven IIoT with intelligent edge and the proposed framework’s design.

3.3.1

AI-driven IIoT Scenario

We first introduce what is meant by an AI task in the IIoT scenario. In a typical IIoT scenario,
different machine learning methods are deployed to process large amounts of data sensing
from IoT devices. In recent years, various deep learning technologies that show advantages
in processing complex data types such as images, videos, and sound have been deployed in
IIoT environments. Deep learning is also the central part of nowadays AI area. Therefore,
AI tasks are considered tasks containing deep learning parts for processing sensing data in
IIoT environments. After that, we use a typical AI-driven IIoT scenario shown in Fig. 3.1 toGreen AI for IIoT: Energy Efficient Intelligent Edge Computing for Industrial Internet of30
Thingsillustrate the issue in processing AI tasks. In this scenario, a factory has applied AI-driven
IIoT applications in the industrial robot, heavy equipment, and transport vehicle for different
purposes. To process AI tasks from different applications in required latency, the factory has
to place high-performance computers as AI task processors physically connected to industrial
equipment. Pre-trained AI models have been stored in AI task processors to support related
AI tasks. Meanwhile, the factory also builds an access network to connect equipment formonitoring and management. The equipment has wireless connections with the accessnetwork through access points. Access points are wired connected to the access network.
Cloud services are applied to support other applications such as databases, management tools,
or information sharing.
The AI task processors usually have high energy consumption due to the performancerequirement. It is not easy to apply traditional energy consumption optimization strategies
in the above scenario. Switching idle devices into a sleep state is a key operation in most
optimizations, which is not appropriate to industrial scenarios because the state changing
will affect the equipment stability. Improving the utilization of computing devices is anotherway to improve energy consumption. However, in the above scenario, AI task processors are
isolated in different wireless access points, and it is slow and unstable to transfer data flowfrom one equipment to another across the access network. Another problem is that dedicatedcomputing devices are applied to process AI tasks from different AI-enabled equipment,
which means migrating tasks across different equipment is not feasible. Therefore, it isnecessary to apply a new structure to support AI-driven IIoT, which can apply the energy
consumption optimization strategies and satisfy the strict latency requirement.
Furthermore, there are several opportunities in the AI-driven IIoT scenario: 1. The
categories or types of tasks in the IIoT scenario are much less than a standard IoT scenario.
Usually, the task categories in an IIoT scenario are determined by some given requirements.
After the IIoT system deployment, very few new task categories will be added to the scenario.
2. IIoT devices in a given environment are knowable and manageable, and the factory orcompany also limits the scale of IIoT devices. 3. The network connections and power
supplement in an IIoT environment are very stable.

3.3.2

Intelligent Edge Computing Framework

Intelligent edge computing is the opportunity to improve the energy efficiency of AI-driven
IIoT and meet the strict requirement from the processing time. As shown in Fig. 3.2, we
design the AI-driven IIoT framework based on intelligent edge computing.
Edge nodes are deployed on the edge of the access network, including the access points
and leaf switches, which are very near to industrial equipment. In a factory environment,

31

3.3 Scenario and Framework

Intelligent Edge Framework

Edge Node
Scheduler

AI ModelConverter

AI ModelStorage

Task Manager

AI Processing
InterfaceManagement
Interface

Original AI
Model

Converted
AI ModelAccess Point

Edge
Node

Edge
Node

Converted
AI ModelEdge
Converted Access Point NodeAI ModelEdge
NodeAccess Point
Edge
Node

Industrial
Robot

Heavy
Equipment

Transport
VehicleFig. 3.2 The intelligent edge computing framework for AI-Drive IIoT

Green AI for IIoT: Energy Efficient Intelligent Edge Computing for Industrial Internet of
32
Thingsit is possible to organize all edge nodes with a centralized controller, which is stable andconvenient to manage multiple edge nodes. The intelligent edge controller consists of theedge node scheduler, AI model converter, AI model storage, task manager, AI processing
interface, and management interface. We introduce these six parts and the working process
between them as follows. The edge node scheduler records all edge nodes’ status andinformation in the network, including the workload, hardware specs, and software settings.
Meanwhile, the resource scheduling strategy is executed to find the best edge node forprocessing the coming AI tasks.
Edge nodes process AI tasks based on delegated pre-trained AI models. However, most
AI models are platform-related and not executed directly in edge environments without
modification. The AI model converter automatically modifies pre-trained AI models to
the feasible models for given edge environments. Meanwhile, the AI model converter
will generate different converted AI models for heterogeneous edge devices. Since model
conversion is a time-consuming task, it is challenging to convert AI-models online. We design
an AI model storage to store and manage all converted AI models for further deployment.
The AI model storage will update deployed AI models in edge nodes after storing pre-trained
AI models with new versions. Therefore, a management table is maintained by the AI modelstorage to record the deployed AI models in each edge node.
The task manager module monitors tasks in all edge nodes, including executing time, edge
node identifications (IDs), task identifications (IDs), and resource consumption. The edge
node scheduler can schedule AI tasks according to the information from the task manager
module. Meanwhile, the task manager module controls the states of tasks and edge nodes
to switch a task or an edge node from one state to another. AI processing interface module
provides a unified command set for the AI model storage. Since different edge nodes andAI platforms have different commands for processing AI tasks, the AI processing interface
module can transfer the device-related or platform-related interface to the unified interface
for upon module. Like the AI processing interface module, the management interface module
also provides the unified interface for the upon module to control edge nodes. After thetask manager module sending the edge node ID, task ID, and operations to the management
interface module, all operations will be transferred to the direct commands to the given edgenode.
In the AI-driven IIoT framework, we add some risk control strategies to avoid potential
failures since the industrial environment needs higher reliability than other IoT environments.
First, the centralized controller is deployed in a company’s private cloud rather than in the
public cloud. Replication of the controller is also considered in the proposed framework toprovide consistent service even if one controller fails. All edge nodes are connected to the3.4 Scheduling Problem Formulation and Solution

33

controller with independent network links, which reduces the influence of the access network.
To avoid the single point of failure, the controller will assign multiple edge nodes to execute
critical tasks.
The scalability of the entire intelligent edge computing framework consists of two parts,
the scalability of AI task categories and the scalability of the edge system. It is easy to add a
new AI task category by adding the pre-trained AI model into the AI Model Storage. For
example, in our demonstration system, converting an open neural network exchange (ONNX)
model to a TensorFlow model only cost several seconds, and the size of most converted
models is from 1 to 300 MB. Usually, a lightweight server can store more than a million of
AI models for large amounts of AI task categories. For the scalability of the edge system,
there is no consistent operation needed by adding or removing edge nodes in the centralized
server. Moreover, since the proposed framework prepared many enough AI models, theframework can deploy standard AI models to a new edge node without model conversion.
Finally, based on the intelligent edge computing framework, the main motivations of ourwork are summarized as follows. First, it is necessary to formulate the scheduling problem ofutilizing scarce computing resources in the edge for executing as many AI tasks as possible
to minimize the energy consumption of the entire AI-driven system. Second, we want todesign an efficient strategy to solve the proposed problem in an online scenario due to the
dynamic workload of most IIoT environments.

3.4

Scheduling Problem Formulation and Solution

In this section, we firstly formulate the problem of resource scheduling in intelligent edge
computing and then describe the scheduling strategy’s details.

3.4.1

Problem Formulation

In the scheduling problem, the edge framework schedules an edge node to process the coming
AI task. According to state of the art in edge computing, the latency between edge nodes ismuch larger than in the cloud environment [40, 65]. Meanwhile, the energy consumption
for transferring most AI models and intermediate data between different edge devices is notaffordable [41]. Meanwhile, due to most AI tasks focus on processing streaming data, theinterruption of migration is not acceptable. Therefore, we assume there is no task migration
between different edge nodes in the scheduling problem. We use set E to denote all edge
nodes and ei to denote an edge node in set E. Let set D denote all equipment in the factory,
and d j denote one device in set D.

Green AI for IIoT: Energy Efficient Intelligent Edge Computing for Industrial Internet of34
ThingsAll AI Tasks are denoted by set A, and one AI task is denoted to ak . We use value X jk todenote whether task ak is processed for device d j , given by(
1, task ak is processed for device d j ,X jk =
0,elsewise.

(3.1)

Usually, the power consumption in processing a given AI task depends on the hardwareof edge devices. Therefore, the energy consumption of task ak in edge node ei is denoted bypik . We also use a value Yik to denote whether edge node ei is scheduled to process task ak ,given by(1, task ak is processed in edge node ei ,
Yik =
(3.2)
0,
elsewise.
We use P to denote the power consumption of the scheduling strategy is given by|E| |A|P = ∑ ∑ Yik · pik .

(3.3)

i=1 k=1

The processing latency of task ak on edge node ei is denoted by likp , and the network
latency from equipment d j to edge node ei is denoted by linj . Considering the communication
limitation of equipment ei , let likd denote the time for transferring data of task ak to edge nodeei , given by|A||E|∑ Yik Dlkd = ∑ Yik · k=1
(3.4)
B
k
i=1
where D is the transferring data size and Bk is the ingress bandwidth of edge node ei .
The total latency for processing task ak is denoted by lkt , given by
|E|

lkt =

∑

|E| |D|

liep ·Yik +i=1∑ ∑ linj · X jk ·Yik + lkd .

(3.5)

i=1 j=1

We use Lt to denote the required processing latency in the scheduling and for a given task
ak , lkt is no more than Lt .
Because edge node ei has a capacity for processing tasks, the resource scheduling strategy
should satisfy|A|∑ Yik · bk ≤ ci
k=1

where ci is the capacity of edge node ei , and bk is the overload of task ak .

(3.6)

35

3.4 Scheduling Problem Formulation and Solution

We consider that the scheduling strategy works on a time series T , which is divided into
time slots. Let t denote a time slot in time series T . We assume the length of each time slot is
the minimal scheduling unit. Therefore, the working time series of task ak , denoted by tk , isgiven bytk = {z1 , z2 , ..., z|T | }(3.7)
and zt is a value to denote whether task tk is working, given by(
1, task ak is working in time tzt =0,elsewise.

(3.8)

Therefore, in time series T , the total energy consumption denoted by E is given by|T |E = ∑ P · tk .

(3.9)

t=1

Therefore, the scheduling problem is summarized asminimize :s.t. :

Elt ≤ Lt(3.10)|A|∑k=1 Yik · bk ≤ ci .

Resource scheduling problem for AI-driven IIoT in intelligent edge: given a set ofintelligent edge nodes and a set of industrial equipment, the edge node scheduler attempts
to assign edge nodes for processing AI tasks submitted by equipment with minimal energy
consumption.

3.4.2

Scheduling Strategy

We first solve the scheduling problem in a static scenario in which AI tasks are submitted at
the beginning and continuously processed until the end of the scheduling. The object of thescheduling problem is simplified tominimize :sub ject :P
lt ≤ Lt(3.11)|A|∑k=1 Yik ≤ ci .

Theorem 1 The scheduling problem that assigning intelligent edge nodes for processing AI
tasks with the static assumption is NP-hard.

Green AI for IIoT: Energy Efficient Intelligent Edge Computing for Industrial Internet of36
ThingsTo solve the edge node scheduling problem, we design a heuristic algorithm shown in
Algorithm 2. We assume that edge nodes are less than equipment devices, i.e. |E| < |D|.
There are two loops in the static scheduling algorithm, a double loop, and a normal loop. The
complexity of the double loop is O(|A||D|) because the first layer of the loop has |A| iterations,and the second layer has no more than |D| iterations. The normal loop has |E| iterations.
Therefore, the time complexity of the proposed algorithm is O(|A||D| + |E|) = O(|A||D|)because |E| is smaller than |A||D|. Moreover, from the analysis in a previous paper [34],the number of edge nodes employed by the static scheduling algorithm is no more than twotimes that in an optimal solution.
Algorithm 2 The static edge node scheduling strategy for AI-driven tasks
Sort A = {a1 , a2 , ..., a|K| } that b1 ≤ b2 ≤ ... ≤ b|A| ;for ak ∈ A dod ← 0;
for d j ∈ D do
if X j k == 1 thend ← d j;Break;
end ifend forE ′ ← ∅;
for ei ∈ E do
if lt < Lt and ci ≥ bk thenS
E ′ ← E ′ ei
end ifend for
Sort E ′ = {e1 , e2 , ..., e|E ′ | } that p1k ≤ p2k , ..., ≤ p|E ′ |kAssign e1 for processing ak ;c1 ← c1 − bk ;end forSince the static scheduling problem is a particular case of the online scheduling problem,the online scenario’s scheduling problem is NP-hard. Therefore, we define a reinforcement
learning-based strategy to find the solution of the online edge node scheduling problem
for AI-driven IIoT. We set the negative value of energy consumption as the reward of thereinforcement learning and use rn to denote the single-step reward at time n, given by|E| |A|rn = − ∑ ∑ Yik · pik · zt .
i=1 k=1(3.12)

3.4 Scheduling Problem Formulation and Solution

37

Let Rn denote the total future discounted reward that is defined as|T |Rn = ∑ γ t−n · rn

(3.13)

t=nwhere γ < 1 is the discounted factor.
We use set O = {O1 , O2 , ..., O|T | } and Z = {Z1 , Z2 , ..., Z|T | } to denote actions and statesof the scheduling strategy, respectively. We also use set O = {O1 , O2 , ..., O|T | } and Z ={Z1 , Z2 , ..., Z|T | } to denote entire admissible action and state space. Let Q∗ (·) denote theaction-reward function with maximum reward in state Zn and an optimal action Zn . From the
Bellman optimality equation [44], the value of Q∗ (Zn , On ) can be formulated byQ∗ (Zn , On ) = EZn+1 [rn + γ · max[Q∗ (Zn , On )]|Zn , On ]On+1

(3.14)

where E is the expectation.
The Q∗ (·) function can be estimated iteratively with the Bellman equation as
Qi+1 (Zn , On ) = EZn+1 [rn + γ · max[Qi (Zn , On )]|Zn , On ]On+1

(3.15)

where i is the iterator. When i ← ∞, the optimal Q∗ is equal to Qi .
We use θ to denote the sequence of approximation weights, and the Q∗ (Zn , On ) can be
estimated byQ∗ (Zn , On ) ≈ Q(Zn , On ; θ ).
(3.16)
In our scheduling strategy, we adopt the deep reinforcement learning method and let
Li (θi ) denote the loss function, given by
Li (θi ) =Eρ(Zn ,On ) [(EZn+1 [rn+ γ · Q(Zn+1 , On+1 ; θi−1 )|Zn , On ](3.17)− Q(Zn , On ; θi ))2 ]where ρ(Z, O) denote the behavior distribution of Z and O.
We apply the stochastic gradient descent (SGD) to optimize (3.17) with the gradientgiven by
∇θi Li (θi ) = Eρ(Zn ,On ) [(rn + γ · Q(Zn+1 , On+1 ; θi−1 )− Q(Zn , On ; θi )) · ∇θi · Q(Zn , On ; θi )](3.18)

Green AI for IIoT: Energy Efficient Intelligent Edge Computing for Industrial Internet of38
Things!"#$%&'"$%()*$"+,-.#%/012%5/012%4/012%3

!

!

!

!

!"#

!"#

!"#

!"#

!"$

!"$

!"$

!"$

!"$

!"$

!"$

!"$

%!

%"

%#
B.C+>
678!9%:"!;
678!9%:#!;
678!9%:$!;<'.=',+>-'.0,%,0?$@%5<'.=',+>-'.0,%,0?$@%4 A+,,?%)'..$)>$"%,0?$@%3

Fig. 3.3 An example of input states, including the edge node scheduling and three tasks
We build the scheduling strategy states as distinct images shown in Fig. 3.3, including thescheduled edge nodes and tasks. The image of the scheduled edge nodes is a table of edge
capacities, in which each column is one unit of the processing resource and each row means
the time slot. Colored cells represent allocated resources, and different color means different
task. The image of a task is also a table in which the width is the needed number of resource
units, and the height represents the time series. The action space is given by {ea1 , ea2 , ..., ea|A| }where eak means the edge node assigned to task ak and Xeak k = 1. If the task ak ends or does
not begin, the value of eak is zero. If the action for one task is valid, the value of eak is ∅. In
each time slot, the scheduler will find a valid action or an invalid action. After the scheduler
finds an action, all tasks will put into the table for edge node scheduling with different colors.
If eak = ∅ exists, the scheduler will shift up by a one-time slot and find another action space.
We also define the training algorithm shown in Algorithm 3 to train the edge nodescheduler. In the training phase, we use an ε-greedy strategy to generate action On in each
iteration. Also, we build a replay buffer for the mini-batch sampling to accelerate the SGD
procedure.

39

3.5 Performance Evaluation
Algorithm 3 Training algorithm for the scheduler
Initial replay buffer
for m form 1 to M do
Z0 is set as an empty state;
for n from 1 to |T | dop ← random(0, 1)
if p ≤ 1 − ε thenOn ← arg maxOn Q(Zn , On ; θn );else
Randomly select On from On ;end ifCalculate rn from (3.12);
Assign edge nodes with On and get Zn+1 ;Store (Zn , On , rn , Zn+1 ) in the replay buffer;
Sample batch B from the replay buffer;
for n′ from 1 to |B| do
Update θn by performing SGD;
end forθn−1 ← θn ;end forend for

3.5

Performance Evaluation

3.5.1

Demonstration System

We first build a demonstration system to evaluate the latency and energy consumption of
AI-driven applications. The demonstration system is developed in a small testbed consists ofdevices shown in Table 3.1. All devices are connected to a gigabit Ethernet Network Switch
with wired connections.
Table 3.1 Hardware setting of the testbed
Abbreviation

Device Name

Operating System

CPU

MemoryGPU

AGX

Jetson AGX Xavier

Ubuntu 18.04

8-core ARM v8.2

32GB 256-Bit LPDDR4x

512-core Volta

NX

Jetson Xavier NX

Ubuntu 18.04

6-core ARM v8.2

8 GB 128-bit LPDDR4x

384-core Volta

Nano

Jatson Nano

Ubuntu 18.04

4-core ARM A57

4 GB 64-bit LPDDR4

128-core Maxwell

RPi4

Raspberry Pi 4B

Raspberry Pi OS 1.5.0

4-core ARM v8

8GB 64-bit LPDDR4VideoCore 4

We test the power consumption and average processing time of three deep learningmodels, YOLOv3, YOLOv3-tiny, and VGG16. In each test, the edge devices process selected
50 images from COCO 2017 image dataset, and we record the model loading time andGreen AI for IIoT: Energy Efficient Intelligent Edge Computing for Industrial Internet of
40
ThingsFig. 3.4 Power consumption of four edge devices in the testbed
processing time. We also converted the models to TensorRT-enabled models with the AI
model converter. The intelligent edge framework is developed in a Macbook Pro 15 2018
computer with Python 3.9.2. The AI model converter is based on the TensorRT 7.1.3 library,
and all GPU-based computing is based on CUDA 10.2.
As shown in Fig. 3.4, we test the power consumption in processing AI tasks and idle stateof four edge devices. Since the Jetson AGX Xavier has the highest performance, its power
consumption is more than 30 watts. Jetson Xavier NX and Jeston Nano’s power consumption
is similar, while the former is more powerful than the latter. The weakest Raspberry Pi 4 also
has the lowest power consumption.
The results of the processing latency in the demonstration system are given by Fig. 3.5
and 3.6. Before processing AI tasks in edge devices, it is necessary to load the pre-trained
and converted models to the edge nodes. From the results shown in Fig. 3.5, becauseof its highest performance, Jeston AGX Xavier loads the AI models in the shortest time.
Meanwhile, loading AI models for GPU computing is slower than models for CPU computing.
Loading converted models is slower than loading models for general GPU computing. The

41

3.5 Performance Evaluation

Fig. 3.5 Model loading time

Fig. 3.6 Average processing time

Green AI for IIoT: Energy Efficient Intelligent Edge Computing for Industrial Internet of
42
ThingsFig. 3.7 Energy consumption with increasing tasks
Raspberry Pi 4B is only able to process AI tasks with the CPU, but its model loading time islonger than Jeston AGX Xavier.
As shown in Fig. 3.6, the average processing time in edge nodes is different from modelloading time. As the quickest device, Jeston AGX Xavier can finish the task in about 50 ms
with the converted YOLOv3-tiny model. The processing time with the converted YOLOv3
model in Jeston AGX Xavier is about 280 ms. Without model converting, the processing timewith the original YOLOv3 in Jeston AGX Xavier is more than 5800 ms which is more than
20 times longer than the converted model. Although the Raspberry Pi 4 has the lowest power
consumption, its processing time with the original YOLOv3 is more than 3640 times than
Jetson Nano with converted YOLOv3 model. Even with YOLOv3-tiny, the processing timein Raspberry Pi 4 is 160 times longer than Jetson Nano with converted YOLOv3-tiny. The
average processing time of VGG16 is very similar with the time of YOLOv3 in all devices.
The model converting can improve then energy efficiency in both high-spec and low-specedge devices. GPU accelerated AI processing is still essential to process AI tasks with lessenergy consumption.

3.5 Performance Evaluation

Fig. 3.8 Energy consumption with different number of edge nodes

Fig. 3.9 Energy consumption with increasing active ratio

43

Green AI for IIoT: Energy Efficient Intelligent Edge Computing for Industrial Internet of
44
ThingsFig. 3.10 Energy consumption with different latency requirments

3.5.2

Experimental Result Analysis

In a large environment, we apply simulations for the performance evaluation of the proposedscheduling strategy. We develop a simulator with Python 3.9.2. The training and evaluation
is executed in a server that equips an Intel Core™ i9 10900X CPU, 128GB memory, a 1TB
SSD and two NVIDIA GeForce TITAN RTX graphics cards. The operating system is Ubuntu
18.04 LTS. We adopt Keras 2.4.0 as the deep leaning platform. In the experiments, we apply
the RMSProp optimizer for Keras library. We set the size of mini-batches to 64. For the
ε-greedy in the scheduler training, we set the value of ε to 0.1.
As shown in Table 3.2, we set the parameters of all simulations according to the experimental results. The tasks are randomly generated in the time series, and the power
consumption for processing tasks is evenly distributed in [5, 25] from Fig. 3.4. The required
processing latency is evenly distributed in [2, 600] ms from Fig. 3.5 and 3.6. According
to the general industrial environment, [18], the network latency between devices and edge
nodes is evenly distributed in [1, 200] ms. The capacity of edge nodes is based on thememory size of each device and evenly distributed in [1, 16]. According to memory usage,
the required resource units of tasks are evenly distributed in [1, 2]. We use a small factorywith 500 employees as the simulation environment. Therefore, the number of edge nodes

45

3.5 Performance Evaluation
Table 3.2 Parameter settings in all simulations
Parameter
Power Consumption for processing a task
Required processing latency
Network latency
Capacity of edge nodes
Required resource units of a task
Number of edge nodes
Number of equipment devices
Number of time-slots
Length of a time slot
Task activity ratio

Value[5, 25][2, 600] ms[1, 200] ms[1, 16][1, 2]
from 100 to 400
from 200 to 500
3600
1s[0.8, 1]and equipment devices is set from 100 to 400 and from 200 to 500, respectively. Usually,
since one device will send at most two tasks, the number of tasks is set from 200 to 800. We
test the energy consumption in one hour with 3600 time-slots and 1 second per slot. The
ratio that tasks are active in the total time series is evenly distributed [0.8, 1].
We compare the results between the deep reinforcement learning-based online schedulingstrategy (Online), the static scheduling algorithm (Static), and the FIFO strategy in all
simulations. All simulations are executed 20 times, and the average values are recorded witha confidence level of 95%. All results of the performance evaluation are represented in Fig.
3.7,3.8,3.9 and 3.10. As shown in Fig. 3.7, we change the number of tasks from 200 to 500
and test the total energy consumption in one hour. The online scheduling strategy shows thelowest energy consumption with the different number of tasks. When the number of tasks isless than 300, the FIFO method performs better than the results with the static schedulingalgorithm. When the number of tasks increases to more than 300, the static schedulingalgorithm brings less energy consumption than the FIFO method.
We also test the energy consumption with different numbers of edge nodes. The numberof tasks is set to 400, and the number of edge nodes increases from 100 to 400. From Fig.
3.8, when the number of edge nodes is less than 200, three methods perform similarly, andthe total energy consumption is more than 5200 Wh. After more edge nodes begin to process
AI tasks, the total energy consumption decreases. The online scheduling method still has thelowest energy consumption in the three methods, and the other two methods perform very
similarly.
The active ratio of the task in the time series also affects the total energy consumption.
In Fig. 3.9, we adjust the active ratio from 0.5 to 1.0 and test the total energy consumption.
The number of edge nodes is set to 400, and the number of total tasks is set to 800. The

Green AI for IIoT: Energy Efficient Intelligent Edge Computing for Industrial Internet of46
Thingsresults show that a higher active ratio will bring more energy consumption, and the online
scheduling strategy performs better than the other two algorithms.
Finally, we test the energy consumption with different required latency. We set the
number of tasks to 400 with 400 edge nodes, and the required latency increases from 100 ms
to 600 ms. As shown in Fig. 3.10, the total energy consumption decreases with increased
required latency. When the required latency is less than 300 ms, the online schedulingstrategy performs much better than the other two methods. When the latency requirement
becomes loose, three scheduling methods perform similarly, and the FIFO has less energy
consumption than the Static algorithm.

Chapter 4
Multiple-layer Intelligent Edge
Computing in Early Cancer Diagnosis
4.1

Motivation

In healthcare and medicine, scientists, governments, and companies focus on artificial
intelligence (AI) in various applications such as medical imaging, electronic health record
data, robotic-assisted surgery, and genomics [8, 17]. In addition, scientists have applied deeplearning in detecting early-stage lung cancer from computed tomography (CT) screening
in medical imaging [60]. Since early screening is so important to improve the survival rate
greatly, it is necessary to identify cancers as early as possible [23].
By processing tasks in the edge near the data source rather than in remote cloud servers,
intelligent edge computing improves the efficiency of AI applications in analyzing a large
amount of data in healthcare and medicine [35]. According to a report from the Massachusetts
Institute of Technology, 75% of medical data will be generated at the edge by 2025, whichmeans edge computing will be the significant architecture for processing AI tasks in the
future [66]. Due to the development of intelligent edge computing, including new hardware
and AI models, it is possible to diagnose cancers in early imaging screening [76].
However, the low specs hardware of most edge devices limited the capability in processing
large computer vision models for cancer detection [4]. Only lightweight deep learning(DL) models without high precision can be performed smoothly in most low specs edge
devices [29], which is not acceptable for cancer diagnosis. For performing high precision DL
models, edge servers need high-spec hardware such as high-performance graphics processing
units (GPUs) or specific tensor processing units (TPU) [31]. Only a part of edge servers canequip high specs GPUs or TPUs due to rack space or power supply systems limitations.

48

Multiple-layer Intelligent Edge Computing in Early Cancer Diagnosis

Therefore, we first study the modern public healthcare system hierarchy in cancer diagnosis and then design a multiple-layer intelligent edge computing (MLIEC) system todiagnose cancers from medical data. In the MLIEC system, we put different hardware
specs edge servers into different edge layers according to the network infrastructure. High
spec edge servers only process cases with high positive probabilities, while other cases areprocessed with low specs hardware. Therefore, the processing capability of the entire system
is improved with hierarchically processing. We also apply the early lung cancer diagnosis as
the case study in the MLIEC system to show the efficiency of the hierarchical design.
This article consists of the following parts. We first introduce three typical applicationsof DL models for early cancer diagnosis. The following describes the healthcare systemhierarchy and the route to cancer diagnosis in the hierarchy. After that, we present the mainidea of the MLIEC system and the processing steps for early cancer diagnosis. The next
part is the case study discussed in a testbed-based intelligent edge computing environment.
Finally, we conclude this article and give the future work.

4.2

DL for Early Cancer Diagnosis

DL begins playing an important role in early cancer diagnosis. We discuss breast cancer,lung cancer, and ovarian cancer diagnosis as typical deep learning applications for earlycancer diagnosis.
DDeep learning models have shown efficiency in breast cancer diagnosis from someexisting works. Sun et al. [58] fuse the inference results from three independent deep neuralnetworks (DNNs) to predict breast cancers. Three homogeneous DNNs with two hidden
neural layers learn the features from multi-dimensional data, including gene expression,
copy number, and clinical data, respectively. DNNs are trained by the Cancer Genome Atlas
Breast Invasive Carcinoma (TCGA-BRCA) dataset, and the accuracy of the proposed work
is near to 80% with a 95.0% confidence interval.
Detecting or classifying lung cancer is another application of DL models in cancer
diagnosis. Coudray et al. [12] train a convolutional neural network (CNN) model by the
TCGA dataset to classify histopathology images into the adenocarcinoma (LUAD), squamouscell carcinoma (LUSC), and normal lung tissues. The authors apply a standard Inception V3
model [62] which shows a good performance for the classification of non-small cell lungcancer. Meanwhile, the CNN-based approach can be applied to other cancer diagnoses.
Detecting cancers from CT images is another application of DL models in cancer diagnosis. Wang et al. [67] train a DL model with 8917 CT images to predict the individual riskand 3-year recurrence probability of high-grade serous ovarian cancer (HGSOC) patients.

49

4.3 Routes to Cancer Diagnosis• First In-hospital
Visit• Initial Diagnosis
Local
Hospital

Referral toDistrict
Hospital

Referral to Local Hospital

Potential
Patients

Clinic• First Contact
with Physician• Investigation ofCancerDistrict
Hospital• Firm Diagnosis• Initial
Treatment

Referral to Specialized Hospital• Specialized
Treatment
Specialized
HospitalFig. 4.1 A route to cancer diagnosis in the healthcare hierarchy
The authors integrate a deep neural network and Cox proportional hazard regression into the
DL model. The experimental results show that the average area under the curve (AUC) of
the 3-year recurrence probability prediction is nearly 80%.
The DL-based methodology has shown enough performance in early cancer diagnosis
from the above research works. Since the models applied in these applications are not
up-to-date DL technologies, the performance of the newest DL models will be improvedin the next decade. However, these DL models still need high specs hardware for training
and inference. For example, as a typical hardware spec for edge computing, a Raspberry
Pi 3 Model B single-board computer needs 10 seconds to infer a 299x299 image with the
Inception V3 model, which is still very slow for practical usage. In the following section, we
will introduce the healthcare system’s hierarchy, which is a possible solution to improve thecapability for low specs edge hardware.

4.3

Routes to Cancer Diagnosis

Healthcare systems in most countries have a hierarchical structure to reduce the publichealthcare resources. There are four layers for disease diagnosis and treatment in Japan,
including clinics, local hospitals, district hospitals, and specialized hospitals. A clinic usuallyfocuses on the care of outpatients and provides fundamental healthcare services. A local
hospital in the local community will have more medical appliances and specialized physicians.

50

Multiple-layer Intelligent Edge Computing in Early Cancer Diagnosis

Access Edge
Layer
Potential
Patients
Lightweight DL
ModelsAggregation Edge Layer

High Precision
DL Models

Cloud Layer/Macro
Edge Layer
HighestPrecision DL
ModelsMedical Appliances
Medical Data
Rapid Diagnosis Results(Positive/Negative)
Physicians

Low Spec
Edge Device

High Spec
Edge Server

Cloud Server/Macro
Edge ServerHigh Precision Results (Cancer Stages, Types, Grades)

Fig. 4.2 Structure of the multiple-layer intelligent edge computing for early cancer diagnosis
Meanwhile, a local hospital has beds for long-term care. A district hospital located in a major
city provides adequate services to patients with enough healthcare resources. A specialized
hospital focuses on a few medical specialties, such as rehabilitation or children’s health. A
cancer treatment hospital is also considered a specialized hospital.
To explain the hierarchy in early cancer diagnosis, we use an example to show the route
between the above four types of medical facilities, as shown in Fig. 4.1. When potential
patients have cancer symptoms, they should first come to the clinic to contact physicians forthe investigation of cancers. If a patient has positive investigation results, the clinic physician
will refer the patient to a local hospital for the diagnosis. The patient will have the firstin-hospital visit in the local hospital and get the initial diagnosis from various tests. If thepositive result of the initial cancer diagnosis is confirmed, the physician in the local hospitalwill refer the patient to a district hospital for further diagnosis and treatment.
In the district hospital, the patient will get a firm cancer diagnosis and determined initial
treatment such as surgery or chemotherapy given by the physician. If the cancer is detectedin the early stage, the treatment will be finished in the district hospital. When the cancer isdetected in an advanced stage, the physical of the district hospital usually refer the patient
to a cancer treatment hospital for a personalized treatment, which can improve the overall
survival of most cancers.

4.4 MLIEC for Early Cancer Detection

51

The hierarchy of the healthcare system brings delays to cancer diagnosis due to primary
healthcare. However, the hierarchy reduces the number of patients in higher facilities byfiltering most patients with light diseases. The hierarchical public healthcare is an efficientsystem and has successes in many countries. In the following section, according to thehierarchy healthcare system, we introduce the MLIEC system to reduce the edge computing
resources in the early cancer diagnosis.

4.4

MLIEC for Early Cancer Detection

Before presenting the MLIEC system, we first introduce the typical structure of an intelligent
edge computing system. An intelligent edge computing system is also an edge computing
system in the network edge. A network usually has several layers from the devices to theexternal network, including the access layer, aggregation layer and core layer. Therefore, thenetwork edge consists of the access edge layer, aggregation edge layer and macro edge layer.
The access edge layer is the nearest layer to the device, which has very limited space andpower supply. In the MLIEC system, we can deploy low specs hardware in the access edgelayer with lightweight DL models. Meanwhile, the MLIEC system can have as many edge
servers as access points to serve a lot of patients. The access edge layer is very similar withthe clinic layer in the healthcare hierarchy.
The aggregation edge layer locates between the access and the macro edge layer, which
has adequate space and power supply for deploying high specs edge hardware in the MLIEC
system. The aggregation layer is similar to the local hospital layer in the healthcare hierarchy
since it has less service capability and higher diagnosis precision based on high precision DL
models.
The high-end hardware can be deployed in the macro edge layer with large space andpower supply. The MLIEC system deploys DL models with highest precision in the macroedge servers. However, considering the number of core network nodes is very limited,
another solution is to apply cloud computing and place highest precision DL models in cloudservers. The cloud layer or macro edge layer can be considered as the district hospital layer
in the healthcare hierarchy.
We use Fig. 4.2 to show the early cancer diagnosis in the MLIEC system. Whenpotential patients come to a clinic or local hospital with some symptoms, physicians will
apply different medical appliances to collect medical data from patients. Medical data, suchas cytopathology, CT scan, and X-ray images, will be transferred to low specs edge servers
in the access edge layer. Finally, the MLIEC system will give rapid diagnosis results tophysicians with the most negligible computing resources.

52

Multiple-layer Intelligent Edge Computing in Early Cancer Diagnosis
Table 4.1 Hardware settings of the testbed

Abbreviation

Device Name

Operating System

GPU

FP32 Performance

TDP

Edge Layer

Nano

Jatson Nano

Ubuntu 18.04

128-core Maxwell

235.8 Giga

10 W

Access Layer

NX

Jetson Xavier NX

Ubuntu 18.04

384-core Volta

1.058 Tera

15 W

Aggregation Layer

A6000

RTX A6000 Server

Debian 1010752-core Ampere

38.71 Tera

300 W

Macro Layer

However, the lightweight DL models limit the precision of the rapid diagnosis results.
Therefore, the aggregation edge layer will send medical data with positive or weak negative
results to the high specs edge servers. After processing the medical data, the high specs edge
server with high precision DL models sends back high precision results to the physicians.
When the cancer diagnosis needs higher precision in some cases, the medical data withpositive results will be sent to the cloud or macro edge servers for further inference with thehighest DL models. The final high-precision results will be sent back to the physician andother layers. Since the access and aggregation layer filters most medical data with negative
results, the cost of the high-end cloud or macro edge servers is reduced by the MLIEC
structure.

4.5

Case StudyIn the case study, we first introduce the testbed of the intelligent edge computing system andthen show the performance for predicting lung cancers from medical data.
As shown in Table 4.1, we build a testbed with three edge devices with different hardware
specs. First, we list the abbreviation, device name, operation system, GPU model, singleprecision floating-point format (FP32) performance, thermal design power (TDP), and theproper layer of each device in the testbed. The hardware specs show that Jeston Xaver NX
has four times higher FP32 performance while the power consumption is about 1.5 times
higher than Jetson Nano. A6000 GPU is one of the fastest GPUs manufactured by NVIDIA,
and its FP32 performance and TDP are nearly 38 and 20 times higher than Jeston Xavier
NX, respectively. Considering the launch time of A6000 is the latest hardware, its power
efficiency is also the best in the testbed.
We choose the lung cancer diagnosis task from work in [12] with Inception V3 andInception ResNet V2 model [61]. Inception ResNet V2 is larger than Inception V3 withhigher precision and longer time for training and inference. Since the performance of Jetson
devices is too weak for training DL models, we train the DL model in A6000. We train twomodels with the TCGA-LUAD dataset for predicting the 3-year recurrence probability. We
applied CUDA 10.2 in Jetson devices and CUDA 11.1 in A6000.

53

4.5 Case Study

300
250

100
Idle
Working
10

Volume

200
150
1

100
50
0

0.1
Nano

NX
Edge Hardware

A6000

Fig. 4.3 Power consumption and volume of different edge hardware

Volume (Liter)

Power Consumption (Watt)

350

54

Multiple-layer Intelligent Edge Computing in Early Cancer Diagnosis

25
20

80.5

Nano
NX
A6000
Accuracy

80
79.5
79

15

78.5
78

10

Accuracy

Service Capability (per second)

30

77.5

5

77

0

76.5
InceptionV3

InceptionResNetV2
DL modelsFig. 4.4 Service capability of different DL models with different hardware
We first test the power consumption of three devices in idle and working statuses. We
also list the size in volume (without power adapters and for Jeston devices and power cables).
As shown in Fig. 4.3, different devices have different power consumption and size. The RTX
A6000 server has the best energy efficiency and largest size, which is difficult for deployment
in base stations and aggregation racks. Jeston devices are small and consume very low power,
appropriate for base stations and general device racks. The volume for deploying an RTX
A6000 server is 130 times that of Jeston Xavier NX.
We then test the service capability of each device for the lung cancer diagnosis. From
Fig.4.4, the RTX A6000 server shows the largest service capability even with the Inception
ResNet V2 model, which can process more than 19 images per second. Jetson Xavier NX
can process 3.2 images per second with Inception ResNet V2 and 4.6 with Inception V3.
Jetson Nano has a service capability that processes 0.7 images per second with Inception
ResNet V2 and 1.1 with Inception V3. The inference accuracy of Inception ResNet V2 andInception V3 is 77.9 % and 80.3 %, which is very similar.
According to the above results, we also calculate the service capability of the given
network infrastructure on our campus. There are 107 access points, 24 aggregation routers,
and one core route to connect the networks inside and outside our campus. We consider theaccess points as the access edge layer, the aggregation routes as the aggregation layer, andthe core route as the macro layer. We assume edge servers are deployed with all access points
and routes. Therefore, as shown in Fig. 4.5, the maximum service capability of the MLIEC

55

4.5 Case Study

140
Device Number

120

Service Capacity (per second)100

Total Volume (Liter)

80
60
40
20
0
Access Edge Layer

Aggregation Edge Layer
Edge Layers

Macro Edge Layer

Fig. 4.5 Service capability and volumes in different edge layers

56Multiple-layer Intelligent Edge Computing in Early Cancer Diagnosis

system will be near 213 images per second. The capability of three layers is 118, 77, and 19
images per second with total volumes of 24.8, 7.6, and 41.3 liters, respectively.

Chapter 5
Conclusions and Future Directions
For the first challenge, the proposed intelligent edge computing framework shows that a welldesigned system will deeply improve the energy efficiency in processing AI tasks for AIoT
applications. Meanwhile, a novel scheduling strategy is also very important to reduce energy
consumption in a heterogeneous environment. From the results of performance evaluation,
the reinforcement learning-based method is a good solution for the online scheduling problem,
especially in the multilevel framework. Moreover, since most commercial edge products
show good enough performance in processing AI tasks with very low energy consumption,
intelligent edge computing will be a very important opportunity for AIoT applications. In
the future, we will continue focusing on intelligent edge computing and introduce othertechnologies such as parallel computing and virtualization in our framework to improvesystem availability and scalability.
For the second challenge, intelligent edge computing shows its potential in the energyefficient AI-driven IIoT scenario. Depending on the development of AI frameworks, it
is possible to concentrate heterogeneous resources on edge to support different AI tasks.
Meanwhile, novel resource scheduling is also important to the energy efficiency in the IIoT
environments. The demonstration system developed the small testbed shows that a welldesigned edge system can outperform traditional cloud-based solutions with short processing
latency and very low energy consumption. Finally, the proposed work is a feasible solution
to the AI-driven IIoT applications from the performance evaluation results. With the online
scheduling strategy, energy consumption in different settings is less than 20% and 30% of
the number with the static scheduling strategy and FIFO algorithm. In the future, we will try
to enlarge the scale of our testbed and develop more deep learning applications such as NLP,
time series forecasting, and audio recognition. We also plan to build intelligent edge service
in robots and UAVs to support autonomous driving and flying.

58

Conclusions and Future Directions

Finally, we present the MLIEC system for cancer diagnosis to improve the total service
capability with limited computing resources for the third challenge. The multiple-layer
concept in the MLIEC system comes from the success hierarchy in modern public healthcare
systems. We also show the route to a cancer diagnosis with the Japanese healthcare system.
Finally, the results from the case study in our testbed-based edge environment show that it isnecessary to build the MLIEC system in normal network infrastructures.

References[1] Alvarez-Aparicio, C., Guerrero-Higueras, A. M., Rodriguez-Lera, F. J., Olivera, M.
C. C., Olivera, V. M., Clavero, J. G., and Rico, F. M. (2019). LIDAR-based people
detection and tracking for @home competitions. In 2019 IEEE International Conference
on Autonomous Robot Systems and Competitions (ICARSC). IEEE.
[2] Antwarg, L., Miller, R. M., Shapira, B., and Rokach, L. (2019). Explaining anomalies
detected by autoencoders using shap.
[3] Arunkumar, A., Bolotin, E., Cho, B., Milic, U., Ebrahimi, E., Villa, O., Jaleel, A., Wu,
C.-J., and Nellans, D. (2017). MCM-GPU. ACM SIGARCH Computer Architecture News,45(2):320–332.
[4] Barbalace, A., Karaoui, M. L., Wang, W., Xing, T., Olivier, P., and Ravindran, B. (2020).
Edge computing. In Proceedings of the 16th ACM SIGPLAN/SIGOPS International
Conference on Virtual Execution Environments. ACM.
[5] Bhattacharya, S. and Lane, N. D. (2016). Sparsification and separation of deep learninglayers for constrained resource inference on wearables. In Proceedings of the 14th ACM
Conference on Embedded Network Sensor Systems CD-ROM. ACM.
[6] Bisio, I., Garibotto, C., Grattarola, A., Lavagetto, F., and Sciarrone, A. (2018). Exploiting
context-aware capabilities over the internet of things for industry 4.0 applications. IEEE
Network, 32(3):101–107.
[7] Buda, T. S., Caglayan, B., and Assem, H. (2018). DeepAD: A generic framework basedon deep learning for time series anomaly detection. In Advances in Knowledge Discovery
and Data Mining, pages 577–588. Springer International Publishing.
[8] Chen, Y., Yu, L., Ota, K., and Dong, M. (2018). Robust activity recognition for aging
society. IEEE Journal of Biomedical and Health Informatics, 22(6):1754–1764.
[9] Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Catanzaro, B., andShelhamer, E. (2014). cudnn: Efficient primitives for deep learning.
[10] Chiu, T.-C., Shih, Y.-Y., Pang, A.-C., Wang, C.-S., Weng, W., and Chou, C.-T. (2020).
Semisupervised distributed learning with non-IID data for AIoT service platform. IEEE
Internet of Things Journal, 7(10):9266–9277.
[11] Cong, Y., Tian, D., Feng, Y., Fan, B., and Yu, H. (2019). Speedup 3-d texture-lessobject recognition against self-occlusion for intelligent manufacturing. IEEE Transactions
on Cybernetics, 49(11):3887–3897.

60

References[12] Coudray, N., Ocampo, P. S., Sakellaropoulos, T., Narula, N., Snuderl, M., Fenyö,D., Moreira, A. L., Razavian, N., and Tsirigos, A. (2018). Classification and mutation
prediction from non-small cell lung cancer histopathology images using deep learning.
Nature Medicine, 24(10):1559–1567.
[13] Cui, H., Zhang, H., Ganger, G. R., Gibbons, P. B., and Xing, E. P. (2016). GeePS. In
Proceedings of the Eleventh European Conference on Computer Systems. ACM.
[14] Dong, M., Ota, K., and Liu, A. (2016). RMER: Reliable and energy-efficient datacollection for large-scale wireless sensor networks. IEEE Internet of Things Journal,
3(4):511–519.
[15] Erfani, S. M., Rajasegarar, S., Karunasekera, S., and Leckie, C. (2016). Highdimensional and large-scale anomaly detection using a linear one-class SVM with deeplearning. Pattern Recognition, 58:121–134.
[16] Erol, B. A., Majumdar, A., Lwowski, J., Benavidez, P., Rad, P., and Jamshidi, M.
(2018). Improved deep neural network object tracking system for applications in homerobotics. In Computational Intelligence for Pattern Recognition, pages 369–395. Springer
International Publishing.
[17] Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui,
C., Corrado, G., Thrun, S., and Dean, J. (2019). A guide to deep learning in healthcare.
Nature Medicine, 25(1):24–29.
[18] Ferrari, P., Flammini, A., Sisinni, E., Rinaldi, S., Brandão, D., and Rocha, M. S. (2018).
Delay estimation of industrial iot applications based on messaging protocols. IEEE
Transactions on Instrumentation and Measurement, 67(9):2188–2199.
[19] Ghosh, I. (2020). Aiot: When artificial intelligence meets the internet of things.
https://www.visualcapitalist.com/aiot-when-ai-meets-iot-technology/. Accessed: 202205-14.
[20] Gupta, K., Stuart, J. A., and Owens, J. D. (2012). A study of persistent threads style
GPU programming for GPGPU workloads. In 2012 Innovative Parallel Computing(InPar). IEEE.
[21] Han, R., Li, S., Wang, X., Liu, C. H., Xin, G., and Chen, L. Y. (2020). Accelerating gossip-based deep learning in heterogeneous edge computing platforms. IEEE
Transactions on Parallel and Distributed Systems, pages 1–1.
[22] Han, T., Muhammad, K., Hussain, T., Lloret, J., and Baik, S. W. (2021). An efficient
deep learning framework for intelligent energy management in IoT networks. IEEE
Internet of Things Journal, 8(5):3170–3179.
[23] Hanna, N. (2008). Survival of patients with stage i lung cancer detected on CT screening.
Yearbook of Oncology, 2008:149–151.
[24] Hannun, A. Y., Rajpurkar, P., Haghpanahi, M., Tison, G. H., Bourn, C., Turakhia,
M. P., and Ng, A. Y. (2019). Cardiologist-level arrhythmia detection and classification in
ambulatory electrocardiograms using a deep neural network. Nature Medicine, 25(1):65–
69.

References

61[25] Haslgrübler, M., Fritz, P., Gollan, B., and Ferscha, A. (2017). Getting through. In
Proceedings of the Seventh International Conference on the Internet of Things. ACM.
[26] Hasselt, H. v., Guez, A., and Silver, D. (2016). Deep reinforcement learning with double
q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence,
AAAI’16, page 2094–2100, Phoenix, Arizona. AAAI Press.
[27] Hiller, J., Henze, M., Serror, M., Wagner, E., Richter, J. N., and Wehrle, K. (2018).
Secure low latency communication for constrained industrial IoT scenarios. In 2018 IEEE
43rd Conference on Local Computer Networks (LCN). IEEE.
[28] Hong, C.-H., Spence, I., and Nikolopoulos, D. S. (2017). GPU virtualization andscheduling methods. ACM Computing Surveys, 50(3):1–37.
[29] Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto,
M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile
vision applications. CoRR, abs/1704.04861.
[30] Huang, S., Guo, Y., Liu, D., Zha, S., and Fang, W. (2019). A two-stage transfer
learning-based deep learning approach for production progress prediction in IoT-enabledmanufacturing. IEEE Internet of Things Journal, 6(6):10627–10638.
[31] Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S.,
Bhatia, S., Boden, N., Borchers, A., Boyle, R., luc Cantin, P., Chao, C., Clark, C., Coriell,J., Daley, M., Dau, M., Dean, J., Gelb, B., Ghaemmaghami, T. V., Gottipati, R., Gulland,W., Hagmann, R., Ho, C. R., Hogberg, D., Hu, J., Hundt, R., Hurt, D., Ibarz, J., Jaffey,A., Jaworski, A., Kaplan, A., Khaitan, H., Killebrew, D., Koch, A., Kumar, N., Lacy,
S., Laudon, J., Law, J., Le, D., Leary, C., Liu, Z., Lucke, K., Lundin, A., MacKean, G.,
Maggiore, A., Mahony, M., Miller, K., Nagarajan, R., Narayanaswami, R., Ni, R., Nix,
K., Norrie, T., Omernick, M., Penukonda, N., Phelps, A., Ross, J., Ross, M., Salek, A.,
Samadiani, E., Severn, C., Sizikov, G., Snelham, M., Souter, J., Steinberg, D., Swing, A.,
Tan, M., Thorson, G., Tian, B., Toma, H., Tuttle, E., Vasudevan, V., Walter, R., Wang,
W., Wilcox, E., and Yoon, D. H. (2017). In-datacenter performance analysis of a tensor
processing unit. In Proceedings of the 44th Annual International Symposium on Computer
Architecture. ACM.
[32] Kanawaday, A. and Sane, A. (2017). Machine learning for predictive maintenance ofindustrial machines using IoT sensor data. In 2017 8th IEEE International Conference on
Software Engineering and Service Science (ICSESS). IEEE.
[33] Kühnel, C., Westermann, T., Hemmert, F., Kratz, S., Müller, A., and Möller, S. (2011).
I’m home: Defining and evaluating a gesture set for smart-home control. International
Journal of Human-Computer Studies, 69(11):693–704.
[34] Li, H., Li, P., Guo, S., and Nayak, A. (2014a). Byzantine-resilient secure softwaredefined networks with multiple controllers in cloud. IEEE Transactions on Cloud Computing, 2(4):436–447.
[35] Li, H., Ota, K., and Dong, M. (2018a). Learning IoT in edge: Deep learning for theinternet of things with edge computing. IEEE Network, 32(1):96–101.

62

References[36] Li, H., Ota, K., and Dong, M. (2019). Deep reinforcement scheduling for mobile
crowdsensing in fog computing. ACM Transactions on Internet Technology, 19(2):1–18.
[37] Li, H., Ota, K., Dong, M., and Chen, H.-H. (2017). Efficient energy transport in 60 ghz
for wireless industrial sensor networks. IEEE Wireless Communications, 24(5):143–149.
[38] Li, H., Ota, K., Dong, M., and Guo, M. (2018b). Learning human activities through wifi channel state information with multiple access points. IEEE Communications Magazine,
56(5):124–129.
[39] Li, K., Tang, X., and Li, K. (2014b). Energy-efficient stochastic task schedulingon heterogeneous computing systems. IEEE Transactions on Parallel and Distributed
Systems, 25(11):2867–2876.
[40] Liu, H., Liu, B., Yang, L. T., Lin, M., Deng, Y., Bilal, K., and Khan, S. U. (2018a).
Thermal-aware and dvfs-enabled big data task scheduling for data centers. IEEE Transactions on Big Data, 4(2):177–190.
[41] Liu, H., Yang, L. T., Lin, M., Yin, D., and Guo, Y. (2018b). A tensor-based holistic edge
computing optimization framework for internet of things. IEEE Network, 32(1):88–95.
[42] Liu, Y., Garg, S., Nie, J., Zhang, Y., Xiong, Z., Kang, J., and Hossain, M. S. (2020).
Deep anomaly detection for time-series data in industrial IoT: A communication-efficient
on-device federated learning approach. IEEE Internet of Things Journal, pages 1–1.
[43] Liu, Y., Yang, C., Jiang, L., Xie, S., and Zhang, Y. (2019). Intelligent edge computing
for IoT-based energy management in smart cities. IEEE Network, 33(2):111–117.
[44] Luus, R. (2019). Dynamic programming. In Iterative Dynamic Programming, pages
67–80. Chapman and Hall/CRC.
[45] Mao, Y., Zhang, J., and Letaief, K. B. (2016). Dynamic computation offloading formobile-edge computing with energy harvesting devices. IEEE Journal on Selected Areasin Communications, 34(12):3590–3605.
[46] Minovski, D., Ahlund, C., and Mitra, K. (2020). Modeling quality of IoT experience inautonomous vehicles. IEEE Internet of Things Journal, 7(5):3833–3849.
[47] Mohammadi, M., Al-Fuqaha, A., Sorour, S., and Guizani, M. (2018). Deep learning
for IoT big data and streaming analytics: A survey. IEEE Communications Surveys &Tutorials, 20(4):2923–2960.
[48] Nguyen, K. T. and Medjaher, K. (2019). A new dynamic predictive maintenanceframework using deep learning for failure prognostics. Reliability Engineering & System
Safety, 188:251–262.
[49] NVIDIA Corporation. Bring your next-gen products to life with the world’s mostpowerful ai computer for energy-efficient autonomous machines. https://www.nvidia.com/
en-us/autonomous-machines/embedded-systems/jetson-orin/. Accessed: 2022-05-14.
[50] NVIDIA Corporation. The most powerful ai platform for edge computing. https://www.nvidia.com/en-us/deep-learning-ai/solutions/ai-at-the-edge/. Accessed: 2021-0714.

References

63[51] NVIDIA Corporation. NVIDIA A100 GPUs power the modern data center. https://www.nvidia.com/en-us/data-center/a100/. Accessed: 2021-03-10.
[52] NVIDIA Corporation. NVIDIA EGX Platform for AI computing. https://www.nvidia.
com/en-us/data-center/products/egx/. Accessed: 2021-03-10.
[53] NVIDIA Corporation. NVIDIA Jetson Xavier NX for Embedded & edge systems. https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-xavier-nx/. Accessed: 2021-03-10.
[54] Pasteris, S., Wang, S., Herbster, M., and He, T. (2019). Service placement with provableguarantees in heterogeneous edge computing systems. In IEEE INFOCOM 2019 - IEEE
Conference on Computer Communications. IEEE.
[55] Qiu, H., Zheng, Q., Zhang, T., Qiu, M., Memmi, G., and Lu, J. (2021). Toward secureand efficient deep learning inference in dependable IoT systems. IEEE Internet of Things
Journal, 8(5):3180–3188.
[56] Romero-Lopez, A., i Nieto, X. G., Burdick, J., and Marques, O. (2017). Skin lesion
classification from dermoscopic images using deep learning techniques. In Biomedical
Engineering. ACTAPRESS.
[57] Schlegl, T., Seeböck, P., Waldstein, S. M., Schmidt-Erfurth, U., and Langs, G. (2017).
Unsupervised anomaly detection with generative adversarial networks to guide marker
discovery. In Lecture Notes in Computer Science, pages 146–157. Springer International
Publishing.
[58] Sun, D., Wang, M., and Li, A. (2019). A multimodal deep neural network for humanbreast cancer prognosis prediction by integrating multi-dimensional data. IEEE/ACM
Transactions on Computational Biology and Bioinformatics, 16(3):841–850.
[59] Sundaravadivel, P., Kesavan, K., Kesavan, L., Mohanty, S. P., and Kougianos, E. (2018).
Smart-log: A deep-learning based automated nutrition monitoring system in the IoT.
IEEE Transactions on Consumer Electronics, 64(3):390–398.
[60] Svoboda, E. (2020). Artificial intelligence is improving the detection of lung cancer.
Nature, 587(7834):S20–S22.
[61] Szegedy, C., Ioffe, S., and Vanhoucke, V. (2016). Inception-v4, inception-resnet andthe impact of residual connections on learning. CoRR, abs/1602.07261.
[62] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2015). Rethinking theinception architecture for computer vision. CoRR, abs/1512.00567.
[63] Teerapittayanon, S., McDanel, B., and Kung, H. (2017). Distributed deep neuralnetworks over the cloud, the edge and end devices. In 2017 IEEE 37th International
Conference on Distributed Computing Systems (ICDCS). IEEE.
[64] Tzafestas, S. G. (2018). Synergy of IoT and AI in modern society: The robotics andautomation case. Robotics & Automation Engineering Journal, 3(5).

64References[65] Tziritas, N., Koziri, M., Bachtsevani, A., Loukopoulos, T., Stamoulis, G., Khan, S. U.,
and Xu, C.-Z. (2017). Data replication and virtual machine migrations to mitigate network
overhead in edge computing systems. IEEE Transactions on Sustainable Computing,2(4):320–332.
[66] van der Meulen, R. (2018). What edge computing means for infrastructure and operations leaders. Smarter with Gartner.
[67] Wang, S., Liu, Z., Rong, Y., Zhou, B., Bai, Y., Wei, W., Wei, W., Wang, M., Guo, Y.,
and Tian, J. (2019). Deep learning provides a new computed tomography-based prognostic
biomarker for recurrence prediction in high-grade serous ovarian cancer. Radiotherapy
and Oncology, 132:171–177.
[68] Wang, Y., Wang, Q., Shi, S., He, X., Tang, Z., Zhao, K., and Chu, X. (2020). Benchmarking the performance and energy efficiency of AI accelerators for AI training. In 2020
20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing(CCGRID). IEEE.
[69] Wulsin, D. F., Gupta, J. R., Mani, R., Blanco, J. A., and Litt, B. (2011). Modeling electroencephalography waveforms with semi-supervised deep belief nets: fast classification
and anomaly measurement. Journal of Neural Engineering, 8(3):036015.
[70] Xie, X., Liu, X., Qi, H., Xiao, B., Li, K., and Wu, J. (2020a). Geographical correlationbased data collection for sensor-augmented rfid systems. IEEE Transactions on Mobile
Computing, 19(10):2344–2357.
[71] Xie, X., Liu, X., Zhao, X., Xue, W., Xiao, B., Qi, H., Li, K., and Wu, J. (2020b).
Implementation of differential tag sampling for cots rfid systems. IEEE Transactions on
Mobile Computing, 19(8):1848–1861.
[72] Xu, J., Ota, K., and Dong, M. (2018). Saving energy on the edge: In-memory cachingfor multi-tier heterogeneous networks. IEEE Communications Magazine, 56(5):102–107.
[73] Xu, J., Ota, K., and Dong, M. (2020). Big data on the fly: UAV-mounted mobile
edge computing for disaster management. IEEE Transactions on Network Science andEngineering, 7(4):2620–2630.
[74] Yan, H., Zhang, Y., Pang, Z., and Xu, L. D. (2014). Superframe planning and accesslatency of slotted MAC for industrial WSN in IoT environment. IEEE Transactions on
Industrial Informatics, 10(2):1242–1251.
[75] Zhang, C., Cao, Q., Jiang, H., Zhang, W., Li, J., and Yao, J. (2018). FFS-VA. In
Proceedings of the 47th International Conference on Parallel Processing. ACM.
[76] Zhang, Z., Chen, P., McGough, M., Xing, F., Wang, C., Bui, M., Xie, Y., Sapkota, M.,
Cui, L., Dhillon, J., Ahmad, N., Khalil, F. K., Dickinson, S. I., Shi, X., Liu, F., Su, H., Cai,J., and Yang, L. (2019). Pathologist-level interpretable whole-slide cancer diagnosis withdeep learning. Nature Machine Intelligence, 1(5):236–245.
[77] Zhu, J., Pande, A., Mohapatra, P., and Han, J. J. (2015). Using deep learning for energy
expenditure estimation with wearable sensors. In 2015 17th International Conference on
E-health Networking, Application & Services (HealthCom). IEEE.

References

65[78] Zikria, Y. B., Afzal, M. K., Kim, S. W., Marin, A., and Guizani, M. (2020). Deeplearning for intelligent IoT: Opportunities, challenges and solutions. Computer Communications, 164:50–53.

Publications
Journals
1. Sha Zhu, Kaoru Ota, Mianxiong Dong, "Energy Efficient Artificial Intelligence ofThings with Intelligent Edge," IEEE Internet of Things Journal, vol. 9, no. 10, pp.
7525-7532, May 2022.
2. Sha Zhu, Kaoru Ota, Mianxiong Dong, "Green AI for IIoT: Energy Efficient Intelligent
Edge Computing for Industrial Internet of Things," IEEE Transactions on Green
Communications and Networking (TGCN), vol. 6, no. 1, pp. 79-88, March 2022.

Proceeding of International Conference
1. Sha Zhu, Kaoru Ota, Mianxiong Dong, "More Power to Save Lives: Distributed Power
Scheduling in Smart Microgrid for Disaster Management," IEEE Global Communications Conference (GLOBECOM 2021), Madrid, Spain, December 7-11, 2021.

Under Review
1. Sha Zhu, Kaoru Ota, and Mianxiong Dong, "Multiple-layer Intelligent Edge Computing
in Early Cancer Diagnosis," IEEE Intelligent Systems, Under Review