# Toward Holistic Integration of Computing and Wireless Networking

Kwang-Cheng Chen, Yingze Wang, Zixiang Nie, Qimei Cui

► To cite this version:

Kwang-Cheng Chen, Yingze Wang, Zixiang Nie, Qimei Cui. Toward Holistic Integration of Computing and Wireless Networking. 2nd IFIP International Internet of Things Conference (IFIPIoT), Oct 2019, Tampa, FL, United States. pp.219-234, 10.1007/978-3-030-43605-6_13. hal-03371598

HAL Id: hal-03371598

https://inria.hal.science/hal-03371598

Submitted on 8 Oct 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Toward Holistic Integration of Computing and Wireless Networking *

Kwang-Cheng Chen¹, Yingze Wang¹,², Zixiang Nie¹, and Qimei Cui²

¹ University of South Florida, Tampa, FL 33620, USA kwangcheng@usf.edu

² Beijing University of Posts and Telecommunications, Beijing 100876, China

**Abstract.** To systematically cop complex systems engineering in Inter-net of Things, this paper looks into a technological challenge to effectively and efficiently integrate computing and wireless networking. One aspect is how machine learning and artificial intelligence to influence wireless networking, and another aspect is how wireless networking to enhance artificial intelligence computing. Finally, a holistic computing and net-working architecture is introduced to examine implementation of holistic computing and wireless networking.

**Keywords:** Artificial Intelligence · Machine Learning · Wireless Net-works · Internet of Things · Edge Computing

## 1 Introduction

Internet of Things (IoT) is well known to involve technology of computing, con-trol, and wireless networking, under wide-range of application scenarios. Com-plex systems engineering of IoT complicates efficient realization of IoT systems. While control of intelligent systems such as robots is widely implemented by artificial intelligence (AI) computing, holistic integration of computing and net-working emerges a critical technology for future IoT. This article presents a unique aspect to look into this new technological paradigm, we first review the applications of machine learning (ML) in the wireless networking of IoT systems (in Section 2), then turn to a particular scenario of applying AI and ML into smart factory of networking (in Section 3). By investigating the facilitation of ML in wireless networks and the requirements of wireless networking into multi-agent AI systems, the possible picture of holistic integration of AI computing and wireless networking in IoT, particularly involving mobile agents such as au-tonomous vehicles, mobile robots, smart factory, has been initially investigated in this paper.

* K.-C. Chen and Z. Nie appreciate the grant from the Cyber Florida to conduct this research.
## 2 Applications of ML in IoT Wireless Networking

Currently, a vary of remarkable developments of the network, notably wireless networks, are breaking the boundaries between virtual and reality and evolve narrow cyber towards the Internet of Things (IoT). The terminals of the internet are no longer limited to PCs, mobile phones. TVs, lamps, mirrors, water heaters, air conditioners, taxis, elevators, workstations and so on are all surprisingly connected to the Internet. The interconnection of all things dramatically facilitates people's work and life. And wireless sensor networks (WSN), the main part of IoT, extend human physiological perception limits. These vast changes also pose profound challenges to theoretical research.

Unlike a series of “excellent” features of traditional wireless communication systems, IoT has been a significant impact on wireless communication technologies. For example, the number of network nodes has increased dramatically, the network topology has become more dynamic, links and interference have become denser, and network transmission has demand fluctuates sharply, etc. Recently, Machine Learning, particularly Reinforcement Learning, has been used as an emerging tool to effectively address the above problems and challenges. Different from classic convex optimization or optimization methods, ML shows exciting performance in solving complicated mathematical structures and instability decisions.

According to our investigation of published articles, the current research directions of applying ML in IoT can roughly divide into three categories: MAC protocol design, cache and offload in MEC scenarios, and network security. Besides, there are some explorations for unique scenarios and particular problems that not included in this list. (The framework is shown in Figure 1)

### 2.1 MAC protocol: wireless access, routing and others

Modern networks, particularly IoT, become more decentralized and ad-hoc and more dynamic in topology and routing. In IoT, entities such as sensors and mobile users need to make independent decisions, e.g., multi-routing selections, channel selections, to achieve their own goals, e.g., throughput maximization. However, this is challenging due to the dynamic and the uncertainty of network status. The reinforcement learning represented by Q-learning and Deep Q-learning Network (DQN) can intuitively adapt to the design requirements of such mac protocols. Therefore, extensive research has been carried out in this direction, such as dynamic wireless access schemes, dynamic routing, multi-point cooperative communication, etc.

Focus on multiple Access for Heterogeneous Wireless Networks, [54] proposed and investigated a MAC protocol based on deep reinforcement learning for heterogeneous wireless networking, referred to as Deep-reinforcement Learning Multiple Access (DLMA). A salient feature of DLMA is that it can learn to achieve an overall objective (e.g., α-fairness objective) by a series of state-action-reward observations while operating in the heterogeneous environment. In particular, it can achieve near-optimal performance with respect to the objective without
<figure><img src="image_2.png" /><figcaption>Fig. 1. The appli of Machine Learning for IoT wireless network</figcaption></figure>

knowing the detailed operating mechanisms of the coexisting MAC. For dynamic
routing implementation issues, [49] devises the off-policy $Q_{ssp}$ algorithm and the
on-policy SARSA$_{ssp}$ algorithm to solve the routing problem in Wireless Sensor
Networks. Specifically, the authors tackle the stochastic shortest path problem
using reinforcement learning schemes by modeling the path searching procedure
as an appropriate discounted Markov decision process. [41] considers the multi-
hop routing algorithm plays an important role in the exploration and monitoring
of deep-sea environments. For this A proposed a routing algorithm based on the
Q-learning for 3D under water WSN. Combined with defined distance and energy
paths, the researchers derived the iterative formula of the Q-table. The proposed
QL-EDR algorithm can extend the network lifetime and improve the efficiency
of the data collection, compared to the conventional protocol. In addition, the
authors defined a regulatory factor to adjust the network performance. Accord-
ing to the realistic demands, choose appropriate values of factor to improve the
network throughput, to reduce the average end-to-end delay or to prolong the
network lifetime.

[62] works on improving the packet transmission efficiency using Cognitive
networks. A Q-learning-based transmission scheduling mechanism using deep
learning for the cognitive radio-based IoT is proposed to solve the problem of
how to achieve the appropriate strategy to transmit packets of different buffers
through multiple channels to maximize the system throughput. [43] [23] have
also tried machine learning applications in the field of IoT congestion control,
and has gained some valuable experience.[57] proposed a cooperative spectrum
sensing algorithm for cognitive radio networks. By implementing DQN based
on upper confidence bounds with Hoeffding-style to improve the exploration efficiency, the proposed algorithm can achieve better reward performance with faster convergence speed than the conventional algorithms based on Q-learning with $\epsilon$-greedy.

We note that most of the research in this direction models the network state of IoT as MDP. In addition, the DQN method receives more attention than other algorithms. The future network will involve multiple network slides, and these network slides have multiple conflicting goals, which brings many challenges to the traditional resource management mechanism and network standard formulation that are worthy of in-depth study.

## 2.2 MEC: caching and offloading

MEC is one of the key scenarios of IoT, and the intra-network cache can effectively reduce duplicate content transmission. Research on wireless caching shows that by caching content in wireless devices, you can significantly reduce access latency, energy consumption, and overall traffic. This direction has also attracted many studies. As each node's storage, computing, and energy consumption capabilities are limited, how to coordinate collaboration between nodes, such as decision cache content, has become a focus of attention.

Most recent studies focus on "attention" tagging of content and tasks and caching and calculation offload allocation based on importance. But there are still many attempts to provide new ideas for research. [25] cares about that rare wireless network resources are difficult to meet the influx of a huge number of terminal devices. Specifically, the authors use two potential recurrent neural network approaches, the echo state network (ESN) and the long short-term memory (LSTM) network, to make predictions about user mobility and content popularity. Finally, use DQN algorithm to make cached decisions for prediction results.[61] formulated the cache replacement problem as a MDP problem and propsed a DRL-based caching policy. In the model *S*, *A* and *reward* are defined as values of information about cached/arrived data items, the caching action selected by the edge node and the sum utility of all data items which are requested, respectively. [44] tries to simultaneously tackle the issues of content caching strategy, computation offloading policy, and radio resource allocation, in fog computing. Authors use the actor-critic reinforcement learning framework to solve the joint decision-making problem with the objective of minimizing the average end-to-end delay, due to wireless signals and service requests have stochastic properties. The deep neural network (DNN) is employed as the function approximator to estimate the value functions in the critical part due to the extremely large state and action space in the problem. The actor part uses another DNN to represent a parameterized stochastic policy and improves the policy with the help of the critic.

At the same time, some scholars have also noticed the connectivity of the MEC devices in this scenario. [29] focus on the connectivity solutions especially for those covering the wide remote areas in the scale of kilometer squares. Although many low-power wide-area network technologies are supposed to sup-
port long-range low-power wireless communication, underneath star topology
limits the scalability of the networks due to the need for a central hub. To
provide connectivity to a wider area, the authors propose to build the mesh
topology upon these LPWAN technologies and propose a distributed as well as
energy-efficient reinforcement learning based routing algorithm for the wide-area
wireless mesh IoT networks. [16]'s goal is to acquire an online algorithm that
optimally adapts task offloading decisions and wireless resource allocations to
the time-varying wireless channel conditions. This requires quickly solving hard
combinatorial optimization problems within the channel coherence time, which
is hardly achievable with conventional numerical optimization methods. the au-
thors propose a Deep Reinforcement learning-based Online Offloading (DROO)
framework that implements a deep neural network as a scalable solution that
learns the binary offloading decisions from the experience. It eliminates the need
for solving combinatorial optimization problems, and thus greatly reduces the
computational complexity especially in large-size networks.

MEC scenarios involve very complicated system analysis, which is due a uni-
fied study on caching, offloading, networking, and transmission control. Strong
couplings among mobile users with heterogeneities in application demand, QoS
provisioning, mobility pattern, radio access interface, and wireless resources also
cause for above. A model-free reinforcement learning approach becomes a promis-
ing candidate to manage huge state space and optimization variables.

## 2.3 IoT security and reliability

In IoT, physical devices, sensors, appliances, and other different objects can com-
municate with each other without the need for human intervention in IoT. And
IoT has many critical and non-critical applications. The security of IoT became
a crucial problem. Future networks become more decentralized and ad-hoc in
nature which is vulnerable to various attacks such as Denial-of-Service (DoS)
and cyber-physical attacks. Recently, the DQL has been used as an effective
solution to avoid and prevent the attacks. [31].

In [52], the Markov game framework is employed to model and analyze the
anti-jamming defense problem. Based on Q-learning, the authors development a
collaborative multi-agent anti-jamming algorithm. As machine learning and arti-
ficial intelligence can be used for the protection of devices by analyzing traffic or
devices behavior, the [48] development a model of increasing security of wireless
environment for IoT appliance through creating a fingerprint by Machine Learn-
ing algorithm. The experimental result shows that the model is able to detect
anomaly flooding traffic in Wi-Fi networks based on characteristic patterns that
separate normal traffic from malicious activity.

A wide variety of low-cost radio technologies, that being used to enable wire-
less communication in IoT, brings a security problem due to the fact that it
is very easy for a malicious user to perform passive wireless signal scanning on
these networks and use this information to launch identity-based attacks. In
[34], the authors propose a learning-based strategy to detect spoofing attacks
in wireless sensor networks. Based on detailed analytical models for the mobile
radio channel, the proposed algorithm combines two classifiers to process and analyze the instant samples of received signal strength to detect attacks. In [11], a watermarking algorithm is proposed for dynamic authentication of IoT signals to detect cyber-attacks. The proposed watermarking algorithm, based on a deep learning long short-term memory structure, enables the IoT devices to extract a set of stochastic features from their generated signal and dynamically watermark these features into the signal. This method enables the IoT gateway, which collects signals from the IoT, to effectively authenticate the reliability of the signals.

The sharp increase in interference caused by the dense network is also an aspect that needs to be explored in depth. [14] considers the optimization of the cache-enabled opportunistic interference alignment network as a so complex problem. The results in the literature were demonstrated that the performance of cache-enabled opportunistic IA networks can be significantly improved by using the proposed deep reinforcement learning approach.

We found that the research on reinforcement learning applied to network security is mainly focused on anomaly detection and identity authentication, and research on interference in network transmission needs to be further promoted.

## 2.4 Low-power operation and sensor networks

In addition to the above directions in which the studies are concentrated, scholars have also conducted extensive explorations on the application of machine learning in wireless networks. Such as power supply problems in low power networks, network structure update problems, etc.

Sensing devices operating in the upcoming IoT are likely to rely on the radio frequency (RF) transmissions of a hybrid access point (HAP) for energy [51]. The HAP is also responsible for setting the sampling or monitoring time of these devices according to their harvested energy. A challenging issue is that setting the HAP's charging time and also the sampling time of each device with imperfect channel gains information. [51] also propose a scheme, through the improvement of Actor-Critic algorithm, to minimize the sampling time of the device. Wireless sensor network has the characteristics of scattered network requirements and uneven information. [46] study the WSN-based field sensing and reconstruction problem. The authors establish a two-layer learning framework based on reinforcement learning, and present the detailed design for an adaptive sampling policy which can actively determine the most informative sensing location and thus significantly reduce the communication cost.

# 3 Machine Learning in Smart Factory

A smart factory is an IoT system of particular interest. Factories, especially manufacturing factories are embracing the notion of integrating cyber resources such as computation, networking and physical processes together to drive the development of smart factory, which is Cyber-Physical Systems (CPS) [42] [38].
Several technologies are believed to bring evolutionary changes to the traditional factories in industry, they are: Internet of Things (IoT), Wireless Sensor Networks (WSN), Cloud Services and Artificial Intelligence [42] [38] [47]. The main goal is to accommodate product variants and production number variance to fulfill the demands from major customers down to individual customers. That requires the real-time collection of relevant information, a fast reintegration of resources within the factory and an optimized re-setup or reconfiguration solution of physical entities within the factory [40]. In addition to that, the call for sustainability requirement smart factory to improve the utility of raw materials and energy [40], even a higher efficiency of supply chain, product packing and logistics among factories.

The integration of cyber resources and physical entities happens on top of automated manufacturing equipped factories. By regarding each physical entity after integration a system, the whole CPS is actually a system of systems. The integration of cyber resources and physical entities is considered in two ways: vertical integration that emphasis on the real-time information collection and control, and horizontal integration that emphasis on cooperation among physical entities [40] [59].

<figure><img src="image_3.png" /><figcaption>Fig. 2. A horizontal integration towards smart factory.</figcaption></figure>

**3.1 Computing and Networking Systems in Smart Factories**

By regarding each physical entity after integrating a system, the whole CPS is actually a system of systems. As shown in figure 2, illustrative systems including product design, materials supply (raw materials), energy use to drive the manufacturing, entities in manufacturing process, product test (inspection),
maintenance of the factory, logistic (local logistic within the factory dealing
with semi-products, packing and shipping of the finished products), local fog
nodes(including storage, computing and etc.), cloud service (including cloud
storage, cloud computing and etc.) and backbone network (with internet access)
are considered to be the import systems in smart factories [38]. Paper [4] [38]
expounded data powered “smart design” in future industry. The sensing devices
integrated to the traditional factory brings tremendous amount of data from ev-
ery stage of manufacturing. After proper data processing and visualization, the
designers could refine the manufacturing processes, part design and etc towards
more flexible and energy saving manufacturing. Paper [38] [4] [17] discussed
the supply chain in smart factory. Since the fact that the production require-
ment is dynamic in the smart factory age, the short-term supply system paves a
solid foundation of smart manufacturing. Paper [17] proposed a multi-objective,
multi-stage flexible flow-shop scheduling model for fast response supply chain
and manufacture agent collaboration. Resource and energy efficiency is another
important indicators toward next generation industry in that it's directly related
to the profit and environment. Smart energy supply gives energy consumption
data as a feedback to the designer and management to improve the plant organi-
zation and production design [42] [19] [59]. A lot of paper in literature focus on
Multi-agent System (MAS) in manufacturing system. Paper [39] [17] proposed
algorithms for scheduling in MAS considering efficiency and dynamic. Paper [10]
[36] [27] adopt Machine Learning (ML), Reinforcement Learning (RL) and Deep
Learning (DL) to give solutions to MAS task allow cation and scheduling con-
sidering the load balancing and efficiency. Paper [24] proposed a deep learning
based inspection system with high accuracy, which can find the possible defective
products. Paper [35] gives a good vision of smart factory maintenance consid-
ering the task offloading, path planning, and access point selection in mobile
scenario. Besides, paper [45] introduced ML based mechanical tool wearing pre-
diction, which is a good addition to the smart maintenance. Local logistics is also
important part of smart factory. Paper [40] [38] discussed about raw material
distribution and (semi-)product collection and delivery within smart factory. In
the framework proposed in figure 2, the edge devices, local fog nodes and cloud
all have the capability of computing. Paper [24] and [27] give a possible solution
that utilize the edge and fog computation. The good side is edge/fog computing
delivers lower latency than cloud computing in practical application. Of course,
data intensive and complicated deep learning algorithms may still good to be
executed on the cloud, but edge computing and fog computing are more in line
with the needs of smart factories for real-time environment and requirement
changes.

3.2 Vertical Integration

As shown in figure 3, seven important elements in traditional factory could all
get integrated with WSN, actuators, computing and AI to become a system.
Each one of four technology is a layer put on top of traditional element, so it's
called vertical integration. After equipped with WSN, the traditional element in
<figure><img src="image_4.png" /><figcaption>Fig. 3. A vertical integration towards smart factory.</figcaption></figure>

addition to the backbone network and fog or cloud becomes a IoT system. After equipped with actuator (either software or physical part that could control the entity), the traditional element becomes a cloud controlled IoT system. After the integration of computing capabilities and AI algorithms (software), the traditional element becomes a complete local AI agent. For example, the energy meter in the energy supply system will keep monitoring the energy consumption data and then upload to cloud storage. The related designer could utilize those data for further refining the design of the product in purpose of green manufacturing. Another example is, after collecting maintenance data from the cloud service, the AI on the cloud send an instruction that one of the robots in manufacturing need to get maintenance. Therefore, the other robots will get more tasks to make up that change for the overall goal. In this example, the actuator is the software running on manufacturing robots. Like just mentioned, the IoT integration brings data that hard to access in traditional factory. This makes machine learning, deep learning and reinforcement learning based AI integration possible to facilitate the smart manufacturing.

## 3.3 Horizontal Integration

As shown in figure 2, after the vertical integration, seven systems are connected to the backbone network of the smart factory. Also, they play a part of the smart factory network. The connections indicate the physical interaction that could happen in the smart factory, of course, along with data exchange. For example, the AI agent within the manufacturing system find the materials are running out. It could send material requirement to the material supply system.
The material supply system then prepares the materials and send requirement to the local logistic system. The logistic system then initiates a material distribution. Another example is all the machine related systems (manufacturing, product test, logistics and etc.) may need daily maintenance. They send their daily running statistics and AI agent in maintenance system will analysis those data and gives a cost-optimized and energy efficient maintenance schedule. This kind of integration emphasis the interaction among systems in smart factory, making a better quality, higher efficiency automation possible. Horizontal integration is vital especially when there are human involved in the overall work flow or the vertical integration is incomplete considering the cost. The reason is that, when there are human involved or incompleteness of vertical integration, some data are not available to the cloud service. For example, in a factory, the local logistics has to be done by human. However, the output of human part, that is, the delivery of the raw material is dynamic because of the variance of working efficiency. At the same time, no measurement will be acquired directly from human in terms of, for example, the working efficiency considering the privacy. Thus, those measurement could be acquired from the next, fully integrated process of the manufacturing by interaction: the manufacturing robot received the raw materials, as a variable in the whole manufacturing. Therefore, the horizontal integration requires sensing and data exchange among systems in the smart factory, which will need extra sensors or related parts. Other reasons such as latency of a centralized cloud based control, failure of the data collection system and etc. brings the necessarily of the horizontal integration. Paper [3] introduces a Parallel Reinforcement Learning (PRL) based IoT system to reduce the learning time of Reinforcement Learning (RL) considering the communication overhead. The simulation based on the multi-agent system in smart factory gives a good vision of horizontal integration.

# 4 Future Networking and Computing Architecture

The holistic networking and computing architecture can be facilitated from two aspects: (1) machine learning for communications and networks (2) networking for AI agents to form a networked multi-agent system, which will be detailed in the following two sub-sections.

## 4.1 State-of-the-Art Applications of Machine Learning to Future Wireless Network Architecture

Future wireless network architecture accommodating machine learning (ML) emerges as an important technology for next decades, while ITU-T forming a focus group (FG) to study from 2018 to 2020. When incorporating ML functionalities into network architecture, there are two mechanisms to execute ML algorithms: online ML and offline ML. The online ML computing means the ML functionality is embedded into networking algorithms or protocols, and thus must be implemented into the corresponding network entities. On the other hand, if
the ML functionality is executed then used to assist network functionalities, it
is known as offline ML computing that can be executed in a co-locating com-
puting facility connected to the corresponding network entities. The offline ML
can be also computed in another far-away computing facility and then transfer
the model of learning to the target network entity. As shown in Figure 4, the
ML computing can be executed and co-located with the user equipment (UE)
or agents, radio access network (RAN), or core network (CN), in addition to the
cloud. The emerging edge computing or edge artificial intelligence (AI) [33] can
be considered co-locating with RAN.

<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Alternatives, agent computing with UE, edge computing co-locating with RAN, and cloud computing through CN to implement machine learning or AI in the wireless network architecture</figcaption></figure>

Generally speaking, ML can be applied in a few possible networking and
communication scenarios:

- Channel State Information (CSI): CSI is critical to air-interface technology for networking algorithms and physical layer communication, which has been considered to be inferred or estimated with the aid of deep learning [53, 18], or calibration the channel models for preferred CSI [1].

- User Behavior: User behavior such as human/vehicular mobility patterns can be useful to network management and mobility management functionalities [7], and autonomous system operation [8], through big data analysis by ML or reinforcement learning [28].

- Traffic Prediction: Deep packet inspection, network intelligence, and user mobility patterns, can be used to predict wireless network traffic for more efficient network/radio resource allocation [55, 20].

- Cybersecurity: ML might be one of the most attractive tools to enhance network security, detect attacks and intrusions to networks [13, 56].

- Anticipatory Networking Mechanism: Except using reinforcement learning or multi-armed bandit mechanisms for radio resource or network resource
allocation [32, 58], existing applications of ML to wireless networking is gen-
erally offline learning to assist or enhance existing solutions. However, an-
other advantage of ML is to develop predictive networking mechanism via
online learning such that ML can enable networking functionalities that are
nor possible before. One of such rare examples is the anticipatory mobility
management using Naive Bayesian and recursive belief update [26, 6] as on-
line learning to enable proactive communication and virtual cell for ultra-low
latency wireless networking, where anticipatory is widely adopted in AI.

## 4.2 Networked Multi-Agent Systems

In addition to apply ML to wireless networks, another question of AI computing
and wireless networking arises: what is the desirable wireless networking for
agents using ML? More precisely, how to design a wireless network for agents of
machine intelligence, say a multi-robot system (MRS) or a multi-agent system
(MAS).

Legg and Hutter gave an informal definition of machine intelligence in [22]:
*Intelligence measures an agent's ability to achieve goals in a wide range of environments.* Distributed artificial intelligence (DAI) has been brought into attention in AI research well over 3 decades [2], which has two common sub-disciplines: distributed problem solving (DPS) and multi-agent system (MAS). DPS typically decomposes task into several not completely independent sub-problems that can be executed on different processors and then synthesizes a solution. On the other hand, MAS considers an agent is an intelligent entity, which can be a robot or an AV, with goals and actions in an operating environment. In state-of-the-art CPS/IoT that are highly parallel in computing, a MAS typically represents a complex system of multiple agents and the mechanism for coordination of agents' behaviors. Please recall that Demazeau inspiringly defined MAS consisting of four major aspects: agents, environments, interactions, and organization [9]. When RL deals with agent's action and environment, communication for decisions was brought into MAS of agents using RL by modeling as partially observed MDP [50] [37]. Though communication or exchange of actions by agents has been studied in MAS and DAI for a long time, the features of wireless networks have been hardly considered in literature. In [37], finite number of communication channels with so-called fast communication was considered for information sharing among a team of cooperative agents. [12] [60] [30] indirectly considered the communication in MAS. However, realistic wireless communications and networking has not well taken into consideration, nor impacts on ML mechanisms.

An interesting study looks into collective behavior of autonomous vehicles
moving across a region of Manhattan streets, by treating the behavior of each
autonomous vehicle as an agent using reinforcement learning. It is shown that
wireless networking reduces average delay [21]. In such wireless networking, dif-
ferent from human-to-human personal communication, the reward map and pol-
icy of another autonomous vehicle in the interaction range would be useful infor-
mation to exchange. The age of such information is critical and thus ultra low-
latency wireless networking is highly preferred, in which the real-time ALOHA
has been considered as multiple access. For collaborative robots that each has
own machine learning algorithms (such as moving actions and planning) to exe-
cute toward a common goal, wireless networking would be extremely beneficial
to collective efficiency [5]. A lot of issues remain open in such networked MAS,
such as network topology [15] and innovative machine learning for networked
MAS.

5 Conclusions

Holistic integration and interaction of AI computing and wireless networking for IoT systems still has a long way to develop. This paper initially brings up the literature survey and in-depth discussions toward this ultimate goal. Many open issues still require remarkable technological innovations in the future.

References

1. Adeogun, R.: Calibration of stochastic radio propagation models using machine learning. IEEE Antennas and Wireless Propagation Letters **18**(12), 2538–2542 (2019)

2. Bond, A.H., Gasser, L.: Readings in distributed artificial intelligence. Morgan Kaufmann (2014)

3. Camelo, M., Claeys, M., Latré, S.: Parallel reinforcement learning with minimal communication overhead for iot environments. IEEE Internet of Things Journal (2019)

4. Chen, B., Wan, J., Shu, L., Li, P., Mukherjee, M., Yin, B.: Smart factory of industry 4.0: Key technologies, application case, and challenges. IEEE Access **6**, 6505–6519 (2017)

5. Chen, K.C., Hung, H.M.: Wireless robotic communication for collaborative multi-agent systems. In: ICC 2019-2019 IEEE International Conference on Communications (ICC). pp. 1–7. IEEE (2019)

6. Chen, K.C., Zhang, T., Gitlin, R.D., Fettweis, G.: Ultra-low latency mobile networking. IEEE Network **33**(2), 181–187 (2018)

7. Chih-Lin, I., Sun, Q., Liu, Z., Zhang, S., Han, S.: The big-data-driven intelligent wireless network: Architecture, use cases, solutions, and future trends. IEEE Vehicular Technology Magazine **12**(4), 20–29 (2017)

8. Cui, Q., Wang, Y., Chen, K.C., Ni, W., Lin, I.C., Tao, X., Zhang, P.: Big data analytics and network calculus enabling intelligent management of autonomous vehicles in a smart city. IEEE Internet of Things Journal **6**(2), 2021–2034 (2018)

9. Demazeau, Y.: From interactions to collective behaviour in agent-based systems. In: In: Proceedings of the 1st. European Conference on Cognitive Science. Saint-Malo. Citeseer (1995)

10. Elango, M., Nachiappan, S., Tiwari, M.K.: Balancing task allocation in multi-robot systems using k-means clustering and auction based mechanisms. Expert Systems with Applications **38**(6), 6486–6491 (2011)

11. Ferdowsi, A., Saad, W.: Deep Learning for Signal Authentication and Security in Massive Internet-of-Things Systems. IEEE Transactions on Communications **67**(2), 1371–1387 (Feb 2019)
12. Ge, X., Han, Q.L.: Distributed formation control of networked multi-agent systems using a dynamic event-triggered communication mechanism. IEEE Transactions on Industrial Electronics 64(10), 8118-8127 (2017)

13. He, D., Liu, C., Quek, T.Q., Wang, H.: Transmit antenna selection in mimo wiretap channels: A machine learning approach. IEEE Wireless Communications Letters 7(4), 634-637 (2018)

14. He, Y., Zhang, Z., Yu, F.R., Zhao, N., Yin, H., Leung, V.C.M., Zhang, Y.: Deep-Reinforcement-Learning-Based Optimization for Cache-Enabled Opportunistic Interference Alignment Wireless Networks. IEEE Transactions on Vehicular Technology 66(11), 10433-10445 (Nov 2017)

15. Hsiao, J.H., Chen, K.C.: Communication methodology to control a distributed multi-agent system. In: ICC 2019-2019 IEEE International Conference on Communications (ICC). pp. 1-6. IEEE (2019)

16. Huang, L., Bi, S., Zhang, Y.J.: Deep Reinforcement Learning for Online Computation Offloading in Wireless Powered Mobile-Edge Computing Networks. IEEE Transactions on Mobile Computing pp. 1-1 (2019)

17. Ivanov, D., Dolgui, A., Sokolov, B., Werner, F., Ivanova, M.: A dynamic model and an algorithm for short-term supply chain scheduling in the smart factory industry 4.0. International Journal of Production Research 54(2), 386-402 (2016)

18. Jiang, Z., He, Z., Chen, S., Molisch, A.F., Zhou, S., Niu, Z.: Inferring remote channel state information: Cramér-rae lower bound and deep learning implementation. In: 2018 IEEE Global Communications Conference (GLOBECOM). pp. 1-7. IEEE (2018)

19. Kang, H.S., Lee, J.Y., Choi, S., Kim, H., Park, J.H., Son, J.Y., Kim, B.H., Do Noh, S.: Smart manufacturing: Past research, present findings, and future directions. International Journal of Precision Engineering and Manufacturing-Green Technology 3(1), 111-128 (2016)

20. Kim, J., Hwang, G.: Adaptive bandwidth allocation based on sample path prediction with gaussian process regression. IEEE Transactions on Wireless Communications 18(10), 4983-4996 (2019)

21. Ko, E., Chen, K.C.: Wireless communications meets artificial intelligence: An illustration by autonomous vehicles on manhattan streets. In: 2018 IEEE Global Communications Conference (GLOBECOM). pp. 1-7. IEEE (2018)

22. Legg, S., Hutter, M.: Universal intelligence: A definition of machine intelligence. Minds and machines 17(4), 391-444 (2007)

23. Lei, L., Xu, H., Xiong, X., Zheng, K., Xiang, W., Wang, X.: Multiuser Resource Control With Deep Reinforcement Learning in IoT Edge Computing. IEEE Internet of Things Journal 6(6), 10119-10133 (Dec 2019)

24. Li, L., Ota, K., Dong, M.: Deep learning for smart industry: Efficient manufacture inspection system with fog computing. IEEE Transactions on Industrial Informatics 14(10), 4665-4673 (2018)

25. Li, L., Xu, Y., Yin, J., Liang, W., Li, X., Chen, W., Han, Z.: Deep Reinforcement Learning Approaches for Content Caching in Cache-Enabled D2d Networks. IEEE Internet of Things Journal 7(1), 544-557 (Jan 2020)

26. Lin, C.Y., Chen, K.C., Wickramasuriya, D., Lien, S.Y., Gitlin, R.D.: Anticipatory mobility management by big data analytics for ultra-low latency mobile networking. In: 2018 IEEE International Conference on Communications (ICC). pp. 1-7. IEEE (2018)

27. Lin, C.C., Deng, D.J., Chih, Y.L., Chiu, H.T.: Smart manufacturing scheduling with edge computing using multi-class deep q network. IEEE Transactions on Industrial Informatics (2019)
28. Liu, X., Liu, Y., Chen, Y., Hanzo, L.: Trajectory design and power control for multi- uav assisted wireless networks: A machine learning approach. IEEE Transactions on Vehicular Technology (2019)

29. Liu, Y., Tong, K.F., Wong, K.K.: Reinforcement learning based routing for energy sensitive wireless mesh IoT networks. Electronics Letters 55(17), 966–968 (Aug 2019)

30. Liu, Z., Dai, J., Wu, B., Lin, H.: Communication-aware motion planning for multi-agent systems from signal temporal logic specifications. In: 2017 American Control Conference (ACC). pp. 2516–2521. IEEE (2017)

31. Mamdouh, M., I. Elrukhsi, M.A., Khattab, A.: Securing the Internet of Things and Wireless Sensor Networks via Machine Learning: A Survey. In: 2018 International Conference on Computer and Applications (ICCA). pp. 215–218. IEEE, Beirut (Aug 2018)

32. Nguyen, D.D., Nguyen, H.X., White, L.B.: Reinforcement learning with network-assisted feedback for heterogeneous rat selection. IEEE Transactions on Wireless Communications 16(9), 6062–6076 (2017)

33. Park, J., Samarakoon, S., Bennis, M., Debbah, M.: Wireless network intelligence at the edge. Proceedings of the IEEE 107(11), 2204–2239 (2019)

34. Pinto, E.M.d.L., Lachowski, R., Pellenz, M.E., Penna, M.C., Souza, R.D.: A Machine Learning Approach for Detecting Spoofing Attacks in Wireless Sensor Networks. In: 2018 IEEE 32nd International Conference on Advanced Information Networking and Applications (AINA). pp. 752–758. IEEE, Krakow, Poland (May 2018)

35. Rahman, A., Jin, J., Cricenti, A.L., Rahman, A., Kulkarni, A.: Communication-aware cloud robotic task offloading with on-demand mobility for smart factory maintenance. IEEE Transactions on Industrial Informatics 15(5), 2500–2511 (2018)

36. Shiue, Y.R., Lee, K.C., Su, C.T.: Real-time scheduling for a smart factory using a reinforcement learning approach. Computers & Industrial Engineering 125, 604–614 (2018)

37. Stein, S., Williamson, S.A., Jennings, N.R.: Decentralised channel allocation and information sharing for teams of cooperative agents. In: Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume 1. pp. 231–238. International Foundation for Autonomous Agents and Multiagent Systems (2012)

38. Tao, F., Qi, Q., Liu, A., Kusiak, A.: Data-driven smart manufacturing. Journal of Manufacturing Systems 48, 157–169 (2018)

39. Tereshchuk, V., Stewart, J., Bykov, N., Pedigo, S., Devasia, S., Banerjee, A.G.: An efficient scheduling algorithm for multi-robot task allocation in assembling aircraft structures. arXiv preprint arXiv:1902.08905 (2019)

40. Thoben, K.D., Wiesner, S., Wuest, T.: “industrie 4.0” and smart manufacturing-a review of research issues and application examples. International Journal of Automation Technology 11(1), 4–16 (2017)

41. Wang, S., Shin, Y.: Efficient Routing Protocol Based on Reinforcement Learning for Magnetic Induction Underwater Sensor Networks. IEEE Access 7, 82027–82037 (2019)

42. Wang, S., Wan, J., Li, D., Zhang, C.: Implementing smart factory of industrie 4.0: an outlook. International Journal of Distributed Sensor Networks 12(1), 3159805 (2016)
43. Wang, Z., Zhang, J., Zhang, X., Wang, W.: Reinforcement Learning Based Congestion Control in Satellite Internet of Things. In: 2019 11th International Conference on Wireless Communications and Signal Processing (WCSP). pp. 1–6. IEEE, Xi’an, China (Oct 2019)

44. Wei, Y., Yu, F.R., Song, M., Han, Z.: Joint Optimization of Caching, Computing, and Radio Resources for Fog-Enabled IoT Using Natural Actor-Critic Deep Reinforcement Learning. IEEE Internet of Things Journal 6(2), 2061–2073 (Apr 2019)

45. Wu, D., Jennings, C., Terpenny, J., Gao, R.X., Kumara, S.: A comparative study on machine learning algorithms for smart manufacturing: tool wear prediction using random forests. Journal of Manufacturing Science and Engineering 139(7), 071018 (2017)

46. Wu, H., Zhang, Z., Jiao, C., Li, C., Quek, T.Q.S.: Learn to Sense: A Meta-Learning-Based Sensing and Fusion Framework for Wireless Sensor Networks. IEEE Internet of Things Journal 6(5), 8215–8227 (Oct 2019)

47. Wuest, T., Weimer, D., Irgens, C., Thoben, K.D.: Machine learning in manufacturing: advantages, challenges, and applications. Production & Manufacturing Research 4(1), 23–45 (2016)

48. Xia, F., Song, H., Xu, C.: Securing the wireless environment of IoT. In: 2018 IEEE International Conference of Safety Produce Informatization (IICSPI). pp. 315–318. IEEE, Chongqing, China (Dec 2018)

49. Xia, W., Di, C., Guo, H., Li, S.: Reinforcement Learning Based Stochastic Shortest Path Finding in Wireless Sensor Networks. IEEE Access 7, 157807–157817 (2019)

50. Xuan, P., Lesser, V., Zilberstein, S.: Communication decisions in multi-agent cooperation: Model and experiments. In: Proceedings of the fifth international conference on Autonomous agents. pp. 616–623. ACM (2001)

51. Yang, C., Chin, K.W., He, T., Liu, Y.: On Sampling Time Maximization in Wireless Powered Internet of Things. IEEE Transactions on Green Communications and Networking 3(3), 641–650 (Sep 2019)

52. Yao, F., Jia, L.: A Collaborative Multi-Agent Reinforcement Learning Anti-Jamming Algorithm in Wireless Networks. IEEE Wireless Communications Letters 8(4), 1024–1027 (Aug 2019)

53. Ye, H., Li, G.Y., Juang, B.H.: Power of deep learning for channel estimation and signal detection in ofdm systems. IEEE Wireless Communications Letters 7(1), 114–117 (2017)

54. Yu, Y., Wang, T., Liew, S.C.: Deep-Reinforcement Learning Multiple Access for Heterogeneous Wireless Networks. IEEE Journal on Selected Areas in Communications 37(6), 1277–1290 (Jun 2019)

55. Zhang, C., Zhang, H., Qiao, J., Yuan, D., Zhang, M.: Deep transfer learning for intelligent cellular traffic prediction based on cross-domain big data. IEEE Journal on Selected Areas in Communications 37(6), 1389–1401 (2019)

56. Zhang, L., Restuccia, F., Melodia, T., Pudlewski, S.M.: Taming cross-layer attacks in wireless networks: a bayesian learning approach. IEEE Transactions on Mobile Computing 18(7), 1688–1702 (2018)

57. Zhang, Y., Cai, P., Pan, C., Zhang, S.: Multi-Agent Deep Reinforcement Learning-Based Cooperative Spectrum Sensing With Upper Confidence Bound Exploration. IEEE Access 7, 118898–118906 (2019)

58. Zhao, P., Tian, H., Cheny, K.C., Fan, S., Nie, G.: Context-aware tdd configuration and resource allocation for mobile edge computing. IEEE Transactions on Communications (2019)
59. Zheng, P., Sang, Z., Zhong, R.Y., Liu, Y., Liu, C., Mubarok, K., Yu, S., Xu, X., et al.: Smart manufacturing systems for industry 4.0: Conceptual framework, scenarios, and future perspectives. *Frontiers of Mechanical Engineering* **13**(2), 137–150 (2018)

60. Zhou, L., Yang, P., Chen, C., Gao, Y.: Multiagent reinforcement learning with sparse interactions by negotiation and knowledge transfer. *IEEE transactions on cybernetics* **47**(5), 1238–1250 (2016)

61. Zhu, H., Cao, Y., Wei, X., Wang, W., Jiang, T., Jin, S.: Caching Transient Data for Internet of Things: A Deep Reinforcement Learning Approach. *IEEE Internet of Things Journal* **6**(2), 2074–2083 (Apr 2019)

62. Zhu, J., Song, Y., Jiang, D., Song, H.: A New Deep-Q-Learning-Based Transmission Scheduling Mechanism for the Cognitive Internet of Things. *IEEE Internet of Things Journal* **5**(4), 2375–2385 (Aug 2018)