# Deep and Reinforcement Learning Technologies on Internet of Vehicle (IoV) Applications: Current Issues and Future Trends

Authors: Lina Elmoiz Alatabani; Elmustafa Sayed Ali; Rania A. Mokhtar; Rashid A. Saeed; Hesham Alhumyani; Mohammad Kamrul Hasan
Journal: Journal of Advanced Transportation
 (2022)

---

## Abstract

Recently, artificial intelligence (AI) technology has great attention in transportation systems, which led to the emergence of a new concept known as Internet of Vehicles (IoV). The IoV has been associated with the IoT revolution and has become an active field of research due to the great need, in addition to the increase in the various applications of vehicle communication. AI provides unique solutions to enhance the quality of services (QoS) and performance of IoV systems as well. In this paper, some concepts related to deep learning networks will be discussed as one of the uses of machine learning in IoV systems, in addition to studying the effect of neural networks (NNs) and their types, as well as deep learning mechanisms that help in processing large amounts of unclassified data. Moreover, this paper briefly discusses the classification and clustering approaches in predicative analysis and reviews their abilities to enhance the performance of IoV application systems.


---

## Body

## 1. Introduction

Deep learning (DL) is a technology scheme used to carry out the intelligent machine-learning concept, which processes the data and creates patterns that make decisions according to the input data [1]. The term “deep” represents the multihidden layers in the learning network, which consists of several connected processing nodes. In recent years, DL has great evolution in helping to enhance many IoT applications [2]. DL improves the IoV quality of services for different related vehicular applications such as autonomous driving and transportation traffic control and smart cities. AI applications are used in many communication fields and in IoV particularly to meet the needs and expectations of users. Attributing the importance of using AI, this paper provides details about deep learning techniques to improve services in IoV applications.The paper is organized as follows: Section2 provides a background to the DL approach and its evolution . The concept of using machine learning (ML) algorithms in IoV systems are briefly introduced in Section 3. In Section 4, some of the DL applications in IoV networks are discussed considering issues such as security and collision prediction. In Section 5, the performance of deep reinforcement learning (DRL) in IoV is presented showing its impact to enhance some aspects related to energy efficiency, resource management, and performance optimization. Moreover, the performance of DL-based IoV network is reviewed and different DL-driven IoV network levels related to network control, data analysis, and regression are discussed. Finally, in Section 7, the paper reviews the future directions and challenges related to the deployment of the ML and DL algorithms in IoV applications. And then, the paper is concluded in Section 8.


## 2. Deep Learning Evolution

The concept of the DL scheme is related to the human brain, where the DL network enables the creation of a computation model accordingly. In the sixties’ era, DL was developed and released as an algorithm to contribute to developing data processing and forecasting methods [3]. The seventies era was the beginning of the deep learning revolution which show up the ability of convolutional neural networks (CNNs), to create a computer model with learning capabilities to recognize visual patterns [4]. In the nineties’ era, the back propagation (BP) for the learning model was used in practice, and later, the deep learning approach has been developed significantly by taking the speed abilities of computers and processors. With the entry of the millennium era, what is known as artificial intelligence was launched, which is used to train neural networks [5].Currently, there is a growing need to increase the speed of processing to enable rapid data analysis by deep learning, in addition to prediction purposes [6]. By using DL, many IoT applications can be improved and contributed to enhancing different related IoT aspects such as signal analysis, pattern recognition, and quality optimization [7]. The DL structure is consisting of several parts beginning from data input, passed through feature extraction, learning, and ending by predicted output data, as shown in Figure 1. The concept of a DL model is located within the learning process using the simulation of the human brain function to recognize patterns.Figure 1
Deep leaning neural network approach.Current IoT applications depend on portable and embedded devices and sensors, in addition to services that are provided through these devices. The use of deep learning on IoT devices introduces a new type of application that has the ability of complex sensing and task recognition, in addition to linking the devices with humans interactively [8]. Different DL frameworks have been proposed. An example of such frameworks is a deep sense which consists of recurrent neural network (RNN) and convolution neural network (CNN) integrated into one model to provide a feasible solution for the problematics related to learning multisensor blending tasks. The deep sense enables to split the input neural network layer into time intervals for processing to provide time-series data estimation and classification [9].The deployment of DL in IoT as deep IoT is facing challenges due to resource constraint, which will compress the deep neural network. In the deep learning IoT framework, the structure drops the hidden elements in the neural network according to the regularization scheme of deep learning (see Figure2). The dropping process is known as dropout, which represents the probability of dropping hidden elements during the learning process [10]. The applications of DL in IoV enable to achieve an intelligent capability for vehicle systems and to provide smart services. Moreover, DL helps to improve the processes related to self-driving and automation for vehicles in the IoV network, in addition to optimizing the data streaming quality of services between IoV networks and existing systems [11].Figure 2
Deep IoT main framework.


## 3. Machine Learning-Based Systems

In the past decade, there has been an increasing need to use machine learning (ML), especially with the continuous development of various interactive communication devices. The rapid and increasing growth of data and the need to analyze it has helped in ML algorithm development for analysis and building decision-making systems. ML algorithms helped to enhance different applications such as driving automation and behavior prediction [12]. In some applications, the ML process may take a long time for learning calculations; hence, the use of distributed systems enables to enhance the parallelization processing and bandwidth to reduce the long runtime of training models [13]. However, sometimes, the centralized solution is not feasible because of the big data storing impact to the machine. To train large datasets using machine learning, algorithms have developed with the ability to perform parallel computations and distribute data, with the flexibility to handle failure [14].There is a challenge facing the design of a comprehensive system that allows machine learning distribution effectively. The challenge is due to the unique communication pattern of the algorithms [15]. As shown in Figure 3, there are two phases related to the ML to represent the training and prediction processes. The training phase is responsible for training a large amount of data by using a suitable ML algorithm based on the specific set of parameters and application purposes. After doing the process of the training phase, the ML extracts the exact output trained model; then, in the prediction phase, the predicted results come out [16].Figure 3
General machine learning overview.In distributed ML, the parallel approach takes two phases, known as the data and model, which are applied simultaneously [17]. The data-parallel phase allows dividing the data in equal for working nodes during the same algorithm operation on different datasets, while in the model parallel phase, the nodes process a copy of entire datasets [18, 19]. Distributed machine learning systems can be divided into three main categories; they are, database, general, and purpose.


### 3.1. Database Systems

In ML algorithms, the use of ordinary database management systems (DBMSs) based on SQL cannot be served. Recent studies are trying to develop a database system that a user can execute machine learning within the DBMS [20]. Such studies are related to BISMARCK and MADlib databases. In BISMARCK, the execution of ML is based on gradient descent. MADlib extends the extensions to SQL to allow users to perform machine-learning procedures built into databases [21].


### 3.2. General Systems

For distributed computing, the use of message passive-interface (MPI) framework enables to provide high performance of computations. MPI provides many primal operations, such as broadcasting and scattering, in addition to sending and receiving. Moreover, it enables to let users execute many applications for ML. However, it can be susceptible to error and labor exhaustive [22]. Another popular system in operation is Hadoop, which is an open-source system designed to execute large clusters of commodity machine workflows. Hadoop allows services such as automatic fault tolerance and simple programming, allowing users to analyze the large scale of data across many machines. However, it cannot support or provide iterative workflows, requiring the submission of a single job for every application [23].


### 3.3. Purpose-Built Systems

The purpose-built systems provide one of two options, machine learning domain-specific languages or algorithm optimization [4]. System ML enables a high-level language to provide R-like syntax programming used in data analysis, in addition to allowing built-in operators to perform matrix operations. Workflows are transformed into map-reduce jobs and rearranged to avoid multiple passes over input data. The Opti-ML provides a linear algebra-based language for scale embedded and domain specific [24]. It includes vector, matrix, and graph data types along with subdata types that allow additional optimization [25]. This is to summarise the available systems and is not intended to include all systems. Distributed machine learning is intended to allow users to conclude their desired results from massive datasets in a noticeably short time with the goal of resource optimization [26].


## 3.1. Database Systems

In ML algorithms, the use of ordinary database management systems (DBMSs) based on SQL cannot be served. Recent studies are trying to develop a database system that a user can execute machine learning within the DBMS [20]. Such studies are related to BISMARCK and MADlib databases. In BISMARCK, the execution of ML is based on gradient descent. MADlib extends the extensions to SQL to allow users to perform machine-learning procedures built into databases [21].


## 3.2. General Systems

For distributed computing, the use of message passive-interface (MPI) framework enables to provide high performance of computations. MPI provides many primal operations, such as broadcasting and scattering, in addition to sending and receiving. Moreover, it enables to let users execute many applications for ML. However, it can be susceptible to error and labor exhaustive [22]. Another popular system in operation is Hadoop, which is an open-source system designed to execute large clusters of commodity machine workflows. Hadoop allows services such as automatic fault tolerance and simple programming, allowing users to analyze the large scale of data across many machines. However, it cannot support or provide iterative workflows, requiring the submission of a single job for every application [23].


## 3.3. Purpose-Built Systems

The purpose-built systems provide one of two options, machine learning domain-specific languages or algorithm optimization [4]. System ML enables a high-level language to provide R-like syntax programming used in data analysis, in addition to allowing built-in operators to perform matrix operations. Workflows are transformed into map-reduce jobs and rearranged to avoid multiple passes over input data. The Opti-ML provides a linear algebra-based language for scale embedded and domain specific [24]. It includes vector, matrix, and graph data types along with subdata types that allow additional optimization [25]. This is to summarise the available systems and is not intended to include all systems. Distributed machine learning is intended to allow users to conclude their desired results from massive datasets in a noticeably short time with the goal of resource optimization [26].


## 4. Deep Learning Applications on Internet of Vehicles

In recent years, IoV gained attention as a revolution in intelligent transport systems (ITSs) since it provides a great service in our lives [27]. The IoV deployment is very complicated and needs special considerations for special characteristics related to high mobility and dynamic change in topology. Due to this complication, standardization in communications is very important. Figure 4 shows the architecture standardization for VANET applications by the European Telecommunications Standards Institute (ETSI) [28].Figure 4
The ETSI architecture.In the ETSI architecture, the accommodation layers are accountable for handling the VANET-associated applications such as cooperative awareness messages (CAMs), decentralized notification messages (DENMs), and local dynamic maps (LDMs) as well as the communication process. Network and transport layers are combined in one layer; in addition, two additional layers were added to represent the management and security. Moreover, ITS dedicated stack integrating the geonetworking and addressing is updated [29, 30]. Nevertheless, such architecture does not clarify the interaction handle in case of handoff and other components that include outside gadget interaction. Figure 5 shows the four Cisco layer architecture [31].Figure 5
Cisco 4-layer architecture.IoV contains different communication application scenarios such as device to device (D2D), vehicle to vehicle (V2V), vehicles to the roadside unit (V2R), vehicle to infrastructure (V2I), roadside to roadside (R2R), vehicle to everything (V2X), vehicle to sensor (V2S), and vehicles to personal devices (V2P), in addition to manyother structures [32, 33]. Much novel research has been conducted using AI approaches and DL mechanisms in IoV-based services for different IoV aspects such as security, cloud-based IoV, and collision production, which were discussed in brief in the following sections.


### 4.1. Internet of Vehicle Security

Protecting IoV from attacks became noticeably important because of the considerable IoV growth. Different studies have been proposed for the use of ML in IoV security. Most recent studies propose the combined system containing AI approaches with IoV [34]. ML and DL are the most famous algorithms that provide a secure IoV. They enable the optimization of the IoV security system for security applications and services [34]. In IoV computing, the area security assurance within the versatile edge computing environment is very important and needs mechanisms to secure the scheduling collaborative resources [35, 36].The use of deep learning methods enables the development of IoV networks decide the legitimacy of the watched information stream and to recognize potential security dangers [37]. For security purposes, blockchain in IoV enables progressing vehicular GPS situating precision, framework vigor, and security. Security thinking is concentrated on strength and security in terms of vehicular situating, data exchanging, and sharing [38]. Table 1 summarizes the most important studies related to AI approaches to use in IoV applications.Table 1
Summary of deep learning applications in secure internet of vehicles.
YearSourceSecurity approachesFeaturesAdvantagesCitations2022ElsevierTrust management in vehicular environmentsEnhances the users’ quality of experienceA comprehensive survey on trust management for IoVHbaieb et al. [39]2021ArXivCybersecurity in cognitive IoV (CIoV)Cognitive design mechanisms to avoid potential security issuesTechnical concepts about security, privacy, and trust in CIoVHasan et al. [40]2021IJSSEIntrusion prevention system (IPS) for cognitive IoV (CIoV)Malicious packet identificationHigher accuracy closed to 99.57%Praneeth et al. [41]2021HindawiSecured IoV communicationImproves IoV edge computing offloading modelReviewing the most recent AI approaches used in IoV securitySayed Ali et al. [42]2020IEEEBlockchain-enabled IoV frameworkImproves vehicular securityEfficient accuracy of securityYanxing et al. [43]2020SpringerLocation privacy protection in mobile edge computing IoVDeep reinforcement learning to secure MEC serversHighest secured computing task and effective scheduling strategyPang et al. [44]2020Computers, MDPIBlockchain base IoV design for security issuesEvaluates different AI algorithmsOut performance of IoV cybersecurity solutionsMecheva and Kakanakov [45]2019SpringerSecurity against intrusion attacksDL methods for validity and security threatsSecure observed data stream and Internet connectivityBerger et al. [46]2018IEEEDetection of cyberattacks in IoVDL and NN for different attack learningEfficient detection latency against attacksLoukas et al. [43]2018IEEESmart spectrum utilization (SSU) with IoVMachine learning for secure IoVSecure IoV applications and servicesAli et al. [47]George et al. demonstrated an application to offload an IoV system through a persistent deep learning outage detection task. The creators utilize both a profound multilayer perceptron and repetitive neural arrange engineering to learn the worldly setting of distinctive assaults. The proposed study determines when the computation can be discharged with useful considering parameters related to network operation and processing requirements of the deep learning model.Sharma et al. introduced the concept of V2X infrastructure to provide optimal and reliable communication services in smart cities. The paper touched on the possibility of using smart spectrum security systems (SSU) in IoV networks. The authors also presented a security system using deep learning that develops secure applications with high reliability. Deep learning works in the IoV system to monitor security threats. The proposed system has led to high performance in terms of monitoring accuracy and security. In a study proposed by Berger et al., the researchers presented several machine learning and deep learning methods and used them in the development of smart vehicle systems to improve the protection capabilities of the observed data flow and to identify the capabilities of potential security threats.In a study by Pang et al., the researchers demonstrated a mechanism for collaborative scheduling of computing resources for Internet of vehicles so that privacy is adopted and protected in the edge computing environment during the movement of vehicles. The researchers adopted the use of a multiuser and multiregion MEC server system to enable vehicles to offload computing tasks to MEC servers in different regions. Researchers focused on developing a solution to the problem of revealing location privacy to the vehicle user in which the dual DQN algorithm was used to solve the optimal scheduling strategy to reduce the total cost of system consumption. Through experiments, the researchers concluded that the proposed system gives optimum performance by comparison with other scheduling algorithms.In a study by Yanxing et al., the researchers presented several current collaborative positioning techniques that help improve the accuracy of vehicle positioning, where a new model of Internet of vehicles based on blockchain technology has been proposed to improve the accuracy of positioning in addition to increasing the degrees of safety. In this study, a deep neural network (DNN) algorithm was used to correct the positions of vehicles, and then, a blockchain system was used to protect the communications between smart vehicles, CoVs, and roadside units. The proposed model showed high accuracy in the identification and security of information transmission of the vehicle network.Teodora and Nikolay showed the common diagrams of the ITS design and security issues. In expansion, the creators explore inventive approaches such as blockchain, sprout channel, haze computing, manufactured insights, diversion hypothesis, and ontologies. Those creators recently tried to figure out some intelligent ITS security procedures.A study presented by Elmustafa et al. provides details about the security issues in the IoV edge computing offload model and the considerations related to the QoS requirements. The paper reviews most artificial intelligence approaches such as ML and DL algorithms and their impact on securing the IoV network. Praneeth et al. presented an intrusion prevention system (IPS) model based on the DL approach in the networks of the cognitive IoV by using binary classification that enables the identification of malicious packets. The model trains and tests packages in a cloud service for an open platform and validates the proposed prevention classifier model using a simulation dataset. Through the results, the researchers concluded that the proposed model gave an accuracy of 99.57% of the experiment compared to other smart models such as RNN and CNN.Hasan et al. presented several concepts related to the use of smart cloud computing technology and its uses in decisions to improve road transport safety. The researchers also discussed what is known as cognitive Internet of vehicles (CIoV). The research discussed several axes related to the possibility of avoiding cybersecurity problems in IoV and cognitive design mechanisms to overcome potential security, privacy, and trust concerns. A study provided by Hbaieb et al. provided a comprehensive review of IoV management resources used to enhance the users’ QoS. The review gives technical concepts related to different IoV communication models and in addition to security issues.


### 4.2. Cloud-Based Mobile IoV Framework

Deep learning studies have been converted to be a very popular technology in image and information processing, in addition to other applications where smart data manipulation is needed [48]. Mobile cloud computing gives an improvement in results related to the training process and model repository in cloud platform transferring. It enhances the computational processes to the cloud making it faster. Moreover, it provides more security in data gathering processes that are transferred to mobile devices [49]. Deep learning topics have become a powerful technology for mobile cloud frameworks. Table 2 summarizes some of the studies related to the use of AI algorithms for cloud-based IoV frameworks.Table 2
Summary of AI applications for cloud-based IoV frameworks.
WorkYearSourceFramework approachesFeaturesAdvantagesZhang et al. [50]2021SpringerResource allocation scheme based on deep RL schemeMinimum total computing cost for IoV in an edge environmentEffectively allocate the computing resources of IoVGrigorescu et al. [51]2020SensorsSelf-driving autonomous vehicles to forecast future traffic demands in IoVAI inference engines for autonomous driving applicationsMitigating privacy issues and effective path predictionClaudio et al. [52]2020IEEEML-based framework for IoV edge networkAvoid the abuse of IoV edge computing activitiesImprove the performance of control mechanisms in IoVNing et al. [53]2019IEEEDeep RL for cloud computing-based offloading framework for IoVMinimize the overall energy consumptionReduce energy consumption by 60%Razi et al. [54]2018IEEEData analytics framework for fog-based IoV architectureAnalysis based on DL and deep RL (DRL)Context-aware services in an IoV environmentChen et al. [55]2018SpringerMobile cloud computing frameworkFramework-based deep learning approachOutperform objects detection rateIn a study presented by Zhang et al., the authors presented a framework for scientific planning of allocating computing resources in edge computing for mobile IoV networks. The researchers used a deep reinforcement-learning (RL) network to improve the computing power of the service nodes and the speed of movement of the vehicle [50]. They also used Q Learning on a deep RL network to improve the stability of the neural network, where through the analysis, the researchers concluded that the proposed model gives an effective performance in managing the computing resources of IoV.Saeed et al. provided a framework for the scientific planning of computing resource allocation in the edge computing of mobile IoV. The study presented a new model to enable smart resource allocation for mobile IoV networks based on smart computing. In this study, deep RL was used to improve computing efficiency, as the presented model gave a significant improvement in the resource management process.According to what was presented by Grigorescu et al., the architecture of cloud computing systems based on deep learning is subject to cloud-level preparation and storage processes. ML empowers acknowledgment preparation and information gathering in versatile gadgets. The creators proposed such a system and communications to guarantee the victory of information transmission in unsteady arrange situations [51]. The structure identifies objects in recorded recordings amid driving, and the ML will give outflank discovery rate.The proposed system by Claudio et al. offers context-aware genuine time and group administrations at the IoV edge [52]. The study creates a common ML-based system that leverages manufactured insights to estimate future activity requests and characterize activity highlights. This system can avoid the abuse of IoV edge computing activity requests and take strides in implementing basic controls in IoV arrangement. The proposed system can coordinate ML to move forward two distinctive organized instruments.Ning et al. proposed a new offloading framework based on three layers to reduce the energy consumption in IoV. The study uses deep reinforcement learning to provide an optimal solution for offloading decisions. The framework is evaluated in real applications in China. The results show that the proposed idea is able to reduce the consumption of energy by 60% compared to other baseline algorithms. Razi et al. explored the information at the edge hubs to empower fog-based benefit. The creators emphasize the challenges included in advertising the context-aware administrations in an IoV environment [56]. The concept introduces an information analysis system for haze frameworks at the haze layer of conventional IoV engineering.For self-driving cars and independent vehicles, Chen et al. displayed a novel system for creating AI deduction motors for independent driving applications based on learning modules. The proposed system empowers to prepare assignments that are sent flexibly over both cloud and edge assets, with the reason of lessening the desired organized transmission capacity, as well as moderating security issues [55]. The system demonstrated a compelling of AI deduction motors for independent vehicles, for environment discernment, and most likely way expectation.


### 4.3. Collision Prediction

Deep learning permits computational models to memorize representations of information with numerous levels of reflection. Table3 shows the comparison between different DL applications for IoV collision prediction. In a study presented by Gupta et al., applications of deep learning for object detection and scene perception to avoid collisions in self-driving vehicles are shown in [57]. The authors discuss the gap between deep learning and self-driving cars and solutions to address image perception problems in real-time driving in autonomous driving applications.Table 3
Summary of deep learning applications for IoV collision prediction.
WorkYearSourceApproachesFeaturesAdvantagesGupta et al. [57]2021ElsevierAvoid collisions in self-driving vehiclesDL for self-driving in a real-time environmentDriving alerts to prevent accidentsSantos et al. [58]2021Computers, MPDISmart traffic accident detectionPredicts future accidentsHigh vehicle accident detection accuracy rateAadil et al. [59]2020Journal of advanced transportationComplex interaction precision in IoV systemDeep learning for traffic crash predictionFine tuning and superior traffic crash predictionsChang et al. [60]2019IEEECollusion detection in IoV system suing DL modelDeep crash system to detect collusionHigh traffic collision detection accuracy rate 96%Chunjiao et al. [61]2018HindawiTraffic accident detection in IoV systemDeep learning for traffic accident collision detection and emergency announcementHigh traffic accident detection accuracy rate 84.58%In the study by Santos et al., several factors affecting the use of accident data in the Portugal region were identified as an example, where the researchers discussed how to address and develop smart models that classify the severity of traffic accidents [58]. The researchers also presented a proposal for a model that predicts future accidents based on antecedent data by using supervised ML and DL mechanisms.In some IoV applications, irregular tunnel environments require an intelligent vehicle driving system. Fayyad et al. presented a real-time location strategy for autonomous vehicle driving on short range. The study reviews the use of the DL learning strategy to identify the vehicle locale within local vehicle environments.Chang et al. presented a profound learning-based Web of Vehicle (IoV) framework known as deep crash [58]. The proposed framework enables the identification of collusion and the transmission of unfortunate location data to the cloud-based database server to enable autonomous collision vehicle fault acknowledgment. The evaluation of the deep crash system results in high traffic collision detection accuracy up to 96%, with a 7-second average response time.In research by Chunjiao et al., the authors pretested a DL framework for the IoV system. The DL helps to detect traffic accident collisions in the IoV network and provides emergency announcements [61]. The study introduces the concept of implementing DL for driver assistance, which helps to avoid drowsiness driving accidents. The authors are benefited from the theory that shows the relationship between the feeling of sleepiness during long trips and the concentration of carbon dioxide in vehicles. The proposed approach gives up to 84.58% detection accuracy for traffic accident collisions.


## 4.1. Internet of Vehicle Security

Protecting IoV from attacks became noticeably important because of the considerable IoV growth. Different studies have been proposed for the use of ML in IoV security. Most recent studies propose the combined system containing AI approaches with IoV [34]. ML and DL are the most famous algorithms that provide a secure IoV. They enable the optimization of the IoV security system for security applications and services [34]. In IoV computing, the area security assurance within the versatile edge computing environment is very important and needs mechanisms to secure the scheduling collaborative resources [35, 36].The use of deep learning methods enables the development of IoV networks decide the legitimacy of the watched information stream and to recognize potential security dangers [37]. For security purposes, blockchain in IoV enables progressing vehicular GPS situating precision, framework vigor, and security. Security thinking is concentrated on strength and security in terms of vehicular situating, data exchanging, and sharing [38]. Table 1 summarizes the most important studies related to AI approaches to use in IoV applications.Table 1
Summary of deep learning applications in secure internet of vehicles.
YearSourceSecurity approachesFeaturesAdvantagesCitations2022ElsevierTrust management in vehicular environmentsEnhances the users’ quality of experienceA comprehensive survey on trust management for IoVHbaieb et al. [39]2021ArXivCybersecurity in cognitive IoV (CIoV)Cognitive design mechanisms to avoid potential security issuesTechnical concepts about security, privacy, and trust in CIoVHasan et al. [40]2021IJSSEIntrusion prevention system (IPS) for cognitive IoV (CIoV)Malicious packet identificationHigher accuracy closed to 99.57%Praneeth et al. [41]2021HindawiSecured IoV communicationImproves IoV edge computing offloading modelReviewing the most recent AI approaches used in IoV securitySayed Ali et al. [42]2020IEEEBlockchain-enabled IoV frameworkImproves vehicular securityEfficient accuracy of securityYanxing et al. [43]2020SpringerLocation privacy protection in mobile edge computing IoVDeep reinforcement learning to secure MEC serversHighest secured computing task and effective scheduling strategyPang et al. [44]2020Computers, MDPIBlockchain base IoV design for security issuesEvaluates different AI algorithmsOut performance of IoV cybersecurity solutionsMecheva and Kakanakov [45]2019SpringerSecurity against intrusion attacksDL methods for validity and security threatsSecure observed data stream and Internet connectivityBerger et al. [46]2018IEEEDetection of cyberattacks in IoVDL and NN for different attack learningEfficient detection latency against attacksLoukas et al. [43]2018IEEESmart spectrum utilization (SSU) with IoVMachine learning for secure IoVSecure IoV applications and servicesAli et al. [47]George et al. demonstrated an application to offload an IoV system through a persistent deep learning outage detection task. The creators utilize both a profound multilayer perceptron and repetitive neural arrange engineering to learn the worldly setting of distinctive assaults. The proposed study determines when the computation can be discharged with useful considering parameters related to network operation and processing requirements of the deep learning model.Sharma et al. introduced the concept of V2X infrastructure to provide optimal and reliable communication services in smart cities. The paper touched on the possibility of using smart spectrum security systems (SSU) in IoV networks. The authors also presented a security system using deep learning that develops secure applications with high reliability. Deep learning works in the IoV system to monitor security threats. The proposed system has led to high performance in terms of monitoring accuracy and security. In a study proposed by Berger et al., the researchers presented several machine learning and deep learning methods and used them in the development of smart vehicle systems to improve the protection capabilities of the observed data flow and to identify the capabilities of potential security threats.In a study by Pang et al., the researchers demonstrated a mechanism for collaborative scheduling of computing resources for Internet of vehicles so that privacy is adopted and protected in the edge computing environment during the movement of vehicles. The researchers adopted the use of a multiuser and multiregion MEC server system to enable vehicles to offload computing tasks to MEC servers in different regions. Researchers focused on developing a solution to the problem of revealing location privacy to the vehicle user in which the dual DQN algorithm was used to solve the optimal scheduling strategy to reduce the total cost of system consumption. Through experiments, the researchers concluded that the proposed system gives optimum performance by comparison with other scheduling algorithms.In a study by Yanxing et al., the researchers presented several current collaborative positioning techniques that help improve the accuracy of vehicle positioning, where a new model of Internet of vehicles based on blockchain technology has been proposed to improve the accuracy of positioning in addition to increasing the degrees of safety. In this study, a deep neural network (DNN) algorithm was used to correct the positions of vehicles, and then, a blockchain system was used to protect the communications between smart vehicles, CoVs, and roadside units. The proposed model showed high accuracy in the identification and security of information transmission of the vehicle network.Teodora and Nikolay showed the common diagrams of the ITS design and security issues. In expansion, the creators explore inventive approaches such as blockchain, sprout channel, haze computing, manufactured insights, diversion hypothesis, and ontologies. Those creators recently tried to figure out some intelligent ITS security procedures.A study presented by Elmustafa et al. provides details about the security issues in the IoV edge computing offload model and the considerations related to the QoS requirements. The paper reviews most artificial intelligence approaches such as ML and DL algorithms and their impact on securing the IoV network. Praneeth et al. presented an intrusion prevention system (IPS) model based on the DL approach in the networks of the cognitive IoV by using binary classification that enables the identification of malicious packets. The model trains and tests packages in a cloud service for an open platform and validates the proposed prevention classifier model using a simulation dataset. Through the results, the researchers concluded that the proposed model gave an accuracy of 99.57% of the experiment compared to other smart models such as RNN and CNN.Hasan et al. presented several concepts related to the use of smart cloud computing technology and its uses in decisions to improve road transport safety. The researchers also discussed what is known as cognitive Internet of vehicles (CIoV). The research discussed several axes related to the possibility of avoiding cybersecurity problems in IoV and cognitive design mechanisms to overcome potential security, privacy, and trust concerns. A study provided by Hbaieb et al. provided a comprehensive review of IoV management resources used to enhance the users’ QoS. The review gives technical concepts related to different IoV communication models and in addition to security issues.


## 4.2. Cloud-Based Mobile IoV Framework

Deep learning studies have been converted to be a very popular technology in image and information processing, in addition to other applications where smart data manipulation is needed [48]. Mobile cloud computing gives an improvement in results related to the training process and model repository in cloud platform transferring. It enhances the computational processes to the cloud making it faster. Moreover, it provides more security in data gathering processes that are transferred to mobile devices [49]. Deep learning topics have become a powerful technology for mobile cloud frameworks. Table 2 summarizes some of the studies related to the use of AI algorithms for cloud-based IoV frameworks.Table 2
Summary of AI applications for cloud-based IoV frameworks.
WorkYearSourceFramework approachesFeaturesAdvantagesZhang et al. [50]2021SpringerResource allocation scheme based on deep RL schemeMinimum total computing cost for IoV in an edge environmentEffectively allocate the computing resources of IoVGrigorescu et al. [51]2020SensorsSelf-driving autonomous vehicles to forecast future traffic demands in IoVAI inference engines for autonomous driving applicationsMitigating privacy issues and effective path predictionClaudio et al. [52]2020IEEEML-based framework for IoV edge networkAvoid the abuse of IoV edge computing activitiesImprove the performance of control mechanisms in IoVNing et al. [53]2019IEEEDeep RL for cloud computing-based offloading framework for IoVMinimize the overall energy consumptionReduce energy consumption by 60%Razi et al. [54]2018IEEEData analytics framework for fog-based IoV architectureAnalysis based on DL and deep RL (DRL)Context-aware services in an IoV environmentChen et al. [55]2018SpringerMobile cloud computing frameworkFramework-based deep learning approachOutperform objects detection rateIn a study presented by Zhang et al., the authors presented a framework for scientific planning of allocating computing resources in edge computing for mobile IoV networks. The researchers used a deep reinforcement-learning (RL) network to improve the computing power of the service nodes and the speed of movement of the vehicle [50]. They also used Q Learning on a deep RL network to improve the stability of the neural network, where through the analysis, the researchers concluded that the proposed model gives an effective performance in managing the computing resources of IoV.Saeed et al. provided a framework for the scientific planning of computing resource allocation in the edge computing of mobile IoV. The study presented a new model to enable smart resource allocation for mobile IoV networks based on smart computing. In this study, deep RL was used to improve computing efficiency, as the presented model gave a significant improvement in the resource management process.According to what was presented by Grigorescu et al., the architecture of cloud computing systems based on deep learning is subject to cloud-level preparation and storage processes. ML empowers acknowledgment preparation and information gathering in versatile gadgets. The creators proposed such a system and communications to guarantee the victory of information transmission in unsteady arrange situations [51]. The structure identifies objects in recorded recordings amid driving, and the ML will give outflank discovery rate.The proposed system by Claudio et al. offers context-aware genuine time and group administrations at the IoV edge [52]. The study creates a common ML-based system that leverages manufactured insights to estimate future activity requests and characterize activity highlights. This system can avoid the abuse of IoV edge computing activity requests and take strides in implementing basic controls in IoV arrangement. The proposed system can coordinate ML to move forward two distinctive organized instruments.Ning et al. proposed a new offloading framework based on three layers to reduce the energy consumption in IoV. The study uses deep reinforcement learning to provide an optimal solution for offloading decisions. The framework is evaluated in real applications in China. The results show that the proposed idea is able to reduce the consumption of energy by 60% compared to other baseline algorithms. Razi et al. explored the information at the edge hubs to empower fog-based benefit. The creators emphasize the challenges included in advertising the context-aware administrations in an IoV environment [56]. The concept introduces an information analysis system for haze frameworks at the haze layer of conventional IoV engineering.For self-driving cars and independent vehicles, Chen et al. displayed a novel system for creating AI deduction motors for independent driving applications based on learning modules. The proposed system empowers to prepare assignments that are sent flexibly over both cloud and edge assets, with the reason of lessening the desired organized transmission capacity, as well as moderating security issues [55]. The system demonstrated a compelling of AI deduction motors for independent vehicles, for environment discernment, and most likely way expectation.


## 4.3. Collision Prediction

Deep learning permits computational models to memorize representations of information with numerous levels of reflection. Table3 shows the comparison between different DL applications for IoV collision prediction. In a study presented by Gupta et al., applications of deep learning for object detection and scene perception to avoid collisions in self-driving vehicles are shown in [57]. The authors discuss the gap between deep learning and self-driving cars and solutions to address image perception problems in real-time driving in autonomous driving applications.Table 3
Summary of deep learning applications for IoV collision prediction.
WorkYearSourceApproachesFeaturesAdvantagesGupta et al. [57]2021ElsevierAvoid collisions in self-driving vehiclesDL for self-driving in a real-time environmentDriving alerts to prevent accidentsSantos et al. [58]2021Computers, MPDISmart traffic accident detectionPredicts future accidentsHigh vehicle accident detection accuracy rateAadil et al. [59]2020Journal of advanced transportationComplex interaction precision in IoV systemDeep learning for traffic crash predictionFine tuning and superior traffic crash predictionsChang et al. [60]2019IEEECollusion detection in IoV system suing DL modelDeep crash system to detect collusionHigh traffic collision detection accuracy rate 96%Chunjiao et al. [61]2018HindawiTraffic accident detection in IoV systemDeep learning for traffic accident collision detection and emergency announcementHigh traffic accident detection accuracy rate 84.58%In the study by Santos et al., several factors affecting the use of accident data in the Portugal region were identified as an example, where the researchers discussed how to address and develop smart models that classify the severity of traffic accidents [58]. The researchers also presented a proposal for a model that predicts future accidents based on antecedent data by using supervised ML and DL mechanisms.In some IoV applications, irregular tunnel environments require an intelligent vehicle driving system. Fayyad et al. presented a real-time location strategy for autonomous vehicle driving on short range. The study reviews the use of the DL learning strategy to identify the vehicle locale within local vehicle environments.Chang et al. presented a profound learning-based Web of Vehicle (IoV) framework known as deep crash [58]. The proposed framework enables the identification of collusion and the transmission of unfortunate location data to the cloud-based database server to enable autonomous collision vehicle fault acknowledgment. The evaluation of the deep crash system results in high traffic collision detection accuracy up to 96%, with a 7-second average response time.In research by Chunjiao et al., the authors pretested a DL framework for the IoV system. The DL helps to detect traffic accident collisions in the IoV network and provides emergency announcements [61]. The study introduces the concept of implementing DL for driver assistance, which helps to avoid drowsiness driving accidents. The authors are benefited from the theory that shows the relationship between the feeling of sleepiness during long trips and the concentration of carbon dioxide in vehicles. The proposed approach gives up to 84.58% detection accuracy for traffic accident collisions.


## 5. Analysis of IoV Network Based on Deep Reinforcement Learning

In general, accidents happen by driver’s miscalculation in driving and mistakes. ML in IoVs provides solutions to raise the level of the driving experience as one of the most important requirements for smart city applications. The technology of IoV is continuously developing technologies and architectures to reduce traffic congestions and accidents transforming vehicles from regular driving tools to intelligent tools [60]. Reinforcement learning provides an architecture where a network learns from previous experience to increase some reward signal. In learning to control, an agent directly receives high-dimensional input such as voice and video in an extremely monotonous task known as the curse of dimensions. Many suggested solutions have been proposed for this dimension’s issue such as the use of linear function approximation, hierarchical representation, and state aggregation [62].For decision making of multistage process in a probabilistic environment, the control process must be accurate. The use of the Markov decision process (MDP) enables the provision of descriptive formal standard methodology as a discrete-time stochastic control process. Accordingly, the process is in a statex at each time step so that it allows a decision maker to move the process to the new state x′ to give a corresponding decision to reward rx,a,x′. The possibility of transition of the process to the new state x′ is determined by the given act according to the state transition function Tx′|x,a to fulfil the Markov property.During the MDP process, a problem of finding decision maker policy may appear because of the total independence of past states concerning the current statex and next state x′ [31]. By defining the function π for a specific action πx to the decision maker for choosing when the state is x, we can derive the purpose of MDP that has to find a policy for π to maximize the cumulative function by the following equation [63]:(1)R=∑n=0∞γnrxn,an,where n represents the time step. γ Denotes discount factor between 0 and 1. By setting the γ factor, the process of the decision can be specified. In the case of γ  = 0, it will make the agent short sighted, and if the γ value is 1, this makes it look to the future high reward.The operation of classic MDP enables to enhance the decision through two approaches, the value iteration or policy iteration. However, this enhancement takes place according to the assumptions that the decision maker has an accurate value for the transition function and reward for all states in the application environment [64]. While in the actual operation, the decision maker cannot recognize the transition function. Accordingly, the use of Q learning helps to overcome such problems.Q-learning (QL) is a form of RL that enables to teach the decision maker how to behave in an MDP environment when the transition function and/or reward are unknown [65]. In this approach, each state is assigned as an initial Q value, which can be estimated by an online incremental update stochastic QL algorithm, as shown in the following equation:(2)Qxn,an≔Qxn,an+anrxn,an+γan+1maxQxn+1,an+1−Qxn,an,where rxn,an represents the single-step reward. an denotes step-size learning rate between 0 and 1. In the case of an=0, no Q values are updated, and no learning process is occurred [66].The previous approach is known as the value-iteration algorithm which fulfils the optimal action-value functionQxn,an⟶Q∗xn,an as n⟶∞. Dimensional curse in the classical QL algorithm is occurred due to a large amount of time to converge. To overcome this issue in MDPs, functional approximation techniques can be used.The use of neural networks helps to find good features for high-dimensional input data. It can be represented within the value of action function by considering the current system state and action in input to obtain suitable Q-value output. This approach is representing the concept of deep RL. The QL with weightθ is trained to learn the θ parameter of the action function Qxn,an,θ by reducing the sequence of the loss function Liθi. The Liθi is stated in the following equation [67]:(3)Liθi=Ern+maxan+1Qxn+1,an+1,θi−1−Qxn,an,θ2,where θi is the neural network parameter. The part rn+maxan+1Qxn+1,an+1,θi−1 is the target for iteration i.The goal is to save the cost expression as small as possible by using a gradient descent algorithm, which repeats the computation of the gradient∇θiLiθi expressed in the following equation [68]:(4)∇θiLiθi=Ern+maxan+1Qxn+1,an+1,θi−1−Qxn,an,θ∇θiQxn,an,θ.The gradient descent algorithm becomes slow when processing huge datasets; however, the use of stochastic gradient descent (SGD) will help to reduce the problem of slow processing [69]. The following sections show the impact of the RL algorithms in IoV networks related to energy efficiency and overall performance optimization.


### 5.1. Scheduling Algorithms for Energy Efficiency

The ITS applications based on vehicular communications and IoV may consist of different vehicular networking architectures such as V2V or V2I which are possibly taking information exchange by the roadside units (RSUs). The use of the V2I communication structure enables the provision of a reliable and efficient driving experience and safety for vehicular communications [70]. The data exchanged in such a structure are scheduled between vehicles and RSUs with a policy to enable V2I communication management to preserve the batteries in the RSUs and prolong the lifetime of the network. Accordingly, the V2I communication will ensure better quality-of-service (QoS) levels [71]. The deployment of RSUs depends on the battery recharge in time intervals using energy harvesting via solar or by physically recharging.Deep reinforcement learning (DRL) based-scheduling scheme enables to manage and organize the communications in RSUs, and to enhance the lifetime of IoV as well. Scheduling DRL approach is similar to action value function (AVF) and Q-learning algorithms in terms of simplicity and robustness. The use of a scheduling policy framework helps to optimize the energy efficiency in IoV, and accordingly, the task scheduling system, as shown in Figure6, enables to provide an energy-efficient overall system that increases the lifetime of the network [72].Figure 6
Network lifetime evaluation based on deep reinforcement agent scheduling algorithms.In Figure6, the task scheduling is defined according to the time until the IoT gateway is cutoff considering ρ vehicle speed. As observed from the figure, a deep Q network (DQN) enables the balance of the power consumption in IoV networks [73]. Compared with other algorithms, DQN gives better performance. The performance of the random vehicle selection algorithm and prioritizing departing vehicles (PDVs) is constant for all considered ρ vehicle speed. The two other algorithms greedy power conservation (GPC) algorithm and greedy prioritize departing vehicle (GPDV) algorithm give an average network lifetime and keep a convergent performance for all ρ vehicle speed. In general, the task decorrelation is the stage responsible for the scheduling policy to organize tasks and reduce the power consumption [47].


### 5.2. RL Algorithms for Resource Management

The provisions quality of service (QoS) and quality of experience (QoE) are the most urgent metrics for vehicle users, which are required to maintain at acceptable levels. Thus, many resource management algorithms have anticipated for IoV communications to improve the network performance. Software-defined network-based resource management algorithms were introduced to ensure vehicle users’ QoS [74]. There is a need to provide highly efficient and reliable communications such as device-to-device communication using V2V with the possibility of improvements by reducing transmission delay and power consumption while providing local packet distribution, improving spectrum efficiency, and reducing gain characteristics [75].An efficient transfer AC (ETAC) learning method enables the achievement of an intelligent resources’ allocation in the IoV network. Intelligent resource management is important to empower the V2X communication with intelligent decision making and satisfy multiple QoS requirements such as latency and reliability requirements for V2V communication, in addition to ensuring suitable data rate requirements for V2I channels. RL, as a solution that enables the adoption of the Markov decision process and with the actor-critic (AC) framework, will provide an intelligent resource allocation mechanism. This mechanism enables a new Markov reward function to allocate smart resources in the IoV network mentioned in the following equations [44]:(5)r1=c1∑k∈KRkc+∑m∈MnorRmd,nor,f1=−c2∑m∈Munipmdelay+pmoutage,f2=−c3∑k∈Kkc,tar−Rkc+∑m∈MnorRmd,nor,tar−Rmd,nor,where r1 represents the sum data rate, f1 and f2 pare instant cost function. The coefficients ci,i∈1,2,3 are the weights of r1, f1, and f2 and are used to balance the utility and cost. Muni represents the D2D-V2V pair sets. In the IoV networks, each agent chooses a policy π to capitalize the reward Qπs,a representing that the state-action function is a summed discount return for starting the network s with an assumed policy π stated in the following equation [46]:(6)Qπs,a=E∑t=1∞γtrtst,at|s0=s,π.Intelligent resource management aims to find the policyπ best describe the maximization of network objective reward, stated in the following equation:(7)Jπ=EQπs,a=∫Sds∫Aπs,aQπb,adads,where ds represents the state spreading function, and πs,a represents the stochastic policy with the state s with completed the current action a. It reveals the conditional probability density of the action (a) at the state (s). Given the above situation, the policy improved numerically by applying the value iteration methods with the RL algorithms [76].The probability of satisfied V2I and V2V be increased by the contribution of ETAC and obtains absolut speed of the vehicle and considering the variation of device vehicle users within D2D-V2V communications. ETAC outperforms other RL algorithms, as shown in Figure7.Figure 7
ETAC agent performance in the V2V-D2D communications.The AC agent might use to apply intelligent resource management to satisfy the user of the vehicle QoS. The critic process: the aim of the critic process of the AC learning agent is to assess the policy quality that the learning system is supposed to search for [45]. The player process: the stochastic-policy-gradient (PG) technique usually employs the player part to parameterize policies to improve the policy thoroughly to enhance the goal function in the following equations:(8)∇θJπθ=EQπs,a=∫Sds∫Aπθs,aQπθb,adads,πθs,a=expθT.Φs,a∑a′∈AexpθT.Φs,a′,where Φs,a is the future vector. Next, the policy parameter vector is updated based on the gradient of the objective reward given in the following equation [43]:(9)θt+=θt+βa∇θJπθ,where βa is actor-learning rate.ETAC enables the vehicle users to import learned strategies from experience VUE and gives an independent learning process. The employment of the parameter vectorν is rationalized in the critic part. The policy θ is rationalized by the actor [77].The problems with the standard AC approach can be addressed by pursuing qualifications to improve the efficiency of the learning setting. The feature of coordinating the work with the inaccurate parameter reduces the fluctuation with angle. The activity methodology helps to enhance the overall quality of learning by increasing the speed of convergence [56, 78]. Performance evaluation of the ETAC is shown in Figure 8. By observing the varying number of D-VUEs, it is clear that ETAC gives better results compared to other reinforcement learning approaches with an improved sum rate.Figure 8
ETAC performance related to communication link sum rate within the D2D-V2V communication network.


### 5.3. Performance Optimization Algorithms

In the IoV network structure, a huge amount of data is exchanged among different IoV vehicles, RSUs, and infrastructure. The data should be stored and interchanged in a secured environment to enhance traffic safety and efficiency. Two types of operations are considered in IoV network data storage and sharing. A blockchain system with reinforcement learning will help to optimize the performance of the IoV network [79].The blockchain framework consists of two main parts. The first part, block producers, is related to the number of node assumptions, block producers, candidates block, a set of nodes, and computational resources. The second part, consensus models, applies the practical byzantine fault tolerant (PBFT) algorithm for consensus considered as a very robust protocol [61, 77, 80]. The client issues a block with the transaction number stored or shared and validated by broadcasting it to other validators to have consensus. The process involves exchanging and verifying messages, as shown in Figure 9. The time-varying transmission links are modeled through the finite-state Markov chain (FSMC) [80].Figure 9
PBFT performance in terms of latency and average computational resources.Among different schemes applying blockchain systems, the average latency/time to finality (TTF) decreases with the increase of average computational resources because an agreement can become more rapidly among more computationally competent validators [82].


## 5.1. Scheduling Algorithms for Energy Efficiency

The ITS applications based on vehicular communications and IoV may consist of different vehicular networking architectures such as V2V or V2I which are possibly taking information exchange by the roadside units (RSUs). The use of the V2I communication structure enables the provision of a reliable and efficient driving experience and safety for vehicular communications [70]. The data exchanged in such a structure are scheduled between vehicles and RSUs with a policy to enable V2I communication management to preserve the batteries in the RSUs and prolong the lifetime of the network. Accordingly, the V2I communication will ensure better quality-of-service (QoS) levels [71]. The deployment of RSUs depends on the battery recharge in time intervals using energy harvesting via solar or by physically recharging.Deep reinforcement learning (DRL) based-scheduling scheme enables to manage and organize the communications in RSUs, and to enhance the lifetime of IoV as well. Scheduling DRL approach is similar to action value function (AVF) and Q-learning algorithms in terms of simplicity and robustness. The use of a scheduling policy framework helps to optimize the energy efficiency in IoV, and accordingly, the task scheduling system, as shown in Figure6, enables to provide an energy-efficient overall system that increases the lifetime of the network [72].Figure 6
Network lifetime evaluation based on deep reinforcement agent scheduling algorithms.In Figure6, the task scheduling is defined according to the time until the IoT gateway is cutoff considering ρ vehicle speed. As observed from the figure, a deep Q network (DQN) enables the balance of the power consumption in IoV networks [73]. Compared with other algorithms, DQN gives better performance. The performance of the random vehicle selection algorithm and prioritizing departing vehicles (PDVs) is constant for all considered ρ vehicle speed. The two other algorithms greedy power conservation (GPC) algorithm and greedy prioritize departing vehicle (GPDV) algorithm give an average network lifetime and keep a convergent performance for all ρ vehicle speed. In general, the task decorrelation is the stage responsible for the scheduling policy to organize tasks and reduce the power consumption [47].


## 5.2. RL Algorithms for Resource Management

The provisions quality of service (QoS) and quality of experience (QoE) are the most urgent metrics for vehicle users, which are required to maintain at acceptable levels. Thus, many resource management algorithms have anticipated for IoV communications to improve the network performance. Software-defined network-based resource management algorithms were introduced to ensure vehicle users’ QoS [74]. There is a need to provide highly efficient and reliable communications such as device-to-device communication using V2V with the possibility of improvements by reducing transmission delay and power consumption while providing local packet distribution, improving spectrum efficiency, and reducing gain characteristics [75].An efficient transfer AC (ETAC) learning method enables the achievement of an intelligent resources’ allocation in the IoV network. Intelligent resource management is important to empower the V2X communication with intelligent decision making and satisfy multiple QoS requirements such as latency and reliability requirements for V2V communication, in addition to ensuring suitable data rate requirements for V2I channels. RL, as a solution that enables the adoption of the Markov decision process and with the actor-critic (AC) framework, will provide an intelligent resource allocation mechanism. This mechanism enables a new Markov reward function to allocate smart resources in the IoV network mentioned in the following equations [44]:(5)r1=c1∑k∈KRkc+∑m∈MnorRmd,nor,f1=−c2∑m∈Munipmdelay+pmoutage,f2=−c3∑k∈Kkc,tar−Rkc+∑m∈MnorRmd,nor,tar−Rmd,nor,where r1 represents the sum data rate, f1 and f2 pare instant cost function. The coefficients ci,i∈1,2,3 are the weights of r1, f1, and f2 and are used to balance the utility and cost. Muni represents the D2D-V2V pair sets. In the IoV networks, each agent chooses a policy π to capitalize the reward Qπs,a representing that the state-action function is a summed discount return for starting the network s with an assumed policy π stated in the following equation [46]:(6)Qπs,a=E∑t=1∞γtrtst,at|s0=s,π.Intelligent resource management aims to find the policyπ best describe the maximization of network objective reward, stated in the following equation:(7)Jπ=EQπs,a=∫Sds∫Aπs,aQπb,adads,where ds represents the state spreading function, and πs,a represents the stochastic policy with the state s with completed the current action a. It reveals the conditional probability density of the action (a) at the state (s). Given the above situation, the policy improved numerically by applying the value iteration methods with the RL algorithms [76].The probability of satisfied V2I and V2V be increased by the contribution of ETAC and obtains absolut speed of the vehicle and considering the variation of device vehicle users within D2D-V2V communications. ETAC outperforms other RL algorithms, as shown in Figure7.Figure 7
ETAC agent performance in the V2V-D2D communications.The AC agent might use to apply intelligent resource management to satisfy the user of the vehicle QoS. The critic process: the aim of the critic process of the AC learning agent is to assess the policy quality that the learning system is supposed to search for [45]. The player process: the stochastic-policy-gradient (PG) technique usually employs the player part to parameterize policies to improve the policy thoroughly to enhance the goal function in the following equations:(8)∇θJπθ=EQπs,a=∫Sds∫Aπθs,aQπθb,adads,πθs,a=expθT.Φs,a∑a′∈AexpθT.Φs,a′,where Φs,a is the future vector. Next, the policy parameter vector is updated based on the gradient of the objective reward given in the following equation [43]:(9)θt+=θt+βa∇θJπθ,where βa is actor-learning rate.ETAC enables the vehicle users to import learned strategies from experience VUE and gives an independent learning process. The employment of the parameter vectorν is rationalized in the critic part. The policy θ is rationalized by the actor [77].The problems with the standard AC approach can be addressed by pursuing qualifications to improve the efficiency of the learning setting. The feature of coordinating the work with the inaccurate parameter reduces the fluctuation with angle. The activity methodology helps to enhance the overall quality of learning by increasing the speed of convergence [56, 78]. Performance evaluation of the ETAC is shown in Figure 8. By observing the varying number of D-VUEs, it is clear that ETAC gives better results compared to other reinforcement learning approaches with an improved sum rate.Figure 8
ETAC performance related to communication link sum rate within the D2D-V2V communication network.


## 5.3. Performance Optimization Algorithms

In the IoV network structure, a huge amount of data is exchanged among different IoV vehicles, RSUs, and infrastructure. The data should be stored and interchanged in a secured environment to enhance traffic safety and efficiency. Two types of operations are considered in IoV network data storage and sharing. A blockchain system with reinforcement learning will help to optimize the performance of the IoV network [79].The blockchain framework consists of two main parts. The first part, block producers, is related to the number of node assumptions, block producers, candidates block, a set of nodes, and computational resources. The second part, consensus models, applies the practical byzantine fault tolerant (PBFT) algorithm for consensus considered as a very robust protocol [61, 77, 80]. The client issues a block with the transaction number stored or shared and validated by broadcasting it to other validators to have consensus. The process involves exchanging and verifying messages, as shown in Figure 9. The time-varying transmission links are modeled through the finite-state Markov chain (FSMC) [80].Figure 9
PBFT performance in terms of latency and average computational resources.Among different schemes applying blockchain systems, the average latency/time to finality (TTF) decreases with the increase of average computational resources because an agreement can become more rapidly among more computationally competent validators [82].


## 6. Deep Learning-Based IoV Network Performance

IoV networks that provide high mobility broadband access have gained much attention from industry and scientists. Due to the increasing number of traffic accidents, there is an urgent need for faster and smarter analytical systems; thus, many scientists provide much research in this field, employing deep learning techniques for its proven results and high performance in areas of collision detection, analysis, and notification [83]. Like the OSI model, the IoV architecture consists of multiple layers such as network and application layers, which depend on the intelligent transportation system (ITS).


### 6.1. Deep Learning-Based IoV Network-Level Data Analysis

DL approach enables to the development of a multilayer architecture that includes intelligent computing and big data analytics with IoV dimensions. IoV dimension is divided into layers to support the operations related to data analysis [84]. These layers are stated as follows:(i)
Perception layer consists of most IoV devices such as sensors, actuators, vehicles, and smart mobiles(ii)
Infrastructure network layer contains the main IoV infrastructure components such as RSUs and base stations, which are considered as the backbone of the IoV network(iii)
Artificial intelligence layer enables the procedure related to computational intelligent algorithms and architectures(iv)
Communication layer provides the required communication technologies such as 5G and 4 G/LTE [85].Various DL techniques are used for data analysis image classification. Deep learning proposing inception v3 network architecture is adopted for binary image classification to achieve the best image classification results. [35]. Inception v3 employs two deep learning network architectures, densely connected convolutional networks (DensNet) which increase the depth of high-dimensional neural networks. In addition, squeeze-and-excitation networks (SENet) are responsible for filtering the last output feature to cancel or remove features that are not needed for the current task [85].


### 6.2. Deep Learning-Driven App-Level IoV Data Analysis

The function of the application layer is to provide data and control the APIs to allow applications to use the distributed data store and to determine the configuration of the deployment. This layer is responsible for supplying the IoV network with applications through the common communication architectures in IoV [87]. Real-time data gathering and analysis are required in IoV networks to ensure immediate and appropriate actions; for example, an ambulance needs real-time data on traffic information to avoid traffic jams and save lives.Maintaining the QoS requirements is crucial in the IoV applications to improve the operation of IoV services. At the application layer, the web services in the service layer provide applications by subscribing data deployment strategies [88]. Such strategies are published/subscribe strategy access, which lets applications register their interest to notify about the service updates. In general, the applications can be categorized into three main sets, real time (RT), batch information required-based application, and near real time (NRT).The strategy of push and publish is considered the best choice for RT and batch applications. While for the near real time (NRT), pull or subscribe is the best fit. The mentioned strategies used with the DL approach enable to enhance the adaptive-traffic message-scheduling application. Historic data is required by the batch data requirement; thus, it has a low QoS requirement in terms of latency [89]. Process offloading and safety applications as real-time QoS requirement are needed for real-time applications; these applications help in complex data processing such as video processing on the IoV network.


### 6.3. Deep Learning-Driven IoV Network Control

In 5G-based IoV communication, the networks consist of both wired and wireless networks that needs an intelligent method to deal with the huge growth in the data traffic of the network for their varying nature of network sceneries such as cloud computing, mobility, and big data manipulation [90]. To make IoV networks smarter, DL enables to learn how to route network traffic between routers and optimize the network performance. DL has shown promising results in network control, by dividing the framework into phases. Phase 1 is the initial phase, where the related data are obtained for training the deep system. Phase 2 is the training phase that depends on collected data by the previous phase, applying supervised learning algorithms to train the DL-based architecture. Phase 3 represents the running phase where we put the learned algorithm into action where traffic patterns are used as an input that outputs a new path or route after learning the network behaviour based on deep learning algorithms [91].The virtualization approach is a new advancement to the IoV network to have more control over the network resources and to elevate the network performance. The sole source input/output virtual (SS-IOV) environment is proposed to allow I/O devices utilized by VMs with no decrement in the run time throughput of the overall network. SR-IOV is capable of creating virtual functions that allow guests to have direct access [35]. It is an efficient network device that provides the benefit of input/output performance and decreases central processing unit (CPU) employment though significant scalability increment and the sharing abilities of the device [92].


### 6.4. Predictive Analytics “Regression” Problem in Deep Learning

ML uses supervised learning in having an input variablex and output variable y using a learning algorithm in between to study the input/output function of mapping y=fx. The ML process simply having inputs and the output is predicted after the learning process is completed.. Approaches to supervised machine learning include linear and logistic regression, classification, decision tree, and support vector machine [93]. Regression states the operation of modelling the association among one or more autonomous variables and a dependent variable; its main objective is prediction. Problems usually occur when regression is supposed to predict numerical values such as prices, the number of vehicles, predicting demand forecasting, and the length of waiting at a queue.


#### 6.4.1. Simplified Dependent and Independent Variable Regressions

These models give conventions about the average functions expressed byEy|u and the variance function, with the assumption that for some transformation, u=ux can be written as(10)Ey|u=β0+β1u.Stating the relation betweeny and u, simple linear regression is operating with only one predictor.


#### 6.4.2. Multiple Regression

Problems of regression that occurred have triggered the need to consider operating with more than one predictor, having the dependent variabley that depends simultaneously on predictors x1, x2, …, xn. We begin with n predictors x1, …, xn building a set of k terms based on the predictors ui-1, ui, …, uk-i to have the multiple linear regression mean function [93].(11)Ey|ui,…,uk−i=βi−1ui−1+βiui+⋯+βk−iuk−i.DL architectures enable to solve the regression problem, proven for its demonstrated execution in the vision of computer errands such as picture categorization, question discovery, and picture examination [95]. The common engineering comprises several convolutional layers, taken after a few fully connected layers and a classification delicate max layer [96, 97]. Deep regression algorithms have given results in traditional vision relapse issues such as human posture estimation and a facial point of interest discovery.


## 6.1. Deep Learning-Based IoV Network-Level Data Analysis

DL approach enables to the development of a multilayer architecture that includes intelligent computing and big data analytics with IoV dimensions. IoV dimension is divided into layers to support the operations related to data analysis [84]. These layers are stated as follows:(i)
Perception layer consists of most IoV devices such as sensors, actuators, vehicles, and smart mobiles(ii)
Infrastructure network layer contains the main IoV infrastructure components such as RSUs and base stations, which are considered as the backbone of the IoV network(iii)
Artificial intelligence layer enables the procedure related to computational intelligent algorithms and architectures(iv)
Communication layer provides the required communication technologies such as 5G and 4 G/LTE [85].Various DL techniques are used for data analysis image classification. Deep learning proposing inception v3 network architecture is adopted for binary image classification to achieve the best image classification results. [35]. Inception v3 employs two deep learning network architectures, densely connected convolutional networks (DensNet) which increase the depth of high-dimensional neural networks. In addition, squeeze-and-excitation networks (SENet) are responsible for filtering the last output feature to cancel or remove features that are not needed for the current task [85].


## 6.2. Deep Learning-Driven App-Level IoV Data Analysis

The function of the application layer is to provide data and control the APIs to allow applications to use the distributed data store and to determine the configuration of the deployment. This layer is responsible for supplying the IoV network with applications through the common communication architectures in IoV [87]. Real-time data gathering and analysis are required in IoV networks to ensure immediate and appropriate actions; for example, an ambulance needs real-time data on traffic information to avoid traffic jams and save lives.Maintaining the QoS requirements is crucial in the IoV applications to improve the operation of IoV services. At the application layer, the web services in the service layer provide applications by subscribing data deployment strategies [88]. Such strategies are published/subscribe strategy access, which lets applications register their interest to notify about the service updates. In general, the applications can be categorized into three main sets, real time (RT), batch information required-based application, and near real time (NRT).The strategy of push and publish is considered the best choice for RT and batch applications. While for the near real time (NRT), pull or subscribe is the best fit. The mentioned strategies used with the DL approach enable to enhance the adaptive-traffic message-scheduling application. Historic data is required by the batch data requirement; thus, it has a low QoS requirement in terms of latency [89]. Process offloading and safety applications as real-time QoS requirement are needed for real-time applications; these applications help in complex data processing such as video processing on the IoV network.


## 6.3. Deep Learning-Driven IoV Network Control

In 5G-based IoV communication, the networks consist of both wired and wireless networks that needs an intelligent method to deal with the huge growth in the data traffic of the network for their varying nature of network sceneries such as cloud computing, mobility, and big data manipulation [90]. To make IoV networks smarter, DL enables to learn how to route network traffic between routers and optimize the network performance. DL has shown promising results in network control, by dividing the framework into phases. Phase 1 is the initial phase, where the related data are obtained for training the deep system. Phase 2 is the training phase that depends on collected data by the previous phase, applying supervised learning algorithms to train the DL-based architecture. Phase 3 represents the running phase where we put the learned algorithm into action where traffic patterns are used as an input that outputs a new path or route after learning the network behaviour based on deep learning algorithms [91].The virtualization approach is a new advancement to the IoV network to have more control over the network resources and to elevate the network performance. The sole source input/output virtual (SS-IOV) environment is proposed to allow I/O devices utilized by VMs with no decrement in the run time throughput of the overall network. SR-IOV is capable of creating virtual functions that allow guests to have direct access [35]. It is an efficient network device that provides the benefit of input/output performance and decreases central processing unit (CPU) employment though significant scalability increment and the sharing abilities of the device [92].


## 6.4. Predictive Analytics “Regression” Problem in Deep Learning

ML uses supervised learning in having an input variablex and output variable y using a learning algorithm in between to study the input/output function of mapping y=fx. The ML process simply having inputs and the output is predicted after the learning process is completed.. Approaches to supervised machine learning include linear and logistic regression, classification, decision tree, and support vector machine [93]. Regression states the operation of modelling the association among one or more autonomous variables and a dependent variable; its main objective is prediction. Problems usually occur when regression is supposed to predict numerical values such as prices, the number of vehicles, predicting demand forecasting, and the length of waiting at a queue.


### 6.4.1. Simplified Dependent and Independent Variable Regressions

These models give conventions about the average functions expressed byEy|u and the variance function, with the assumption that for some transformation, u=ux can be written as(10)Ey|u=β0+β1u.Stating the relation betweeny and u, simple linear regression is operating with only one predictor.


### 6.4.2. Multiple Regression

Problems of regression that occurred have triggered the need to consider operating with more than one predictor, having the dependent variabley that depends simultaneously on predictors x1, x2, …, xn. We begin with n predictors x1, …, xn building a set of k terms based on the predictors ui-1, ui, …, uk-i to have the multiple linear regression mean function [93].(11)Ey|ui,…,uk−i=βi−1ui−1+βiui+⋯+βk−iuk−i.DL architectures enable to solve the regression problem, proven for its demonstrated execution in the vision of computer errands such as picture categorization, question discovery, and picture examination [95]. The common engineering comprises several convolutional layers, taken after a few fully connected layers and a classification delicate max layer [96, 97]. Deep regression algorithms have given results in traditional vision relapse issues such as human posture estimation and a facial point of interest discovery.


## 6.4.1. Simplified Dependent and Independent Variable Regressions

These models give conventions about the average functions expressed byEy|u and the variance function, with the assumption that for some transformation, u=ux can be written as(10)Ey|u=β0+β1u.Stating the relation betweeny and u, simple linear regression is operating with only one predictor.


## 6.4.2. Multiple Regression

Problems of regression that occurred have triggered the need to consider operating with more than one predictor, having the dependent variabley that depends simultaneously on predictors x1, x2, …, xn. We begin with n predictors x1, …, xn building a set of k terms based on the predictors ui-1, ui, …, uk-i to have the multiple linear regression mean function [93].(11)Ey|ui,…,uk−i=βi−1ui−1+βiui+⋯+βk−iuk−i.DL architectures enable to solve the regression problem, proven for its demonstrated execution in the vision of computer errands such as picture categorization, question discovery, and picture examination [95]. The common engineering comprises several convolutional layers, taken after a few fully connected layers and a classification delicate max layer [96, 97]. Deep regression algorithms have given results in traditional vision relapse issues such as human posture estimation and a facial point of interest discovery.


## 7. Machine Learning-Based IoV Future Trends and Deployment Challenges

By 2030, it is expected that the services related to autonomous vehicles will become fully automated, covering most of the applications related to intelligent driving and transportation safety [98]. This expectation will soon become a reality by using machine and deep learning mechanisms, which play an important role in developing most applications related to automated driving. These mechanisms have the ability to build algorithms that carry out intelligent learning and predictions related to network security, control, and resource management.With the increase of technologies in the future, vehicle networks and the links between communications, computing, and resource management become more complex, and most recent studies have tended to use deep learning techniques to improve the quality-of-service requirements [42]. When considering how to apply deep learning mechanisms to IoV applications, researchers face a few challenges that represent a rich environment for future research. These challenges are related to complications related to big data computations, control processing, and resource management, which require extensive improvements to obtain a coherent system that meets service quality and user quality requirements as well.


## 8. Conclusion

The fast development of DL approaches, models, and frameworks with the thought of quick development of IoV applications and administrations activated more and broad inquires about to progress the way of life. Profound learning scheme for IoV is discussed in this paper counting convolutional, deep reinforcement learning (DRL), conventional neural network (CNN), and recurrent neural networks (RNNs). These schemes have demonstrated a tall execution when connected to IoV applications and administrations, i.e., in independent driving, activity checking, mishap avoidance, activity direction frameworks, secure route, electronic fee collection, and secure route. In the future, the ML guarantees to optimize the arrangements to the Qo/QoS for perceptive future IoV systems. The enhancements will upgrade information-gushing quality for amusements and activity administrations. [99] In any case, the sending of ML on IoV will confront diverse challenges due to the huge sum of traded information and distinctive asset availability.



---

*Source: 1947886-2022-04-15.xml*