# Anti-Attack Scheme for Edge Devices Based on Deep Reinforcement Learning

Authors: Rui Zhang; Hui Xia; Chao Liu; Ruo-bing Jiang; Xiang-guo Cheng
Journal: Wireless Communications and Mobile Computing
 (2021)

---

## Abstract

Internet of Things realizes the leap from traditional industry to intelligent industry. However, it makes edge devices more vulnerable to attackers during processing perceptual data in real time. To solve the above problem, we use the zero-sum game to build the interactions between attackers and edge devices and propose an antiattack scheme based on deep reinforcement learning. Firstly, we make thekNN-DTW algorithm to find a sample that is similar to the current sample and use the weighted moving mean method to calculate the mean and the variance of the samples. Secondly, to solve the overestimation problem, we develop an optimal strategy algorithm to find the optimal strategy of the edge devices. Experimental results prove that the new scheme improves the payoff of attacked edge devices and decreases the payoff of attackers, thus forcing the attackers to give up the attack.


---

## Body

## 1. Introduction

Internet of Things (IoT) [1, 2] integrates various sensors or controllers with sensing and monitoring capabilities as well as advanced technologies (e.g., mobile communication technology and intelligent analysis technology) into all aspects of industrial production, realizing the leap from traditional industry to intellect industry. It has been widely used in logistics [3, 4], transportation [5], energy [6], and so on. During the application process of IoT, mass perception data is produced in end devices, which requires the edge devices to have higher real time, security [7, 8], and privacy [9, 10]. However, edge devices are usually located in a nearby user or on a routing path to the cloud, making them more vulnerable to attackers. For example, machine learning models on edge devices during the training period are vulnerable to well-designed adversarial examples [11, 12]. UPGUARD, an American cybersecurity firm, found that hundreds of millions of Facebook user records stored on Amazon’s cloud computing servers could be easily accessed by anyone. Tens of thousands of private Zoom videos are uploaded to the public web page that anyone can watch online. The above threats can cause network penetration, personal data theft, and the epidemic spread of intelligent computer viruses. Therefore, preventing attacks and ensuring data security are the key to improve the efficient application of this system.Currently, resisting malicious attackers mostly adopts traditional skills in IoT, such as encryption method and identity management technology. The encryption method is the most common traditional skill [13, 14]. However, due to the limited resources of edge devices in IoT, making the lightweight encryption program becomes one of the biggest challenges. Identity management technologies are the first line of resisting malicious attackers. However, the existing identity management technology cannot achieve identity authentication between multi-layer architectures. In recent years, a few emerging safety precaution technologies are widely used in IoT [15–17], such as trusted execution environments and machine learning technologies. However, most machine learning technologies, which are based on the assumption that training data remains constant during training, are incompatible with the environment where the data changes dynamically in real-time in IoT [18, 19].Inspired by the above schemes, from the point of view of attacker payoffs, we build the interactions between edge devices and attackers as the zero-sum game and propose an antiattack scheme for edge devices based on deep reinforcement learning. The structure diagram of the proposed scheme is shown in Figure1. The major contributions are as follows:
(1)
To find the optimal strategy of edge devices, we propose thekNN-DTW algorithm to find a similar sample to the current sample and then use the weighted moving mean method to calculate the mean and the variance of the samples;(2)
To weaken the influence of time series’ irregularity, we emphasize the influence of the latest data on forecast value and then set weight for samples by the law that the object is big when near and small when far;(3)
To overcome the overestimation problem of the optimal strategy, we design an optimal strategy algorithm to find the edge device’s optimal strategy by maximizing their accumulated payoff and then achieve the purpose of defending against attackers.Figure 1
The structure diagram of the proposed scheme.The structure of this paper is as follows: in Section2, we define problems that we seek to solve in this article. In Section 3, we discuss the antiattack scheme for edge devices. In Section 4, we verify the effectiveness of the antiattack scheme for edge devices. Section 5 contains conclusions and future research.


## 2. Related Work

This section introduces the latest development and research of antiattack schemes from two aspects: traditional security protection schemes and emerging security protection schemes in IoT.


### 2.1. Traditional Safety Precaution Technologies

Homomorphic encryption, differential privacy, and identity authentication are three traditional protection technologies. Homomorphic encryption can process sensitive data without decryption to protect data privacy. Lu et al. [20] encrypted structured data by using homomorphic Paillier cryptographic system technology. Tan et al. [21] used the technique of finite field theory and proposed a private comparison algorithm based on full homomorphic encryption for encrypted integers. Differential privacy technology is used to ensure the privacy of any single item in the data set under the statistical query. Wang [22] proposed a data-driven spectrum trading solution that could maximize the income of PUS and retain SU’s privacy differences. However, the computing resources on the edge devices are quite limited and cannot support the huge computing power consumed by using encryption schemes. Identity management technologies can set access authority by identity management and access control to prevent illegal user intrusion. Alizadeh et al. [23] summarized authentication technology in mobile cloud computing. Malik et al. [24] proposed an identity authentication and expeditious revocation framework based on the blockchain, which can quickly update the status of revoked vehicles in the shared blockchain. Zhang et al. [25] proposed a smart contract framework comprised of several access control contracts, a judge contract, and a registered contract, which gave a trusted access control strategy. However, using an access control strategy for precautions makes it hard to clear different users’ roles and their rights in IoT.


### 2.2. Emerging Safety Precaution Technologies

The improvement of traditional security schemes can be used to enhance the security of edge devices in the IoT [26]. With the rise of artificial intelligence, some emerging security prevention technologies, such as trusted execution environment and machine learning technology, are gradually used to improve the security of edge devices in the IoT. The trusted execution environment [27] can be used to ensure the security of the running environment of the software. Running an application in a trusted execution environment can guarantee the security of data even if edge devices are compromised. Trusted execution environment, such as trustzone, intel management engine, and ARM trustzone, are quite popular. Han et al. [28] built a complete framework that supports visibility into encrypted traffic and can be used in secure and functional networks. However, trusted execution environment usually has its loophole, such as Qualcomm loopholes, and trustonic loopholes. Ghaffarian and Shahriari [29] proposed a neural vulnerability analysis method based on custom intermediate graph representation of the program for software vulnerability analysis. Scandariatio et al. [30] explored machine learning-based text mining to predict security loopholes in a software source code. However, most machine learning methods assume that statistical data remain unchanged during the training process, but the data in the IoT changes dynamically in real time.


## 2.1. Traditional Safety Precaution Technologies

Homomorphic encryption, differential privacy, and identity authentication are three traditional protection technologies. Homomorphic encryption can process sensitive data without decryption to protect data privacy. Lu et al. [20] encrypted structured data by using homomorphic Paillier cryptographic system technology. Tan et al. [21] used the technique of finite field theory and proposed a private comparison algorithm based on full homomorphic encryption for encrypted integers. Differential privacy technology is used to ensure the privacy of any single item in the data set under the statistical query. Wang [22] proposed a data-driven spectrum trading solution that could maximize the income of PUS and retain SU’s privacy differences. However, the computing resources on the edge devices are quite limited and cannot support the huge computing power consumed by using encryption schemes. Identity management technologies can set access authority by identity management and access control to prevent illegal user intrusion. Alizadeh et al. [23] summarized authentication technology in mobile cloud computing. Malik et al. [24] proposed an identity authentication and expeditious revocation framework based on the blockchain, which can quickly update the status of revoked vehicles in the shared blockchain. Zhang et al. [25] proposed a smart contract framework comprised of several access control contracts, a judge contract, and a registered contract, which gave a trusted access control strategy. However, using an access control strategy for precautions makes it hard to clear different users’ roles and their rights in IoT.


## 2.2. Emerging Safety Precaution Technologies

The improvement of traditional security schemes can be used to enhance the security of edge devices in the IoT [26]. With the rise of artificial intelligence, some emerging security prevention technologies, such as trusted execution environment and machine learning technology, are gradually used to improve the security of edge devices in the IoT. The trusted execution environment [27] can be used to ensure the security of the running environment of the software. Running an application in a trusted execution environment can guarantee the security of data even if edge devices are compromised. Trusted execution environment, such as trustzone, intel management engine, and ARM trustzone, are quite popular. Han et al. [28] built a complete framework that supports visibility into encrypted traffic and can be used in secure and functional networks. However, trusted execution environment usually has its loophole, such as Qualcomm loopholes, and trustonic loopholes. Ghaffarian and Shahriari [29] proposed a neural vulnerability analysis method based on custom intermediate graph representation of the program for software vulnerability analysis. Scandariatio et al. [30] explored machine learning-based text mining to predict security loopholes in a software source code. However, most machine learning methods assume that statistical data remain unchanged during the training process, but the data in the IoT changes dynamically in real time.


## 3. Problem Definition

In this article, we build the interactions between edge devices and attackers as a zero-sum game [31]. Namely, the payoff of attackers equals the loss of edge devices. In each round of interaction, the attacker attacks the sample from edge devices to gain illegal payoff, and the edge device plays his strategy to defend against attackers. Previous studies usually determine the optimal strategy of edge devices by calculating the Nash equilibrium in the handcrafted abstraction of the domain [32]. Currently, some researchers introduce the recursion technique to the neural network to determine players’ optimal strategy by predicting human action in a strategic environment. From [33], the payoff function of edge devices can be defined as
(1)Ua,μ,C=logaTμ−12∗aTμ2aTCa,where a is the strategy vector of edge devices, that is, a=a1,a2,⋯,anT; μ, μ=μ1,μ2,⋯,μnT is sample mean payoff; and C is the covariance matrix of sample payoff. As can be seen from Equation (1), if we know the sample mean payoff and the covariance matrix, we can determine the optimal strategy of edge devices by maximizing the payoff function. That is,
(2)aopt∈argmaxaUa,μ,C.However, the sample mean payoff and the covariance matrix are unknown. But, we can take advantage of the similarity between the historical sample and current sample to predict the sample mean payoff and the covariance matrix and then determine the optimal strategy of the edge device.When we determine the optimal strategy of the edge device, we should try to resolve the following three problems: (1) during the process of measuring similarity between samples, we need to avoid using improper measuring methods which might cause the disappearance of optimal solution; (2) during the process of calculating the sample mean payoff and covariance matrix, we need to weaken the influence of time series’ irregularity; (3) during the process of finding the optimal strategy of edge devices, we need to break the correlation between training samples and solve the problem of overestimation.


## 4. Antiattack Scheme for Edge Devices

This section introduces the following three problems that we seek to solve: how to find the sample that are similar to the current sample, how to calculate the mean value and the covariance matrix, and how to calculate the optimal strategy of edge devices to resist attackers.


### 4.1. Measuring Similarity of Sample

To find a sample similar to the current sample, we propose thekNN-DTW algorithm to determine the category of the current sample and then find the similar sample with the current sample. The kNN-DTW algorithm is a combination of the k-nearest neighbor algorithm and the dynamic time warping method (DTW); the kNN algorithm classifies the current sample and the DTW method finds the similar sample in the same category samples with the current sample.In thekNN algorithm, the choice of k has a significant impact on the classification results. We use the bootstrap method to find the optimal value of k. Assuming the value of k and the probability of time series being correctly classified ρ satisfy the following regression model,:
(3)ρi=hki,β+ε,i=1,2,⋯,n,where h· is the mapping from k to ρ, β is a coefficient vector, and εi is a numerical vector, i.e., Fx. We use the least square method to estimate β, i.e., β^=gρ1,⋯,ρn, make the regression residual empirical distribution function to estimate Fx, and apply the bootstrap method to estimate the covariance matrix Varβ^ of β. If the estimation error of each coefficient (the square root of the diagonal element in Varβ^) meets the threshold ε0, the value of k can be determined by maximizing ρ.After the category of the current sample is determined bykNN, we use DTW to measure the distance between the current sample and the historical sample. DTW method locally scales two samples on the time axis to make the morphology of the two sets, so that the DTW method can measure the distance between time samples that have different lengths. Comparing with Euclidean distance, the DTW method is more elastic and supports local time shifts and in the length of time series, but the time and special complexity of this method is Onm, where n and m are the lengths of two time series, respectively. To decrease the space and time complexity of the DTW method, we apply early abandoning method to optimize the computations of the DTW method. The detailed process is as follows:Step 1. Given two time seriesX and Y,
(4)X=x1,x2,⋯,xn,Y=y1,y2,⋯,ym,wheren is the length of time series X, and m is the length of time series XStep 2. Define the warping path asP=p1,p2,⋯,pK, where maxn,m<K<m+n+1, pk=i,j is the kth element in warping path P, i is the ith cell of time series X, and j is the jth cell of time series Y, the i and j of pk=i,j are monotonically increasing,
(5)pk=i,j,pk+1=i′,j′,i≤i′≤i+1,j≤j′≤j+1.Specially, when calculating warping path, it must ensure that every coordinate in the time seriesX and Y is involved. That is, the calculation starts from p1=1,1 and ends atpk=n,mStep 3. Find the warping path between two time series with the shortest cumulative distance,(6)P=argminpk∈P∑k=1Kpk.To obtain the warping path with the shortest cumulative distance, Eq. (6) can be solved iteratively by using the dynamic programming method
(7)Di,j=Disti,j+minDi−1,j,Di,j−1,Di−1,j−1Step 4. Set the distance thresholdε,ε>0, if the distance Di,j>ε in cell i,j, the calculation of the distance between two time series on the path will be terminatedStep 5. Determine the distance between two time seriesDn,m


### 4.2. Calculating the Mean Value and the Covariance Matrix

To weaken the influence of time series’ irregularity, we emphasize the influence of the latest data on forecast value and set weight for samples by the law that the object is big when near and small when far. Namely, the sample elements that are close to the prediction period will be given a relatively big weight. We use the weighted moving average method to calculate sample’s mean value. That is,(8)μt=ytwt+yt−1wt−1+⋯+y1w1wt+wt−1+⋯+w1,where wt refers to the weight of sample data yt; it follows the rule that weight decreases as the distance increases, i.e., wt>wt−1>⋯>w1. Accordingly, the covariance matrix C can be calculated as
(9)C=EX−μXY−μY.


### 4.3. Preventing Malicious Attacks

After finding the similar sample, we first take the mean payoff and covariance matrix of the similar sample as the mean payoff and covariance matrix of the current sample, respectively. And then, to weaken the influence of time series irregularity, we emphasize the influence of the latest data on forecast value and set weight for samples by the law that the object is big when near and small when far. Finally, we find the solution to the optimal strategy of edge devices by maximizing the payoff function. The detailed process is shown in Algorithm1.Algorithm 1: Optimal strategy.
Input: Similarity sample setW;Output: Optimal strategyamax;1: forn=2: N do2: Calculatesimilarity with DTW;3: ifsimilarity<1 then4:  Continue;5: else:6:UP=maxaUa,μ,C;7: end for8: returnamaxHowever, the above method is prone to overestimation. To solve the above problem, we design Algorithm2 to find the optimal strategy of the edge devices by maximizing their accumulated payoff.Reinforcement learning is aimed at maximizing the reward for the long term to find the payoff maximum of the agent. Thus, players of the game are transformed into separate agents. We useDeep Q Network to find the agent’s optimal strategy. In this algorithm, the state set S of agent is defined as S=s1,s2, where s1 means that the current data is normal (not attacked) and s2 means that the current data is abnormal (already attacked); the above states can be described by the Markov decision process. The action set A is defined as A=a1,a2, where a1 means that the agent accepts the current data set, a2 means that the agent rejects the current data set, and action reward R is defined as
(10)R=1,a=a1,s=s1ora=a2,s=s2,−1,others.The agent interacts with its fellow agents and stores its experience of strategy transitionssj,aj,rj,sj+1 in replay memory R. To break the correlation between training samples, during the process of training, we select samples randomly from replay memory R to train model for finding the optimal strategy of the agent. The detailed process is shown Algorithm 2, where Q^ is the payoff when the agent adopts the optimal strategy.Algorithm 2: Optimal strategy.
01: Initialize replay memoryR;02: Initialize anticipatory parametersη;03: Initialize target functionQ with weight ϑ;04:forepisode=1, Mdo05: Set policyσ←aopt,ηβ,1−η;06: Receive initial observation states1 and reward r1;07:fort=1, Tdo08:  Select action at from policyσ;09:  Execute action at and observe rewardrt+1 and observe new state st+1;10:  Store transition (st,at,rt+1,st+1) in R;11:  Sample random minibatch of transition (sj,aj,rj,sj+1) from R12:ift terminates at step t+1then13:Qt=rt;14:else15:Qt=rt+rmaxat+1Qηat+1−ηβ∣st;16:end if17:  Perform a gradient descent step onQ^−Qt2 with respect to network parameters;19:  Periodically update the target networksQ;20:end for21:end for


## 4.1. Measuring Similarity of Sample

To find a sample similar to the current sample, we propose thekNN-DTW algorithm to determine the category of the current sample and then find the similar sample with the current sample. The kNN-DTW algorithm is a combination of the k-nearest neighbor algorithm and the dynamic time warping method (DTW); the kNN algorithm classifies the current sample and the DTW method finds the similar sample in the same category samples with the current sample.In thekNN algorithm, the choice of k has a significant impact on the classification results. We use the bootstrap method to find the optimal value of k. Assuming the value of k and the probability of time series being correctly classified ρ satisfy the following regression model,:
(3)ρi=hki,β+ε,i=1,2,⋯,n,where h· is the mapping from k to ρ, β is a coefficient vector, and εi is a numerical vector, i.e., Fx. We use the least square method to estimate β, i.e., β^=gρ1,⋯,ρn, make the regression residual empirical distribution function to estimate Fx, and apply the bootstrap method to estimate the covariance matrix Varβ^ of β. If the estimation error of each coefficient (the square root of the diagonal element in Varβ^) meets the threshold ε0, the value of k can be determined by maximizing ρ.After the category of the current sample is determined bykNN, we use DTW to measure the distance between the current sample and the historical sample. DTW method locally scales two samples on the time axis to make the morphology of the two sets, so that the DTW method can measure the distance between time samples that have different lengths. Comparing with Euclidean distance, the DTW method is more elastic and supports local time shifts and in the length of time series, but the time and special complexity of this method is Onm, where n and m are the lengths of two time series, respectively. To decrease the space and time complexity of the DTW method, we apply early abandoning method to optimize the computations of the DTW method. The detailed process is as follows:Step 1. Given two time seriesX and Y,
(4)X=x1,x2,⋯,xn,Y=y1,y2,⋯,ym,wheren is the length of time series X, and m is the length of time series XStep 2. Define the warping path asP=p1,p2,⋯,pK, where maxn,m<K<m+n+1, pk=i,j is the kth element in warping path P, i is the ith cell of time series X, and j is the jth cell of time series Y, the i and j of pk=i,j are monotonically increasing,
(5)pk=i,j,pk+1=i′,j′,i≤i′≤i+1,j≤j′≤j+1.Specially, when calculating warping path, it must ensure that every coordinate in the time seriesX and Y is involved. That is, the calculation starts from p1=1,1 and ends atpk=n,mStep 3. Find the warping path between two time series with the shortest cumulative distance,(6)P=argminpk∈P∑k=1Kpk.To obtain the warping path with the shortest cumulative distance, Eq. (6) can be solved iteratively by using the dynamic programming method
(7)Di,j=Disti,j+minDi−1,j,Di,j−1,Di−1,j−1Step 4. Set the distance thresholdε,ε>0, if the distance Di,j>ε in cell i,j, the calculation of the distance between two time series on the path will be terminatedStep 5. Determine the distance between two time seriesDn,m


## 4.2. Calculating the Mean Value and the Covariance Matrix

To weaken the influence of time series’ irregularity, we emphasize the influence of the latest data on forecast value and set weight for samples by the law that the object is big when near and small when far. Namely, the sample elements that are close to the prediction period will be given a relatively big weight. We use the weighted moving average method to calculate sample’s mean value. That is,(8)μt=ytwt+yt−1wt−1+⋯+y1w1wt+wt−1+⋯+w1,where wt refers to the weight of sample data yt; it follows the rule that weight decreases as the distance increases, i.e., wt>wt−1>⋯>w1. Accordingly, the covariance matrix C can be calculated as
(9)C=EX−μXY−μY.


## 4.3. Preventing Malicious Attacks

After finding the similar sample, we first take the mean payoff and covariance matrix of the similar sample as the mean payoff and covariance matrix of the current sample, respectively. And then, to weaken the influence of time series irregularity, we emphasize the influence of the latest data on forecast value and set weight for samples by the law that the object is big when near and small when far. Finally, we find the solution to the optimal strategy of edge devices by maximizing the payoff function. The detailed process is shown in Algorithm1.Algorithm 1: Optimal strategy.
Input: Similarity sample setW;Output: Optimal strategyamax;1: forn=2: N do2: Calculatesimilarity with DTW;3: ifsimilarity<1 then4:  Continue;5: else:6:UP=maxaUa,μ,C;7: end for8: returnamaxHowever, the above method is prone to overestimation. To solve the above problem, we design Algorithm2 to find the optimal strategy of the edge devices by maximizing their accumulated payoff.Reinforcement learning is aimed at maximizing the reward for the long term to find the payoff maximum of the agent. Thus, players of the game are transformed into separate agents. We useDeep Q Network to find the agent’s optimal strategy. In this algorithm, the state set S of agent is defined as S=s1,s2, where s1 means that the current data is normal (not attacked) and s2 means that the current data is abnormal (already attacked); the above states can be described by the Markov decision process. The action set A is defined as A=a1,a2, where a1 means that the agent accepts the current data set, a2 means that the agent rejects the current data set, and action reward R is defined as
(10)R=1,a=a1,s=s1ora=a2,s=s2,−1,others.The agent interacts with its fellow agents and stores its experience of strategy transitionssj,aj,rj,sj+1 in replay memory R. To break the correlation between training samples, during the process of training, we select samples randomly from replay memory R to train model for finding the optimal strategy of the agent. The detailed process is shown Algorithm 2, where Q^ is the payoff when the agent adopts the optimal strategy.Algorithm 2: Optimal strategy.
01: Initialize replay memoryR;02: Initialize anticipatory parametersη;03: Initialize target functionQ with weight ϑ;04:forepisode=1, Mdo05: Set policyσ←aopt,ηβ,1−η;06: Receive initial observation states1 and reward r1;07:fort=1, Tdo08:  Select action at from policyσ;09:  Execute action at and observe rewardrt+1 and observe new state st+1;10:  Store transition (st,at,rt+1,st+1) in R;11:  Sample random minibatch of transition (sj,aj,rj,sj+1) from R12:ift terminates at step t+1then13:Qt=rt;14:else15:Qt=rt+rmaxat+1Qηat+1−ηβ∣st;16:end if17:  Perform a gradient descent step onQ^−Qt2 with respect to network parameters;19:  Periodically update the target networksQ;20:end for21:end for


## 5. Stimulation Results

We use the Anaconda-integrated development tool to validate the proposal. First, we analyze the feasibility of weakening the influence of time series irregularity to prove the reasonableness of setting sample data’s weight according to the rule of the object being big when near and small when far. Second, we compare theDTW method with seven classical distance methods like correlation distance, Jaccard distance, and cosine distance to verify the reasonableness of kNN-DTW. Finally, we apply optimal strategy to the rock-paper-scissors game [34] to verify the practicability of optimal strategy by comparing the winner (choose winner’s strategy) and opponent (choose opponent’s strategy) strategies.


### 5.1. Feasibility Analysis of Weakening the Influence of Time Series Irregularity

Tables1–3 analyze the influence of weighting weights on each parameter in the target payoff function according to the law that the object is big when near and small when far. To better describe the complete process, we set each sample dataset only has 20 data and make the weight for each sample, as shown in Table 1. Table 2 shows the prediction error between the weighted mean and mean under the same strategy profile 1,1,1,0, where 1 means that the edge device accepts the data, and 0 means that the edge device rejects the data.Table 1
Sample dataset.
NumberData 1Data 2Data 3Data 4Weight122.4421.9521.9615.820.099066223.0121.9422.6716.690.08585323.122.422.416.430.082904423.221.721.8815.840.07981521.5120.8521.4415.660.078681622.0821.321.4516.20.073956721.8621.1521.7915.970.07361821.3920.7721.1116.650.07244921.4320.8521.2316.450.0673821021.4820.9621.2515.890.0510611121.1920.821.116.560.046455122221.242216.540.0457421322.4721.521.6416.280.039781421.620.7420.8116.440.033061520.4819.5119.8816.110.0288711620.1919.7519.815.90.0194831720.0618.1618.5915.10.0101271818.7718.0618.3114.510.0048141918.3517.618.214.060.0035912018.4117.7317.9913.880.003319Table 2
Player’s mean payoff comparison.
ValueData 1Data 2Data 3Data 5Mean payoff21.25120.44820.77515.849Weight mean payoff21.993721.21921.543516.1693Error0.7426950.7710.76850.3203Action1110Table 3
The influence of time series irregularity.
ParameteraTμCaTCalogaTμNo-weight62.472.152.102.070.992.102.182.141.032.072.142.151.030.991.031.030.6919.091.80Weight64.760.490.480.480.350.480.460.470.340.480.470.480.350.350.340.350.254.291.81Table3 shows the effects of weakening the influence of time series irregularity on the parameters of the objective function (e.g., C). According to the table, setting the weight of the data according to the rule of the object is big when near and small when far has a greater influence on C and a weaker influence on logaTμ. Therefore, the proposed method can weaken the irregularity of the time series.


### 5.2. Verification of the Reasonableness ofkNN-DTW

To verify the reasonableness of combiningDTW method and kNN algorithm, Figure 2 shows the DTW method with seven classical distance methods like correlation distance, Jaccard distance, and cosine distance. From Figure 2, we can see that the cosine distance and Chebyshev distance are the worst. For example, when the ratio of the same elements in the range from 20% to 33%, the results of the Cosine distance are all 0.79 while the ratio of the same elements in the range from 6% to 67%; the results of the Cosine distance are all 0.66. Therefore, cosine distance and Chebyshev distance are not suitable for measuring the distance between the samples in this paper. Although other methods also produce the same results, the number of the same results is less than that of cosine distance and Chebyshev distance. For example, the results of the DTW method are the same if and only if the ratio of the same elements is 80% or 87%. And the results of Euclidean distance are the same if and only if the ratio of the same elements is 47% or 53%.Figure 2
Similarity comparison.
(a)
Similarity comparison (I)(b)
Similarity comparison (II)By comparing Figures2(a) and 2(b), it can be seen that DTW method works best, followed by Jaccard distance. For example, when the number of the same elements is 80%, 87%, and 93%, the results of Jaccard distance are 0.2, 0.13, and 0.07; the results of Euclidean distance are 0.52, 0.41, and 0.01; the results of Manhattan distance are 0.74, 0.42, and 0.01; and the results of DTW method are 0.27, 0.17, and 0.0004, respectively. From the above results, we can draw a conclusion that Euclidean distance and Manhattan distance have a similar impact on measuring the distance between samples, while the results measured by these two methods varied greatly when the data in the two samples varied from 87% to 93%. While the Jaccard distance and DTW method have a similar impact measuring the distance between samples, the results measured by these two methods varied slightly when the data in the two samples varied from 87% to 93%. The DTW method is more suitable for measuring the distance between samples; this is because the DTW method can measure the distance between samples of different lengths. Therefore, we combine the DTW method with the kNN algorithm to measure the distance between samples.


### 5.3. Application of Antiattack Scheme

First, we need to define the rock-paper-scissors game’s payoff matrix, as shown in Table4. The rock-paper-scissors game is a typical example of zero-sum game. In the game, two players have the same strategy set, which is (rock, paper, scissors). If two players play the same strategy, then both of them get 0 for a draw; otherwise, the winner gets 2 and the loser gets -2.Table 4
Payoff matrix.
Player1/2RockScissorsPaperRock0, 02, -2-2, 2Scissors-2, 20, 02, -2Paper2, -2-2, 20, 0Figure3 shows the changing trend of payoff that player 1 and player 2 play optimal strategy, winner strategy, and opponent strategy in the initial states S0=scissors,rock and S1=rock,scissors. In the figure, winner vs. optimal means that player 1 plays winner strategy and player 2 players optimal strategy in the game. Similarly, we can know the meaning of opponent vs. optimal and winner vs. opponent. Figures 3(a)–3(d) show the payoff of player 1 and player 2 in S0 and S1 states, respectively. From Figure 3(a), we can see that due to player 1 adjusts scissors strategy to rock strategy when starting the second round of the game, the payoff of player 1 is -2 in the first round, and the payoff of player 1 is 2 in the second round. It is worth noting that the payoff trend of player 1 and player 2 is the same in winner vs. optimal and opponent vs. optimal because the strategies adjusted by winner strategy and opponent are the same in the initial state S0. According to Figures 3(a)–3(d), we can draw a conclusion that the optimal strategy is optimal in state S0, while in state S1, optimal strategy is superior to winner strategy and inferior to opponent. As can be seen Table 5, the overall payoff of players is same in opponent strategy and optimal strategy. To sum up, optimal strategy scheme can help to determine the player’s strategy and maximize the player’s payoff.Figure 3
Payoff comparison.
(a)
Player 1 payoff in (scissors, rock)(b)
Player 2 payoff in (scissors, rock)(c)
Player 1 payoff in (rock, scissors)(d)
Player 2 payoff in (rock, scissors)Table 5
Total payoff comparison.
PlayersWinnerOpponentOptimalTotal-1688


## 5.1. Feasibility Analysis of Weakening the Influence of Time Series Irregularity

Tables1–3 analyze the influence of weighting weights on each parameter in the target payoff function according to the law that the object is big when near and small when far. To better describe the complete process, we set each sample dataset only has 20 data and make the weight for each sample, as shown in Table 1. Table 2 shows the prediction error between the weighted mean and mean under the same strategy profile 1,1,1,0, where 1 means that the edge device accepts the data, and 0 means that the edge device rejects the data.Table 1
Sample dataset.
NumberData 1Data 2Data 3Data 4Weight122.4421.9521.9615.820.099066223.0121.9422.6716.690.08585323.122.422.416.430.082904423.221.721.8815.840.07981521.5120.8521.4415.660.078681622.0821.321.4516.20.073956721.8621.1521.7915.970.07361821.3920.7721.1116.650.07244921.4320.8521.2316.450.0673821021.4820.9621.2515.890.0510611121.1920.821.116.560.046455122221.242216.540.0457421322.4721.521.6416.280.039781421.620.7420.8116.440.033061520.4819.5119.8816.110.0288711620.1919.7519.815.90.0194831720.0618.1618.5915.10.0101271818.7718.0618.3114.510.0048141918.3517.618.214.060.0035912018.4117.7317.9913.880.003319Table 2
Player’s mean payoff comparison.
ValueData 1Data 2Data 3Data 5Mean payoff21.25120.44820.77515.849Weight mean payoff21.993721.21921.543516.1693Error0.7426950.7710.76850.3203Action1110Table 3
The influence of time series irregularity.
ParameteraTμCaTCalogaTμNo-weight62.472.152.102.070.992.102.182.141.032.072.142.151.030.991.031.030.6919.091.80Weight64.760.490.480.480.350.480.460.470.340.480.470.480.350.350.340.350.254.291.81Table3 shows the effects of weakening the influence of time series irregularity on the parameters of the objective function (e.g., C). According to the table, setting the weight of the data according to the rule of the object is big when near and small when far has a greater influence on C and a weaker influence on logaTμ. Therefore, the proposed method can weaken the irregularity of the time series.


## 5.2. Verification of the Reasonableness ofkNN-DTW

To verify the reasonableness of combiningDTW method and kNN algorithm, Figure 2 shows the DTW method with seven classical distance methods like correlation distance, Jaccard distance, and cosine distance. From Figure 2, we can see that the cosine distance and Chebyshev distance are the worst. For example, when the ratio of the same elements in the range from 20% to 33%, the results of the Cosine distance are all 0.79 while the ratio of the same elements in the range from 6% to 67%; the results of the Cosine distance are all 0.66. Therefore, cosine distance and Chebyshev distance are not suitable for measuring the distance between the samples in this paper. Although other methods also produce the same results, the number of the same results is less than that of cosine distance and Chebyshev distance. For example, the results of the DTW method are the same if and only if the ratio of the same elements is 80% or 87%. And the results of Euclidean distance are the same if and only if the ratio of the same elements is 47% or 53%.Figure 2
Similarity comparison.
(a)
Similarity comparison (I)(b)
Similarity comparison (II)By comparing Figures2(a) and 2(b), it can be seen that DTW method works best, followed by Jaccard distance. For example, when the number of the same elements is 80%, 87%, and 93%, the results of Jaccard distance are 0.2, 0.13, and 0.07; the results of Euclidean distance are 0.52, 0.41, and 0.01; the results of Manhattan distance are 0.74, 0.42, and 0.01; and the results of DTW method are 0.27, 0.17, and 0.0004, respectively. From the above results, we can draw a conclusion that Euclidean distance and Manhattan distance have a similar impact on measuring the distance between samples, while the results measured by these two methods varied greatly when the data in the two samples varied from 87% to 93%. While the Jaccard distance and DTW method have a similar impact measuring the distance between samples, the results measured by these two methods varied slightly when the data in the two samples varied from 87% to 93%. The DTW method is more suitable for measuring the distance between samples; this is because the DTW method can measure the distance between samples of different lengths. Therefore, we combine the DTW method with the kNN algorithm to measure the distance between samples.


## 5.3. Application of Antiattack Scheme

First, we need to define the rock-paper-scissors game’s payoff matrix, as shown in Table4. The rock-paper-scissors game is a typical example of zero-sum game. In the game, two players have the same strategy set, which is (rock, paper, scissors). If two players play the same strategy, then both of them get 0 for a draw; otherwise, the winner gets 2 and the loser gets -2.Table 4
Payoff matrix.
Player1/2RockScissorsPaperRock0, 02, -2-2, 2Scissors-2, 20, 02, -2Paper2, -2-2, 20, 0Figure3 shows the changing trend of payoff that player 1 and player 2 play optimal strategy, winner strategy, and opponent strategy in the initial states S0=scissors,rock and S1=rock,scissors. In the figure, winner vs. optimal means that player 1 plays winner strategy and player 2 players optimal strategy in the game. Similarly, we can know the meaning of opponent vs. optimal and winner vs. opponent. Figures 3(a)–3(d) show the payoff of player 1 and player 2 in S0 and S1 states, respectively. From Figure 3(a), we can see that due to player 1 adjusts scissors strategy to rock strategy when starting the second round of the game, the payoff of player 1 is -2 in the first round, and the payoff of player 1 is 2 in the second round. It is worth noting that the payoff trend of player 1 and player 2 is the same in winner vs. optimal and opponent vs. optimal because the strategies adjusted by winner strategy and opponent are the same in the initial state S0. According to Figures 3(a)–3(d), we can draw a conclusion that the optimal strategy is optimal in state S0, while in state S1, optimal strategy is superior to winner strategy and inferior to opponent. As can be seen Table 5, the overall payoff of players is same in opponent strategy and optimal strategy. To sum up, optimal strategy scheme can help to determine the player’s strategy and maximize the player’s payoff.Figure 3
Payoff comparison.
(a)
Player 1 payoff in (scissors, rock)(b)
Player 2 payoff in (scissors, rock)(c)
Player 1 payoff in (rock, scissors)(d)
Player 2 payoff in (rock, scissors)Table 5
Total payoff comparison.
PlayersWinnerOpponentOptimalTotal-1688


## 6. Conclusion

InIoT, defending against attacks by determining the optimal strategy of the edge device for ensuring data security is the key to improve its effectiveness. In this article, we propose an antiattack scheme for edge devices based on deep reinforcement learning to solve this issue. And the core of this scheme is he optimal strategy algorithm. Detailed simulation experiment verified the effectiveness of this new scheme.In future studies, we will focus on creating a new methodology to determine the similarity between data samples and use machine learning approaches to solve more data security problems.



---

*Source: 6619715-2021-04-15.xml*