DEIM Forum 2019 I5-1

モバイルセンサーネットワークにおける学習能力を持つ攻撃端末の発見
Boqi GAO† , Takuya MAEKAWA† , Daichi AMAGATA† , and Takahiro HARA†† Osaka University, Graduate School of Information Science and Technology
1–5, Yamadaoka, Suita, Osaka 565–0871 Japan
E-mail: †{gao.boqi,maekawa,amagata.daichi,hara}@ist.osaka-u.ac.jp
Abstract Mobile wireless sensor networks (WSNs) are facing threats from malicious nodes which disturb packet
transmissions, leading to the poor performance of mobile WSNs. Existing studies provided some methods, suchas decision tree based classification, to detect these malicious nodes. However, these malicious nodes are assumed
to follow only pre-defined attack models and do not have any learning ability. This limits the malicious nodes inregard of development of machine learning technologies. For this reason, in this study, we design a reinforcement
learning based malicious nodes, and define a novel observation space and a sparse reward function for them. We also
propose an adaptive learning method to detect these malicious nodes. Extensive experiments show that comparedwith existing attack models, our malicious nodes can make networks perform worse while not being detected. We
also investigate the performance of our detecting method and confirm that our method significantly outperforms
the state-of-the-art methods in terms of detection accuracy and false detection rate.
Key words Mobile WSN, reinforcement learning, security

1. Introduction

Thus, this study assumes that malicious nodes have learning
ability, and they can learn from the countermeasures to avoid

Driven by extensive growth of Internet of Things (IoT) andbeing detected. Similarly, the above attack countermeasures

artificial intelligence (AI) technologies, as an important ap-

cannot deal well with an attack model which is capable ofplication of them, a mobile wireless sensor network (WSN)

learning, because preparing a static rule for an attack modelhas demonstrated its feasibility, which requires long range

which can change its behavior is impractical. Therefore, thecommunications and capability of monitoring. However, due

attack countermeasures should also be have learning ability.

to the self-organized characteristic, malicious nodes can eas-

Contribution. To address the problem of designing an grey

ily join a mobile WSN. That is, mobile WSNs are inherently

hole attack model with learning ability, we utilize reinforce-

vulnerable to malicious nodes, and the malicious nodes dis-ment learning to let the malicious nodes learn by themselves

rupt communications in mobile WSNs, causing packet trans-

in a mobile WSN. In our method, malicious nodes obtain

mission failures; all of which presents security challenges toa positive reward when they deteriorate the performance ofmobile WSN environments.

the mobile WSN. However, if these nodes behave too mali-

Existing studies have proposed a variety of routing at-

ciously, they may be easily detected by a countermeasure oftack models [6]. These attack models mainly aim at dis-the mobile WSN. We assume that they receive a large nega-

turbing packet transmissions and causing redundant traﬃc.

tive reward if they are detected within a time threshold. In

One of the most aggressive attacks, grey hole attack [22],a large number of episodes, the malicious nodes will finallyhas been proved to have the ability to destroy the rout-learn how to perform maliciously as much as possible whileing procedure and data transmissions of a sensor network.

not being detected. In addition, we design a method that

Rule-based countermeasures have been proposed for avoid-robustly detects the above attack model. Over a certain pe-

ing attacks and/or detecting grey hole attacks in mobile

riod, the detection method is updated adaptively to find the

WSNs [25], [26]. However, the existing grey hole attack fol-rule of attack detection. The principal contributions of ourlows only pre-defined routines, which means that they cannot

work are:modify their attack patterns. This leads to the result that•We design a novel reinforcement learning based grey

the malicious nodes are easy to be detected by correspond-hole attack model in mobile WSNs. To our knowledge, this is

ing countermeasures [1], [24]. Moreover, the assumption thatthe first work that applies reinforcement learning algorithms

malicious nodes have no learning ability is extremely unreal-

to build a smart grey hole attack model in mobile WSNs.

istic in a world with rapid development of machine learning.

•

We propose a method for discrete-time adaptively up-

dating the countermeasure. This method can update countermeasures to detect our novel attack model.
•

We conduct extensive experiments to investigate theperformance of our attack model and our correspondingity.

3. Assumption
3. 1 Network model

countermeasure. From the results of these experiments, we

Our network environment is assumed to be a mobile WSN

confirm that our attack model is hardly detected by exist-

consisting of n wireless nodes with unique identifiers (i.e.,

ing countermeasures. Moreover, our countermeasure outper-

node ID). These nodes can move without restriction, whileforms the state-of-the-art.

directly communicating with other nodes if they are within

Organization. We review related works in Section 2. Sec-the communication range. We assume that all nodes havetion 3 introduces the assumption in this paper. Our proposed

the same communication range, and if a given node is within

method is described in Section 4. The experimental results

the communication range of other nodes, it is a neighboringare illustrated in Section 5. Section 6 summarizes our con-node of them.

clusion.

2. Related work

As a routing protocol, AODV [19], which is a standard
routing protocol in mobile WSNs, is employed. In AODV,
when a source node s wants to send a data packet to a des-Reinforcement learning. Reinforcement learning (RL)

tination node d, if there is no valid route existing in routing

has become an increasing popular research area due to its

table of s, s broadcasts a route request (RReq) to create aeﬀectiveness in various tasks. As a model-free reinforcement

packet transmission route, and this message is transmitted

learning technique, the Q-learning algorithm can derive theby some intermediated nodes. When node d receives thisoptimal strategy with probability one if all the feasible ac-

RReq, it sends a route reply (RRep) toward s to create ations are repeatedly sampled over all the states in a Marko-

valid route, then data packet will be transmitted throughvian decision process [20]. Mnih et al. [17] firstly successfully

this route between s and d. To maintain the route, route er-

approximated Q function with a deep convolutional neuralror (Rerr) and hello messages are also utilized. When a node

network, and enabled their agent to beat a human expert

detects a link failure in an active route, it sends an Rerr

in several Atari games. Later on, RL algorithms have been

to notify other nodes of the link failure. Each node checks

applied to many WSN-related applications such as schedul-

whether it has sent a message or not within an interval called

ing of energy harvesting node [3] and detecting the roughhello interval. If not, it broadcasts a hello message to notify

edge of a VANET [15]. In the domain of WSN security, Li

other nodes of its existence (see [19] for other details).

et al. [14] used game theory to protect packet transmissions

3. 2 Mobile nodes

against smart attacks. Xiao et al. [27] utilized RL against

In this paper, we consider a mobile WSN contains two

smart jamming. These works assume that malicious nodes

categories of nodes: malicious nodes, which do grey hole at-follow existing patterns, and the proposed methods are only

tack, and normal nodes. We assume that each category ofeﬀective for pre-defined attack models. Therefore, for a ma-

nodes hold their own servers (respectively normal server andlicious node that can learn, simply extending these existing

malicious server) with machine learning functions. In ad-

methods is not trivial.

dition, we assume that malicious nodes conspire together toGrey hole attack model and countermeasures indestroy the regular routing procedure and disturb the proper

WSNs. Security of mobile WSNs is a challenging issue, anddata transmissions in a mobile WSN. Furthermore, equipped

many existing studies concerned about it. Existing studies

with computation power, caching resources, and the mali-

have proposed plenty attack models [13]. Grey hole attack iscious server, these malicious nodes can use reinforcementa special one among them. In the grey hole attack model,

learning to learn not to be detected while executing mali-

malicious nodes can selectively (randomly) forward packets

cious behaviors. On the other hand, the normal nodes at-or reply requests to drag routes to themselves. The ordi-

tempt to update their detecting method to follow the latest

nary grey hole attack is a smart version of the famous black

behavioral pattern of the malicious nodes. The normal nodes

hole attack [23], and it is hard to be detected by ordinary

work together to detect malicious nodes with a specific data

reputation based detection methods [22]. Some works havecollecting method (e.g., flooding through the network).

developed techniques for avoiding the attack with multipath

3. 3 Deep Q-learning for malicious server

approaches [26], analyzing the impact of the attack [25], andWe assume that malicious nodes employ deep Q-learning

preventing the attack [21]. However, the existing grey hole[17] as the reinforcement learning algorithm.

attack model and corresponding countermeasures do not as-present a brief summary of deep Q-learning.

sume that the malicious nodes can equip with learning abil-

Below we

Reinforcement learning deals with learning an optimal pol-icy π ∗ for an agent interacting in an unknown environment.
At each time step t, an agent observes the current state st(1) Malicious Server

Training environmentof the environment, decides on an action at according to afit

policy π, and observes a reward signal rt . The goal of theselectactionagent is to find a policy that maximizes the expected sum of
discounted rewards Rt
Rt =

T
∑

DQN agent′γ t −t rt′ ,state reward(1)(2) Normal Servert′ =twhere T is the time at which the episode terminates, and γ∈ [0,1] is a discount factor that determines the importance of
future rewards. The Q-function of a given policy π is defined

update…

as the expected return from executing an action a in a stateClassifiers:InstancesQπ (s, a) = E[Rt |st = s, at = a](2)
Figure 1 Overview of the training phase of the proposed method.

Finally, the updating rule for the Q-values for an action selection policy π is as follows:Qπ (st , at ) = (1 − α)Qπ (st , at )+π(1) Malicious server monitors the mobile WSN, and utilizes our method to obtain state and reward. Then themalicious server updates the deep Q-learning agent and(3)α[Rt (st , at ) + γmaxQ (st+1 , a)]selects an action. (2) Normal server collects instancesfrom the normal nodes then updates the classifier.

a

where
0<α<1

overview of the training phase of the proposed method. In(4)the training phase, we simulate various environments with
diﬀerent environmental parameters, e.g., initial node po-is the learning rate. It has been proved that through suf-

sitions, and train the malicious server with reinforcement

ficiently large number of learning iterations, the Q-learninglearning algorithms in an on-line manner, using data ob-

algorithm converges and returns the optimal policy π ∗ [11].

tained from the malicious nodes in the environments. Si-

Instead of performing the Q-learning updates in an on-

multaneously, the normal server is also trained by the dataline learning fashion, it is popular to use experience replay

collected from the normal nodes. It is important to recall

to break the correlation between successive samples [2]. At

that in the training phase, all the malicious and normal nodes

each time step, an agent experience (st , at , rt , st+1 ) is stored

respectively share their own server. In other words, both thein a replay memory, and the Q-learning updates are done onservers can monitor the whole mobile WSN, and can utilize

batches of experiences randomly sampled from the memory.

all the data from their nodes because the training phase is

In the training of the deep Q-learning agent, we assume thatbased on simulations.

malicious nodes share the same malicious server, so note that

The reason why we train servers on diﬀerent network en-

the st and st+1 in one experience tuple (st , at , rt , st+1 ) are

vironments is derived from the fact that only training on athe state and next state of a particular malicious node.

single network environment is not general [10], i.e., the nor-At every training step, the next action is generated by us-

mal server may overfit the behavioral pattern of the malicious

ing an ϵ-greedy strategy: with a probability ϵ, the next action

nodes in a single network environment. For example, assume

is selected randomly, and with probability 1 − ϵ, the nextthat we have network environments A and B, and we trained

action is the best one obtained by the deep Q-learning algo-

the normal server in network environment A. This would eas-rithm. In our method, we start with ϵ = 1 and progressively

ily detect malicious nodes in network environment A because

decay ϵ.

the normal server fits the malicious data perfectly. However,

4. Proposed method

it may not work well in network environment B, as the normal server has no training data of network environment B.

4. 1 Overview

In the test phase, each normal/malicious node is equipped

Here we describe the overview of our method. The method,with a fixed normal/malicious server obtained in training

which is based on machine learning techniques, consists ofphase. Normal nodes use the trained normal server to detect

a training phase and a test phase. Figure 1 presents anTable 1 Information observed by each node

tination. ii) When the malicious node receives a data packet,

Information Definition

it may also perform normally, or just drop it. The percent-

NReqRec# RReq overheard from one neighborage that the node performs normally or not is decided by a

NRepRec# RRep overheard from one neighbor

NRepSen# RRep sent to one neighbor

NRerRec# Rerr overheard from one neighbormalicious ratio (e.g., 50%).
However, this static pre-defined malicious ratio makes the

NRerSen# Rerr sent to one neighborgrey hole malicious nodes easily detected by some machine

NHelRec# hello messages overheard from one neighborlearning based detection methods. Therefore, in this study,

NDatRec# data packets overheard from one neighbor

we employ reinforcement learning to enable the malicious

NDatSen# data packets sent to one neighbornode to learn the most proper behavior. The actions of our

TReqRec

Total # RReq overheard by observing node

smart grey hole attack model are the same as ordinary grey

TReqSenTotal # RReq sent by observing nodehole attack model.

TRepRec

Total # RRep overheard by observing node

TRepSenTotal # RRep sent by observing node

TRerRec

Total # Rerr overheard by observing node

TRerSenTotal # Rerr sent by observing node

THelRec

Total # hello messages overheard by observing nodetions, and in each episode, we employ diﬀerent network pa-

THelSenTotal # hello messages sent by observing node

rameters. During each training episode, the malicious server

TDatRec

Total # data packets overheard by observing nodemonitors the mobile WSN and employs our method to ob-

TDatSenTotal # data packets sent by observing nodetain a reward and a state after a malicious node receives any

ActNei

Active # neighbors met from start

packet. The malicious server then updates itself, and selects

TPckSenTotal # packets sent

an action for this malicious node. When all malicious nodes

TPckRec

Total # packets received

4. 2. 2 Overview of smart grey hole attack model
We describe the overview of smart grey hole attack model.
In the training phase, we run suﬃcient episodes of simula-in the mobile WSN are detected or the simulation time goesover the threshold, an episode ends.

malicious nodes, whose action is decided by their malicious

In the test phase, each malicious node is installed with aservers. Note that each server is no more updated in the

trained malicious server obtained in the training phase. The

test phase because, (i) for the normal server, the category

malicious server of each node is no more updated. However,of a neighboring node is unknown, and (ii) for the malicious

it still utilizes our method to obtain state and selects action

server, the reward is unknown. Therefore, in the test phase,

for its malicious node.

both malicious and normal nodes do not need to collect datafrom other nodes.

4. 2. 3 State and reward engineering
As introduced in Section 3. 3, reinforcement learning uti-

Moreover, we assume that both malicious and normal

lizes a series of tuples (st , at , rt , st+1 ) for updating the policy

nodes observe the behavior of its neighboring node. Beforeof an agent. However, diﬀerent from existing simple game-we describe the data collected for the normal and malicious

like learning tasks (e.g., Cart-pole game from OpenAI Gym

servers, we present the information related to the neighbor-（注 1）

ing node observed by each node (Table 1). Note that, inAtari game missions, which can utilize convolutional neu-

AODV, messages are transmitted by broadcasting, and thus

ral networks to conveniently extract features from game pic-

observing nodes can overhear the messages.

ture frames, the malicious server does not have an existing

4. 2 Smart grey hole attack model

method to obtain state and reward from monitoring data.

Here we describe our smart grey hole attack. This novelThus, we design a method that obtains the state and reward.

attack model is based on reinforcement learning. Malicious

State engineering. A malicious node observes a state, thennodes equipped with this attack model aim at disturbing theselects an action according to the decision of the malicious

routing and data transmission of a mobile WSN while notserver. To enable the malicious nodes to understand its sit-

being detected.

uation, we design the state (as a vector to input into neural), which have obvious state and reward declaration, and4. 2. 1 Ordinary grey hole attack model

networks) for the malicious nodes by extracting information

Before we illustrate our smart grey hole attack model, wefrom the most recent received packets. In particular, we ex-

first describe the details of ordinary grey hole attack model.

tract packet category ID and neighboring node ID. Packet

The ordinary grey hole attack model contains mainly two

category ID shows the category of this received packet (e.g.,parts. i) When a grey hole malicious node receives an RReq,

RReq is 1 and RRep is 2), and node ID represents that ID

it may perform normally (follow the routing protocol of thisof the node who sends this packet. Note that the packet

mobile WSN), or reply an RRep with one hop count, whichmeans that the malicious node states that it is near the des-

（注 1）：https://gym.openai.com/

Network environment

Table 2 States of malicious nodes
Definition

Malicious node receives a packet

States

Received packet:

Packet category

Packet category ID

Node ID

received packet category ID

Time

...

Hop count

One-hot vector of most recently

Neighbor nodeOne-hot vector of most recently
received neighboring node ID

Node ID : k. Node number: n.

ActNeiRatio

1/ActNei

One-hot embedding vector of Node ID:

ReqTotSenRatio

TReqSen / TPckSen

RepTotSenRatio

TRepSen / TPckSen

RerTotSenRatio

TRerSen / TPckSen

HelTotSenRatio

THelSen / TPckSen

DatTotSenRatio

TDatSen / TPckSen

ReqTotRecRatio

TReqRec / TPckRec

RepTotRecRatio

TRepRec / TPckRec

RerTotRecRatio

TRerRec / TPckRec

HelTotRecRatio

THelRec / TPckRec

category ID and the node ID are not numeric data, but cat-

DatTotRecRatio

TDatRec / TPckRecegorical data. For instance, compared with node ID 3, node

ReqRecTotSenRatio

TReqRec / TPckSen

k
0

0

0 … 0

1

0

0 … 0

0

0

n

Figure 2 One hot encoding of node ID

RepRecTotSenRatio TRepRec / TPckSen

ID 2 and node ID 8 have just the same diﬀerences. However,
if we just utilize these node IDs (2, 3, 8) as input of the neural networks, the neural networks of malicious server treat 2

RerRecTotSenRatio

TRerRec / TPckSen

HelRecTotSenRatio

THelRec / TPckSen

DatRecTotSenRatio

TDatRec / TPckSen

more similar to 3 than 8 [7]. As a result, we employ one-hot

ReqSenTotRecRatio

TReqSen / TPckRec

encoding [8] to encode the node ID and packet category ID,

RepSenTotRecRatio TRepSen / TPckRecfor the purpose of appropriately processing categorical data.

RerSenTotRecRatio

TRerSen / TPckRec

Figure 2 illustrates our one-hot encoding for node ID. In a

HelSenTotRecRatio

THelSen / TPckRecmobile WSN of n nodes, the node ID k of a received packet

DatSenTotRecRatio

TDatSen / TPckRec

is transformed into a vector with length n, with the k − thdimension of the vector is 1, and the rest is 0. The one-the values of these processed ratios are all between [0, 1],

hot encoding of packet category ID is the same as that ofwhich means our state design does not need normalization,

the node ID. A malicious node can learn which neighbor-and can converge rapidly.

ing node sends this packet, and learns the behaviors of its
neighboring nodes from these two settings.
However, the malicious nodes cannot recognize their sit-

We finally concatenate one-hot vector of packet category
ID, one-hot vector of neighboring node ID, and extracted
ratios together as our state vector.

uations clearly based only on these two states, because theReward shaping. The design of reward in reinforcement

received packet category ID and neighboring node ID may

learning is important because it describes how the agent

always be similar during the training simulation. Therefore,

ought to behave. Therefore, we explicitly design the reward

we also extract ratios that can well describe the malicious

for the malicious server.

nodes’ situations. Table 2 presents a summary of states. In

Our smart grey hole attack model aims at i) dropping data

this table, the first and second lines are the one-hot vectors

packets while ii) trying not to be detected. The result ofof packet category ID and neighboring node ID. Then wedropping data packets can be represented by how much thecalculate the connection between sent and received numbers

transmission rate (total # of data packets received by desti-of each category of packet and total sent and received num-nation node/total # of data packets sent by source node) is

bers. These ratios can not only represent the node density

lowered, and the result of trying not to be detected can be(ActNeiRatio) near a malicious node, but also can illustrate

represented as how long the malicious nodes keep not being

the packets sent and received frequency of a particular cat-detected. As a result, the following rewards for shaping theegory of packet. For example, if a malicious node sends too

reward function of the malicious server are considered.

many RReps, its RepTotSenRatio is extremely high. Under•

Positive reward for lowering the transmission rate.

such a situation, it will be soon detected by some machine•

Positive reward for not being detected before a pre-learning based detecting method. Consequently, this mali-

defined time threshold.

cious node will try to send fewer times of RReps not to be•

detected. Moreover, another advantage of the state is that

Although we form the principle of our reward function, set-

Negative reward for being detected.

ting magnitudes and frequencies of rewards is diﬃcult, andcollects data (features and categories of neighboring nodes)

they often depend on applications. For instance, in Atari

from all the normal nodes, then the normal server uses thisPong game, the rewards are bounded by −1 and +1, whiledata to update itself.

in Atari Mr. Pac-Man eating a single ghost can yield a re-As the features of the normal server, we extract inherent

ward of up to +1600. To overcome this, we employ clippedfeatures by the method in [10]. We also add another feature,

reward [18], where if the malicious nodes are all detected be-

the ActNeiRatio, which is calculated by 1/ActNei, to enable

fore the pre-defined threshold, the reward is −1. Otherwise,

the normal nodes to figure out the number of surrounding

the reward is +1. For lowering the transmission rate, clipping

active neighboring nodes. Note that we utilize SMOTE [5]all the rewards to +1 cannot show the decrease of transmis-

to over-sample the minority class so that the proportion ofsion rate, so we bound the reward, as 1 − transmisson rate,

training instances from each class is equal.

to reduce imbalance. We then add the positive and negative

4. 3. 3 Pre-training of normal classifier

reward as a final reward.

In our method, we train a neural network classifier and uti-

Note that in our setting, we utilize the sparse reward func-

lize it to detect malicious nodes. However, if we randomly

tion. That is, the malicious server does not receive any non-initialize the neural network classifier, it will have low accu-

zero reward until an episode is done. Note that an episode isracy and may converge slowly. Therefore, before the start ofdone when i) all the malicious nodes are detected or ii) the

training phase, for the purpose of fast adaption and accel-

simulation time goes over the threshold. We finally summa-

erating training speed, we pre-train the classifier with data

rize our reward:


0





 1 + 1 − transmissionratereward =not detected)−1 + 1 − transmissionratedetected)

obtained in diﬀerent network environments. In particular,(not end)(end &

we run simulations with normal nodes and malicious nodes
with ordinary grey hole attack models to obtain data. Then
we utilize the data to construct a pre-trained classifier. Fi-(end &

nally, we run the training phase on this pre-trained classifier
instead of a randomly initialized classifier.

4. 3 Countermeasure

5. Experiments

This section describes our countermeasure for smart grey

5. 1 Setting

hole attack model.

We used Qualnet 7.4 network simulator（注 2）. Each node

4. 3. 1 Overview of countermeasuretransmitted messages and data packets, whose payload sizes

In our countermeasure, we aim at building a robust clas-

were 256 bytes, using an IEEE 802.11b device. The commu-

sifier to detect smart grey hole malicious nodes. This neuralnication range of each node was adjusted to roughly 100 me-

network classifier in the normal server is built to classify

ters, and the network bandwidth was 11Mbps. As with exist-

the neighboring nodes of each normal nodes. To obtain thising works [9], [12], we used the random way point model [4],classifier, in the training phase, the classifier is trained si-

with a maximum movement speed of vmax and pause time

multaneously when the malicious server is updated. That is,of 0. (The velocity of each node was randomly chosen fromin the large number of episodes in the training phase, the(0,vmax ].) When there were n nodes in a network, there wereclassifier is also trained in each episode. Finally, a trainedn · m (m ∈ [0.1, 0.4]) malicious nodes in the network. The

classifier is obtained after the training phase.

normal nodes contact with the normal server to detect neigh-In the test phase, we install this trained normal server onboring malicious nodes and update the classifier in every 0.4

each normal node, and normal nodes utilize majority votes

seconds. The threshold time for giving positive was set as

to decide the category of a neighboring node. For instance,

10 seconds. We randomly chose a pair of source node andif one node a has 3 neighboring normal nodes, and two ofdestination node every f seconds. If the source node has an

the normal nodes classify this node as malicious node, thenactive route to the destination node, the source node sends

this node a will be decided as malicious node.

a data packet to the destination node directly. Otherwise

4. 3. 2 Frequent adaption in the training phasethe source node broadcasts an RReq to find a route to the

In the training phase, because the malicious nodes employ

destination node. The network parameters are described inreinforcement learning to adjust their behavior pattern, theTable 3.

normal nodes, simultaneously, need to update their server

Evaluation method. We run simulation of 1500 episodes

frequently. We thus make the normal nodes connect to thein the training phase. We trained both malicious and nor-normal server to update the classifier and detect malicious
nodes frequently. Before each adaption, the normal server（注 2）：http://web.scalable-networks.com/qualnet-network-simulator-software

Table 4 Detection timeP-detection MLP

Table 3 Parameter configuration
Parameter

Values20

P-attack [sec]

10.4

28.8

m of training

0.1, 0.2, 0.4

O-attack [sec] 2.4

9.6

m of testing

0.3

vmax [m/sec]

4.0nTable 5 Results on O-attack
P-detection MLP

Network size [m2 ] 300 × 300
f [sec]0.2

Accuracy

0.95

0.85

Detection rate

0.94

0.86

Mis-detection rate 0.02

0.14

mal servers simultaneously with random initial node positions and random node IDs. We also selected m from 0.1,
0.2, and 0.4 for each episode in the training phase, while intest phase, m was 0.3.
To investigate the eﬀectiveness of the proposed counter-

Table 6 Results on P-attack
P-detection MLP
Accuracy

0.76

0.58

Detection rate

0.88

0.60

Mis-detection rate 0.23

0.41

measure (P-detection for short), we used MLP [16] as acompetitor.

This is a state-of-the-art technique utilizing

MultiLayer Perceptron for classification. The method employs the following features: TReqRec, TReqSen, TRepRec,

Table 7 Decrease of transmission rate
Avg first 50 episodes Avg last 50 episodes
Transmission rate 71.7

64.7

TRepSen, TRerRec, TRerSen, TDatRec, TDatSen（注 3）, number of neighboring nodes (NeiNum), ratio of routing tablemuch faster than MLP. This is because MLP employs diﬀer-

update with respect to entries (PCR), and ratio of routing

ent features from the P-detection. In particular, MLP onlytable update with respect to hop counts (PCH). Note thatutilizes numbers of packets, which is hard for the neural net-

we also update MLP in the same way as the proposed coun-works to process when the numbers grow larger. However,termeasure.

our countermeasure utilizes the ratios, and the neural net-

Furthermore, to investigate the eﬀectiveness of our pro-works can process them more smoothly with gradient de-posed grey hole attack model (P-attack for short), we com-scent. As a result, it is obvious that our features can de-pared P-attack with ordinary grey hole attack model (O-scribe the malicious behaviors better. Moreover, comparedattack for short) with a malicious percentage of 50%.

with O-attack, normal nodes spend much more time to de-

Criteria. We focus on the following criteria to measure thetect P-attack. This is because malicious nodes of P-attack

performance of MLP and P-detection.

can learn from the detecting method and change their be|Tnor→nor,mal→mal |,|T |haviors, and they are hardly detected. We can see from thiswhere Tnor→nor,mal→mal and T are respectively the set ofresult that P-attack can exist longer than an O-attack withcorrectly classified instances and the set of all instances.

a detecting method employed by the mobile WSN.

•

Accuracy: This is represented by|Tmal→mal |,|Tmal |

Accuracy, detection rate, and mis-detection rate. Ta-

where Tmal→mal and Tmal are respectively the set of cor-bles 5 and 6 show the experimental results of accuracy, de-rectly classified instances describing malicious nodes andtection rate, and mis-detection rate of P-detection and MLP

the set of all instances describing malicious nodes.

on both P-attack and O-attack.

•

Detection rate: This is represented by

In Table 5, we can see|
Mis-Detection rate: This is represented by |Tnor→mal,|Tnor |

that our P-detection has good performance for detecting

where Tnor→mal and Tnor are respectively the set of wrongly

O-attack, and outperforms MLP. This is because O-attack

classified instances describing normal nodes and the set of

cannot adaptively change its performance ratio, while P-

all instances describing normal nodes.

detection learns its behavioral pattern rapidly. MLP also has••

Detection time: This shows how long time normal

good result because although MLP utilizes packet numbers,

nodes spend to detect all the malicious nodes in a mobile

it can still learn from the training data when the numbers

WSN.

are small. In Table 6, P-detection outperforms MLP on all

5. 2 Result

the three criteria with regard to detecting P-attack. How-

Detection time. Table 4 shows the detection time of P-

ever, compared with results in Table 5, performance of P-

detection and MLP for P-attack and O-attack. For both thedetection and MLP decrease. This is because P-attack tries

attack models, P-detection detects all the malicious nodes

to update itself by changing behaviors. These results showthat P-attack can actually learn to avoid being detected, and（注 3）：MLP does not convert these listed features to ratios, which

diﬀer from our method.

our proposed detection method is eﬀective and can be em-ployed for detecting this form of attack in mobile WSN.
Decrease of detection rate. To show the performance of[9]

P-attack, we present the decrease of transmission rate of mobile WSN in the training phase (Table 7). This table shows
the result of the average transmission rate of the first and[10]

the last 50 episodes. From this result, we can see that thetransmission rate drops after the training phase. This resultshows that P-attack can learn to disturb the transmission of[11]data packet in mobile WSNs.
[12]

6. Conclusion
This paper presented a smart grey hole attack model and[13]

its countermeasure in mobile WSN. We constructed a reinforcement learning based attack model and detect it by our[14]

adaptive server. The experiment results revealed that theattack can learn from the state-of-art countermeasures and[15]

extend the lifetime of malicious nodes. Our countermeasure,
as well, outperformed the state-of-art and detected the malicious nodes rapidly. As a part of future work, we plan to[16]

investigate on super all-round attack with not only the greyhole attack function, but can harm a whole mobile WSN in[17]

diﬀerent ways, and we likewise plan to design a countermeasure of it.
[18]

Acknowledgment
This research is partially supported by JSPS Grant-in-Aid
for Scientific Research (A) Grant Number JP26240013.

[19]
[20]

[21]

References[1] S. Abbas, M. Merabti, D. Llewellyn-Jones, and K. Kifayat.
Lightweight sybil attack detection in manets. IEEE Systems
Journal, 7(2):236–248, 2013.
[2] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong,
P. Welinder, B. McGrew, J. Tobin, O. P. Abbeel, andW. Zaremba. Hindsight experience replay. In NIPS, pages
5048–5058, 2017.
[3] H. Ayatollahi, C. Tapparello, and W. Heinzelman. Reinforcement learning in mimo wireless networks with energy
harvesting. In ICC, pages 1–6, 2017.
[4] C. Bettstetter, H. Hartenstein, and X. Pérez-Costa.
Stochastic properties of the random waypoint mobility
model. Wireless Networks, 10(5):555–567, 2004.
[5] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P.
Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321–
357, 2002.
[6] X. Chen, K. Makki, K. Yen, and N. Pissinou. Sensor network security: A survey. IEEE Communications Surveys
and Tutorials, 11(2):52–73, 2009.
[7] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau,
F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase
representations using rnn encoder–decoder for statisticalmachine translation. In EMNLP, pages 1724–1734, 2014.
[8] A. Coates and A. Y. Ng. The importance of encoding ver-

[22]

[23]

[24]

[25]

[26]

[27]

sus training with sparse coding and vector quantization. In
ICML, pages 921–928, 2011.
W. Galuba, P. Papadimitratos, M. Poturalski, K. Aberer,
Z. Despotovic, and W. Kellerer. Castor: Scalable securerouting for ad hoc networks. In INFOCOM, pages 1–9,
2010.
B. Gao, T. Maekawa, D. Amagata, and T. Hara.
Environment-adaptive malicious node detection in manetswith ensemble learning. In ICDCS, pages 556–566, 2018.
S. Gu, T. Lillicrap, I. Sutskever, and S. Levine. Continuous
deep q-learning with model-based acceleration. In ICML,
pages 2829–2838, 2016.
B. Karaoglu and W. Heinzelman. Cooperative load balancing and dynamic channel allocation for cluster-based mobile
ad hoc networks. TMC, 14(5):951–963, 2015.
C. Karlof and D. Wagner. Secure routing in wireless sensor
networks: Attacks and countermeasures. In SNPA, pages
113–127, 2003.
Y. Li, L. Xiao, H. Dai, and H. V. Poor. Game theoretic
study of protecting mimo transmissions against smart attacks. In ICC, pages 1–6, 2017.
X. Lu, X. Wan, L. Xiao, Y. Tang, and W. Zhuang. Learningbased rogue edge detection in vanets with ambient radio
signals. In ICC, pages 1–6, 2018.
A. Mitrokotsa and C. Dimitrakakis. Intrusion detection in
manet using classification algorithms: The eﬀects of costand model selection. Ad Hoc Networks, 11(1):226–237, 2013.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves,
I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing
atari with deep reinforcement learning. arXiv preprint
arXiv:1312.5602, 2013.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K.
Fidjeland, G. Ostrovski, et al. Human-level control throughdeep reinforcement learning. Nature, 518(7540):529, 2015.
C. E. Perkins and E. M. Royer. Ad-hoc on-demand distancevector routing. In WMCSA, pages 90–100, 1999.
E. Rodrigues Gomes and R. Kowalczyk. Dynamic analysis of multiagent q-learning with ε-greedy exploration. In
ICML, pages 369–376, 2009.
N. Schweitzer, A. Stulman, R. D. Margalit, and A. Shabtai. Contradiction based gray-hole attack minimization for
ad-hoc networks. TMC, 16(8):2174–2183, 2017.
J. Sen, M. G. Chandra, S. Harihara, H. Reddy, and P. Balamuralidhar. A mechanism for detection of gray hole attack
in mobile ad hoc networks. In ICICS, pages 1–5, 2007.
S. Shahabi, M. Ghazvini, and M. Bakhtiarian. A modified algorithm to improve security and performance of aodv
protocol against black hole attack. Wireless Networks,
22(5):1505–1511, 2016.
S. K. Shandilya and S. Sahu. A trust based security scheme
for rreq flooding attack in manet. International Journal ofComputer Applications, 5(12):4–8, 2010.
T. Shu and M. Krunz. Privacy-preserving and truthful detection of packet dropping attacks in wireless ad hoc networks. TMC, 14(4):813–828, 2015.
F.-H. Tseng, L.-D. Chou, and H.-C. Chao. A survey of blackhole attacks in wireless mobile ad hoc networks. Humancentric Computing and Information Sciences, 1(1):4, 2011.
L. Xiao, Y. Li, C. Dai, H. Dai, and H. V. Poor. Reinforcement learning-based noma power allocation in the presenceof smart jamming. TVT, 67(4):3377–3389, 2018.