Master’s Thesis

Title

Anomaly Detection in Smart Home Network
by Inspecting Exceptional Operations on IoT Devices
using User’s Daily-Life Behavior Learning

Supervisor
Professor Masayuki Murata

Author
Masaaki Yamauchi

February 8th, 2019

Department of Information Networking
Graduate School of Information Science and Technology
Osaka University

Master’s Thesis

Anomaly Detection in Smart Home Network
by Inspecting Exceptional Operations on IoT Devices
using User’s Daily-Life Behavior Learning
Masaaki Yamauchi

Abstract
Many devices, such as air conditioners, heaters, and refrigerators, are now being connected to the Internet. These devices are called “IoT” devices. As the number of these
IoT devices increases, they have become targets of cyberattacks. Especially, the operations by attackers can cause serious problems, which may harm users or make users unsafe.
However, such attacks are diﬃcult to detect using security software or IDS, because theattackers use the same protocol as legitimate operations by users.
In this thesis, we propose a method to detect such attacks based on user behavior. We
model user behavior as a sequence of user’s events, which includes the operation of IoT
devices and other behavior monitored by any sensors such as user’s entering and leaving.
Our method learns the sequences of events for each one of a set of conditions predefinedby the time and sensor values such as room temperature and humidity. After learning, ourmethod detects attacks by comparing the sequences of the events including the current
operation with the learned sequences.
For the evaluation of our method, we constructed a network of home IoT devices in ourlaboratory. We selected four subject students, let them use the devices as they like, andcaptured all packets of the home network. We obtained times of users’ entering/leaving
and operating the home IoT devices, using the captured packets, and evaluated our method
using the obtained data. When the method observed sequences of events related to theoperations, our method detected around 95–100% of anomalous operations with less than
23% of misdetection ratio. Based on the results, we demonstrate the accuracy of our
method and discuss the limitations of our method.
1

Keywords
Anomaly Detection
IoT
Security
Smart Home
Behavior Pattern
Operation by Attackers

2

Contents
1 Introduction

7

2 Anomaly Detection Based on User Behavior

9

2.1

2.2

2.3

2.4

Learning Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.1.1

Condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.1.2

User Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Learning User Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.1

Generation of Event Sequences . . . . . . . . . . . . . . . . . . . . . 11

2.2.2

Selection of Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.2.3

Updating the Tree of Event Sequences . . . . . . . . . . . . . . . . . 12

Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3.1

Generation of Event Sequences . . . . . . . . . . . . . . . . . . . . . 12

2.3.2

Decision-Making Method . . . . . . . . . . . . . . . . . . . . . . . . 13

Parameter Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.4.1

Setting the Parameter T . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.4.2

Setting the Parameters αi and nd . . . . . . . . . . . . . . . . . . . . 16

3 Evaluation

21

3.1

Evaluation Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.2

Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.3

Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.3.1

Misdetection Ratio . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.3.2

Detection Ratio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.4

Hyperparameters Setting

3.5

Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.6

. . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.5.1

Evaluation Results for One Month . . . . . . . . . . . . . . . . . . . 26

3.5.2

Evaluation Results for Three Months . . . . . . . . . . . . . . . . . . 28

Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4 Conclusion and Future Work

31

3

Acknowledgments

32

References

34

4

List of Figures
1

Overview of Anomaly Detection for Smart Home Based on User Behavior .

2

Overview of the Detection Method . . . . . . . . . . . . . . . . . . . . . . . 10

3

Generated Event Sequences when Events A, B, and C Occur . . . . . . . . 11

4

Experimental Home Network Environment . . . . . . . . . . . . . . . . . . . 21

5

9

List of Tables
1

List of Deployed IoT Devices . . . . . . . . . . . . . . . . . . . . . . . . . . 22

2

List of Deployed IoT Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3

Hyperparameters for Evaluation4

Detection Results for January 2017 . . . . . . . . . . . . . . . . . . . . . . . 26

5

Detection Results for April, June, and August 2017. . . . . . . . . . . . . . 28

6

Detection Results for May, July, and September 2017. . . . . . . . . . . . . 28

. . . . . . . . . . . . . . . . . . . . . . . . 25

6

1

Introduction

Recently, home appliances and healthcare devices, such as refrigerators, air conditioners,
and pacemakers, have started to be connected to the Internet in addition to personal
computers and smartphones. These devices are called “IoT (Internet of Things)” devices.
Users can obtain information from the IoT devices and can operate the IoT devices withusing a smartphone, a tablet, or an AI speaker, such as Google Home [1] or Amazon
Echo [2].
Currently, seven billion IoT devices are connected to the network. 215 billion IoT
devices will be connected to the network in 2025 [3]. As the number of devices connectedto the Internet increases, the risk that these devices become the target of cyberattacks isincreasing [4–7] and, in fact, direct attacks and malware targeting IoT devices [8, 9] havealready been observed.
Most of the current attacks targeting IoT devices are designed to compromise IoT
devices to create botnets [10, 11]. Such attacks may be detectable by methods basedon an analysis of the behavior of the attackers [12–14] or in comparison with legitimate
usage [15, 16].
However, because IoT devices are closely related to everyday life, there is a risk ofattacks having an immediate and personal eﬀect on users [17]. In particular, the operationof IoT devices by attackers may make the users unsafe, and may even harm them, bychanging the set temperature of an air conditioner, the setting of a healthcare device, orsimilar. In addition, in a case that high wattage IoT devices simultaneously operated,
power demand would change greatly and the change may cause a large scale blackout [18].
Therefore, methods to detect and prevent operations initiated by attackers are necessary.
Conventionally, security software and intrusion detection systems are used to detectcyberattacks [19]. They detect attacks by pattern matching, comparing the packets withpredefined rules or detecting outliers of statistical information of observed traﬃc. However,
because packets sent by attackers to operate IoT devices are the same as the packets sentby legitimate users and packets of operating IoT devices are small, existing methods havediﬃculty in detecting anomalous operations. In particular, if an attacker sends packets
via the compromised smartphone of a legitimate user, the packets cannot be distinguished7

from packets sent by legitimate users.
In this thesis, we propose a method to detect the anomalous operation of home IoT
devices based on learning users’ behavior related to the operation of IoT devices. In thismethod to detect the anomalous operation, a home gateway that can monitor all packets
between the home network and the Internet learns users’ behaviors and detect anomalous
operations. A home gateway that connects to the home IoT devices, home IoT sensors
and smartphones can collect environment information in home obtained from sensors suchas room temperature and humidity. Also, a home gateway can get information whether
a user is at home based on the information, using information that whether a user’ssmartphone is connected to the home network [20]. In addition, a home gateway cangrasp operations of home IoT devices from the home and via the network, because thehome gateway relays packets that operate the home IoT devices via the Internet and fromhome smartphone. Using such information, a home gateway learns the behaviors of users
for each condition defined by the time of day and the information observed by the sensors
in the home network. Then, when a command arrives, the gateway checks whether itmatches the learned behavior. If the command does not match the learned behavior, thegateway classifies the command as an anomaly.
The rest of this thesis is organized as follows. We describe our method to detect
anomalous operations in section 2. Then, we evaluate our method in section 3. Finlay, we
conclude this work and discuss future issues in section 4.

8

2

Anomaly Detection Based on User Behavior

Legitimate Operation
1. Operate
DeviceAnomalous Operation
5. Drop or not

User’s Tablet

2. Collect
Operation Info.
and Sensor ValueONON

4. Judge
Anomaly or
Legitimate
3. Matchingwith Learned
Behaviors
LearnerSmart Home

Compromised Tablet

Home Gateway

Figure 1: Overview of Anomaly Detection for Smart Home Based on User Behavior
We propose a method to detect the anomalous operation of home IoT devices byattackers, shown in Figure 1. It is based on that users have their own patterns of behavior
depending on conditions. For example, when a user returns home and feels cold, theyturn on a heater first and then turn on a humidifier; on the other hand, when a user feels
warm, they never turn on a heater. In addition, the sequence of operations reflects thecharacteristics of users: one user first turns on a heater and then turns on a humidifier,while another user first turns on a humidifier.
Our method learns such condition-depend operation sequences as behavior patterns
and classifies deviations from these learned patterns as anomalous operations.
In this section, we first explain the model of user behavior used in our method. Then,we explain how user behavior is learned and how anomalous operations are detected.

9

Learning Model

8VHU(QWHU

7HPSHUDWXUH

0DWFKHGWLPHV
2SHUDWH
'HYLFH$WLPHV
2SHUDWH'HYLFH& WLPHV
2SHUDWH'HYLFH%

1RW
0DWFKHG/HDUQHG(YHQW6HTXHQFHV

1R'HWHFWLRQ

2SHUDWH
'HYLFH&

7LPHRI'D\&RQGLWLRQV

8VHU(QWHU
2SHUDWH
'HYLFH&

0DWFKLQJ

2.1'HWHFWLRQ
2SHUDWH
'HYLFH$0RQLWRUHG(YHQW6HTXHQFHVFigure 2: Overview of the Detection Method
Figure 2 shows the model of user behavior in our method.
2.1.1

Condition

We define a condition as a combination of the time of day and values obtained from sensors.
The variables representing the various components of a condition will be denoted ci , where
the index i runs from 1 to some maximum value imax . For example, as in Figure 2, c1
could represent time of day, while c2 represents the temperature of a room, and so on.
Of course, continuous data have to be discretized for tractability. So, we use multiple(j)thresholds for each type of value. A value of ci that satisfies ci(j+1)≤ ci ≤ ci(j)where ciis the jth threshold for the ith variable, is classified into jth region for that variable.
2.1.2

User Behavior

We define an event as any monitored behavior of a user, including the operation of IoT
devices and any other behavior monitored by the sensors, such as entering or leaving aroom. We define an event sequence to be a sequence of events that occurs within T seconds
of a previous event. Our method learns a user’s behavior by learning the event sequences
for each condition. The set of event sequences corresponding to a condition is modeled as
multiple trees whose roots are the first events and whose leaves are the last events. We

10detect anomalous event sequences by checking whether the sequence is included in trees.
If the sequence is not included in the tree, we classify the sequence as an anomaly.

2.2

Learning User Behavior

Our method learns user behavior from observed information by monitoring events in thehome and generating event sequences. Then, the condition corresponding to an event
sequence is selected from the predefined set. Finally, the observed sequence is used to
update the model for the corresponding condition.
2.2.1

Generation of Event Sequences

Our method regards events observed within T seconds of one another as belonging to thesame event sequence, constructing the sequence at the home gateway. However, this meansthat some event sequences may include events from more than one user because a typicalsmart home has multiple users. To remove these events, which should be treated as noise,and learn the essential event sequence, we generate multiple event sequences by removingsome events from the observed sequence. Figure 3 shows an example of the generationof event sequences. By considering these event sequences, we discover the essential event
sequences that include only the events generated by a single user.

$*HQHUDWHG
HYHQW
VHTXHQFHV

%

$

$

%
&

&

%

$

%

&

&

%

$

$%

$&

%&

&
5HPRYHG HYHQWV 1RQH

Figure 3: Generated Event Sequences when Events A, B, and C Occur

11

2.2.2

Selection of Conditions

Our method selects the condition whose corresponding model is to be updated basedon the condition of the first event in a sequence. However, to make eﬀective use of event
sequences and learn users’ behaviors even when only a small number of event sequences foreach condition are observed, we update not only the model corresponding to the conditionof its first event but also the models for similar conditions. When the condition of the{}root
first event is croot
1 , . . . , cimax , the sequence is used to update the model for the region forwhich the variable ci satisfies croot− αi ≤ ci ≤ croot+ αi for some value αi , which can beiidiﬀerent for each i.
2.2.3

Updating the Tree of Event Sequences

When the home gateway observes an event sequence, we add nodes and links to the treecorresponding to the selected conditions so that the event sequence is included in thetree, the first event of the sequence is the root, and the final event of the sequence isthe leaf. Then, we increment a counter for each link on the route corresponding to theevent sequence. By repeating this procedure, the count for links related to frequent event
sequences increases. Thus, we detect anomalies by using the pruned tree which contains
only the links whose counters exceed a threshold nd × Lnum where nd is a parameter, d isthe depth of the link, and Lnum is the total number of learned operations for the device
targeted for detection. By doing so, we eliminate the noise included in the event sequence.

2.3

Detection

When the home gateway observes a new operation, it generates event sequences related
to the operation. Then, it compares these sequences with learned behaviors.
2.3.1

Generation of Event Sequences

When the home gateway observes a new operation, it generates multiple event sequences,in the same way as during learning, by removing some events from the sequence of events
that occurred within T seconds after previous event and uses these to check whether theoperation is anomalous.

122.3.2

Decision-Making Method

We check whether the new operation is anomalous or not by comparing the generated
event sequences with the learned behaviors. We first select the condition corresponding
to the first operation of the generated sequences. Then, we compare each of the generated
sequences with the learned sequences for the selected condition by finding the events inthe tree. First, we find the tree whose root is the first event of the event sequence. Then,we find the node that is connected to the found root and is the second event of theevent sequence. We repeat searching for nodes until nodes corresponding to all events inthe sequence are found or no corresponding nodes are found. We repeat the check for
all generated event sequences using the above steps. If all sequences have events whose
corresponding nodes are not found in the learned behavior, we classify the operation as
an anomaly. On the other hand, if corresponding nodes are found for all events in one of
the sequences and the node corresponding to the final event is the leaf, we decide that all
the events in the sequence are legitimate. If corresponding nodes are found for all eventsin one of the sequences but the node corresponding to the final event is not the leaf, we
wait for the next event. If the next event does not occur for T seconds, we finish searchingfor the sequence. If the next event occurs, we generate event sequences including the next
event and compare the sequences with learned behaviors using the same steps as describedabove.

13

2.4

Parameter Setting

In this section, we describe the method used to set parameters. Our method has threekinds of parameter, T , αi , and nd . T aﬀects the generation of event sequences, while αi andnd aﬀect the model used for the detection. Thus, we set the parameter T independently
from the other parameters so as to generate suitable event sequences. To set parameters,
we use events monitored for P weeks. The information on the monitored events includes
the kind of event, the time, and the sensor values defining the conditions corresponding
to the events.
2.4.1

Setting the Parameter T

First, we set the parameter T . A user may operate multiple devices simultaneously or
within a small interval of time. In this thesis, we set T so that the operations of such
related devices are included in the event sequences.
Algorithm 1 shows the steps used to set T . In these steps, T is incremented until
the generated event sequences become stable. We check whether the generated event
sequences are stable by counting the number of pairs of events included in the same
event sequence using Algorithm 2. The number of pairs of events included in the same
event sequence (NumOfPair(T )) initially increases with T . However, the number becomesstable when T is large enough that the operations of related devices are included in thesame sequences. Algorithm 2 regards the generated event sequences as stable when thediﬀerence between NumOfPair(T ′ ) and NumOfPair(T ′ + Tinc ) is less than N for all T ′stable , where T stable is a preset parameter.
satisfying T ≤ T ′ < T + Tincinc

14

Algorithm 1 Setting the parameter T
Input: P : Period of the parameter setting (weeks)
Output: T
T ⇐ Tinit
while IsNumberOfSequencesStable(T ) == False do
T ⇐ T + Tincend while
return T

Algorithm 2 Whether generated event sequences are stable
Input: T
Output: True | False
function IsNumberOfSequencesStable(T )T′ ⇐ T
stable dowhile T − T ′ ≤ Tinc

T ′ ⇐ T ′ + Tinc
if NumOfPair (T ′ + Tinc ) - NumOfPair (T ′ ) ≥ N thenreturn False
end ifend while
return Trueend function

15

2.4.2

Setting the Parameters αi and nd

After setting the parameter T , we set the parameters α and n, where α = {α1 , · · · , αimax }and n = {n1 , · · · , ndmax }. imax is the number of values used to define the condition,and dmax is a predefined value indicating the depth of the tree considered when setting
parameters. For the links whose depth d′ is larger than dmax , we use ndmax as nd′ .
The parameters α and n have impacts on the detection ratio (D), that is the proportion of anomalous operations detected as anomalies, and misdetection ratio (M ), thatis the proportion of legitimate operations detected as anomalies. In this thesis, we set
these parameters to obtain the highest detection ratio from among the values that given
an acceptable misdetection ratio, M ≤ M goal . In order to calculate the detection andmisdetection ratio, we perform the “Detection Test” shown as Algorithm 4. The detection
test calculates the detection and misdetection ratio when anomalous operations of IoT devices are added to event datasets by using cross-validation. At first, we use event datasetsto learn behaviors except for a certain day. Then, we add 100 anomalous operations in
the omitted day and count the number of detected anomalous operations and misdetected
legitimate operations. After doing this for each day, we sum up the numbers of both andcalculate the detection and misdetection ratio.
Algorithm 3 shows the steps followed to set parameters α and n, and we obtain optimal parameters by calling Optαn(Null, Null, k = 1, M goal ). This function finds optimal
parameter values by recursively calling itself with k incremented by 1. This function finds
optimal parameters by fixing the k ′ th parameters for k ′ < k. As αi becomes large, themisdetection ratio becomes small, because the observed event sequences are used to update a larger range of conditions and similar event sequences are regarded as legitimate
even if the condition is slightly diﬀerent from that of the learned sequence. Therefore, if
we cannot achieve a misdetection ratio less than M goal by setting αi = αi′ , an acceptablemisdetection ratio cannot be achieved for αi ≤ αi′ . Similarly, as nd becomes small, more
event sequences are used as legitimate behavior, and the misdetection ratio becomes small.
That is, if we cannot achieve a misdetection ratio less than M goal by setting nd = n′d , anacceptable misdetection ratio cannot be achived for nd ≥ n′d . Therefore, to reduce thecalculation time, we avoid considering such parameters in Algorithm 3.

16

If no parameters satisfy the target misdetection ratio, our algorithm cannot find suitable parameters. When such cases occurred in our evaluation, we increased the misdetection ratio until suitable parameters could be found.

17

Algorithm 3 Optimal α, n, and Detection-Misdetection Ratio
Input: αin , nin , k, M goal
Output: D, M, αbest , nbest
function Optαn(αin , nin , k, M goal )for 0 < k ′ < k do
if k ′ ≤ imax thenαk′ ⇐ αkin′elsenk′ −imax ⇐ nin
k′ −imax
end ifend for(Dbest , M best ) ⇐ (0.0, 1.0)
if k ≤ imax thenαk ⇐ αkmax
else
nk−imax ⇐ nmink−imax
end ifwhile True do
if k < imax + dmax then(D′ , M ′ , α′ , n′ )⇐Optαn(α, n, k + 1, M goal )else(D′ , M ′ ) ⇐ DetectionTest(α, n)(α′ , n′ ) ⇐ (α, n)
end if
if M ′ > M goal thenBreakelse if D′ > D then(Dbest , M best , αbest , nbest )⇐(D′ , M ′ , α′ , n′ )end if
if k ≤ imax then18

αk ⇐ αk − αkdec
if αk < αkmin thenBreakend ifelse
nk−imax ⇐ nk−imax + ninck−imaxif nk−imax > nmax
k−imax thenBreakend ifend ifend while
return (Dbest , M best , αbest , nbest )end function

19

Algorithm 4 Detection Test
Input: α, n
Output: D, M
function DetectionTest(α, n)for each Day ∈ Period of parameter setting do
Insert 100 anomaly operations into the Day
Learn users’ behavior without the Day with α
Detect the Day’s operations with n
end forSum up the results of each Day
return Calculate D, M
end function20

3

Evaluation

3.1

Evaluation Environment

Internet
Gateway
SwitchMirroring

Wireless
Access Point

Capturing
PacketOperating Tablet

Home IoT Devices

Figure 4: Experimental Home Network Environment

To evaluate our method, we constructed a network of home IoT devices in our laboratory, shown in Figure 4, and captured all packets. Because the “Gateway Switch” inFigure 4 relays all packets of home IoT devices, we can capture every operation packets
of home IoT devices. We deployed 13 kinds of connectable home appliances, shown in
Table 1 and nine kinds of sensors, shown in Table 2. We selected four subjects of studentsof the laboratory and let them use the home IoT devices as they liked for a month. In
addition, we ask them to record operating logs including time of entering and leaving thelaboratory and time of operating home IoT devices. We confirmed the times of opera-

21

Table 1: List of Deployed IoT Devices
DevicesNumber of Set Devices

Air Purifier

2

Automatic Cooker

1

Coﬀee Maker

1

Electric Fan

4

Heater1

Humidifier

1

Lighting Equipment

6

Microwave

1

Monitor Camera

1

Recorder4

Refrigerator

1

Robot Vacuum

1

TV

4

tions of home IoT devices and users’ entering/leaving the laboratory can be obtained from
captured packets, comparing the time of operations obtained from captured packets withlog recorded by subjects. We also captured sensor data. The sensor data includes thetemperature, humidity, noise values, and so on.

22

Table 2: List of Deployed IoT Sensors
Sensors

Number of Set Devices

CO2 Sensor

2

Humidity Sensor

3

Noise Sensor

1

Particulate Matter Sensor

1

Pressure Sensor

1

Rainfall Sensor

1

Temperature Sensor

3

VOC Sensor

1

Wind Sensor

1

23

3.2

DatasetWe recorded the times of operations of the home IoT devices from captured packets by
the increasing or decreasing of the number of packets of each device. From informationthat whether smartphones are connected to the home network, we generate the events
that a user enters or leaves the home. From these information of events and behavior
logs recorded by the user, we evaluated our method. However, the value of sensors that
includes temperature, humidity, and noise, did not change significantly in our environment.
Therefore, in this evaluation, only the time of day is used to define the condition.

3.3

Procedure

We evaluated three datasets; the dataset obtained for one month in January 2017, andthe datasets obtained for the three months of April, June, and August 2017 and the threemonths of May, July, and September 2017. In each period, the same subjects were using
the devices. The first dataset was obtained in the winter, and the subjects used the heater.
The other datasets were obtained from the spring to the summer, and the subjects did
not use the heater but used the electronic fans.
In this section, we use two metrics: the detection ratio and the misdetection ratio.
3.3.1

Misdetection Ratio

To evaluate the misdetection ratio in a straightforward way, we would need a suﬃcientnumber of legitimate operations in the test data. However, we have only a limited numberof legitimate operations in our collected data. Therefore, we evaluate the misdetection
ratio by Leave-One-Out Cross-Validation (LOOCV) [21]. LOOCV separates the dataset
into multiple small datasets and verifies one of them after training using the others. Afterverifying all patterns of the test dataset, we summarize the results.
For this thesis, we separated the data by day and used the data for one day as the test
data and used the data for the other days to learn legitimate behaviors.

24

3.3.2Detection Ratio

In our evaluation, we added anomalous operations into observed legitimate operations
to obtain the detection ratio. We used a strategy similar to that for the evaluation of
misdetection to obtain the detection ratio: we separated the dataset by day and used thedata for one day for the test and the remaining data for learning.
We added 100 anomalous operations into the test data for each day and tried to detectthem. Then, we obtain the detection ratio by counting the total number of detected
anomalous operations for all days and dividing by 100 × number of days of observations.

3.4

Hyperparameters Setting

When determining optimal parameter values, we set hyperparameters as Table 3, where
Pnum is the number of operations used for parameter setting for the device targeted for
detection.
Table 3: Hyperparameters for Evaluation
HyperparameterValue

M goal

0.10

stableTinit , Tinc , Tinc0, 10, 120
10 × PN
imax

1

α1min , α1dec , α1max

600, 600, 43200

dmax

2

inc max
nmin
1 , n1 , n10.00, 2/Pnum , 1.00

inc max
nmin
2 , n2 , n20.00, 1/Pnum , 1.00

25

3.5
3.5.1

Evaluation Results
Evaluation Results for One Month
Table 4: Detection Results for January 2017
Detection

Detected

Misdetection

Misdetection

Ratio/TotalRatio/TotalCoﬀee Maker

0.157

346/2200

0.000

0/48

Heater

0.959

2110/2200

0.182

2/11

Humidifier

0.080

176/2200

0.000

0/38

TV A

1.000

2200/2200

1.000

8/8

TV B

1.000

2200/2200

0.000

0/2

First, we used the dataset obtained in January 2017. In this evaluation, we set theparameters by using the data monitored in the first week. After setting parameters, weevaluate our method by LOOCV using the data monitored in the last three weeks. In thisevaluation, we also use the data monitored in the first week as training data to learn thelegitimate behaviors.
Table 4 shows the results of our evaluation in this case. Over 95% of anomalous
operations of the heater were detected. The high detection rate is achieved by using thelearned event sequences. Our method learned three patterns of event sequences related tothe heater; the patterns that a user operates a humidifier before operating a heater, a user
operates a heater before operating a coﬀee maker, and the user01 leaves the room after
operating a heater on the afternoon. Even if an anomalous operation command arrives,
we detect anomaly unless the operation matches the above-learned behaviors.
The number of misdetection of legitimate operations of the heater is two, which isslightly larger than the predefined target ratio M goal . The misdetected operations are
related to the event sequence in which a user operates the heater before operating thehumidifier. Such event sequences occurred only a few times in one month and our method
did not learn such a sequence as legitimate.
Table 4 shows that the detection ratios for operations of the coﬀee maker or the26humidifier are small. This is caused by single operation: for most of the operations of
the humidifier or the coﬀee maker, previous or subsequent events were not observed. In
addition, they were operated at various times of day. We set parameters so that suchsingle operations at various times of day cannot be detected as anomalies. As a result,most of such operations are regarded as legitimate.
Table 4 also indicates that all anomalous operations of TVs are detected, but themisdetection ratio for TV A is quite high. This is caused by the low number of operations
used to learn legitimate behavior. Furthermore, the monitored event sequences related to
TV A vary quite a lot. As a result, an event sequence cannot be learned as legitimate byits similarity to other event sequences. The misdetection ratio of TV B is small because
many event sequences are used for detection.

27

3.5.2

Evaluation Results for Three Months
Table 5: Detection Results for April, June, and August 2017.
Detection

Detected

Misdetection

Misdetection

Ratio/TotalRatio/TotalCoﬀee Maker

0.611

3908/6400

0.058

3/52

Electric Fan A

0.998

6384/6400

0.000

0/9

Electric Fan B

0.999

6399/6400

0.000

0/6

TV A

0.996

6377/6400

0.171

7/41

TV B

1.000

6400/6400

0.400

4/10

TV C

0.999

6397/6400

0.000

0/10

TV D

0.999

6398/6400

0.111

1/9

Table 6: Detection Results for May, July, and September 2017.
Detection

Detected

Misdetection

Misdetection

Ratio/TotalRatio/TotalCoﬀee Maker

0.057

368/6500

0.000

0/89

Electric Fan A

0.991

6439/6500

0.074

2/27

Electric Fan B

0.986

6409/6500

0.302

13/43

TV A

0.998

6485/6500

0.231

3/13

TV B

0.999

6497/6500

0.077

1/13

TV C

0.999

6496/6500

0.000

0/3

We also evaluated our method by using datasets for two sets of three months: April,June, and August 2017 and May, July, and September 2017. We used data monitored in
the first week of each month to set parameter values and to learn legitimate behaviors,
and then performed LOOCV using the remaining data in a similar way and with same
parameters as for the 1-month data.
Tables 5 and 6 show the results of the evaluation in these cases. Our method detected
more than 98% of all anomalous operations, except for those of the coﬀee maker. This is28because our method successfully learns legitimate event sequences that are repeated manytimes, such as the sequence that electronic fans are often turned on immediately after
entering the room.
Tables 5 and 6 show that the misdetection ratios for operations of the TVs are smaller
than for the dataset for one month. This is because we used more event sequences tolearn legitimate behavior. As a result, when a new legitimate operation arrives, theoperation can be regarded as legitimate, because similar event sequences have been learned.
However, even when we use more data, we cannot detect the anomalous operations of thecoﬀee maker accurately. In addition, the misdetection ratios for TV B in Table 5 andfor Electronic Fan B and TV A in Table 6 are also large. This is caused by their single
operation. Therefore, to improve the detection of anomalous operations and reduce themisdetection of the legitimate operations, we need a method to accurately identify suchsingle operations as legitimate, which is one of our future research topics.

29

3.6

Discussion

Our method could detect around 95–100% of anomalous operations if the events related tolegitimate operations can be monitored. However, our method also produced several misdetections. These are either single operations, for which related events are not observed,
or rare operations whose corresponding event sequences are rare.
Our method achieves accurate detection of anomalous operations by comparing event
sequences. However, we cannot utilize a sequence to check a single operation and thecondition of the operation is the only information we have to determine whether such an
operation is legitimate.
One approach to improving the accuracy of identification for single operations is todeploy sensors that can monitor events related to these operations. For example, single
operations of the humidifier occurred after the water tank became empty. If we canmonitor the water tank of the humidifier, we can obtain events related to its operation.
Consequently, such operations of the humidifier are no longer single operation. Anotherapproach to improving the accuracy of identification of legitimate single operations is touse more information to define the conditions. In our evaluation, we used only the timeof day to define the condition. However, if we define a condition in such a way as todistinguish conditions where legitimate operations are performed from other conditions,
we can identify legitimate single operations accurately. Methods to identify the legitimate
single operations are included in our future research topics.
The mitigation of misdetection of rare legitimate operations is another challenge, as
we cannot obtain a suﬃcient amount of training data to accurately identify such rare
operations in each home. One approach to obtaining more training data is to use data
monitored at other homes. However, several issues to be solved before this could be done.
One issue is the diﬀerence between homes; information about users whose behaviors whenusing the devices are diﬀerent may be useless. Thus, we need a method to obtain datafrom other homes that have users with similar behaviors. Privacy is another issue. Thatis, we need a method that uses the data from other homes without exchanging private
information. This is also one of our future research topics.

30

4Conclusion and Future WorkIn this thesis, we proposed a method to detect the anomalous operation of home IoT
devices. This method learns sequences of user behaviors for each condition that is definedby time of day, sensed temperatures, humidity and so on. Then, when an operationcommand arrives, the method compares the current sequence with the learned sequences
for the condition corresponding to the current condition. If they do not match, our methodclassifies the operation as an anomaly.
We constructed a network of home IoT devices in our laboratory and let four subjects
use the devices. We recorded the times at which the devices were operated along withsensor data. Using this data, we evaluated the detection and misdetection ratios for our
method. The results demonstrate that our method detects around 95–100% of anomalous
operations if events related to legitimate operations are observed. However, our method
cannot accurately identify single operations for which related events are not observed.
Rare legitimate operations are also diﬃcult to identify, which results in misdetection. The
development of methods to accurately identify rare or single operations is something wewill address in future research.

31

Acknowledgments
This thesis would not accomplish without a lot of great support from many people. First,
I would like to express my deepest gratitude to Professor Masayuki Murata of Osaka
University, for providing me with the opportunity to research with a talented team ofresearchers. His creative suggestions, insightful comments, and patient encouragement
have been essential for my research activity.
Furthermore, I would like to show my greatest appreciation to Associate Professor
Yuichi Ohsita of Osaka University. He devoted a great deal of time for me and gave me a
lot of advice about my research. Without his continuous support, my work would not be
accomplished. In addition, he enthusiastically taught me the way of thinking and writing.
They must be invaluable skills in the future of my life.
Moreover, I would like to show my appreciation to Associate Professor Shin’ichi Arakawa,
Assistant Professor Daichi Kominami, Assistant Professor Naomi Kuze, and Specially Appointed Assistant Professor Tatsuya Otoshi of Osaka University for appreciated comments
and support. Their kindnesses on my behalf were invaluable, and I am forever in debt.
I am grateful to Dr. Yoshiaki Kato, Dr. Akira Ishihara, Dr. Takeshi Yoneda, Dr.
Shotaro Miwa, Dr. Isao Otsuka, Mr. Tsutomu Yoshikawa, Mr. Masao Noguchi, and Mr.
Kensuke Ueda of Mitsubishi Electric Corporation for providing me many valuable comments. The opportunity of discussion at the Mitsubishi Electric Cybersecurity Research
Alliance Laboratories improved my explanation capability.
I am also grateful to Professor Toru Fujiwara, Assistant Professor Naoto Yanai of Osaka
University, and Assistant Professor Yuya Tarutani of Okayama University, for their adviceof my research and experimental environment. Their advice from diﬀerent perspective
points of view was always informative and helpful.
I would like to thank subjects who participated in the experiment. Also, I sincerely
appreciate a lot of support of Mr. Masahiro Tanaka, Ms. Mari Otokura, and Mr. Takahiro
Higuchi of Osaka University for deploying and operating the experimental environment.
I would like to thank all the members of Advanced Network Architecture Research Laboratory at the Graduate School of Information Science and Technology, Osaka University,
for their inciting discussions and fellowship.

32

Last, but not least, I thank my parents and my sister for their invaluable support andconstant encouragement during my master studies.

33References[1] “Google Home,” https://home.google.com.
[2] “Amazon Echo,” https://www.amazon.com/echo.
[3] K.
vices

L.

Lueth,
nowat“Stateof

7B

Market

-

the

IoT

2018:

accelerating,”

Number

of

IoT

de-

https://iot-analytics.com/

state-of-the-iot-update-q1-q2-2018-number-of-iot-devices-now-7b/,

IoT

Analyt-ics, Aug. 2018.
[4] I. Lee and K. Lee, “The Internet of Things (IoT): Applications, investments, andchallenges for enterprises,” Business Horizons, vol. 58, no. 4, pp. 431–440, Jul. 2015.
[5] M. U. Farooq, M. Waseem, A. Khairi, and S. Mazhar, “A critical analysis on the
security concerns of internet of things (IoT),” International Journal of Computer
Applications, vol. 111, no. 7, pp. 1–6, Feb. 2015.
[6] B. L. R. Stojkoska and K. V. Trivodaliev, “A review of Internet of Things for smart
home: Challenges and solutions,” Journal of Cleaner Production, vol. 140, no. 3, pp.
1454–1464, Jan. 2017.
[7] M. Capellupo, J. Liranzo, M. Z. A. Bhuiyan, T. Hayajneh, and G. Wang, “Security
and Attack Vector Analysis of IoT Devices,” in Proceedings of International Conference on Security, Privacy and Anonymity in Computation, Communication andStorage, vol. 10658.

Springer International Publishing, Dec. 2017, pp. 593–606.

[8] M. Antonakakis, T. April, M. Bailey, M. Bernhard, E. Bursztein, J. Cochran, Z. Durumeric, J. A. Halderman, L. Invernizzi, M. Kallitsis, D. Kumar, C. Lever, Z. Ma,
J. Mason, D. Menscher, C. Seaman, N. Sullivan, K. Thomas, and Y. Zhou, “Understanding the Mirai Botnet,” in Proceedings of the 26th USENIX Security Symposium.
Vancouver, BC: USENIX Association, Aug. 2017, pp. 1093–1110.
[9] D.
sirai

Palmer,botnet“120,000say

IoT

camerasresearchers,”

vulnerable

to

new

Per-

https://www.zdnet.com/article/

120000-iot-cameras-vulnerable-to-new-persirai-botnet-say-researchers/, May 2017.
34

[10] Y. M. P. Pa, S. Suzuki, K. Yoshioka, T. Matsumoto, T. Kasama, and C. Rossow,“IoTPOT: A Novel Honeypot for Revealing Current IoT Threats,” Journal of Information Processing, vol. 24, no. 3, pp. 522–533, May 2016.
[11] M. Lyu, D. Sherratt, A. Sivanathan, H. H. Gharakheili, A. Radford, and V. Sivaraman, “Quantifying the Reflective DDoS Attack Capability of Household IoT Devices,”in Proceedings of the 10th ACM Conference on Security and Privacy in Wireless andMobile Networks, ser. WiSec ’17. New York, NY, USA: ACM, Jul. 2017, pp. 46–51.
[12] V. Martin, Q. Cao, and T. Benson, “Fending oﬀ IoT-hunting attacks at home networks,” in Proceedings of the 2nd Workshop on Cloud-Assisted Networking.

ACM,

Dec. 2017, pp. 67–72.
[13] K. Xu, F. Wang, and X. Jia, “Secure the Internet, one home at a time,” Security andCommunication Networks, vol. 9, no. 16, pp. 3821–3832, Jul. 2016.
[14] S. Shirali-Shahreza and Y. Ganjali, “Protecting Home User Devices with an SDNBased Firewall,” IEEE Transactions on Consumer Electronics, vol. 64, no. 1, pp.
92–100, Feb. 2018.
[15] K. Xu, F. Wang, R. Egli, A. Fives, R. Howell, and O. Mcintyre, “Object-oriented
big data security analytics: A case study on home network traﬃc,” in Proceedingsof International Conference on Wireless Algorithms, Systems, and Applications, vol.
8491. Springer International Publishing, Jun. 2014, pp. 313–323.
[16] K. Xu, F. Wang, L. Gu, J. Gao, and Y. Jin, “Characterizing Home Network Traﬃc:
An Inside View,” Personal Ubiquitous Comput., vol. 18, no. 4, pp. 967–975, Apr.
2014.
[17] N. Komninos, E. Philippou, and A. Pitsillides, “Survey in Smart Grid and Smart
Home Security: Issues, Challenges and Countermeasures,” IEEE Communications
Surveys Tutorials, vol. 16, no. 4, pp. 1933–1954, Apr. 2014.
[18] S. Soltan, P. Mittal, and H. V. Poor, “BlackIoT: IoT Botnet of High Wattage Devices
Can Disrupt the Power Grid,” in Proceedings of 27th USENIX Security Symposium(USENIX Security 18). Baltimore, MD: USENIX Association, Aug. 2018, pp. 15–32.
35

[19] B. B. Zarpelao, R. S. Miani, C. T. Kawakani, and S. C. de Alvarenga, “A survey of
intrusion detection in internet of things,” Journal of Network and Computer Applications, vol. 84, pp. 25–37, Apr. 2017.
[20] N. Apthorpe, D. Reisman, and N. Feamster, “A smart home is no castle: Privacy
vulnerabilities of encrypted iot traﬃc,” arXiv preprint arXiv:1705.06805, pp. 1–6,
May 2017.
[21] C. M. Bishop, Pattern Recognition and Machine Learning (Information Science andStatistics).

Secaucus, NJ, USA: Springer-Verlag New York, Inc., Feb. 2006.

36