Real-time Analysis of Privacy-(un)aware IoT Applications 146 

IoT app at runtime. The collected data is used by IoTWatcH to classify the app information into user-defned 

(also customizable) privacy labels through Natural Language Processing (NLP) techniques. The privacy labels 

provide the users with a more intuitive mechanism to 

understand how IoT apps handle their private information. Also, IoTWatcH analyzes the privacy preferences 

of users to uncover sensitive data leaks. Finally, IoTWatcH notifes the users about the sensitive data-leaks 

when they validate the users’ privacy preferences, allowing the users to make informed decisions about their 

privacy. 

To evaluate IoTWatcH, we trained an NLP model 

with taint-sink data strings extracted from 380 SmartThings market apps. The model was used to classify taint-sink content (e.g., “the door is locked”, and 

“kitchen lights are turned o ̇”) to user privacy preferences. Then, we analyzed 160 di ̇erent Samsung SmartThings apps to evaluate the accuracy of IoTWatcH

at runtime. IoTWatcH successfully classifed 146 IoT 

strings to privacy preferences with an average accuracy 

of 94.25% and a precision of 95%. Additionally, IoTWatcH identifed 35 IoT apps that leaked sensitive data 

to unauthorized recipients. IoTWatcH yielded minimal 

overhead to the IoT apps execution, introducing on average of 105 ms additional instrumentation latency. 

Summary of Contributions. The contributions of 

this work are as follows: 

– We conducted a comprehensive IoT privacy survey 

with 123 IoT users through a set of structured questions. 

– We developed IoTWatcH, a dynamic privacy analysis tool for IoT apps. IoTWatcH provides users 

with a privacy interface and collects app data at 

runtime based on user preferences. Then, it analyzes the collected data and informs users when a 

data leak matches the user’s privacy preferences. 

– We collected data from 380 real IoT apps to train 

an NLP model. We used the model to evaluate IoTWatcH’s performance on other 120 market and 

40 malicious IoT apps. IoTWatcH classifed 146 

IoT taint-sink content into correct privacy labels 

with 94.25% accuracy while imposing an additional 

105ms latency to app execution on average. Additionally, IoTWatcH identifed 62 sensitive data 

leaks to unauthorized third parties in 35 IoT apps 

(29 via messaging and 33 via Internet taint-sinks). 

– We made IoTWatcH freely available to the community at https://iotwatch.appspot.com/. 

Organization. In Section 2, we articulate the privacy 

issues in IoT apps through a use case, and present the 

defnitions and threat model. In Section 3, we present 

the results of the IoT privacy survey with 123 users. In 

Sections 4 and 5, we give an overview of IoTWatcH, 

and present IoTWatcH’s architectural details. In Section 6, we provide the implementation details. Then, 

we evaluate IoTWatcH and show its e ̇ectiveness and 

performance in Section 7. Finally, we discuss the related 

work in Section 8, and conclude the paper in Section 9. 

2 Background and Threat Model 

2.1 Anatomy of an IoT App 

IoT applications from di ̇erent platforms may use different programming languages. For instance, Samsung 

SmartThings apps are written in Groovy [31], while 

OpenHAB [36] applications use Domain Specifc Language (DSL) developed in Java. In other examples, Apple Homekit [66] apps can be written in Swift or Objective C, while Windows IoT [67] use C#, C++, or even 

Python. Despite this diversity, IoT applications share 

a similar structure [10]. In general, IoT apps have a 

description block that informs the users specifcs of the 

app itself (e.g., the app functionality, devices it controls, 

etc.). We expect that developers use these description 

blocks to also inform the users regarding the specifc 

types of sensitive information that the app will have access to on the IoT systems and who this information will 

be shared with. In that way, the users can take informed 

decisions regarding their privacy before installing the 

apps. The apps also include a permission block where 

the user adds what she desires to control with the app 

(e.g., smart thermostat). Also, the permission block is 

used to defne the information used to specify device 

settings (e.g., min and max thermostat temperatures) 

and app notifcation recipients (e.g., phone numbers for 

push notifcations). We study this permission block during our analysis as it contains highly private information 

from users and the IoT systems, including characteristics of the smart devices and their settings. Also, we 

correlate information defned by the user in these blocks 

to verify that the sensitive information is being sent 

to authorized recipients (e.g., user-defned phone numbers). The IoT apps also include an event subscription 

block, where developers match the specifc functions that 

are triggered by di ̇erent device states (e.g., the function f1 is executed when the presence sensor triggers a 

“present” state). Finally, the event handlers defne these 

functions and the app logic that is executed whenever 
Real-time Analysis of Privacy-(un)aware IoT Applications 147 

1: // smart-lock-control app: “controls the smart lock” 2: // Permissions Block 3: 1 Device: smart_lock sl, presence pr 4: User-defined inputs: phone 5: // Events Subscription 6: 2 subscribe(pr, “present”, f1) 7: subscribe(pr, “not present”, f2) 8: //Events Handler 9: 3 f1(){...} 10: f2(){ sl.lock() 11: notifyUser(sl.state, phone) 12: leakInfo()} 13: // Sends notification to the user 14: notifyUser(state, number){ 15: sendSMS(“Your lock is: ” + state, phone) 16: POST(“http://support.com”, sl.getLocation()) 17: // Leaks sensitive data to attacker 18: 4 leakInfo(){ 19: sendSMS(“Nobody is Home”, 123-456-7890)} 20:

Fig. 1. An example IoT app leaking sensitive data to a hardcoded phone number and implementing insecure HTTP calls. 

they are called. In our work, we study the event handlers as they defne the way how sensitive information 

is handled within IoT apps. Also, event handlers specify 

the sink functions that send the sensitive data out of the 

apps to external parties. As a summary, we target description blocks, permission blocks, event subscription 

blocks, and event handlers as the most important app 

constructs to investigate for any sensitive data leakage 

from IoT apps. 

2.2 Problem and Challenges 

We use an example IoT app source code derived from 

a smart-lock-control app (Figure 1) to illustrate the 

privacy concerns in IoT apps. The expected behavior of 

the app is to lock the door and notify to user-defned 

contacts that the door is locked when the user leaves the 

house. At install-time, the user grants permissions to the 

smart lock, presence sensor, and enters a phone number 

for messaging notifcations ( 1 ). The app subscribes to 

two event handlers f1 and f2 to implement the app 

functionality. The event handlers are invoked based on 

the presence sensor’s state (user-present and user-notpresent) ( 2 ). When the user leaves home, “not-present” 

event handler (i.e., f2) locks the door, sends a message 

notifcation, and transmits out the door lock state to a 

remote server (i.e., http://support.com) ( 3 ). However, 

the actual behavior of the app adds a piece of code that 

invokes a function (i.e., leakinfo()) sending a string 

that contains “Nobody is Home” to a hard-coded phone 

number ( 4 ). This string is highly private and informs 

an adversary that the house is empty. 

Requirements for IoT Privacy Analysis. The diversity of information handled by IoT (e.g., location, 

user behavior, user preferences) makes these systems 

vulnerable to privacy risks, hence the need for a specifc 

privacy analysis tool for IoT. However, IoT platforms 

do not provide the means for such an analysis. There 

exist no security mechanisms or methodologies to analyze and understand the specifc types of information 

that an IoT application handles and discloses to support their logic and functionalities. Previous works have 

also noted that IoT systems do not explicitly inform the 

users regarding their utilization of the sensitive information, and researchers are trying to propose solutions to 

these issues [45]. 

There exist systems to identify sensitive data-fows 

in IoT apps. For instance, SainT is a static tool that 

uses taint analysis to identify sensitive data-fows in IoT 

apps [10]. FlowFence, a dynamic system, uses quarantined modules to enforce data-fow policies on the use 

of sensitive data [19]. Albeit useful, these approaches 

are limited in precision and the number of privacy policies enforced as they both focus on taint data. Hence, 

SainT and FlowFence are oblivious if an app leaks sensitive data through implicit programming constructs (i.e., 

description blocks, event handlers) via developer- or 

user-defned strings. For instance, the string “Nobody 

is home” leaked through leakinfo()). Meanwhile, our 

analysis of 540 IoT market apps showed that 64% of 

apps potentially leaked sensitive data through strings 

that did not include any tainted data, yet the string was 

sensitive. Indeed, static systems like SainT, iRuler [54], 

and privacy tools for trigger-action platforms [27] fail 

to detect sensitive data leaks through dynamic method 

invocation [31]. Additionally, there are no approaches 

that allow users to closely examine their sharing and 

privacy preferences over individual IoT apps. For instance, a user may desire to share her energy usage data 

with a third party in a specifc IoT app yet she wants 

to restrict sharing all other sensitive information with 

third-parties. This requires a personalized privacy setting for each app that gives control to the users over 

what to share. Finally, previous research [9, 10] have 

demonstrated that app analysis tools that may be e ̇ective for other systems like Android cannot be directly 

applied to IoT as they lead to substantial false positives due to important architectural and programming 

di ̇erences between their apps. 

Challenges of an IoT Privacy Solution. In contrast to the previous approaches [9, 10, 19, 27, 54], IoTWatcH tracks runtime data-fows, and process the fowcontent to determine whether a data-fow constitutes a 

privacy concern or not. Such a runtime analysis overcomes the limitations of static tools [10] that do not 

consider dynamically generated taint variables or hide 

them via implicit constructs. Additionally, IoTWatcH

provides a user with an interface that allows them to 

confgure their privacy settings for each app and informs 

the users about its fndings. 
Real-time Analysis of Privacy-(un)aware IoT Applications 148 

We provide a detailed comparison of IoTWatcH

with other privacy tools for IoT and Android apps in 

Section 8. However, to support its capabilities, IoTWatcH needs to overcome several technical challenges. 

First, it needs to capable of analyzing the source code 

of IoT apps and identifying privacy-relevant information beyond taint data, which is a deeper investigation 

than infexible static approaches. Second, it needs to 

implement mechanisms to be able to collect the privacy 

information for further analysis. Static tools like SainT

may be a good starting point for source code analysis 

in IoTWatcH. However, such static tools require extensive modifcations to identify non-taint privacy data 

within apps. Similarly, SainT or other static analysis 

tools do not support app instrumentation, which is required for real time dynamic analysis. Also, IoTWatcH

analysis requires the implementation of NLP models to 

classify privacy data into simpler privacy labels that can 

be easily understood by the users. This process requires 

building specifc IoT corpus for training purposes. Finally, as IoTWatcH implements real-time analysis and 

notifcation systems for the user, it requires of specifc 

APIs to support and further secure the communications 

between IoT apps and IoTWatcH’s servers without imposing a considerable overhead. 

2.3 Defnitions 

Taint-Sinks. IoT programming platforms defne specifc call-sites to transmit data [36, 51]. We focus on 

two main call-site types; messaging and Internet. These 

call-sites require two types of information: the recipient 

and the content. The recipients defne where the content 

is being sent to, and the content defnes the data sent 

in the form of strings. 

Privacy Labels. We defne four di ̇erent privacy labels 

(i.e., date-time, device-info, location, and user-behavior) 

to classify the content of the taint-sinks. These privacy 

labels are selected based on users responses in our privacy survey detailed in Section 3. 

Privacy Profle. The collection of privacy and notifcation preferences identifes the users’ privacy profles. 

These profles helps IoTWatcH to selectively instruments the apps to collect data for privacy analysis. 

Privacy Leaks. We consider a privacy leakage happens 

when data is sent to external parties, and it is neither 

informed to nor acknowledged by the user. For instance, 

if an app transmit location information and a user is not 

informed, then it constitutes a privacy leak. 

2.4 Threat Model 

We consider IoT apps that access sensitive data without 

user consent. Second, we consider data leaks, through 

Internet and messaging taint-sinks, transmitted out of 

an app to unauthorized recipients, and due to the carelessness of developers. Lastly, we consider app transmitting sensitive data in clear-text (through HTTP) to 

notify users. 

We do not consider safety and security violations in 

IoT apps as they were already studied before [12, 13]. 

Additionally, we do not track data-fows via push notifcations or taint-sinks that are authorized via OAuth 

(e.g., a user authorizes a third-party service through 

OAuth protocol to share the device states for data visualization) as they are considered a di ̇erent external 

capability that supports the IoT functionality. Finally, 

we do not consider apps using obfuscation or encryption techniques to hide the content of the data fows; 

yet, we discuss potential measure for these techniques 

in Section 5.1.1). We note that we did not fnd one single app out of 540 IoT apps we analyzed that encoded 

the content of their messaging or Internet taint-sinks. 

3 Privacy Survey 

In this work, we frst conducted a comprehensive survey to understand the privacy concerns of users when 

they use various IoT devices and apps. The entire survey 

was authorized by the institutional ethics review board 

(IRB) and occurred between April 2019 and May 2019. 

Survey Goals. With this survey, we aimed to answer 

the following questions: (1) what are the privacy concerns of IoT users?, (2) is there a need for privacy analysis tools designed for IoT?, and (3) what are the user 

expectations, in terms of usability requirements, for privacy analysis tools? Specifcally, we expected to understand the types of privacy sensitive information that 

is most relevant to the IoT users so that it can be refected in the design of the privacy choices o ̇ered by 

IoTWatcH. Also, we expected to gain valuable understanding about the users’ desire to have a fexible design 

that could be captured by the customizable privacy profle o ̇ered by IoTWatcH at install time rather than enforced fxed privacy policies for all the data. In addition, 

we wanted to understand the importance of prompt privacy feedback to the users so that we can incorporate 

real-time analysis into our approach. Finally, the survey 

would reveal the specifc types of app communications 

and potential leaks that most users would want to track 

with our privacy tool. For all these, we created 26 dif-
Real-time Analysis of Privacy-(un)aware IoT Applications 149 

ferent questions organized into three categories (sample 

questions in Appendix A). These categories align with 

three specifc privacy survey goals: (1) the characterization of the participants, (2) privacy concerns of IoT 

users, and (3) the need for IoT privacy analysis tools and 

their usability requirements. We present the profles of 

participants and our key fndings below. 

Survey Overview and Recruitment. We made the 

survey available to participants for four weeks. The users 

could access the survey and submit their responses via 

an online questionnaire hosted on Google Forms [22]. 

The questionnaire included single choice questions (e.g., 

yes, no), multiple-choice questions, and free-form questions. We made all the questions required except for the 

ones requesting an additional explanation from users in 

the form of free-text input. Finally, we recruited the 

participants via recruitment emails in our institutions. 

The emails included a brief description of the survey 

and a link to the online form. Finally, the survey was 

designed so the participants would spend an average of 

15 minutes from start to fnish. 

Participant Demographics. We recruited 123 participants. In the group, 100 (81.3%) were male, and 23 

(18.7%) were female. Also, 69 participants (56.1%) were 

in the range of 18-25 and 37 (30.1%) were in the range of 

26-35 years old. The remaining 17 (13.8%) participants 

were 36 years or older. The majority of the participants 

(110 (89%)) had at least completed some bachelor-level 

courses, and 37 (30%) were enrolled in graduate-level 

courses. Finally, 100 participants reported being part of 

an educational institution. A total of 112 (91.05%) users 

shared that they currently use or are planning to use 

IoT devices in their homes. Finally, 19 (15.4%) participants knew how to develop their own IoT apps while 82 

(66.7%) participants had previous experience installing 

apps from an IoT market or via using the source code 

of IoT apps available online. 

Ethics and Analysis Approach. The IRB of our 

institutions approved the privacy survey. The participants had to be over 18 years old, and the survey did 

not collect any personal information from the participants other than an institutional email address that was 

requested for compensation purposes. We did not allow participants to submit multiple responses, but they 

had the chance to change their answers anytime before the survey closing date. No participants changed 

their original replies. We processed and accepted all 

the responses obtained from the participants. Further, 

we directly quantifed the responses from single- and 

multiple-choice questions. Finally, we used two independent researchers to analyze the free-from responses remove answers fagged as potential outliers. 

3.1 Discussion of Survey Results 

Privacy Concerns of Users. The participants were 

concerned about their sensitive information being inadvertently leaked to unauthorized parties. To detail, 

65 (52.8%) participants felt uncomfortable about their 

personal data (e.g., their password to login into edge devices), their behavior and habits (e.g., when they go to 

sleep), location (e.g., whether they are home or away), 

device’s settings (e.g., heating value of a thermostat) 

and time confguration (e.g., when kids leave home), and 

device states (e.g., whether the door is locked or not) 

being handled by IoT systems. Also, at least 89 (72.4%) 

participants expressed that they are aware of IoT apps 

collecting their sensitive information and sending it to 

remote servers for data analytics such as profling their 

energy usage and for advertisement purposes [2]. Finally, 103 (83.7%) participants expressed privacy concerns on the use of IoT systems, and 88 (71.5%) mentioned having heard about the privacy issues in IoT systems from the news or other media. 

The Need for Real-time Privacy Tools. A total of 

112 (91.1%) participants raised broad concerns about 

the lack of an existing tool that informs the users regarding the potential privacy risks of IoT systems in realtime. Also, 119 (96.74%) participants found the idea 

of using a tool to uncover privacy risks in IoT highly 

desired and expected. Our participants were happy to 

use automatic tools that instruments IoT apps to enable 

prompt privacy analysis and results. Out of the 123 participants, 119 (96.74%) expressed their support for this 

option if the tool is verifed by the IoT platform. 

Users’ Expectations. To understand the characteristics of an easy-to-use privacy tool, we asked participants 

to identify their preferences on di ̇erent types of sensitive information they desire to be informed. The participants noted four di ̇erent privacy labels, which we later 

use them in our design and analysis of IoTWatcH. The 

“Device-info” was approved by 109 (88.6%) participants 

to defne device information (e.g., device states or device 

type). The “User-behavior” received 103 (83.7%) positive responses to identify information related to user 

options and activities (e.g., what the user does, how the 

user confgure his/her IoT system). The “Location” was 

approved by 110 (89.4%) participants to identify the location of devices and users (e.g., user is at home, the 

kitchen switch is turned on). Lastly, the“Date-time” was 


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

- - - -

Real-time Analysis of Privacy-(un)aware IoT Applications 152 

12:01 

< Back smart lock control Save 

IOTWATCH Privacy Labels: 1

Date-time Information 

Device Information 

Location Information 

User Behavior 

Leaking Mechanisms: 2

Notify Leaks via Messaging 

Notify Leaks via Internet 

Click here to access IoTWatch’s tutorial 

smart lock control 

12:01 

< Back Save 

Privacy Profile Tutorial 

We provide basic definitions of the features included in the privacy profile 

to guide the user through its configuration. If selected by the user (toggle 

buttons showing blue color), IoTWatcH will consider these features into its 

analysis and provide feedback about privacy concerns related to them to 

the user in real time. 

Privacy Labels: 

• Date-time Information: If selected, IoTWatch notifies the user 

anytime scheduling information of device and app activities (e.g., 

the time a door is schedule to automatically unlock) is shared. 

• Device Information: If selected, IoTWatch notifies the user anytime 

information regarding device states (e.g., motion sensor active) or 

device information (e.g., device branch and model) is shared. 

• Location Information: If selected, IoTWatch notifies the user 

anytime information regarding her or the device locations (e.g., 

user is not at home) is shared. 

• User Behavior: If selected, IoTWatch notifies the user anytime 

information regarding user preferences within the IoT system (e.g., 

the user mode changed from home to away) is shared. 

Leaking Mechanisms: 

• Notify Leaks via Messaging: Informs the user whenever the 

previous types of information are leaked using messages. 

IoTWatcH also provides details regarding the recipient of the 

information. 

• Notify Leaks via Internet: Informs the user whenever the previous 

types of information are leaked to external computing systems. 

IoTWatcH also provides details regarding the recipient of the 

information. 

(a) (b) 

Fig. 4. (a) IoTWatcH enables users to defne their privacy preferences, and (b) sample tutorial to help users understand the 

meaning of the privacy profle’s components. 

we detail later in Section 5.2, this data is matched with 

the information extracted from sink-calls (i.e., recipients) to identify, for instance, whether the data sent to 

unauthorized recipients (i.e., not defned by the user), 

which may lead to potential privacy issues for the user. 

Taint-Sink Information. It includes the content and 

recipients of messaging and Internet call sites. Turning 

to app source code depicted in Figure 2, IoTWatcH obtains the content “Your lock is: + state” from the frst 

messaging function in Line 16, and the content “Nobody 

is Home” from the second messaging function in Line 21. 

As per the recipients, it extracts the recipient’s value 

contained in the variable phone from the frst messaging method (Line 16) and the hard-coded phone number “123-456-7890” from the second messaging function 

(Line 21). IoTWatcH uses taint-sink contents to inform 

a user of the sensitive data type. 

Purpose of Collected Data. We implement an NLPbased model to analyze the content of taint sinks 

and classify them into four privacy labels. Furthermore, IoTWatcH matches the recipient information 

extracted from messaging and Internet taint-sinks with 

app information to uncover sensitive data leaks. The 

app information is either entered by the user, or informed by the developer via the app’s description block 

and approved by the user. In cases where the sink-call 

(i.e., messaging or Internet) is executed using unauthorized recipients, a leak is fagged and the user is informed. Additionally, we check whether Internet taintsinks send data in clear-text, confrming sensitive data 

is not accessed by potential eavesdroppers. 

5.1.1 Selective Code Instrumentation 

IoTWatcH performs a selective code instrumentation 

to implement privacy analysis on the sensitive data that 

is of interest to the user, which also reduces the number 

of real-time notifcations. Also, selective instrumentation permits the analysis of encrypted IoT strings. 

Privacy User Interface. The instrumentor adds additional code to implement a UI and create a privacy 

profle of the user. Figure 4(a) shows the privacy user 

interface presented to the user during install-time, which 

illustrates the selective code instrumentation options of 

the IoTWatcH-enabled app. IoTWatcH’s instrumentor does not impact the UI experience of the IoT app at 

runtime, but o ̇ers new privacy features at install-time 

not available in the original app. The instrumented app 

o ̇ers the user the possibility to create a privacy profle and receive notifcations regarding specifc privacy 

labels that are of interest to the user ( 1 ). Also, it allows for selecting which privacy concerns (e.g., option 

to notify leaks from messaging or Internet taint-sinks) 

must be analyzed and informed by IoTWatcH ( 2 ). 

Such a design supports the expectations of the IoT app 

users with (1) confgurable privacy preferences, (2) ondemand privacy controls, and (3) timely privacy notifcations (Table 1) as were summarized in Section 3. The 

user interface also includes a link to a tutorial that explains the di ̇erent settings of the privacy profle. The 

justifcation behind this design rationale is to to facilitate the user experience regarding the setup and use 

of the privacy profles, their components and meanings, 

and how they can be included into IoTWatcH’s analysis, as shown in Figure 4(b). The tutorial can be easily 

accessed from the privacy user interface anytime. Also, 

as we target open-source IoT platforms, the use of selective instrumentation does not change or impact the original user interface of the app which may be considered 

as intellectual property. For closed-source platforms, we 

envision developers using the features o ̇ered in IoTWatcH to evaluate and improve the protection of sensitive information and the privacy of users. Lastly, we note 

that users can update their privacy profles through the 

settings provided in instrumented IoT app. We follow a 

similar principle that a user can update the app settings 

such as when a new device is added. The updated privacy profles then are sent to IoTWatcH’s server and 

the new user notifcations are automatically enabled. 

Evading IoTWatcH Analysis. The use of encryption and obfuscation techniques to hide the sink-call 

content limits the e ̇ectiveness of NLP techniques. For 

instance, NLP models that use plain-text data would 

fail to classify encrypted/obfuscated strings. However, 

selective instrumentation may facilitate the analysis of 

IoT apps that codify or encrypt the data that is sent 

Real-time Analysis of Privacy-(un)aware IoT Applications 154 

App Sink-Call Content Assigned Privacy Labels 

“Thermostat is turned on” device-info 

“The door will remain open for another 5 minutes” device-info, date-time 

“Door is closed since car was not at Home, 15 sec ago” device-info, location, date-time 

“Sleep time set for you as requested” user-behavior 

Table 2. Examples of leaked taint-sink contents from IoT apps 

and their privacy labels. IoTWatcH assigns multiple privacy 

labels to specifc IoT strings to capture their complex semantics. 

an app includes a string, “the user mode changed to 

vacation from home”. 

We note that taint-sink contents may require more 

than one privacy label assigned to guarantee accurate 

classifcation. For instance, the message string “The 

door will remain open for another 5 minutes” is labeled 

with multiple labels of types Device-info and Date-time. 

Similarly, “Garage door is not opening since the car was 

not present at Home, less than 15 sec ago”, is labeled as 

Device-info (the door is not opening so it is still closed), 

Location (the car was not present at Home), and Datetime (15 sec ago). We present examples of IoT taintsink contents assigned to the various privacy labels in 

Table 2. The use of multiple privacy labels to a single 

text includes more complex semantics string structures 

and in turn provide users with more accurate labels. As 

we explain later in this section, we use NLP techniques 

to achieve this goal. 

IoT Application Corpus. We study sink-call contents 

of 540 IoT apps. These contents pose a few unique characteristics compared to those of other domains. First, 

the size of the texts extracted is usually three to four 

shingles on average, yet it contains highly private data 

(e.g., “the door is unlocked”). Second, their linguistic 

structure is minimal regarding semantics (e.g., “mode 

changed to away”), compared to other short documents extracted from popular general-knowledge corpora [21, 33, 56]. Third, their meanings usually are 

closely attached to the app’s context (e.g., “if the user 

is not-present, turn o ̇ the light”). Based on our fndings, we build an NLP model that is e ̇ective at classifying semantic-defcient, but information-rich texts. To 

determine whether strings include sensitive information 

and assign them a privacy label, we frst implemented 

a classifcation model using publicly available data corpora [21, 33, 56]; however, due to the specifc characteristics of IoT strings, we obtained very low classifcation accuracy (detailed in Appendix C). To improve 

this, we constructed an IoT-specifc corpus for classifcation purposes that successfully considers and solves 

these challenges. We frst collected the content of messaging and Internet calls from current IoT market apps. 

We pre-processed the resulting dataset by fltering out 

punctuation and stop words. Further, we manually labeled the IoT strings to the four privacy labels. Here, we 

applied multi-labeling to contents that contained information related to more than one privacy label. Overall, 

the resulting dataset represents the semantics of taintsink contents found in IoT apps. 

NLP Model Construction. We implement an NLP 

model that uses a specifc IoT corpus above for training. 

We use the model to classify unknown sink-call contents 

to privacy labels. To train the model, we propose a supervised learning approach that requires labeled data as 

input. We expect that the supervised approach yields 

better results than other approaches such as keywordsearch. This is because the IoT string containing privacy 

information often does not include suÿcient information to assign the labels through simple keyword-searchbased analysis. For instance, keyword-search would fail 

to identify the user-behavior in a messaging taint-sink 

that leaks a string “the kids left home”. The model 

integrates doc2vec [8] to represent every n-gram IoT 

string (i.e., IoT document) into a multi-dimensional 

vector. Specifcally, we choose the Paragraph VectorDistributed Memory (PV-DM) approach as strings extracted from IoT apps have limited semantic structures, 

yet they include a substantial amount of information. 

By using doc2vec helps IoTWatcH fnd the leaks that 

are context-wise and semantically similar. Thus, we select PV-DM doc2vec over other known approaches like 

bag-of-words (BoW) [30] as it has proven to be e ̇ective 

against incomplete or semantically-limited strings. PVDM considers the entire structure of the text and infers 

what is missing to perform syntax and context analysis (i.e., multi-word expression analysis) [32]. In this 

way, we are able to extract the privacy content from 

semantically-limited IoT strings. We then performed 

topic classifcation of the IoT strings using an automated machine learning approach (i.e., autoML) [6]. We 

note that autoML selects the best classifcation algorithm by trying various classifcation algorithms; thus, 

the best accuracy is always guaranteed. We provide details of the NLP model and IoT corpus in Section 6. 

5.3 User Notifcations 

IoTWatcH implements two di ̇erent privacy notifcation options, as shown in Figure 3(a), Appendix C. 

First, it allows a user to select specifc privacy labels 

(one or multiple) for an app to receive notifcations. For 

instance, if the user is only concerned about the location 

information, she may select the location label so that 

IoTWatcH informs if a sink-call transmits data defned 
Real-time Analysis of Privacy-(un)aware IoT Applications 155 

through location-label. This means that the user would 

not receive any other notifcations of other privacy labels. The second option allows the user to determine 

the type of taint-sink to be notifed, messaging, and Internet, or both. The selective user-specifc notifcations 

provide fexibility, reduces the IoTWatcH latency in instrumented apps, and enhances the user experience by 

reducing the number of notifcations at runtime. Lastly, 

IoTWatcH also check the recipients of Internet taintsink and see whether it uses HTTP or HTTPS to transmit sensitive data. This exposes the content of the Internet message to passive observers (i.e., eavesdroppers). In 

this case, IoTWatcH is able to notify users as well, as 

shown in Figure 3(b), Appendix C. 

We note that IoTWatcH enables all privacy labels 

and taint-sink options by default. Tech-savvy users that 

fully understand the privacy risks of IoT apps may create their own privacy profles by disabling install-time 

options for better user experience. Users that do not entirely understand the privacy labels in IoTWatcH may 

rely on the default options. 

6 Implementation Details 

We implemented IoTWatcH for IoT applications developed for Samsung SmartThings, which is the IoT 

platform that has the highest share of devices and applications in the current commodity IoT market [47, 52]. 

Samsung SmartThings apps are developed in Groovy, 

a dynamic programming language that supports static 

compilation. Static compilation permits for all methods 

and classes in the apps to be annotated at compile time, 

which makes this information fully available to the instrumentation portion of IoTWatcH. 

Code Instrumentor. IoTWatcH traverses the Abstract Syntax Tree (AST) of the IoT app’s Intermediate Representation (IR) using the Groovy’s 

ASTTransformation class [23]. This class, included in 

the Apache Groovy project, o ̇ers several metaprogramming utility methods to build and analyze the app’s 

AST. We used the AST to perform our analysis as 

we targeted open-source applications, and we could 

take advantage of the specifc programming classes that 

Groovy o ̇ers for the compile-time analysis. Specifcally, we use the class ASTNode to build an app’s Intraprocedural Control Flow Graph (ICFG) [10]. From 

there, we used the visit methods to inspect the different ICFG nodes and fag methods that implement 

sink functions (e.g., sendSMS), handle sensitive privacy data (e.g., phone number), subscribe to events, 

or defne event handlers. IoTWatcH involves around 

1700 lines of code written in Groovy to analyze the 

app source code, construct the IR, generate the ICFG, 

and perform the code instrumentation. We implemented 

IoTWatcH’s instrumentor as a web application using Groovy programming language (detailed in Appendix B). We made the instrumentor available online 

at https://iotwatch.appspot.com/. 

App and IoT Data Collection Process. We collected 540 Samsung SmartThings apps to implement 

and evaluate IoTWatcH, which represented 100% of 

the open-source SmartThings apps available at the time 

of developing this project. These IoT apps are from six 

di ̇erent general categories: Convenience, Smart Home 

Automation, Entertainment, Personal Care, Security & 

Safety, and Smart Transportation. Out of the 540 apps, 

380 SmartThings apps were used to build an IoT corpus and train the NLP model, and 160 apps to evaluate its performance (detailed in Section 7). For comprehensiveness, we included in our analysis SmartThings

market apps crawled from oÿcial Samsung repositories [48, 50] and malicious apps crawled from the IoTBench repository [28], an IoT-specifc test corpus used 

to evaluate systems for IoT app security and privacy. 

The IoTBench includes fawed apps that perform various malicious activities, including sensitive data leaks 

via both messaging and Internet taint-sinks. 

To collect the IoT strings from the crawled apps, 

we analyzed and instrumented the total population of 

apps using the IoTWatcH’s code instrumenter. IoTWatcH’s instrumentor adds on average 25% more lines 

of code (LoC) to the apps, which translates into adding 

65 LoC to an IoT app that has an average size of 265 

LoC. We then executed the instrumented apps in the 

SmartThings Simulator IDE [49], a propriety simulation tool provided by Samsung. The IDE permits the 

modeling of trigger-action scenarios in which an instrumented app sends messaging- and Internet-based taint 

sinks that contain the IoT strings to the IoTWatcH

server. The IoT strings are individually labeled through 

app and string ID, and taint sink in the form {ID: 

app_ID, taint_sink, string}. 

Data Split and Model Training. We frst organized 

the strings collected from the 540 IoT apps into the six 

di ̇erent app categories. Then, for proper data balance, 

we randomly selected the strings collected from 70% of 

the apps from each category until reaching 380 training 

apps. The remaining strings from the other 160 apps 

were used later for evaluating IoTWatcH. From the 

selected apps for training, we extracted a total of 2014 

di ̇erent IoT strings. We then labeled these strings according to the four privacy labels. Specifcally, 46.8% 
Real-time Analysis of Privacy-(un)aware IoT Applications 156 

of the strings contained information related to the IoT 

devices, while 20.8% contained relevant information related to Date-time. The remaining 19.2% and 13.2% of 

the IoT strings shared information related to location 

and user-behavior, respectively. We allowed up to 75% 

of inter-labeling assignment to the strings, meaning, up 

to three di ̇erent privacy labels can be assigned to a single string. In total, we applied multi-labeling to 72% of 

the privacy strings in the IoT corpus. Finally, we applied 

k-fold cross validation on the labeled data to train and 

validate the NLP model. We randomly selected 75% of 

the total corpus to train the classifer for approximately 

15 hours. Then, we validated the obtained model with 

the remaining 25% of the data, which is a practice followed by modern autoML [7]. Initial testing results on 

the NLP model showed an average precision of 94.3% 

and recall of 89.6%. 

Classifcation of IoT Privacy Strings. We use Automatic Machine Learning (Auto-ML) tools o ̇ered by 

Google App Engine [6] to perform privacy classifcation of IoT strings. Among its benefts, modern autoML approaches perform neural architecture search and 

transfer learning, enable hyper-parameter optimization, 

and utilize advanced model architectures to classify contents to privacy labels with high accuracy. More particularly, we implemented a custom multi-class multilabel model using the Natural Language API o ̇ered by 

Google (Google-NL) [7]. Google-NL o ̇ers a suite of ML 

algorithms that automatically optimize the algorithm 

parameters based on the specifc algorithm utilized and 

the characteristics of the dataset to guarantee the highest accuracy. Our initial analysis of 2014 IoT strings 

showed remarkable semantic similarities among them; 

thus, the use of labeled data reduces the training time 

considerably. Finally, we implemented our NLP solution 

to classify strings written in English as only two (0.5%) 

IoT strings analyzed were written in Spanish. 

IoTWatcH API. We implemented a REST API to 

enable e ̇ective and secure data exchange between the 

instrumented IoT app and IoTWatcH’s analyzer (Figure 3). From the app to the server, the API constructed 

a JSON object with the user-defned recipients and the 

taint-sink information. From the server to the app, another JSON object was used to send the IoTWatcH’s 

analysis results. The API also handles IoTWatcH’s privacy notifcations to the user. Once the privacy analysis is completed, IoTWatcH sends back to the user 

another JSON object containing its fndings. (aditional 

details in Appendix C). Further, we protected the communications between the apps and the IoTWatcH’s 

server with the asynchttpv1 class of Samsung SmartThings [51], which allows for asynchronous and encrypted HTTPS calls. Also, we guaranteed integrity of 

the data being exchanged by digitally signing the API 

requests with SmartThings x.509 certifcates. Specifcally, the API requests fetch the public keys for signature verifcation from https://key.smartthings.com

+ <aKeyId>. Then, the IoTWatcH computed the validity of the signatures using the resolved key combined 

with the HTTP headers provided on the API callback 

request. Finally, we assumed the IoTWatcH servers 

were secure so the actual NLP analysis as well as the 

user notifcations were not compromised by external attackers. Note that attacks to the server or server-related 

threats were not considered in this work. 

7 Performance Evaluation 

We evaluated the performance and eÿcacy of IoTWatcH based on the three research questions (RQ). 

RQ1 What is the e ̇ectiveness of IoTWatcH in correctly classifying the tain-sink contents into privacy labels? (Section 7.1). 

RQ2 What is the e ̇ectiveness of IoTWatcH in identifying data leaks that confrms privacy preferences of 

users? (Section 7.2). 

RQ3 What is the runtime overhead of IoTWatcH in 

terms of latency and storage? (Section 7.3). 

Evaluation Setup. We analyzed and extracted IoT 

strings from a total of 160 Samsung SmartThings apps 

to evaluate IoTWatcH’s performance. Specifcally, we 

included 120 market and 40 malicious apps in the evaluation. The apps were instrumented and executed in 

the SmartThings IDE to extract the app communication content as explained in Section 6. Finally, the apps 

sent their data to IoTWatcH’s analyzer at execution 

which ran on a Python web server hosted on Google 

App Engine [20]. 

Defning User Privacy Preferences. The ideal case 

to defne user preferences requires executing IoT apps 

with the real users, where each of them installs an app. 

However, this is time-consuming as we have hundreds of 

apps, and each user needs to understand the functionality of each app before they defne their preferences. 

To address this, we assume users set their preferences 

when the IoT data is transmitted out of an app without 

their consent, which allows us to defne privacy preferences of users in our experiments as a ground truth. A 

user is able to give consent to IoT app recipients and 

contents in two ways. First, a user defnes the taint-sink 
157 

1 

0.9 

0.8 

0.7 

0.6 

0.5 

Real-time Analysis of Privacy-(un)aware IoT Applications 

Label TP TN FP FN Accuracy Recall Precision Specifcity 

Device-info 958 232 11 84 0.9260 0.9214 0.9890 0.9547 

Date-time 99 1161 18 36 0.9589 0.7333 0.8918 0.9847 

User-behavior 508 684 26 95 0.9079 0.8425 0.9585 0.9633 

Location 221 1061 10 22 0.9756 0.9095 0.9610 0.9907

Multi-label Average Value

Accuracy 

Recall 

Precision 

Specificity 

Optimal 

Threshold 

Value (th = 0.4) 

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 

Classification Threshold 1 - Specificity Sensitivity 

Area Under the Curve (AUC) 

IoTWatch's Classifier ROC Curve 

0.1 

0.2 

0.3 

0.4 0.5 

0.6 

0.7 0.8 0.9 

Indicates the 

classification 

threshold value 

Table 3. Performance of IoTWatcH in classifying IoT strings to 

all the di ̇erent privacy labels. 

(a) Classifcation Metrics (b) ROC, AUC 

Fig. 6. IoTWatcH classifer performance: (a) average value of 

all performance metrics for the di ̇erent threshold values and 

(b) ROC curve and AUC that illustrate the performance of IoTWatcH’s classifer for all privacy labels. 

recipient at app install-time, for instance, a cell-phone 

number to receive a notifcation or a web page name 

that the content is sent. We are able to automate this 

process through sink-call recipient analysis. IoTWatcH

matches the taint-sink recipients with information defned by the user at install-time. For instance, the user 

defnes an authorized recipient in the feld phone. IoTWatcH extracts this information and checks it against 

the taint-sink recipient. If they match, it means that the 

user defnes the recipient at install-time. This is because 

IoTWatcH recognizes that the user previously authorized this recipient, and the data transmitted to the 

recipient do not violate the user’s privacy preference. 

Defning Leaks. In cases where IoTWatcH sees discrepancies between the recipient of a taint-sink and the 

one that was defned by the user, it identifes it as a data 

leak. Also, we considered a data leak any time an app 

sends information to a third party without informing the 

app user via the app description. As we consider that 

the description block of an app should explicitly specify 

the content transmitted out of an app and its recipients, 

failing to do so makes it impossible for the user to acknolewdge and confrm the app’s intent at install-time. 

Thus, for instance, an Internet taint-sink that sends sensitive data to a server that is not informed to the user 

via an app description block is clearly a leak. 

7.1 Taint-sink Content Classifcation 

IoTWatcH’s analyzer classifes IoT app taint sink content (i.e., IoT strings) into four privacy labels. The classifcation results include the assigned label and the confdence scores through a threshold th. If the classifcation score is over the predefned threshold, the privacy 

label is assigned to the string. For instance, the string 

“the front door at home is unlocked” resulted in classifcation scores of device-info=0.94, user-behavior=0.03, 

location=0.86, and date-time=0.3. By design, the classifcation scores are independent of each other, that is, 

if th value is set to 0.7, IoTWatcH classifes the string 

into privacy information of type device-info and location. In total, IoTWatcH classifed 146 IoT strings extracted from 95 di ̇erent IoT apps. Out of these, 112 

strings were extracted from messaging taint-sinks; 54 

messages from 44 market IoT apps and 58 messages 

from 30 malicious apps. The remaining 34 strings were 

extracted from Internet taint-sink; 12 Internet calls extracted from 10 market IoT apps, and 22 extracted from 

11 malicious IoT apps. 

Performance by Threshold Values. We study how 

IoTWatcH’s classifer performs for di ̇erent threshold 

values th. The goal of this analysis is to fnd the value 

th that leads to the highest performance overall. Figure 6(a) summarizes the average metrics for di ̇erent 

values of th. Overall, IoTWatcH yields the best accuracy for threshold values between 0.1 to 0.5. We observe 

that th = 0.4 yields the best classifcation results for 

all metrics. Finally, IoTWatcH classifes IoT strings to 

correct privacy labels with 94.25%, 85.17%, 95.01%, and 

97.34% average accuracy, recall, precision, and specifcity, respectively (additional details in Appendix D). 

Performance by Privacy Label. We further study 

the sensitivity of IoTWatcH’s classifer to each privacy label. The goal is to determine the e ̇ectiveness of 

IoTWatcH in classifying strings to the di ̇erent privacy labels. Table 3 shows the performance of IoTWatcH in classifying IoT strings to di ̇erent privacy 

labels. IoTWatcH achieves the highest accuracy for 

privacy labels of type location and date-time. This is 

because date, time, and location can be easily inferred 

from semantically-simple strings. Also, it is very common to fnd information related to these privacy labels 

embedded in the same string (e.g., “he arrived home 

5 minutes ago”). In contrast, IoTWatcH obtained the 

lower accuracy results for privacy labels of type userbehavior. This is because user-behavior information is 

harder to infer from simple strings. In spite of these 

results, IoTWatcH achieved the lowest accuracy of 

90.79% for user-behavior labels, which is comparable 

with the best classifcation results of other similar tools 

in the market [37]. In summary, our observation is that 

the classifcation errors of IoT strings into privacy la-
Real-time Analysis of Privacy-(un)aware IoT Applications 158 

bels are due to the highly-limited structure of the IoT 

strings. In most cases, developers try to exfltrate more 

information using short messages lacking proper semantics (e.g., “front door was unlocked 5 min ago”). In this 

case, we note that a straightforward classifcation approach does not yield successful results when extracting complete privacy information related to the device 

(e.g., door unlocked), date-time (e.g., 5 min ago), and 

location (e.g., front of the house) and, hence, would not 

be instrumental in uncovering all the privacy risks. Finally, we present the Receiver Operating Characteristic (ROC) curve and the corresponding Area Under the 

Curve (AUC) in Figure 6(b). These metrics summarize 

the overall performance of the IoTWatcH’s classifer 

for all privacy labels. The AUC fully supports the classifcation metrics detailed in Figure 6(a) and refects the 

high prediction quality of the proposed NLP classifer. 

Our observation is that the classifcation errors are 

due to the highly-limited structure of the IoT strings. 

In most cases, developers try to exfltrate more information using short messages lacking proper semantics (e.g., 

“front door was unlocked 5 min ago”). In this case, we 

note that a straightforward classifcation approach does 

not yield successful results when extracting complete 

privacy information related to the device (e.g., door unlocked), date-time (e.g., 5 min ago), and location (e.g., 

front of the house) and, hence, would not be instrumental in uncovering all the privacy risks. To overcome 

these issues and improve the classifer’s performance 

in IoTWatch, we followed di ̇erent and more fruitful 

design strategies, as stated in Section 5.2.1. First, we 

used a rich IoT corpus to train the model. Second, we 

represented every IoT string with a multi-dimensional 

vector via paragraph vector-distributed memory-based 

(PV-DM) doc2vec. With this approach, the IoTWatch 

classifer could predict the missing words from the 

structurally-limited strings and improve the classifcation results signifcantly. We will clarify this in the fnal 

version of the paper. 

Findings on Privacy Analysis of IoT Strings. IoTWatcH classifes IoT strings to privacy labels with an 

average accuracy of 94.25%. Out of 160 apps, IoTWatcH identifed 50 (31.25%) apps that transmit data 

related to device information via messaging, and 20 

(12.5%) apps that do the same via Internet taint-sinks. 

Also, 11 (6.9%) apps handled data related to date and 

time in messages and only one transmit similar information using the Internet. IoTWatcH also identifed 38 

(23.75%) apps transmitting information related to the 

user behavior in their messages and nine (5.6%) includApp Type 

Market 

Malicious 

No. of Apps 

120 

40 

Mess. 

54 

58 

Leaks 

0 

29 

Cl-T. 

– 

– 

E ̇ec. 

– 

100% 

Total 160 112 29 – 100% 

Mess. - Messaging taint-sinks Analyzed 

Leaks - No. of Data Leaks Found 

Cl-T. - No. of Clear-text Leaks Found 

E ̇ec. - IoTWatcH E ̇ectiveness 

Table 4. E ̇ectiveness of IoTWatcH in detecting sensitive data 

leaks via messaging. 

App Type 

Market 

Malicious 

No. of Apps 

120 

40 

Int. 

12 

22 

Leaks 

11 

22 

Cl-T. 

3 

3 

E ̇ec. 

100% 

100% 

Total 160 34 33 6 100% 

Int. - Internet taint-sinks Analyzed 

Table 5. E ̇ectiveness of IoTWatcH in detecting sensitive data 

leaks via Internet taint-sinks. 

ing similar type of information in Internet taint-sinks. 

Further, 20 (12.5%) apps sent information related to location via messaging, while six (3.7%) apps did the same 

via Internet calls. Finally, we evaluated how the privacy 

analysis of IoT strings benefted from the use of NLP. 

IoTWatcH assigned multiple privacy labels to classify 

more semantically-complex IoT strings, which guaranteed completeness in the privacy analysis. Out of 146 

strings analyzed, IoTWatcH applied multi-labeling to 

68 (46.5%). Specifcally, 54 IoT strings (36.9%) received 

two privacy labels and 14 (9.6%) strings received three 

privacy labels. 

7.2 Data Leakage Analysis 

Findings on Data Leaks via Messaging. Table 4 

shows IoTWatcH’s fndings after analyzing messaging 

recipients. IoTWatcH extracted 54 recipients in messages from market apps and 58 from malicious apps. 

We found no data leaks via messaging in market apps, 

meaning, all recipients were defned (or authorized) by 

the user at install-time. We believe this is due the strict 

review process enforced by SmartThings IoT market. 

Further, we found 29 leaks from 14 di ̇erent malicious 

apps [28], meaning, all these recipients were hard-coded 

by a developer and their intent were not defned in the 

app’s description block. For instance, the User Event 

app [28] leaks privacy labels of type device-info, location, and user-behavior (“Everyone is away and hub ID 

is #”) to a hard-coded phone number. We manually reviewed the app source codes and verifed that all IoTWatcH’s fndings were correct. Also, we verifed that 

all data leaks via messaging were properly fagged. 

Findings on Data Leaks via Internet. Table 5 details the e ̇ectiveness of IoTWatcH’s analyzer in fnd-
Real-time Analysis of Privacy-(un)aware IoT Applications 159 

ing data leaks via Internet taint-sinks. IoTWatcH analyzed 34 recipients from Internet taint-sinks, 12 from 

market and 22 from malicious apps. Our tool fagged 11 

Internet taint-sinks from seven market apps that leak 

privacy data without user consent. That is, the user 

is neither informed about the recipient of the data in 

the app description block nor enters the URL or domain name herself. For instance, the ThingSpeak Logger app [48] transmits a device ID to a remote server 

through an HTTP call, which is identifed through 

a device-info privacy label. For malicious apps, IoTWatcH fagged 22 Internet taint-sinks from 14 di ̇erent 

malicious apps as leaks. We verifed that 100% of data 

leaks via Internet were properly fagged. 

Clear-Text Data Leaks. IoTWatcH is also able to 

verify whether an IoT app sends a clear-text through 

HTTP. IoTWatcH fagged three clear-text leaks from 

three di ̇erent market apps out of twelve Internet taint 

sinks. We found that one of these calls was authorized 

by the user at install-time through the app description 

block. Lastly, IoTWatcH fagged three Internet taintsinks that uses HTTP to leak sensitive data from malicious apps. 

7.3 Overhead Analysis 

We show the performance of IoTWatcH in terms of 

runtime and storage overhead. 

Runtime Overhead. Latency refers to the time 

elapsed from the moment that the app’s data is collected 

to the moment that the user receives IoTWatcH’s notifcations. Latency overhead is calculated as the average di ̇erence in the execution time of the original and instrumented apps. On the one hand, IoTWatcH required on average 75 ms to classify the sinkcall contents. On the other hand, the communication 

latency between the IoT app and the IoTWatcH’s 

server was 30 ms on average. We used the Groovy 

class asynchttp_v1 to implement asynchronous HTTPS

requests to reduce communication latency. We found 

that the total latency introduced by IoTWatcH is on 

average 105 ms. 

Storage Overhead. We measured the storage overhead imposed by IoTWatcH. Our tool does not store 

app information after the analysis is completed; thus, 

the storage cost is determined by the total storage size 

of the JSON object used to exchange information between the IoT apps and IoTWatcH’s analyzer (Section 6). We evaluated the storage overhead imposed by 

the analysis of 160 IoT apps. On average, IoTWatcH

imposes a negligible 1 KB of storage overhead. 

7.4 Discussion and Future Work 

IoTWatcH is the frst dynamic tool that performs NLPbased real-time privacy analysis in IoT apps to (1) 

classify IoT strings into privacy labels that are easy 

to understand by the user, and to (2) fag IoT apps 

that represent privacy concerns for the user. We implemented IoTWatcH for SmartThings IoT platform, 

and we plan to extend our analysis to other IoT platforms. Additionally, we analyzed 380 IoT apps and constructed a dataset to study how these apps use privacysensitive information. While our corpus included IoT 

strings extracted from SmartThings market apps, we 

plan to investigate other IoT platforms to construct similar datasets. 

We designed and built IoTWatcH by frst understanding the privacy needs of IoT users. We plan to conduct an additional study to evaluate the usability of IoTWatcH which is outside the scope of the current work. 

Also, IoTWatcH’s analysis would beneft from mapping 

the app descriptions to privacy labels. However, this is 

challenging as the description block of an IoT app does 

not explicitly state the app’s privacy behavior but its 

functionality. We plan to use more advanced NLP techniques to address this challenge. Finally, IoTWatcH’s execution requires the collection and analysis of privacysensitive information. We use secure HTTPS communications to protect the communication between IoT apps 

and IoTWatcH’s server. In addition, IoTWatcH does not 

keep record of any collected information nor share this 

information with any third party. As a future work, a 

complete privacy assessment of IoTWatcH may be conducted to guarantee that user’s privacy is completely 

preserved. 

8 Related Work 

Data Flow Analysis. Previous works have focused on 

fow data analysis to research the security and privacy 

of the mobile phone [5, 14, 17, 24, 26, 57] and the IoT 

apps [9, 10, 18, 40]. Tools for mobile apps cannot be 

directly applied to the IoT as applications from both 

ecosystems pose di ̇erent architectural challenges. For 

instance, FlowCog [37] establishes data fow dependencies based on Android app view context, which cannot 

be extracted from IoT apps due to specifc architectural 

di ̇erences. On the other hand, previous solutions for 

IoT mostly consider security risks from data fows with 

tainted variables or via inter-rule or cross-app vulnerabilities. For instance, SainT [10] does not consider data 

leaks at runtime or via IoT strings, while FlowFence [19] 

often over-approximates the data leaks, which leads to 
Real-time Analysis of Privacy-(un)aware IoT Applications 160 

Tool Name Domain Code Analysis Dynamic Analysis Semantics Analysis Privacy Analysis NLP/ML Analysis User Awareness Overhead Evaluation Freely Available 

TaintDroid [17] Android 

FlowDroid [5] Android 

FlowCog [37] Android 

FlowFence [19] IoT 

ContextIoT [29] IoT 

Saint [10] IoT 

ProvThings [55] IoT 

iRuler [54] IoT 

SmartAuth [53] IoT 

IoTWatcH IoT 

Table 6. Comparison between IoTWatcH and other data fow analysis tools for Android and IoT apps. 

failure when dependency between taint variables and 

leaks cannot be established. The authors in [18] proposed the use of static analysis to reveal permissionbased security faws. Also, the authors in [40] enforced 

data fow control by applying opacifed computation. 

Lastly, ProvThings [55] uses static and dynamic analysis to collect data provenance and identify the root 

cause of attacks in IoT apps. However, this work is limited to analyze dependencies between events and data 

states and does not o ̇er any built-in privacy analysis. 

iRuler [54] applies NLP techniques to uncover inter-rule 

vulnerabilities parsed not from IoT apps, but IoT services. The SmartAuth tool [53] also uses NLP, this time 

to generate and enforce security policies in IoT apps. 

Similarly, the authors in [27] analyze privacy risks in 

IoT online recipes. Table 6 compares IoTWatcH with 

other the existing data fow analysis tools. 

IoT Privacy. Information exposure in IoT rises concerns among users and researchers. The authors in [58] 

perform a qualitative analysis of the type of sensitive information IoT devices expose. Also, the authors in [59] 

study IoT privacy focusing on the user’s perception. 

A novel work proposes the use of blockchain [60] or 

attribute-based encryption [63] to provide enhanced security and privacy to smart home systems. Meanwhile, 

other authors propose the use of unifed systems as the 

solution to provide enhanced privacy in IoT [61]. Finally, some works focus on protecting the privacy of IoT 

communications via novel protocols [64]. 

Comparison to Existing Works. Di ̇erent from 

SaINT [10] and ProvThings [55], IoTWatcH uncovers 

privacy risks of IoT apps not only from taint data but 

also from simple IoT strings. Also, IoTWatcH analysis focuses on privacy concerns from IoT apps, something that is missing in other analysis tool for IoT like 

iRuler [54], SmartAuth [53], and ContextIoT [29]. IoTWatcH uses user feedback to performs qualitative analysis on the information handled by IoT applications. 

Such a capability is missing in all other tools included 

in Table 6. Specifcally, IoTWatcH collects app data in 

real time and converts the data into specifc privacy labels that the user can understand and customize. Thus, 

IoTWatcH not only studies the cause and fow of the 

sensitive information, but its meaning to the user and 

the systems. There are some other features that are 

unique to IoTWatcH. Specifcally, IoTWatcH represents the frst security tool that exposes the recipients of 

the sensitive information in real time. Finally, since the 

proposed privacy tool is user-centered, it guides its analysis with customizables privacy profles that the users 

create at install time and change anytime after. That 

way, IoTWatcH minimizes the numbers of notifcations 

and focuses its analysis on privacy features that are of 

interest of every user individually. 

9 Conclusion 

IoT apps access sensitive data that, if leaked, may compromise the privacy of the users. IoT platforms do 

not o ̇er real-time privacy analysis that informs users 

about how the IoT apps handle sensitive information. 

To adress these concerns, in this paper, we introduced 

IoTWatcH, a dynamic analysis tool that uncovers the 

privacy risks of IoT apps in real-time. We developed 

IoTWatcH based on a study of the privacy needs of 

123 users. IoTWatcH enables users to select their privacy preferences, and uses NLP techniques to classify 

IoT strings into user-friendly privacy labels. This allows 

users to make informed decisions about their privacy 

and reject apps. We analyzed 540 IoT apps to train the 

NLP model and evaluate its e ̇ectiveness. IoTWatcH

classifes IoT strings to correct privacy labels with an 

average accuracy of 94.25% and fags 35 apps that leak 

sensitive data. Finally, IoTWatcH yields a minimal 

overhead to an IoT app’s execution, on average 105 ms 

additional latency. 

10 Acknowledgments 

This material is based upon work partially supported by 

the U.S. National Science Foundation under award numbers NSF-CAREER-CNS-1453647 and NSF-1663051. 
Real-time Analysis of Privacy-(un)aware IoT Applications 161 

References 

[1] Acar, A., Fereidooni, H., Abera, T., Sikder, A. K.,

Miettinen, M., Aksu, H., Conti, M., Sadeghi, A.-R.,

and Uluagac, A. S. Peek-a-boo: I see your smart home 

activities, even encrypted! WiSec (2020). 

[2] Aksu, H., Babun, L., Conti, M., Tolomei, G., and

Uluagac, A. S. Advertising in the iot era: Vision and challenges. IEEE Communications Magazine 56, 11 (November 

2018), 138–144. 

[3] Z. B. Celik and P. McDaniel and G. Tan and L.

Babun and A. S. Uluagac Verifying Internet of Things 

Safety and Security in Physical Spaces. IEEE Security Privacy, 17 (September 2019), 30–37. 

[4] Apple’s Home Kit Security and Privacy on iOS. 

https://www.apple.com/business/docs/iOS_Security_ 

Guide.pdf. [Online; accessed 9-January-2020]. 

[5] Arzt, S., Rasthofer, S., Fritz, C., Bodden, E., Bartel, A., Klein, J., Le Traon, Y., Octeau, D., and

McDaniel, P. FlowDroid: Precise Context, Flow, Field, 

Object-sensitive and Lifecycle-aware Taint Analysis for Android Apps. ACM SIGPLAN Notices (2014). 

[6] AutoML. https://www.ml4aad.org/automl/. [Online; 

accessed 10-February-2019]. 

[7] AutoML Natural Language Google. https://cloud. 

google.com/natural-language/automl/docs/. google[Online; 

accessed 10-February-2019]. 

[8] Ayyadevara, V. K. Word2vec. Apress, Berkeley, CA, 

2018, pp. 167–178. 

[9] Babun, L., Sikder, A. K., Acar, A., and Uluagac,

A. S. Iotdots: A digital forensics framework for smart environments, 2018. 

[10] Celik, Z. B., Babun, L., Sikder, A. K., Aksu, H., Tan,

G., McDaniel, P., and Uluagac, A. S. Sensitive Information Tracking in Commodity IoT. In 27th USENIX

Security Symposium (2018). 

[11] Celik, Z. B., Fernandes, E., Pauley, E., Tan, G., and

McDaniel, P. Program Analysis of Commodity IoT Applications for Security and Privacy: Challenges and Opportunities. ACM Computing Surveys (CSUR) (2019). 

[12] Celik, Z. B., McDaniel, P., and Tan, G. Soteria: Automated IoT safety and security analysis. In USENIX Annual

Technical Conference (USENIX ATC) (2018). 

[13] Celik, Z. B., Tan, G., and McDaniel, P. IoTGuard: 

Dynamic enforcement of security and safety policy in commodity IoT. In Network and Distributed System Security

Symposium (NDSS) (2019). 

[14] Clause, J., et al. Dytan: a Generic Dynamic Taint Analysis Framework. In ACM Software Testing and Analysis

(2007). 

[15] Denney, K., Babun, L. and Uluagac, A. S. USBWatch: a Generalized Hardware-Assisted Insider Threat Detection Framework. In in Journal of Hardware and Systems

Security (2020). 

[16] Denney K., Erdin E., Babun L., Vai M., and Uluagac

A. S. USB-Watch: A Dynamic Hardware-Assisted USB 

Threat Detection Framework. In Security and Privacy in

Communication Networks (SecureComm). (2019). 

[17] Enck, W., Gilbert, P., Han, S., Tendulkar, V.,

Chun, B.-G., Cox, L. P., Jung, J., McDaniel, P., and

Sheth, A. N. TaintDroid: An Information-Flow Tracking 

System for Realtime Privacy Monitoring on Smartphones. 

ACM Transaction on Computer Systems (2014). 

[18] Fernandes, E., Jung, J., and Prakash, A. Security 

Analysis of Emerging Smart Home Applications. In IEEE

Security and Privacy (SP) (2016). 

[19] Fernandes, E., Paupore, J., Rahmati, A., Simionato,

D., Conti, M., and Prakash, A. FlowFence: Practical 

Data Protection for Emerging IoT Application Frameworks. 

In USENIX Security (2016). 

[20] Google App Engine. https://cloud.google.com/ 

appengine/. [Online; accessed 26-August-2020]. 

[21] Google Books NGrams. https://aws.amazon.com/ 

datasets/google-books-ngrams/. [Online; accessed 10-

August-2020]. 

[22] Google Forms. https://www.google.com/forms/about/. 

[Online; accessed 26-August-2020]. 

[23] The Apache Groovy. https://groovy-lang.org/ 

metaprogramming.html#_abstractasttransformation. [Online; accessed 26-August-2020]. 

[24] Gordon, M. I., Kim, D., Perkins, J. H., Gilham, L.,

Nguyen, N., and Rinard, M. C. Information Flow Analysis of Android Applications in DroidSafe. In NDSS (2015). 

[25] Gorla, A., Tavecchia, I., Gross, F., and Zeller, A.

Checking App Behavior Against App Descriptions. In Proceedings of the 36th International Conference on Software

Engineering (2014), ICSE 2014, ACM. 

[26] Gu, B., Li, X., Li, G., Champion, A. C., Chen, Z., Qin,

F., and Xuan, D. D2Taint: Di ̇erentiated and Dynamic 

Information Flow Tracking on Smartphones for Numerous 

Data Sources. In INFOCOM (2013). 

[27] IFTTT (if this, then that). https://ifttt.com/, 2017. 

[Online; accessed 26-August-2020]. 

[28] IoTBench. https://github.com/IoTBench, 2017. [Online; 

accessed 26-August-2020]. 

[29] Jia, Y. J., Chen, Q. A., Wang, S., Rahmati, A., Fernandes, E., Mao, Z. M., Prakash, A., and Unviersity,

S. J. ContexIoT: Towards Providing Contextual Integrity to 

Appifed IoT Platforms. In NDSS (2017). 

[30] Le, Q., and Mikolov, T. Distributed Representations 

of Sentences and Documents. In Proceedings of the 31st

International Conference on International Conference on

Machine Learning - Volume 32 (2014), ICML’14. 

[31] Metaprogramming. http://docs.groovy-lang.org/docs/ 

next/html/documentation/core-metaprogramming.html. 

[Online; accessed 26-August-2020]. 

[32] Mikolov, T., Chen, K., Corrado, G. S., and Dean, J.

Eÿcient estimation of word representations in vector space. 

CoRR abs/1301.3781 (2013). 

[33] N. Iderhoff, "nlp-datasets". https://github.com/ 

niderho ̇/nlp-datasets/blob/master/README.md. [Online; accessed 26-August-2020]. 

[34] OpenHAB IoT App Market (Eclipse Market Place). 

http://docs.openhab.org/eclipseiotmarket. [Online; accessed 

26-August-2020]. 

[35] OpenHAB IoT App Submission Guideline. https: 

//marketplace.eclipse.org/content/eclipse-marketplacepublishing-guidelines. [Online; accessed 26-August-2020]. 

[36] OpenHAB: Open Source Automation Software for
Real-time Analysis of Privacy-(un)aware IoT Applications 162 

Home. https://www.openhab.org/. [Online; accessed 26-

August-2020]. 

[37] Pan, X., Cao, Y., Du, X., He, B., Fang, G., Shao, R.,

and Chen, Y. FlowCog: Context-aware Semantics Extraction and Analysis of Information Flow Leaks in Android 

Apps. In 27th USENIX Security Symposium (Baltimore, 

MD, 2018). 

[38] Pandita, R., Xiao, X., Yang, W., Enck, W., and Xie,

T. WHYPER: Towards Automating Risk Assessment of Mobile Applications. In Presented as part of the 22nd USENIX

Security Symposium (Washington, D.C., 2013), USENIX. 

[39] Qu, Z., Rastogi, V., Zhang, X., Chen, Y., Zhu, T.,

and Chen, Z. AutoCog: Measuring the Description-topermission Fidelity in Android Applications. In Proceedings

of the ACM Conference on Computer and Communications

Security (New York, NY, USA, 2014), CCS ’14, ACM. 

[40] Rahmati, A., Fernandes, E., and Prakash, A. Applying the Opacifed Computation Model to Enforce Information Flow Policies in IoT Applications. In IEEE Cybersecurity

Development (SecDev) (2016). 

[41] Sikder, A. K., Aksu, H., and Uluagac, A. S. 6thsense: 

A context-aware sensor-based attack detector for smart devices. In 26th {USENIX} Security Symposium ({USENIX} 

Security 17) (2017), pp. 397–414. 

[42] Sikder, A. K., Aksu, H., and Uluagac, A. S. A 

context-aware framework for detecting sensor-based threats 

on smart devices. IEEE Transactions on Mobile Computing

(2019). 

[43] Sikder, A. K., Babun, L., Aksu, H., and Uluagac,

A. S. Aegis: A context-aware security framework for smart 

home systems. ACSAC (2019). 

[44] Sikder, A. K., Petracca, G., Aksu, H., Jaeger, T.,

and Uluagac, A. S. A survey on sensor-based threats 

to internet-of-things (iot) devices and applications. arXiv

preprint arXiv:1802.02041 (2018). 

[45] Pardis E. and Yuvraj A. and Lorrie F. C. and

Hanan H. Ask the Experts: What Should Be on an IoT 

Privacy and Security Label? arXiv preprint arXiv:2002.04631

(2020). 

[46] SmartThings Classic Documentation: Classes and

JARs. https://docs.smartthings.com/en/latest/gettingstarted/groovy-for-smartthings.html#allowed-classes. [Online; accessed 26-August-2020]. 

[47] SmartThings Code Review Guidelines and Best

Practices. http://docs.smartthings.com/en/latest/codereview-guidelines.html. [Online; accessed 26-August-2020]. 

[48] SmartThings Community Forum for Third-party

Apps. https://community.smartthings.com/. [Online; 

accessed 26-August-2020]. 

[49] SmartThings Groovy IDE. https://graph.api. 

smartthings.com/. [Online; accessed 26-August-2020]. 

[50] SmartThings Official App Repository. https://github. 

com/SmartThingsCommunity. [Online; accessed 26-August2020]. 

[51] SmartThings Official Developer Documentation. 

http://docs.smartthings.com. [Online; accessed 26-August2020]. 

[52] SmartThings Supported IoT Products (Devices). 

https://www.smartthings.com/products. [Online; accessed 

26-August-2020]. 

[53] Tian, Y., Zhang, N., Lin, Y.-H., Wang, X., Ur, B.,

Guo, X., and Tague, P. SmartAuth: User-Centered Authorization for the Internet of Things. In 26th USENIX

Security Symposium (Vancouver, BC, 2017). 

[54] Wang, Q., Datta, P., Yang, W., Liu, S., Bates, A.,

and Gunter, C. A. Charting the Atack Surface of TriggerAction IoT Platforms. In Proceedings of 26th ACM Conference on Computer and Communications Security (2019). 

[55] Wang, Q., Hassan, W. U., Bates, A. J., and Gunter,

C. Fear and logging in the internet of things. In Network

and Distributed Systems Symposium (NDSS) (Feb 2018). 

[56] Wikipedia. https://dumps.wikimedia.org/wikidatawiki/ 

entities/. [Online; accessed 26-August-2020]. 

[57] Zhu, D. Y., Jung, J., Song, D., Kohno, T., and

Wetherall, D. TaintEraser: Protecting Sensitive Data 

Leaks Using Application-level Taint Tracking. SIGOPS Operating Systems Review (2011). 

[58] Ren, J. and Dubois, D. J. and Choffnes, D. and Mandalari, A. M. and Kolcun, R. and Haddadi, H. Information Exposure From Consumer IoT Devices: A Multidimensional, Network-Informed Measurement Approach. Proc.

of the Internet Measurement Conf. (2019). 

[59] A. Dorri and S. S. Kanhere and R. Jurdak and P.

Gauravaram User Perceptions of Smart Home IoT Privacy. 

Proc. ACM Hum.-Comput. Interact. (2018). 

[60] Zheng, S. and Apthorpe, N. and Chetty, M. and

Feamster, N. Blockchain for IoT Security and Privacy: 

The case study of a smart home. IEEE PerCom Workshops

(2017). 

[61] R. Chow The Last Mile for IoT Privacy. IEEE Security

Privacy (2017). 

[62] Google AutoML Natural Language Google Training. https: 

//cloud.google.com/natural-language/automl/docs/prepare. 

Online; accessed 26-August-2020. 

[63] T. Song and R. Li and B. Mei and J. Yu and X. Xing

and X. Cheng A Privacy Preserving Communication Protocol for IoT Applications in Smart Homes. IEEE Internet of

Things Journal (2017). 

[64] X. Wang and J. Zhang and E. M. Schooler and M.

Ion Performance evaluation of Attribute-Based Encryption: 

Toward data privacy in the IoT. 2014 IEEE International

Conference on Communications (ICC) (2014). 

[65] OpenHAB Community, Openhab documentation, 

http://docs.openhab.org/index.html (2017). [Online; accessed 26-August-2020]. 

[66] Apple, Apple homekit documentation, 

https://developer.apple.com/homekit/ (2017). [Online; 

accessed 26-August-2020]. 

[67] Microsoft, Windows IoT core documentation, https://developer.microsoft.com/enus/windows/IoT/explore/IoTcore (2017). [Online; accessed 

26-August-2020]. 

[68] AKM I. Newaz and A. K. Sikder and A. M. Rahman

and A. S. Uluagac Healthguard: A Machine Learningbased Security Framework for Smart Healthcare Systems. 

2019 Sixth International Conference on Social Networks

Analysis, Management and Security (SNAMS). 

[69] AKM I. Newaz and A. K. Sikder and A. M. Rahman

and A. S. Uluagac A Survey on Security and Privacy 

Issues in Modern Healthcare Systems: Attacks and Defenses. 
Real-time Analysis of Privacy-(un)aware IoT Applications 163 

arXiv preprint arXiv:2005.07359. 

[70] AKM I. Newaz and A. K. Sikder and L. Babun and

A. S. Uluagac HEKA: A Novel Intrusion Detection System 

for Attacks to Personal Medical Devices. 2020 IEEE Conference on Communications and Network Security (CNS). 

[71] A. K. Sikder and L. Babun and Z. B. Celik and A.

Acar and H. Aksu and P. McDaniel and E. Kirda

and A. S. Uluagac Kratos: Multi-User Multi-DeviceAware Access Control System for the Smart Home. 13th 

ACM Conference on Security and Privacy in Wireless and 

Mobile Networks, 2020. 

[72] L. Babun and H. Aksu and L. Ryan and K. Akkaya

and E. S. Bentley and A. S. Uluagac Z-IoT: Passive Device-class Fingerprinting of ZigBee and Z-Wave IoT 

Devices. 2020 IEEE International Conference on Communications (ICC) 

[73] J. Myers and L. Babun and E. Yao and S. Helble

and P. Allen MAD-IoT: Memory Anomaly Detection for 

the Internet of Things. 2019 IEEE Globecom Workshops 

(GC Wkshps) 

[74] L. Rondon and L. Babun and K. Akkaya and A. S

Uluagac HDMI-Walk: Attacking HDMI Distribution Networks via Consumer Electronic Control Protocol. ACSAC 

2019 

[75] L. Rondon and L. Babun and K. Akkaya and A. S

Uluagac HDMI-Watch: Smart Intrusion Detection System 

Against HDMI Attacks. IEEE Transactions on Network 

Science and Engineering, 2020 

[76] L. Babun, H. Aksu, A. S. Uluagac, Identifying Counterfeit 

Smart Grid Devices: A Lightweight System Level Framework, 

in: 2017 IEEE International Conference on Communications 

(ICC), 2017, pp. 1–6 (May 2017). doi:10.1109/ICC.2017.

7996877. 

[77] L. Babun, H. Aksu, A. S. Uluagac, A System-level Behavioral Detection Framework for Compromised CPS 

Devices: Smart-Grid, in: ACM Transactions on CyberPhysical Systems, 2019, pp. 1–28 (Nov 2019). http:

//doi.acm.org/10.1145/3355300. 

[78] Babun, Leonardo (Miami, FL, US), Aksu, Hidayet (Miami, 

FL, US), Uluagac, Selcuk A. (Miami, FL, US). 2018. Detection of Counterfeit and Compromised Devices Using System and Function Call Tracing Techniques. (July 2018). 

https://www.osti.gov/biblio/1463864 

[79] Babun, Leonardo (Miami, FL, US), Aksu, Hidayet (Miami, FL, US), Uluagac, Selcuk A. (Miami, FL, US). 2019. 

Method of Resource-limited Device and Device Class Identifcation Using System and Function Call Tracing Techniques, Performance, and Statistical Analysis. (March 2019). 

https://patents.google.com/patent/US10242193B1/en 

[80] L. Rondon and L. Babun and A. Aris and K. Akkaya

and A. S Uluagac PoisonIvy: (In)secure Practices of 

Enterprise IoT Systems in Smart Buildings. accepted at 

BuildSys ’20, 2020 

A Sample Survey Questions 

We present a list of representative IoT privacy survey questions from all the categories due to space 

constraints. The entire user study is available at 

https://anonymous.com. 

A.1 User Characterization 

1. Do you use, have used, or are you planning to use 

any IoT device? 

( ) Yes 

( ) No 

( ) Maybe 

2. What is your technical experience with IoT apps? 

( ) I can build, implement, code my own IoT app 

( ) Installed/can install/confgure an IoT app using 

the source code available online 

( ) Installed/can install/confgure an IoT app’s marketplace (Google Play, App Store, etc.) 

( ) I just know how to press the buttons 

( ) I have no idea how to deal with IoT apps 

A.2 Security and Privacy Concerns in 

Smart Apps 

1. What information would you consider sensitive if 

used in IoT apps/devices? Please check all that 

apply. 

( ) My personal data (e.g., email address, phone 

number, residential address, etc.) ( ) Whatever I 

do, my behavior (e.g., when I arrive home, when I 

leave home, I go to sleep, etc.) 

( ) My (or my devices’) location (e.g., my location 

while using the apps, etc.) 

( ) My device settings/the way I confgure the devices (e.g., Time of the day the lights turn On/O ̇, 

my thermostat temperature settings, etc.) 

( ) My (or my devices’) timing (e.g., the time passed 

since I left home, the time passed since I went to 

sleep, etc.) 

( ) Information from my devices (e.g., device type, 

manufacturer, device IDs, etc.) 

( ) Data from my devices (e.g., door state open or 

close, light on or o ̇, etc.) 

( ) Other 

If you selected "Other", please explain: 

2. Have you heard or personally have privacy concerns 

on the use of the IoT devices and systems? 

( ) Big concerns ( ) Some concerns 

 

 

 

 

 

 

- - - -

-

smart lock control

12:01

< Back Save

IOTWATCH Data Flow Labels:

smart lock control

12:01

< Back Save

IOTWATCH Data Flow Labels:

Real-time Analysis of Privacy-(un)aware IoT Applications 165 

Listing 1. An example of a JSON object sent from an IoT app to 

IoTWatcH for further analysis. 

1 /* An example of a JSON object sent to Daint analytics tool */

2 

3 data "{

4 ’exfiltration’:{

5 ’texttype’:’PLAIN_TEXT’, 6 ’calltype’:’Messaging’, 7 ’phone’:’111-111-1111’, 8 ’content’:’The door was opened for 10 min’ 9 ’userrecipients’:’123-456-7890’,

10 }

11 }" "https://iotwatchanalyticstool.com/classifytext/"

Listing 2. An example IoTWatcH response as a JSON object 

received by an IoT app. 

1 /* An example of a JSON object as response from our analytics tool */

2 

3 data "{

4 ’exfiltration’:{

5 ’texttype’:’PLAIN_TEXT’,

6 ’classification’:[’device-info’, ’date-time’]’,

7 ’risklevel’: ’data leakage’

8 }

9 }"

Sample API Objects Listing 1 illustrates a sample JSON object used to send data from an IoT apps 

to IoTWatcH analyzer. Here, information regarding a 

messaging content and its recipients is collected. The 

fndings are then sent back to the user via another JSON 

object similar to the one depicted in Listing 2. 

LC MwoLoPC IwoLoPC MoLoPC IwLoPC Total % Total 

Device-info 47 12 43 17 119 52.3 

Date-time 7 1 7 0 15 6.6 

User-behavior 21 2 34 10 67 29.3 

Location 8 0 14 5 27 11.8 

Total 83 15 98 32 228 100 

LC - Privacy Label Category 

MwoLoPC - Messaging without Leaks or Privacy Concerns 

IwoLoPC - Internet Calls without Leaks or Privacy Concerns 

MwLoPC - Messaging with Leaks or Privacy Concerns 

IwLoPC - Internet Calls with Leaks or Privacy Concerns 

Table 2. Distribution of privacy labels 

Privacy Labels Table 2 summarizes the distribution 

of the privacy labels during evaluation. One can notice 

that, out of 228 di ̇erent labels assigned, 52.3% corresponds to device-info, followed by a 29.3% of userbehavior information. We also show in Table 2 the number of labels assigned to the di ̇erent call types (i.e., 

messaging and Internet). For instance, 83 labels were 

assigned to strings extracted from messaging that do 

not leak information, while 98 labels were assigned to 

strings extracted from messaging that leaked information. On the other hand, Internet communications without privacy concerns received 15 privacy labels while 

32 were assigned to Internet communications that leak 

IOTWATCH has detected data sent 

containing information related to 

your Location. 

Date-time Information 

Device Information 

Location Information 

User Behavior 

Notification Options: 

Notify Leaks via Messaging 

Notify Leaks via Internet 

Information related to your Device 

was sent to www.support.com in 

clear text format. 

Date-time Information 

Device Information 

Location Information 

User Behavior 

Notification Options: 

Notify Leaks via Messaging 

Notify Leaks via Internet 

(a) (b) 

Fig. 3. IoTWatcH’s fndings are informed to the users through 

push notifcations. The fndings include (a) the privacy labels 

assigned to the taint-sink content, and (b) the clear-text transmitted out of an IoT app. 

data. With this distribution, one can notice that the 

majority of labels were assigned to leaked data. These 

results refect on the fact that privacy leakage constitute 

a serious problem in IoT. 

IoTWatcH Notifcation System We illustrate the 

notifcation system implemented by IoTWatch to inform its fndings to the user. In Figure 3(a), the privacy 

tool instruments a push notifcation to inform the user 

that an IoT string related to the user’s location was 

sent out of the app. In Figure 3(b), the tool informs 

that sensitive information was transmitted in clear-text, 

potentially making the data available to passive eavesdroppers. 

D Additional Evaluation Results 

Table 1 details average metric values in classifying IoT 

strings to privacy labels for every di ̇erent th. Also, we 

present the average metric values after combining results from all considered privacy labels and thresholds. 

One can verify that IoTWatcH obtains the best performance for threshold values of 0.4, which supports the 

results showed in Section 7, Figure 6(a). For th values 

higher than 0.5, the accuracy decreases for the privacy 

labels of device-info and user-behavior. This is mainly 

because the evaluation of semantically-limited strings 

related to these two privacy labels requires more sophisticated analysis. For instance, user-behavior achieved incorrect or lower classifcation scores as the string “IoT 

switched to sleep mode” cannot be easily related to user 

activities. We obtained similar results for recall metrics; 

however, recall values of date-time are lower compared 

to other labels. We found that date-time information 

could be easily missed from short strings, which makes 
Real-time Analysis of Privacy-(un)aware IoT Applications 

Classifcation Thresholds Evaluation Metric 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Average 

Accuracy 0.9537 0.9623 0.9692 0.9743 0.9623 0.9469 0.9296 0.9092 0.8750 0.9425 

Recall 0.9754 0.9400 0.9379 09341 0.8821 0.8419 0.7919 0.7359 0.6263 0.8517 

Precision 0.8470 0.8776 0.9163 0.9587 0.9803 0.9833 0.9929 0.9975 0.9835 0.9501 

Specifcity 0.9283 0.9498 0.9682 0.9772 0.9823 0.9855 0.9876 0.9907 0.9907 0.9734 

Table 1. Evaluation results in classifying IoT strings for all classifcation thresholds. The rightmost column is the average values. 

Type* App Name Leak Type** Recipient Content Behavior Privacy Labels 

Squeeze Box Controller [48] I http://$ip:$port “Open box” Device-info 

StatHat Quick Start [48] I http://api.stathat.com/ez “Thermostat1:thermostat” Device-info 

ThingSpeak Logger [48] I http://api.thingspeak.com “DeviceID:333:Key:123” Device-info 

User Lock Manager [48] M defned by user “User no longer has access to the door” Device-info, User-behavior 

Smart Lock [48] M defned by user “SmartLock disabled. Door unlocked indefnitely” Device-info, User-behavior 

1 

Fire Alarm [28] I http://stmalware/maliciousServer.php – – 

Ransomware [28] I http://stmalware/maliciousServer.php – – 

Remote Command [28] I http://stmalware/maliciousServer.php – – 

Spyware [28] M 123 − 456 − 7890 “Doors locked after everyone departed” Device-info, User-behavior 

User Event [28] M 123 − 456 − 7890 “Everyone is away and Hub ID is 123” Device-info, Location, User-behavior 

2 

* 

166 

1 is for Market IoT apps and 2 is for Malicious IoT apps (Handcrafted) 

** I is for Internet and M is for Messaging. is for Privacy Risk and is for Privacy Preference of a user. 

Table 3. Examples of privacy risks and the use of sensitive information in market and malicious IoT apps. 

the label specially vulnerable to false negative events. – 

The precision and specifcity improves with th for all 

the labels, which denotes a remarkable confdence of 

the model for those results with the highest classifca- – 

tion scores. 

Results from Real-life Apps. Table 3 presents IoTWatcH’s results and fndings after analyzing real mar- – 

ket and handcrafted apps. As can be seen, IoTWatcH

was able to identify leaks happening due to both messaging and Internet communications. In all the cases, – 

it reported back to the user the type of sensitive information leaked (based on the four privacy labels) and 

its recipients (phone number in case of messaging and – 

server URL in case of Internet calls). In some specifc 

cases (“User Lock Manager” and ”Smart Lock” apps), 

no leak was detected as IoTWatcH found the recipients 

were defned by the user at install time, yet IoTWatcH

informed the user the type of information handled by – 

the apps accordingly. Finally, we note that privacy leaks 

were fagged in all the cases as the apps did not report 

handling these types of information nor their recipients 

to the user via app’s description blocks. – 

E Evaluation Metrics 

The performance metrics used during IoTWatcH’s 

evaluation: 

– True Positive (TP) represents the number of times 

a privacy label is correctly applied to an IoT string 

for certain threshold th. 

True negative (TN) represents the number of times 

a privacy label is correctly discriminated for certain 

threshold th. 

False Positive (FP) is the number of times a privacy 

label is incorrectly assigned to certain IoT string for 

certain threshold th. 

False Negative (FN) is the number of times a privacy label is incorrectly discriminated for certain 

threshold th. 

Accuracy is the overall ability of IoTWatcH to correctly apply privacy labels to the IoT string for every di ̇erent th. 

Recall is the ability of the classifer to correctly assign the privacy labels to a specifc IoT string after 

considering both the correctly classifed and the incorrectly ignored privacy labels for every value of 

th. 

Precision is the ability of our classifer to correctly 

apply the privacy labels to a specifc IoT string after considering both the correct and the incorrectly 

applied privacy labels for every value of th. 

Specifcity is the ability of our tool to discriminate 

the privacy labels for every di ̇erent th.