1
Quality of Information in Mobile Crowdsensing: Survey andResearch Challenges

arXiv:1709.03583v1 [cs.CY] 7 Sep 2017

FRANCESCO RESTUCCIA, Northeastern University, USA
NIRNAY GHOSH, SHAMEEK BHATTACHARJEE, and SAJAL K. DAS, Missouri S&T, USA
TOMMASO MELODIA, Northeastern University, USA
Smartphones have become the most pervasive devices in people’s lives, and are clearly transforming the way
we live and perceive technology. Today’s smartphones benefit from almost ubiquitous Internet connectivity
and come equipped with a plethora of inexpensive yet powerful embedded sensors, such as accelerometer,gyroscope, microphone, and camera. This unique combination has enabled revolutionary applications basedon the mobile crowdsensing paradigm, such as real-time road traffic monitoring, air and noise pollution, crime
control, and wildlife monitoring, just to name a few. Differently from prior sensing paradigms, humans are
now the primary actors of the sensing process, since they become fundamental in retrieving reliable andup-to-date information about the event being monitored. As humans may behave unreliably or maliciously,
assessing and guaranteeing Quality of Information (QoI) becomes more important than ever. In this paper, we
provide a new framework for defining and enforcing the QoI in mobile crowdsensing, and analyze in depth thecurrent state-of-the-art on the topic. We also outline novel research challenges, along with possible directionsof future work.
CCS Concepts: • Human-centered computing → Smartphones; Reputation systems; • Computer systems
organization → Sensor networks; • Security and privacy → Trust frameworks;
Additional Key Words and Phrases: Quality, Information, Truth Discovery, Trust, Reputation, Framework,
Challenges, Survey
ACM Reference Format:
Francesco Restuccia, Nirnay Ghosh, Shameek Bhattacharjee, Sajal K. Das, and Tommaso Melodia. 2017. Qualityof Information in Mobile Crowdsensing: Survey and Research Challenges. ACM Trans. Sensor Netw. 1, 1,
Article 1 (August 2017), 41 pages.
https://doi.org/0000001.0000001

1

INTRODUCTION

Smartphones have now become one of the most powerful pervasive technologies. According to theEricsson Mobility Report of 2016, the number of smartphone subscriptions reached the staggering
figure of 3.2 billion in 2015, and by the end of 2021, it is projected that there will be nearly 6.3
billion global mobile users, almost one for every person on the planet [45].
This work is partially supported by the NSF, under grants CNS-1545037, CNS-1545050, DGE-1433659 and IIS-1404673.
Authors’ addresses: F. Restuccia and T. Melodia, Electrical and Computer Engineering Department, Northeastern University,
360 Huntington Ave, Boston, MA 02115, US. Email: {frestuc, melodia}@ece.neu.edu; N. Ghosh, S. Bhattacharjee and S.K. Das,
Department of Computer Science, Missouri University of Science and Technology, 500 W 15th St, Rolla, MO 65409 US.
Email: {ghoshn, shameek, sdas}@mst.edu.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and thefull citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires
prior specific permission and/or a fee. Request permissions from permissions@acm.org.
© 2017 Copyright held by the owner/author(s). Publication rights licensed to Association for Computing Machinery.
1550-4859/2017/8-ART1 $15.00
https://doi.org/0000001.0000001
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:2

F. Restuccia et al.

Not only are today’s smartphones ubiquitous devices, they are also equipped with a plethora ofembedded multi-modal sensors; integrate wireless communication technologies such as 4G/WiFi
Internet connectivity, and possess complex processing capabilities. For example, the cameras on
smartphones can be used as video and image sensors [29], the microphone can be used as anacoustic sensor [39, 154, 206], and the embedded global positioning system (GPS) receiver can beused to gather accurate location information, while gyroscopes, accelerometers, and proximity
sensors can be used to extract contextual information about the users, such as driving or walking
states [75, 122]. Further, additional sensors can be easily interfaced with smartphones via Bluetoothor wired connections, such as temperature, air quality, and humidity [69, 168].
These technological features, combined with the advanced sensing capability of humans, has
spurred a significant amount of research from both academia and industry, which together haveproposed over the last 10 years a myriad of applications based on the emerging mobile crowdsensing
paradigm1 . Mobile crowdsensing empowers ordinary citizens (or users2 ) with the capability toactively monitor various phenomena pertaining to themselves (e.g., health, social connections) or
their community (e.g., environment). This rich information or inference about themselves or thecommunity may also be sent back to the participating or other concerned users to improve theirlife experiences, thus influencing their choices. Real-life applications, which can take advantage ofboth low-level sensor data and high-level user activities, range from real-time traffic monitoring
applications, to environmental pollution monitoring, crime monitoring, and social networking, justto name a few. For a survey on mobile crowdsensing applications, we refer the reader to [92].
Although mobile crowdsensing allows the creation of disruptive technologies and applications, italso comes with novel and unique research challenges, which were absent in less complex sensing
paradigms such as wireless sensor networks. The most important challenges that we will analyze
in this paper can be summarized by the following three questions:
How can we trust that the smartphone-human sensors will send useful information?
How can we enforce the submission of useful information?
How can we estimate the usefulness of the submitted information?
For each question, we have highlighted a key word that epitomizes a research challenge. Below,
we briefly discuss each keyword and introduce the related research challenges that will be discussed
in this paper.
• Useful. Mobile crowdsensing systems can be used for many purposes, all of them very
different from each other and each of them requiring different constraints on the natureof information. For example, in a traffic monitoring application, information about traffic
status on a given road should be refreshed every few minutes to allow motorists to arrange
their routes in real time according to the traffic conditions. On the other hand, in an air
pollution application, such sensing granularity may not be necessary, as information regarding
pollution usually may not change as often. In such diverse and complex context, is it possible
to define the notion of useful information, also referred to as quality of information (QoI)?
Specifically, what are the characteristics that define QoI for the wide variety of the mobile
crowdsensing systems nowadays available? Can traditional definitions of QoI, for example,
in sensor networks [17], be applied to mobile crowdsensing?
1 Henceforth, we will refer to the term mobile crowdsensing to designate applications where participants voluntarily contributesensor data for their own benefit or for the benefit of the community by using their phones. Such a notion therefore includes
participatory sensing [20], mobiscopes [1], opportunistic sensing [23], and equivalent terms such as mobile phone sensing
and smartphone sensing [98], or mobile crowdsensing [65]. It also covers specific terminologies such as urban sensing [23],citizen sensing [20], people-centric sensing [23], [24], and community sensing [97].
2 In the following, we will use the terms “participants” and “users” interchangeably, as well as “system”, “application” and“platform”.
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:3• Enforce. Once the notion of QoI has been defined, a mobile crowdsensing system has toprovide a set of minimum QoI requirements that need to be satisfied. For example, an air
pollution application may require to receive 100 sensing reports from 10 different sources ina given neighborhood of a city during an hour. In traditional sensor networks, this challenge
could be “easily” overcome by deploying more sensors and making sure constraints on the
bandwidth and packet delivery rate are satisfied, usually by solving complex optimization
problems [43]. However, in mobile crowdsensing the control of the sensing process is mainly
delegated to people, who may choose to deliver (or not) relevant information depending on
their willingness and personal constraints, such as time, smartphone battery, available network
bandwidth, etc. Other factors may also influence the QoI of submitted sensing reports. For
example, smartphone sensors are not typically of the same fidelity as task-specific sensors(e.g., a noise pollution monitor vs. smartphone microphone). Also, inadvertent erroneous
handling of the phone (e.g., capturing blurry photos by mistake), or lack of experience inperforming the sensing tasks may adversely influence the QoI, irrespective of the willingnessof the participants. Moreover, the degree of participation is also heavily influenced by theparticipants’ current mobility pattern, which frequently changes over time as most users ofmobile crowdsensing systems are pedestrian, drivers, or people commuting to their workplace[141]. Due to the strong human component in the sensing process, enforcing QoI in mobile
crowdsensing is a peculiar issue and requires considerable research effort.
• Estimate. After having received the sensing information from the participants, but before
using it, the crowdsensing system must be able to make sure the information containedin the sensing reports is actually compliant to the QoI constraints required by the sensing
application. However, it is significantly challenging to estimate the QoI of sensing reports
submitted by smartphone-human sensors in the wild. First, smartphone sensors could be faulty,
uncalibrated, or tampered with. Second, some participants may exhibit malicious behavior,trying to disrupt the sensing application by deliberately sending low-QoI information. For
example, in 2014 a group of students from Technion-Israel Institute of Technology successfully
simulated through GPS spoofing a traffic jam on the Waze [194] road traffic monitoring
application, currently used by more than 50 million people worldwide [173]. This simulation
lasted hours, causing thousands of motorists to deviate from their planned routes [7]. Third,
the information contained in sensing reports coming from the same geographical area at(almost) the same time could be highly conflicting, further complicating the QoI estimation
process as traditionally used techniques such as sensor fusion might become ineffective[101]. The unique combination of these aspects makes the issue of estimating QoI in mobile
crowdsensing a significantly challenging process.
To the best of our knowledge, existing literature lacks a framework encompassing and surveying
the main aspects of QoI for mobile crowdsensing systems, including the definition of existing
and future research challenges in this field. For this reason, within the scope of this manuscript,
we analyze the state-of-the-art in QoI as applied to mobile crowdsensing campaigns; besides
describing the solutions currently applied, we also highlight their limitations, and discuss open
issues as well as their impact on QoI sensing surveys. Specifically, our novel contributions can besummarized as follows. First, by surveying the most relevant mobile crowdsensing applications
currently available, we propose and discuss a unified framework that defines the concept of “qualityof information” (QoI) in mobile crowdsensing systems. Second, we define and analyze in detail the
research challenges in enforcing and estimating the QoI in mobile crowdsensing systems, as well
as discussing the already existing related work on the topic. Finally, by analyzing the limits and
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:4

F. Restuccia et al.

weaknesses of existing work, we provide a roadmap of possible directions of novel future researchthat we hope will inspire additional research in this exciting field.

Summary
Conclusions(Section 8)

Background Concepts
Paper Motivation(Sections 2-3)

Definition of QoI
for Mobile
Crowdsensing(Section 4)

Limitations andResearch Challenges(Section 7)

Survey of Strategies
to Estimate andEnforce QoI(Sections 5-6)

Fig. 1. Paper organization - A top-down approach to QoI.

Figure 1 outlines the organization of the paper – we will present our work by following a topdown approach. Section 2 provides background information on mobile crowdsensing and briefly
surveys related work on QoI. Section 3 summarizes the motivations of this work by highlighting
the differences between mobile crowdsensing and prior sensing paradigms. Section 4 proposes anovel framework that outlines the operations needed to define, enforce and estimate QoI in mobile
crowdsensing systems. We then survey the state of the art pertaining to each component of ourframework in Sections 5 and 6. At the end of these two sections, we discuss in details the limitationsof existing work and provide suggestions on how to improve it. Section 7 presents open researchchallenges toward improving the QoI in mobile crowdsensing, while Section 8 concludes the paper.
2 WHAT IS MOBILE CROWDSENSING?
In this section, we briefly introduce the concept of mobile crowdsensing and discuss related workon QoI in other fields different from mobile crowdsensing.
Mobile crowdsensing is a paradigm that was first proposed in [20]. In a nutshell, it leverages
the widespread usage of smartphones to acquire up-to-date and fine-grained information abouta location (or event) of interest. As far as the architectural components are concerned, most of
the existing mobile crowdsensing systems are generally composed of the participants, the sensing
application (app), and the mobile crowdsensing platform (in short, MCP).
• The core component of the whole architecture are the participants, whose task is to use theirsmart devices to capture different kinds of sensor data, such as location, images, sound samples,
accelerometer data, biometric data, and barometric pressure, just to name a few. However,more complex information about the sensing area or an environmental phenomenon, suchas traffic status or weather information, may also be provided. In practical implementations,participants typically register with the system by providing a username and password [194]that allow their contributions to be uniquely identified.
• The sensing application (app), deployed on the users’ smartphones, is distributed throughcommon application markets like Google Play or App Store, or is retrieved from a mobile cloudcomputing system [46]. It is responsible for providing the users with a friendly interface fordata acquisition and visualization. In particular, data acquisition may be triggered by theusers themselves or elicited by the app, on a one-time on-demand basis or periodically.
• The backend component of the system, and the place where most of the computation is
executed, is the mobile crowdsensing platform (in short, MCP). This platform is usually
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:5

responsible for filtering, elaboration, and redistribution of sensed data, as well as coordinating
every operation performed by the system. This component is usually implemented by a set ofservers dedicated to the processing of sensed data [144]. Furthermore, the MCP also ensuresefficient storage and elaboration of the sensed data coming from the users, which may be
stored in some relational databases [60], or databases specially adapted to the managementof sensor readings. Along with the main elaboration system, the MCP might leverage areputation system and an incentive mechanism. Briefly, the target of a reputation system isto predict the reliability of the data sent by the users based on their past behavior [78, 188]so as to filter out unreliable reports. Conversely, the target of an incentive mechanism is toencourage user participation by appropriately rewarding the users for their contributions to
the mobile crowdsensing campaign [147].
3

WHY IS THE QOI PROBLEM IN MOBILE CROWDSENSING SO CHALLENGING?

Traditionally, the notion of quality in sensor networks has been typically thought of as a derivativeof the traditional quality of service (in short, QoS) in networks, typically assessed along various
QoS attributes, such as bandwidth, delay, delay jitter, and packet loss probability. Therefore, the“quality” level of a network translates into a judgement statement about the network’s ability toattain various levels for these attributes that are appropriate (hence of value) for a particular classof applications, such as best effort and constant or variable bit-rate.
Analogously, one would expect that the “goodness” of a mobile crowdsensing system in performing its tasks of producing and providing sensory information to stakeholders would relate to QoS
attributes, and how well the values of these attributes reflect and accommodate the informationneeds of applications that consume it. However, these network-oriented quality aspects of sensor
networks do not capture in their entirety the application-oriented quality aspects of the system.
To make this point, we report a citation by Peter Drucker, first published in his book “Innovationand Entrepreneurship” (1985). Focused on customer satisfaction, he defined quality as follows:“Quality in a product or service is not what the supplier puts in.
It is what the customer gets out and is willing to pay for. ”
From this simple yet very profound statement, we can derive that quality is something which isusually perceived by the final user, and not what the object is per se. By extending this definition to thecontext of mobile crowdsensing, we derive that the information generated in mobile crowdsensing
systems becomes of value (and therefore, has quality) only when the system (or a portion of it)
depends on that information, and in particular, needs that information to implement its functionality.
Therefore, the standard definition of QoS for networks cannot be applied to define the QoI for
mobile crowdsensing systems.
Furthermore, there are fundamental differences between mobile crowdsensing and other paradigms,
such as traditional crowdsourcing and wireless sensor networks. Below we summarize why QoI in
mobile crowdsensing cannot be defined and attained by traditional methodologies used in otherfields of computer networking.
• Mobile crowdsensing systems handle extremely diverse types of information. Since its inception
in 2006, a plethora of different mobile crowdsensing systems have been proposed for manydifferent purposes, from emotional and health monitoring [134, 139] to noise mapping systems[39, 136], road traffic monitoring [155, 210], discovery of people in distress [2, 148], and so on.
Each system uses different kinds of sensors and may or may not require interaction with theparticipants. Therefore, traditional definitions of QoI hardly apply to mobile crowdsensing
systems. Therefore, the QoI definition must include a broader and more comprehensive viewof QoI to entail the variety of applications currently available.
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:6

F. Restuccia et al.
• We can’t assume there will be a steady flow of information received by the system. In traditional
sensing paradigms, it is reasonable to assume the system will obtain a steady amount of
information generated by the network in a given time frame. This may be achieved byleveraging QoS-aware algorithms and protocols for medium channel access [64, 145, 146],
node discovery [96, 142, 143], routing to the sink node [171], and topology management [203],
among other strategies. Given in mobile crowdsensing humans become the primary actorsof the sensing process, it is not possible to directly apply QoS-aware algorithms to obtain asteady flow of sensed data. This is because such protocols do not account for the fact thathumans may choose to deliver (or not) the information depending on their willingness andconstraints, such as smartphone battery, available network bandwidth, and time to performthe sensing service. Humans are also highly mobile, meaning that their trajectory may not beknown a priori and it is rarely controllable [12]. This aspect is fundamentally different fromtraditional crowdsourcing, where human participants are not generally assumed to be mobile(e.g., Amazon Mechanical Turk). This implies that novel mechanisms must be designed toguarantee a steady flow of information to the system.
• We can’t assume the information collected by the system will be accurate. An abundance of
research work has been dedicated to address the problem of data accuracy, usually throughdata aggregation [72, 151, 163] and trust management [9, 68, 100]. Although these solutions are
significantly effective in the context of sensor networks, it is intuitive that the same techniques
may not be directly applied to the context of mobile crowdsensing. There are several reasonsthat demonstrate this point. First, task-specific sensors deployed in sensor networks have far
better fidelity than the sensors deployed on smartphones, which are designed for simple tasks
such as phone orientation or basic activity recognition (e.g., walking). Second, humans maylack expertise or willingness when performing the sensing tasks. For example, participants
may inadvertently capture blurry photos or place their smartphones in a noisy environment,
thus invalidating their effort in providing meaningful sensor data.
Another crucial issue in mobile crowdsensing, also studied in the context of genericcrowdsourcing, is how to deal with malicious behavior by participants [33, 54, 179]. Indeed,
conversely from prior paradigms, in mobile crowdsensing it becomes extremely simple formalicious users to disrupt the sensing application by sending low-QoI information. Mostoften, malicious participants are motivated by the desire of earning a reward with little
or no effort [53], spamming [150], or influencing other participants’ behavior [7, 71]. The
practical consequence of malicious and unreliable behavior is that the information received
by the system will be highly conflicting, further complicating the QoI estimation process as
traditionally used techniques such as sensor fusion might become ineffective [101].

4

A COMPREHENSIVE FRAMEWORK FOR QOI IN MOBILE CROWDSENSING

Given the wide variety of mobile crowdsensing applications, we need a notion of quality of
information (QoI) flexible and broad enough to entail different applications and existing works on
QoI proposed so far in the literature. This section formulates a coherent and comprehensive QoIframework, as well as briefly discusses related work on QoI.
Definition. The Quality of Information (QoI) in mobile crowdsensing is:(1) the set of constraints specified by the system in terms of Information Quantity andInformation Accuracy, collectively called QoI Constraints;(2) the set of activities in the QoI Loop needed to achieve the QoI Constraints defined by
the system, hereafter referred to as QoI Enforcement and QoI Estimation.
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:7
Data Collection
from Participants (2)Definition of
QoI Constraints (1)Information Accuracy

Information Quantity- Objectivity- Precision- Completeness- Timeliness- Frequency

The QoI Loop

QoI Enforcement (4)QoI Estimation (3)Truth Discovery (Section 5)
Trust Frameworks (Section 6)
Fig. 2. The “QoI loop” of mobile crowdsensing systems.

With the help of the flowchart in Figure 2, let us now discuss in details each of the concepts
pertaining QoI.
(1) Before deployment, the system has to clearly specify the QoI constraints needed by theapplication. Such constraints must be clearly quantified in objective (possibly, numeric) terms.
These constraints depend on the type of data the system is collecting, and the applicationneeds. In details, we define Information Accuracy and Quantity as follows:• As in [8], we use the term Information Accuracy to indicate “the correctness of the information” received by the system. Specifically, we will use this term to encompass boththe elements of intrinsic QoI, which are (i) objectivity and (ii) precision, and contextual
QoI, which are (iii) completeness and (iv) timeliness [185]. Although there might be otherattributes to describe information accuracy, we believe these four attributes are sufficientfor the scope of existing mobile crowdsensing applications.
• Information Quantity defines the amount of information that is received by the system overa period of time, in other words, the frequency of information retrieval [17]. The constraint
on Information Quantity must be specified in numerical terms, e.g., in terms of sensing
reports received in an hour. Such constraint may also vary over time (e.g., more data isneeded during day time than during the night) and over space (e.g., some neighborhoods
are more important than other).
(2) The first (and most fundamental step) in the QoI loop is collecting sensing reports from theparticipants. Given that mobile crowdsensing relies completely on the voluntary participationof citizens, recruiting and incentivizing users’ active participation becomes a key factor. For
this reason, a number of incentive mechanisms have been proposed to increase not only thecollected amount of data, but also its quality. Furthermore, a number of frameworks aimed torecruit participants in order to maximize the coverage of the sensing area have been recently
proposed [30, 61, 85, 86, 104, 182, 183, 208], including detailed reviews of the related literature[58, 81, 147]. Therefore, in this paper we focus on the QoI Estimation and Enforcement stepsof the QoI loop, which has received relatively less attention.
(3) After the data collection phase, but before accepting the information into the system, the QoIof the collected data must be estimated and meaningful information must be extracted fromACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:8

F. Restuccia et al.

the crowdsourced data. This step (called QoI Estimation) is crucial to make sure that the datacollected by the system actually respects the QoI constraints defined by the application. Afterassessing its QoI level, the information can be redistributed to the concerned participants, byusing websites or the sensing app used to transmit the sensed information [57].
(4) After identifying the QoI constraints, the QoI Enforcement step must take place to make surethat the QoI constraints are respected and useful information is inferred from the (most likely
conflicting) sensing reports reported by the participants. Usually, these tasks are accomplishedwith the help of truth discovery [101] or trust frameworks [118]. Given their paramountimportance in the QoI loop, the rest of the paper will be devoted to the discussion of existing
truth discovery (Section 5) and trust frameworks (Section 6).
4.0.1 Related work. Defining and modeling the concepts of quality and quality of information(QoI) has been (and still is) the objective of much work throughout the years. This effort is justified
by the fact that information has always been considered to be an asset that a business unit canplan and exercise control over. This asset aspect of information has given rise to management
processes for ensuring QoI [44], as well as to the industry efforts to establish good data quality
management practices for inter- and intra-business processes [13]. Another example is the ISO
9001 quality standard, which was first proposed in 1987 but is still used to determine quality inmanagement systems. Specifically, ISO 9000 [125] is a family of standards related to the quality
management systems. It defines quality (within its context) as the “degree to which a set of inherent
characteristics fulfills requirements”.
As far as the computerized world is concerned, QoI has been traditionally studied in the context
of enterprise data management systems and processes, the outcome of Web searches [95] and withinformation fusion applications of detection, classification, identification, and tracking for military
applications [34]. The most used definition of QoI in these cases refers to information that is “fit-touse” for a purpose [42]. A plethora of research has also been conducted in the telecommunicationand networking domains to define and enforce QoI. For example, the ITU-T Rec. E.800 [170] deals
with QoS provisioning of digital telecommunication services. Similarly, the IETF RFC 2386 [49]deals with a framework for QoS-based routing in the Internet.
In the sensing domain, the first paper to define and deal with QoI in a holistic way is [17]. In thisarticle, the authors highlight, refine, and extend a two-layer definition of QoI, where the first relates
to context-independent aspects and the latter to context-dependent aspects of an information
product. They also introduce a framework for scoring and ranking information products based on
their value attributes. Building on this novel definition, much work has since been proposed toaddress QoI in networking [28, 51, 111, 164].
5 TRUTH DISCOVERY
In this section, we survey the state of the art in truth discovery in mobile crowdsensing, andalso propose novel research challenges. First, in Section 5.1 we provide an overview of truth
discovery, formalize its problem definition, and discuss related work. Next, in Section 5.2 wediscuss background information necessary to grasp the main concepts of existing truth discovery
frameworks, which are subsequently discussed in detail in Section 5.3. We then conclude the section
by discussing how to improve existing work in Section 5.4.
5.1 Overview, Problem Definition and Related Work
The field of truth discovery has recently attracted enormous attention from the data mining andcrowdsourcing communities [102, 175]. This stems from the fact that the information sent by two
or more participants about the same event is likely to be conflicting. As discussed earlier, thisACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:9

may be due to several reasons, such as erroneous handling of the smartphone, noisy environment,
typing error, or network delay, and so on.
To explain this point, let us assume that three participants, also called sources in the lexicon oftruth discovery, are reporting information regarding the gas price at a gas station at a particulartime instance. The reports submitted by the participants contain the following information: $2.89,$2.57, and $2.85. The fundamental question that one may ask is which quantity best represents
the real status of the event (in this case, the gas price at that particular station), and also what
led the participants to report an unreliable piece of information. In this and many more sensing
scenarios, it is essential to aggregate noisy information about the same event collected from different
participants. One approach could be to use simple strategies, such as majority voting or averagingof the sensing reports. However, such strategies may not be effective as they do not consider thatparticipants may not be equally reliable, or they may collude.
For example, referring to the previous gas station example, we know that one report may be
more reliable than the other as it comes from the owner of the gas station. Therefore, the accuracyof aggregated results can be improved by capturing the reliabilities of participants. The challenge
is that the reliability of participants is usually unknown a priori and has to be somehow inferred.
Let us now introduce some definitions and notations that will be used throughout this section
for clear and consistent discussions.
• Entity (or object): An entity j ∈ E is the object or quantity being monitored. This can be either
air quality measurement, or traffic situation at a particular location, or presence of potholes
on a given street, and so on;• Source (or participant): A source i ∈ S generates a sensing report regarding an entity. Thus,
if entity j is observed by source i, we denote the report as x i, j ;• Source-claim matrix: A source-claim matrix SC is of the order S × E. Each entry Si C j keeps
track of whether a source i has made any claim for the entity j;• Reliability (or weight): The reliability w i of participant i is a quantity that expresses theprobability of participant i to provide reliable information, where higher reliability indicatesthat i is more reliable and the information is more likely to be accurate;• Truth: The truth (also truth value) for an entity j, defined as x j∗ , is the information selected as
the most reliable among the set of candidate observations submitted by different sources.
Based on these definitions, we define the general truth discovery problem as follows.
Definition 1. Truth Discovery Problem. Let S be a set of sources and E the set of entities being
monitored. For each entity j ∈ E, find an optimal truth value {x j∗ }j ∈E , from a set of observations{x i, j }i ∈S, j ∈ E submitted by different sources.
5.1.1 Related work. Research on truth discovery from the machine learning and data mining
communities has provided several interesting results. The first efforts on the topic attempted tosolve a similar problem in information networks using heuristics whose inspiration can be traced
back to Google’s PageRank [19]. PageRank iteratively ranks the credibility of sources on the Web,by iteratively considering the credibility of sources who link to them.
Extensions of PageRank, known as fact-finders, iteratively compute the credibility of sources andclaims. Specifically, they estimate the credibility of claims from the credibility of sources that makethem (source reputation estimation), then estimate the credibility of sources based on the credibilityof their claims (truth computation step). More formally, the reliability (weight) w i of participant i
at the k t h step of the iterative algorithm is calculated in terms of the trustworthiness of its claims∗
on entity j in the previous iteration x i,k −1
j , and the truth of each claim x j in terms of {w i }i ∈S . The
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:10

F. Restuccia et al.

truth computation step and source reputation estimation are repeated until convergence. A generaltruth discovery algorithm is provided in Algorithm 1.
ALGORITHM 1: General Framework for Truth Discovery.
Input: Observations from sources {x i, j }i ∈S, j ∈E
Output: Optimal truths {x j∗ }j ∈E and the estimated source reliability {w i }i ∈S
Initialize: Source weights {w i }i ∈S ;while Convergence not achieved dofor each j ∈ E do
Truth computation: infer truth for entity j based on the current estimation of source weights;
end
Source weight estimation: update source weights {w i }i ∈S based on the current identified truths;
end
Return: {x j∗ }j ∈ E , {w i }i ∈S ;One of the first efforts in this sense was proposed in [202], where the authors introduced
TruthFinder as an unsupervised fact-finder for trust analysis on a providers-facts network. TruthFinder
calculates the “probability” of a claim by assuming that each source’s reputation is the probabilityof it being correct, and then averages reputations to obtain trustworthiness scores for each claim.
In details,P
k −1
Yj ∈S i C j x i, jkx jk = 1 −(1 − w ik )(1)
wi =|{Si C j }j ∈E |i ∈S Cij

Similarly, Pasternack et al. extended such framework in 2010 by proposing three algorithms
on fact-finding, namely Average.Log, Investment, and Pooled Investment [128]. Other fact-finders
enhanced the basic framework by incorporating analysis on properties or dependencies withinassertions or sources [55, 132, 174]. For example, Galland et al. took the notion of hardness offacts into consideration by proposing the algorithms Cosine, 2-Estimates, and 3-Estimates [55].
The Bayesian Interpretation scheme presented by [174] provided an approximation approach totruth estimation. This interpretation lead to a general foundation for using information network
analysis to quantify, in an analytically-founded manner, the probability that each fact or source iscorrect. However, this approach is very sensitive to initial conditions of iterations, as pointed out
by [178]. More recently, new fact-finding algorithms have been designed to address the backgroundknowledge [129], multi-valued facts [209], and multi-dimensional aspects of the problem [204].
There exists a significant amount of literature in the machine learning community to improve data
quality and identify low quality labelers in a multi-labeler environment. For a comprehensive survey,
the reader may refer to [50]. Sheng et al. proposed in [157] a repeated labeling scheme to improvelabel quality by selectively acquiring multiple labels and empirically comparing several models that
aggregate responses from multiple labelers. Dekel et al. applied a classification technique to simulate
aggregate labels and prune low-quality labelers in a crowd to improve the label quality of the
training dataset [36]. However, all of the above approaches made explicit or implicit assumptions
that may not be applicable for mobile crowdsensing applications. For example, [157] assumed
labelers were known a priori and could be explicitly asked to label certain data points. Moreover,[36] assumed most of labelers were reliable and the simple aggregation of their labels would beenough to approximate the ground-truth. In contrast, participants in social sensing usually upload
their measurements based on their own observations and the simple aggregation technique (e.g.,majority voting) was shown in [128] to be inaccurate when the reliability of participant is not
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:11

sufficient. More recently, in [47] Fang et al. formulated the problem of improving labeling quality
by characterizing the knowledge set of each labeler. Tarasov et al. proposed in [166] an algorithm
called dynamic estimation of rater reliability in regression (DER3), which dynamically estimates
the reliability of participants and accepts only those participants who are deemed to be reliable.
Finally, maximum likelihood estimation (MLE) [119, 130, 158, 184] and Kalman Filtering (KF)
[110, 123, 190] frameworks have been widely used in the wireless sensor networks (WSNs) and data
fusion communities. For example, Pereira et al. proposed in [130] a diffusion-based MLE algorithm
for distributed estimation in WSN in the presence of data faults. Sheng et al. developed in [158] a
MLE method to infer locations of multiple sources by using acoustic signal energy measurements.
Msechu et al. proposed in [119] a MLE based approach to aggregate the signals from remote sensor
nodes to a fusion center without any inter-sensor collaborations. Olfati-Saber proposed in [123]three algorithms based on KF that use consensus filters and covariance information for fusion of
the sensor data. Masazade et al. study the problem of target tracking based on energy readings of
sensors in [110], and propose a framework to minimize the estimation error of sensors by using
a Kalman filter. The same problem is tackled by Wang et al. in [190], where the authors use the
combined effect of MLE and KF to remove the sensing nonlinearity and obtain better tracking
performance. The above work on MLE and KF primarily focused on the estimation of continuous
variables from physical sensor measurements, and does not take into account any contextual
information, e.g., the reputation of sources.
5.2 Background Concepts
We briefly provide some background on mathematical models and concepts that will be usedthroughout this section.
5.2.1 Maximum Likelihood Estimation (MLE). Suppose there is a sample X = {x 1 · · · x n } ofindependent and identically distributed (i.i.d.) observations, coming from a distribution with an
unknown probability density function (p.d.f.) f (·). It is assumed that f (·) belongs to a family of
distributions f (·|θ ), θ ∈ Θ called the parametric model, where θ is a vector of parameters for thisfamily such that f = f (· | θ ). The value of θ is unknown and is referred to as the true value of theparameter vector. It is desirable to find an estimator θˆ which would be as close to the true value θas possible. Either or both the observed variables x i and θ can be vectors.
To use the method of maximum likelihood, the densities for all observations must be specified.
For an i.i.d. sample, this joint density function isf (x 1 , x 2 , . . . , x n | θ ) = f (x 1 | θ ) × f (x 2 |θ ) × · · · × f (x n | θ ).

(2)

Now let us look at this function from a different perspective by considering the observed values X
to be fixed “parameters” of this function, whereas θ will be the function’s variable and allowed tovary freely; this same function will be called the likelihood function:L(θ ; x 1 , . . . , x n ) = f (x 1 , x 2 , . . . , x n | θ ) =

n
Y
i=1

f (x i | θ ).

(3)

Note that the semicolon denotes a separation between the two categories of input arguments:
the parameters θ and the observations X . In practice the algebra is often more convenient whenworking with the natural logarithm of the likelihood function, called the “log-likelihood”:
ln L(θ ; x 1 , . . . , x n ) =

n
X
i=1

ln f (x i | θ ),(4)

ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:12

F. Restuccia et al.

or the average log-likelihood ℓˆ = (1/n ) ln L. The hat over ℓ indicates that it is akin to some estimator.
Indeed, ℓˆ estimates the expected log-likelihood of a single observation in the model. The methodˆ ; x ). This method ofof maximum likelihood estimates θ by finding a value that maximizes ℓ(θestimation defines a maximum likelihood estimator (MLE) of θ :ˆ ; x 1 , . . . , x n )},{θˆmle } ⊆ {arg max ℓ(θ

(5)

θ ∈Θ

if a maximum exists. An MLE estimate is the same regardless of whether we maximize the likelihoodor the log-likelihood function, since log is a monotonically increasing function. Above, it is assumed
that the data are independent and identically distributed. The method can be applied however to abroader setting, as long as it is possible to write the joint density function f (x 1 , x 2 , . . . , x n | θ ), andits parameter θ has a finite dimension which does not depend on the sample size n.
An MLE coincides with the most probable Bayesian estimator given a uniform prior distribution
on the parameters. Indeed, the maximum a posteriori estimate is the parameter θ that maximizes
the probability of θ given the data:
P (θ | x 1 , x 2 , . . . , x n ) =f (x 1 , x 2 , . . . , x n | θ ) · P (θ )P (x 1 , x 2 , . . . , x n )(6)where P (θ ) is the prior distribution for the parameter θ and where P (x 1 , x 2 , . . . , x n ) is the probabilityof the data averaged over all parameters. Since the denominator is independent of θ , the Bayesianestimator is obtained by maximizing f (x 1 , x 2 , . . . , x n | θ ) · P (θ ) with respect to θ .
For many models, an MLE can be found as an explicit function of the observed data X . For manyother models, however, no closed-form solution to the maximization problem is known or available,
and an MLE has to be found numerically using iterative mechanisms, such as expectation maximization (EM). For some problems, there may be multiple estimates that maximize the likelihood.
For other problems, no maximum likelihood estimate exists and either the log-likelihood function
increases without ever reaching a supremum value, or that the supremum does exist but is outside
the bounds of Θ, the set of acceptable parameter values.
To make an example with Bernoulli variables, let us suppose that X = x 1 · · · x n represents theoutcomes of n independent Bernoulli trials, each with success probability p. Since the observations
are taken from i.i.d. random variables, the distribution isL(p ; x 1 , . . . , x n ) =The log-likelihood is therefore:nYf (x i ; p) =i=1

ln L(p ; x 1 , . . . , x n ) =

n
Y
i=1

n
X
i=1

P

P

p x i (1 − p) 1−x i = p i x i (1 − p) n− i x ix i · ln p + (n −

n
X
i=1

x i ) · ln(1 − p)

(7)

(8)

Differentiating ln L(p ; x 1 , . . . , x n ) with respect to p and setting the derivative to zero shows thatPP
this function achieves a maximum at p̂ = ni=1 x i /n. Since ni=1 x i is the total number of successes
observed in the n trials, p̂ is the observed proportion of successes in the n trials.

5.2.2 Expectation Maximization (EM). Expectation maximization is a general optimization technique for finding the maximum likelihood estimation of parameters in a statistical model, where
the model depends on unobserved latent variables [37]. It iterates between two main steps (namely,
the E-step and the M-step) until the estimation converges (i.e., the likelihood function reaches the
maximum).
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:13(1) E-step: Calculate the expected value of the log-likelihood function, with respect to the conditional probability distribution of Z given X under the current estimate of the parametersθ (t ) :Q (θ |θ (t ) ) = EZ |X,θ (t ) [log L(θ ; X, Z)](2) M-step: Find the parameter θ (t +1) that maximizes the quantityθ (t +1) = arg max Q (θ |θ (t ) )θ

5.2.3 Maximum-A-Posteriori (MAP) Estimation. Assume that we want to estimate an unobserved
population parameter θ on the basis of observations x. Let f be the sampling distribution of x,
so that f (x |θ ) is the probability of x given the underlying population parameter is θ . Then thelikelihood function is given as:θ 7−→ f (x |θ )(9)
The maximum likelihood estimate of θ is as follows:θˆM L (x ) = arg max f (x |θ )(10)
θLet a prior distribution д over θ exists. This allows us to treat θ as a random variable as in Bayesianstatistics. We can calculate the posterior distribution of θ using Bayes’ theorem:f (x |θ ) · д(θ )θ 7−→ f (θ |x ) = R
(11)
f
(x
|ϑ
)
·
д(ϑ
)dϑ
ϑ ∈Θ

where д is the density function of θ , and Θ is the domain of д.
The method of maximum a posteriori estimation then estimates θ as the mode of the posterior
distribution of this random variable:f (x |θ ) · д(θ )θˆM L (x ) = arg max f (θ |x ) = R= arg max f (x |θ ) · д(θ )
(12)
f
(x
|ϑ
)
·
д(ϑ
)dϑ
θ
θ
ϑ

The denominator of the posterior distribution (so-called marginal likelihood) is always positiveand does not depend on θ and therefore plays no role in the optimization. The MAP estimate of θ
coincides with the ML estimate when the prior д is uniform (that is, a constant function).
In many types of models, such as mixture models, the posterior may be multi-modal. In such acase, the usual MAP-based recommendation is that one should choose the highest mode. However,this is not always feasible as the highest mode may be uncharacteristic of the majority of theposterior. Computation of MAP estimates can be done in several ways: (i) analytically, if the mode(s)of the posterior distribution can be expressed in closed form; (ii) numerical optimization, by using
conjugate gradient method or Newton’s method – this usually requires first or second derivatives,
which have to be evaluated analytically or numerically; (iii) EM, which does not require derivativesof the posterior density; and (iv) Monte Carlo method, which uses simulated annealing.
5.3 Truth Discovery in Mobile Crowdsensing
Over the last five years, a plethora of work has been devoted to develop truth discovery algorithms
for mobile crowdsensing. Figure 3 shows a taxonomy of truth discovery techniques.
The majority of the existing works have leveraged MLE-based Expectation Maximization (EM)algorithms to estimate source reliability and to infer truth value from it. Few works have also usedunsupervised models based on maximum-a-posteriori (MAP), which is closely related to MLE, but
employs an augmented optimization objective which incorporates a prior distribution over thequantity to be estimated. This prior is the additional information available for the entity underconsideration. Thus, MAP estimation can be seen as a regularization of ML estimation. In addition,
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:14

F. Restuccia et al.
Truth Discovery in Mobile Crowdsensing

MLE-based EM
Wang et al. [178]Wang et al. [176]Wang et al. [177]Wang et al. [187]Wang et al. [186]Huang and Wang [76]Marshall et al. [109]Huang and Wang [77]Yao et al. [201]MAP Models
Ouyang et al. [126]Ouyang et al. [127]Yao et al. [199]Optimization
Methods
Su et al. [162]Meng et al. [115]Ma et al. [107]OthersMiao et al. [117]Qiu et al. [133]Meng et al. [116]Fig. 3. Taxonomy of Truth Discovery in Mobile Crowdsensing.

some optimization-based approaches have also been presented, which find source reliability andentity truthfulness by assuming them to be constant in alternative iterations. Privacy-preserving
truth discovery and predicting missing sensed information using matrix factorization are the othermiscellaneous approaches available in the literature. In the following subsections, we will give a
brief introduction to these methodologies and summarize the contributions made in these works.
5.3.1 Maximum Likelihood Estimation-based Expectation Maximization. The authors in [178] are
the first to show that an EM formulation is appropriate to solve the truth discovery problem in mobile
crowdsensing. The optimal solution directly leads to an accurate quantification of measurement
correctness as well as participant reliability. This work assumes that the sources are independentand there exists no correlation among the entities. Other works use the basic EM framework andintegrate different aspects of social sensing paradigm, i.e., dependency among sources [177, 201],
physical constraint among sources and correlation among observed variables [176, 186], different
error distribution [187] and topic expertise of sources [77].
One important aspect of EM algorithm-based truth discovery is to ensure convergence. Moreover,as mobile crowdsensing is a real-time paradigm, it becomes necessary to perform truth estimation
in a reasonable time. Hence, test of convergence and computational efficiency are the two key
performance indicators for any EM algorithm and we have investigated these features in theexisting literature. A comparative analysis of different works which have proposed MLE-based EM
algorithms for source reliability and truth inferring are given in Table 1.
As Table 1 shows, most of existing work has not given proof of convergence or analyzed thecomputational complexity of the proposed EM algorithm. Convergence in terms of error in estimate
and corresponding iteration numbers has been analyzed in [77, 109, 178], where the EM algorithm
attains a stable error rate after 3-6 rounds of iterations. Remarkably, these works have consideredbinary claim in terms of true and false, exception being [187] in which uncertain claim has also
been considered. Thus, for n sources, there can be 2n input claim patterns from which the EM has
to infer reliability and truth value. For a large number of sources, which is expected in a mobile
crowdsensing application, the EM algorithm has to find optimal solution from an exponential
search space, yielding non-trivial computation time in the worst case. However, these papers havenot discussed any approximation to generate sub-optimal solutions in reasonable time.
5.3.2 Maximum A Posteriori (MAP) Estimation. The MAP estimator has been extensively usedin [126, 127]. These works tackle the problem of truth estimation in quantitative crowdsourcing
tasks: percentage annotation and object counting (e.g., people# in an image). The authors identified
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey
Paper
Wang et al. [178]Wang et al. [176]Wang et al. [177]Wang et al. [187]Wang et al. [186]Huang and Wang
[76]
Marshall et al. [109]Huang and Wang
[77]
Yao et al. [201]Summary
Inferring source reliability and true observation value
Exploits physical (location) constraints
among sources and correlation amongobserved variables for better estimation ofreliability and truth value
Inferring correctness of individual observation and reliability of sources considering dependencies among the observations.
Truth estimator for cyber-physical systems
considering different error distributions forsources
Finds truth value and source reliability byconsidering correlation among the entities.
Constructed the correlation by modeling thevariables as Bayesian network (BN).
Addresses the problem of determining thetime (of the claim) and location (of thesource) information towards truth discovery
and source reliability
Assigns true values to claim and reliability
to sources by incorporating the context andchallenges encountered while reporting
Inferred truth values of a claim by considering: (i) topic relevance and truthfulnessand (ii) topic awareness and reliability of thesourceEstimates the veracity of information sharedby different sources over the social sensing
platforms (e.g., Twitter, etc) by considering
source dependency. This is motivated fromthe observation that sources tend to repeat
claims of another.

1:15
Variable
BinaryApproach
EM algorithm

Convergence Proof
Empirical analysis

BinaryEM algorithms (OtO + DV)

Not provided

BinaryEM algorithm

Not provided

Ternary

EM algorithm (EM-VTC)

Not provided

BinaryEM algorithm (EM-CAT)

Not provided

BinaryEM algorithm (ST-EM)

Not provided

BinaryEM algorithm (HA-EM)

Empirical analysis

BinaryEM algorithm (TA-EM)

Empirical analysis

BinaryGibbs sampling has beenused to approximate theerror bound in situation
when the number of observation patterns become intractable.

Not provided

Table 1. Summary of Work using MLE

the difference of these tasks from normal object classification, and then proposed an unsupervised
model, called Quantitative Truth Finders (QTF). The QTF has two variants: (i) QTF-A (annotation),and (ii) QTF-M (counting), which model participants’ bias and confidence as Gaussian and Lognormal distribution, respectively. The MAP estimation has been used to impose an informative
prior distribution on true values and biases to restrict generation of infinite number of optimal
solutions using the MLE.
Truth discovery from noisy and unreliable source claims may be attempted by modeling the
popularity of an event location and personal location visit tendencies. For this purpose, the authorsof [127] proposed two unsupervised models, called Truth Finder for Spatial Events (TSE) andPersonalized Truth Finder for Spatial Events (PTSE), respectively. The novelty is that it preserves
participation privacy by avoiding location tracking and enhances estimation accuracy by notignoring missing reports as negative ones. Both models use the MAP estimator to find the optimal
configuration of the random variables that maximize the posterior probability. Yao et al. [199]
identifies the inefficiency related to batch analysis of offline datasets and proposes an online
recursive state estimator to recover ground truth from the streaming data. The sensing time isdivided into equal-sized windows and posterior belief on source is computed at the end of eachwindow by using the prior belief obtained at the beginning through MAP estimation. A comparative
analysis of different works which have used this MAP estimate to infer source reliability and truth
values are given in Table 2.
5.3.3 Optimization-based methods. In truth discovery, the objective is to minimize the “distance”
between ground truth and the inferred true values over all entities. The distance between the
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:16
Paper
Ouyang et al. [126]Ouyang et al. [127]Yao et al. [199]

F. Restuccia et al.
Summary
Truth estimation in quantitative crowdsourcing tasks:
percentage annotation (e.g.,
occupancy %), object counting (people# in an image).
Rather than exact match withthe ground truth, this work
jointly assesses latent quantitative truths and abilitiesof participants from noisy
crowdsourced claims without
any supervision.
Addresses the problem of uncertainties with participant
mobility and reliability intruth finding by avoiding location tracking and ignoring
missing reports.
Proposes an online recursive
state estimator for recovering
ground truth from streaming
data. The objective is to speed
up batch-like processing of
sensory data generated by social sensing platforms.

Variable
Continuous

Approach
MAP-based unsupervised models:
QTF-A (for annotation), and QTFM (for counting). The QTF-A andQTF-M model participant’s bias asGaussian and Log-normal, respectively.

Comp. Efficiency
Asymptotic upper boundnot provided. However,
empirically it has beenshown that QTF-A andQTF-M take 1.5 seconds toprocess 1000 odd tasks.

BinaryMAP-based unsupervised models:
Truth finder for Spatial Events (TSE),
Personalized Truth finder for Spatial
Events (PTSE).

BinaryMAP-based EM algorithm whichruns at the end of each time window. It combines prior belief withnew samples to generate the maximum likelihood posterior belief
and assertion correctness. For parameter estimation inside the time
window, an interpolative estimatoris invoked.

Both TSE and PTSE havebeen experimentally shownto take higher order polynomial time w.r.t inputs: (i)number of participants and(ii) number of events.
Empirically the algorithm
has been shown to growsub-linearly with the number of claims generated.

Table 2. Summary of Work using MAP Estimate

ground truth and any source observation is weighted by the latter’s degree of reliability. Thisimplies that if a reliable source makes incorrect claim, the truth discovery system will be more
penalized while the penalty for a less reliable source will be low. A general form of the optimization
problem can be formulated as follows:XX
argminw i .δ (x i, j − x j∗ )(13)
{w i }, {x j∗ } i ∈S j ∈ E

where δ (.) is the distance function that measures the difference between the information provided
by source i, called x i, j , and the identified truth, called x j∗ . Usual distance functions are: (i) 0-1 loss
function for categorical data, and (ii) L2-norm can be adopted for continuous data.
From Equation 13 we can infer the optimization problem has two variables, namely (i) source
reliability w i and (ii) identified truth value x j∗ . Both of them contribute to the estimation of eachother in subsequent iterations. The general approach to solve such optimization problem is the block
coordinate descent method [15], where for a particular iteration, one of the variables is assumed
constant. The optimal value for the other variable is computed by setting the partial derivative ofloss/error function with respect to the former to zero. Similarly, in the next iteration, the optimalvalue of the first variable is obtained by considering the second variable as constant, and the processcontinues until a predefined convergence criteria is satisfied. In the context of truth discovery,
this leads to solutions in which the truth computation step and source weight estimation step are
iteratively conducted until convergence.
A Generalized Decision Aggregation (GDA) framework has been proposed in [162] to integrate
multi-modal information from distributed sensors. The framework constructs a sensor-entity belief
graph which represents different observations. The observations for a particular entity may be
categorical and source’s confidence on each such category is designed in the form of a decision
vector. Finally, an optimization problem is formulated which outputs the category that yields
minimum error. In [115], ubiquitous correlation among observed entities has been considered while
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:17

inferring the true value. This work developed an optimization function which minimizes two terms(balanced by a hyper-parameter): (i) the disagreement between observations and truths weighed bysource reliability, and (ii) the distance between correlated entities weighed by a similarity function.
Ma et al. identified in [107] a limitation with existing MLE-based truth discovery approaches, andproposed a Fine Grained Truth Discovery model for crowdsourced data (FaitCrowd), which assigns
topics to questions, estimate topic-specific expertise of each source, and learn the true answers
simultaneously. It proposed a joint optimization function which attempts to maximize two aspects:(i) likelihood of generating question content, and (ii) likelihood of generating answers. A hybrid
inference approach, Gibbs-EM, was adopted for sampling (learn hidden variables) and variational
optimization (inference). A comparative analysis of different works which have used optimization
approach to infer source reliability and truth values are given in Table 3.
Paper
Su et al. [162]Summary
Infers truth value by incorporating source confidence

Variable
Continuous

Approach
Block coordinate descent method
to solve a multi-objective (reliability, truth value) optimization

Meng et al. [115]Optimizes disagreement between claim and unknown
truth, weighted by source’s
reliability, considering ubiquitous (spatial + temporal)correlation among entities.
Estimates true value of observed variables by incorporating topic-specific expertise ofeach source

Continuous

Block coordinate descent method
to solve the multi-objective (reliability, truth value) optimization.
A parallelized version of the algorithm is implemented using the
MapReduce Framework.
Hybrid inference method combining sampling and variational optimization, named Gibbs-EM which
is an inference method alternating
between Gibbs sampling and gradient descent

Ma et al. [107]CategoricalComp. Efficiency
O (mnt ) , where n = number of sources (sensors), t =number of events (observations), and m = number ofclasses for each event.
Use of MapReduce framework on Hadoop cluster
gives linear time complexity w.r.t the number of observed entities.
The computational complexity is linear to thenumber of observations.

Table 3. Summary of Work using Optimization Method

5.3.4 Other truth discovery methods. Few works in the literature have identified the privacy
problem in the truth discovery mechanism. As sources are submitting their observations to a serverin an unprotected manner, it can lead to location tracking and disclosure of other personalized
information. Moreover, the reliability scores of the sources are also available to the sensing platform.
In [117], a cloud-enabled privacy preserving truth discovery (PPTD) framework has been proposed.
It sends encrypted distance value between user observation and estimated aggregated values tothe cloud server and uses homomorphic crypto-system to update user’s weight (reliability) without
decrypting the received distance information. On the other hand, the user calculates the cipher-textof weighted data using the encrypted weight received in the previous step. Finally, the cloud server
estimates truth based on the cipher-texts received from the users. The work in [133] is different
from standard truth discovery, however, the broad objective to improve the quality of information
in social sensing paradigm remained the same. The authors addressed the challenges of privacy andmultimedia data quality preservations in mobile crowdsensing applications and proposed SLICER, ak-anonymous privacy preserving scheme. It integrates erasure data encoding scheme with message
exchanging and ensures that the service provider cannot identify the contributor of each sensing
record from a group of at least k participants.
In [116], the authors identified both sparseness and redundancy in crowdsensing data which affects
the overall truth estimation accuracy. Interpolation infers true values of missing entities basedon entity similarities, however, it considers redundancy and sparsity as separate challenges. The
authors proposed Redundancy and Sparsity Tackling (RST) framework which designs an optimization
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:18

F. Restuccia et al.

problem to minimize error in truth inferring. This optimization integrates three terms in a weighted(given by hyper-parameters) sum form:• Matrix factorization term (MF): It identifies K virtual users such that two latent matrices
U ∈ RM ×K and V ∈ RN ×K are formed from a big source-claim matrix of the order M × N ,where M and N are the numbers of source and entity, respectively. Next, these matrices areused to provide a good approximation for each entry of the source-claim matrix X ≈ UV T .
• Regularization term on entity similarity (R1): This term is used to extrapolate the values
for some entities which have not observed by any virtual user. It essentially incorporates asimilarity relationship based on a basic principle - observations from virtual users should not
differ too much on entities that are similar to each other.
• Regularization term on virtual users (R2): By this regularization, the framework assigns
importance to virtual users. If a virtual user is more important, higher penalty will be received
if its observation is quite different from the aggregated one.
Table 4 presents a comparative analysis of different works which have used miscellaneous approaches to infer user reliability and truth values.
Paper

Summary

Variable

Approach

Miao et al. [117]Performs truth discovery by ensuring
privacy of the user’s
data and reliability score using acrypto-system. Parallel PPTD scheme
implemented using
the
MapReduce
Framework.
Addresses the challenges of privacy
andmultimedia
data quality preservations in mobile
crowdsensing applications.

Categorial,
Continuous

Privacy-preserving truth
discovery (PPTD) iteratively updates reliability
and truth value through an
iterative approach. Information exchange between twophases is protected by acloud-based homomorphic
crypto-system.

Continuous
as records

SLICER: Erasure codingbased algorithm (for data
quality) + two transfer
schemes TMU and MCT(for preserving privacy).

Estimates the true
values of the entities
from redundant andsparse data in crowd
sensing applications.

Continuous

Developed
Redundancy
and Sparsity Tackling (RST)
framework. It uses matrix
factorization,
regularizations on entity similarityand virtual users to predict
missing values and finally
aggregate them to infer
the truth. Block coordinate
descent method is used tofind out the optimal valuesof each parameter.

Qiu et al. [133]Meng et al. [116]Convergence
Proof
Empirical analysis

Erasure-coding:deterministic
algorithm;
transfer
schemes have been
reduced to 0-1 knapsack problem for
NP-completeness.
Greedyapproach
proposed for suboptimality.
Theoretical analysis

Comp. Efficiency
Truthestimation
takes much moretime than weight
update.
However,
total computation
time is linear to thenumber of observed
entities
Computational complexity is linear tothe sizes of the sensing records.

Not provided.

Table 4. Summary of Work using Miscellaneous Approaches

5.4 How to Improve Existing WorkIn this section, we have analyzed the state-of-the-art research on the topic of truth discovery, whichis by far one of the most promising approaches for QoI estimation and enforcement in mobile
crowdsensing. Truth discovery approaches can be divided into four broad categories: (i) MLE-based
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:19

EM algorithms, (ii) MAP-based unsupervised models, (iii) Optimization-based methods, and (iv) a
few assorted approaches related to privacy-preserving and guessing the missing information.
Several limitations still linger in existing work, which we expect will be addressed in the near
future, and are enumerated below.
(1) Most of the existing approaches are iterative and computationally expensive. Moreover,convergence guarantees have also not been provided in majority of the EM algorithm-based
schemes. As crowdsensing is real-time application, there is a need to propose lightweight
QoI estimation mechanism;(2) It is still unclear how to address the problem of malicious participants who may collude to
generate a false consensus regarding an event. Further, reliable users may also be compromisedby an adversary and generate false observations. In such cases, mere consideration of source
weights will not help in inferring correct truths. Most of the state-of-the-art papers haveignored this aspect and assumed ideal attack-free sensing environment;(3) Reliability of sources has been considered to be static. This again undermines the fact thatthe sensors in this paradigm are humans and not dumb nodes. As human does not always
behave rationally, a better model is required to capture dynamism in the reliability scores;(4) Most of the works have considered binary claims to realize their models in a simplistic manner.
However, in reality, crowdsensing data can be multi-model viz., text, audio clip, video, image,and so on. Truth discovery on such data is non-trivial and brings new challenges;(5) Crowdsensing is an “open” system, where any user can get himself registered and participate
in the sensing task. There exists no bounds either on the number of users or the observed
entities in the system. Thus, scalability of the truth finding mechanism is an important
requirement. By scalability, we imply that truth can be inferred in reasonable time event if
the number of participants and the entities increase. Very few works in the existing literature
have considered this issue;(6) With the exception of [200], EM algorithm-based works are mostly designed to run on static
datasets. However, the computation becomes less efficient for streaming data as the iterativealgorithm needs to re-run on the whole dataset from scratch every time it gets updated.
Additional research should address this important issue.
6 TRUST FRAMEWORKSIn this section, we introduce and discuss in details existing trust frameworks for mobile crowdsensing. First, in Section 6.1 we survey the foundations of trust in computer science and distributed
systems, and show their analogy with QoI in mobile crowdsourcing systems. We also introduce the“trust loop” problem in mobile crowdsensing and briefly discuss related work. Then, by referring toexisting work, we discuss the methods and approaches for collection of trustworthiness evidence
in Section 6.2 and computation of trust scores in Section 6.3. We conclude the section by discussing
how to improve existing work in Section 6.4.
6.1 Overview, Problem Definition, and Related Work
The original concept of trustworthiness has its origins in human society. According to the MerriamWebster dictionary [195], trust is the “[...] belief that someone or something is reliable, good, honest,
effective, etc.” In other words, trust is a qualitative way of expressing whether a particular interaction
with an entity is going to be dependable, given evidence of outcomes of prior or current behaviors
and interactions. Over the years, the concept of trust has significantly evolved and has also been
employed in the computing, communication and networking fields. Analogically, each node in a
network can be viewed as a human, whereas the network itself may be seen as a society where
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:20

F. Restuccia et al.

decisions are made based on inputs/contributions from each node. This idea is often termed
as mechanical or technological trust, a quantitative measure of trustworthiness for interactions
between entities [67]. In the context of mobile crowdsensing, services are offered based on thecontributions/information provided by the participating users, either through mobile devices
or automated sensors [92]. Hence, it is reasonable to relate the trustworthiness of the overall
contributions from multiple mobiles and sensors to the Quality of Information (QoI).
In order to represent, quantify, and manipulate trust between different entities, several metrics
have been introduced. More formally, if E defines the set of considered entities, also called astrustees, the trust function may be defined as T : E → D that associates to each entity e ∈ E atrust “value” T (e), also referred to as “score”, where D is the domain of the trustworthiness metric.
Several metrics have been defined to quantify trustworthiness, which are briefly surveyed below.
For a comprehensive survey of the topic, the reader may refer to [16, 152].
• Single-valued. This metric assigns a single trust value in a domain D ⊆ R+ . Generally, thelower the value, the less trustworthy an entity is. The values in D are usually bounded
between +100 and −100 [63], 0 and 1 [120], or −1 and +1 [40, 108], and may be discreteor continuous. Often, single-valued metrics are inspired from Bayesian statistics [14], andrepresent trustworthiness as the probability that a trustee will perform an action that isbeneficial or at least not detrimental to the trustor. In this case, the metric also observes the
Cromwell’s rule [80], which prohibits the assignment of 0 or 1 as trust because if the priorprobability is 0 or 1, then according to Bayes’ theorem, the posterior probability is forced to
be 0 or 1 as well. In this case, no evidence, however strong, could have any influence on theposterior, which may be improper for a system where Bayesian inference is used based onincremental evidence over time.
• Multi-valued. This representation is the most popular when the system is interested in modeling and characterizing the uncertainty of trustworthiness. This is achieved by characterizing
trust not only by a single value in D but instead through a set of values. For example, insome cases, trustworthiness is defined by a fuzzy set in a continuous [26] or discrete [135]space. Thus, the trustworthiness is represented by the shape of the fuzzy set – a wider
fuzzy set means less confidence on that reputation, and vice versa. These fuzzy sets are thenmanipulated and updated through fuzzy aggregation [94]. Another very popular model for
multi-valued representation is the Jøsang’s belief model [87] which uses a quadruplet termed
as opinion to express all components about the interaction with an agent. More formally, an
opinion about entity e is represented as a 4-tuple ωe = (b, d, u, a), where the components
represent degrees of belief, disbelief, uncertainty, and relative atomicity respectively, where{b, d, u} ∈ [0, 1] and b + d + u = 1. The expected opinion T (e) = E(ωe ), is computed asE(ωe ) = b + a · u. The relative atomicity a ∈ [0, 1] determines the extent to which the degreeof uncertainty contributes to E(ωe ). For example, if we are trying to find whether a node in a
network is malicious, then a = 0.5 assuming no other information is available on that node.
Another concept that arises when dealing with trust is reputation. Although very similar inpurpose, the concepts are distinct. Specifically, conversely from trust, reputation does not necessarily
involve two parties only and can represent a multinomial nature of interactions with multiple
parties. Therefore, the term reputation can be loosely defined as, “what is generally said or believed
about a person’s or entity’s character or standing [90]. In other words, it is a combined measureof trustworthiness as seen by all other entities interacting with a particular trustee. Since theinteraction between the mobile crowdsensing system and the participating users can be seen as to
one-to-one interaction (i.e., participants are unaware of each other), for the sake of simplicity wewill use the terms “reputation” and “trust” interchangeably, as well as “system” and “framework”.
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey
Collection ofTrustworthiness
Evidence

1:21
Trust Function
T (p)

Trust
Scores

Trust
Aggregation

The Trust Loop

Fig. 4. Block scheme of interactions in the trust loop.

6.1.1 Problem definition. Similarly to truth discovery, the main purpose of existing trust-based
frameworks is to assess the QoI of the data submitted by a participant p. This is done by inferring
the reliability of p through the computation of his/her trustworthiness. There are several activities
and algorithms involved in such process, whose interactions are defined by us as “the trust loop”.
Figure 4 depicts a block scheme of the activities and interactions involved in the trust loop.
The computation of trust values for each participant p is done through the trust function T (p),which takes as input (i) the trust value of p computed at the previous time step (if any); and (ii)the evidence towards the trustworthiness of p collected during the current time step. The trust
function then computes the current trust score of participant p, which is then used to infer the QoIof the data submitted by p during the current time step. This is usually done through aggregationof trust scores of participants reporting on the same phenomenon.
6.1.2 Related work. Beyond mobile crowdsensing, the notion of computational trust has been
employed to solve many research problems related to the QoI, including e-commerce [93, 124].
Among the fields closely related to mobile crowdsensing, we include sensor networks [9, 83, 84,
137, 163], opportunistic networks [32, 99, 153, 198], and social networks [59, 197], to name a few.
However, establishing trust in mobile crowdsensing poses some unique challenges that cannot be
addressed solely with existing trust frameworks. These challenges include (i) the need of scalability
given by the sheer scale of mobile crowdsensing systems; (ii) the need to collect trustworthinessevidence from likely unreliable sources; and (iii) the need to handle sensing reports of different
types, including multimedia information (e.g., photo or video).
6.2 Collection of Trustworthiness Evidence
Earlier, we have defined trust as the likelihood that a participant will submit reliable information,
based on the past behavior of the participant. Thus, regardless of the trust function being used by
the system, it is mandatory to collect objective measurements (i.e., evidence) that may hint at theparticipants’ degree of reliability.
Arguably, the collection of trustworthiness evidence is the most challenging activity in the trust
loop. Specifically, even if some evidence of reliability is obtained, how can we trust that this very
evidence will itself be reliable? How can we scalably collect such information given the large scaleof existing mobile crowdsensing systems? In order to address these challenges, several approaches
have been proposed over the last few years. Below we summarize the most relevant ones, and also
discuss the advantages and disadvantages of each approach.
• Use of Contextual data. The majority of existing work has focused on the use of contextual
information to collect evidence towards establishing trustworthiness of participants. By contextual, we mean information that (i) is related to the particular event being monitored; and(ii) indicates the suitability of each participant to monitor that event. In most cases, collecting
contextual data does not require additional equipment and can be easily implemented, which
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:22

F. Restuccia et al.
is a major advantage. Unfortunately, some forms of contextual data (e.g., spatio-temporal, personal endorsement) can be forged by malicious users interested in increasing their reputationand thus obtain some form of gain, e.g., monetary rewards [141, 144, 147].
(1) Spatio-temporal location. Several works [140, 189, 191, 193, 211] have proposed the useof spatio-temporal location of sensing reports to collect trustworthiness evidence. The
rationale is that if the report is originated from a location closer to the event being monitored,
and has been submitted before a given deadline, then the report and thus the participant
is trustworthy. Some works have also considered frequency of submissions [144] and thetimeliness, which measures how promptly a participant performs prescribed tasks [5, 140].
For example, in [189] the authors propose a spatio-temporal provenance factor to establish
trustworthiness. Due to privacy reasons, the server does not know the actual location of theparticipant. Instead, a circular cloaking area of diameter D c is reported as the participant’s
location. The server assigns to the center Ls of this cloaking area the approximate sensing
location. The task of interest has a location Lt tagged as the target location. Thus, the closer
Ls is to Lt , the more trustworthy the report is. More formally, the authors in [189] quantify
a location distance factor asθ =e−D c α· 1−e−( |L s −L t |)αfor θ ∈ [0, 1)(14)

where α is the known as the location sensitivity parameter that controls the weight of thelocation factor’s influence. If |Ls − Lt | is larger than a maximum threshold the location isconsidered inappropriate, then the reports in such cases are discarded. In [189], the authors
also introduce a time gap factor similar to the location distance factor. Finally, the base trustof a report is a function of the location distance and time gap factors. Other contextual
pieces of information, known as milieu factors, are also included in the computation. These
milieu factors include type of data, mobile device type, speed of the mobile device, battery
level, weather conditions around the expected location, and so on.
(2) Personal endorsement. A number of works in the literature [6, 105] propose the use of
personal endorsement over social networks to measure the trustworthiness of participants.
The rationale behind this approach is that economically, the participants aim to maximize
their own economic benefit by (optimally) selecting whom to endorse and whom to beendorsed by. The advantage of such approach is that although a participant might generate
fake or noisy data, either by sabotage or by mistake, he tends to refrain from doing so if (a)such conducts may be witnessed by other people, and/or (b) his connections have to bear
the consequences, such as compromised benefit. Such a witness effect and beneficiary effecthave the potential to incentivize participants to improve the quality of their contribution[106]. In [6], the authors proposed an approach based on a preferred participant list to
select the most appropriate individuals to execute the sensing tasks. This list may be
automatically generated by the application (typically containing the participant’s friends
who have demonstrated trustworthy behavior) or manually by the requester (based on
his trust upon these individuals). Similar to the preferred list, a blocked list may also be
available which contains the list of those whom the participants desire not to endorse.
(3) Expertise. The rationale of this approach is that greater trust is placed in participants having
expertise in executing the task. Expertise is defined as the measure of a participant’s knowledge and is particularly important in tasks that require specific knowledge about a particulardomain, such as programming skills, familiarity to a geographical area, proficiency witha particular language, and so on. Expert finding systems such as [18, 103, 181] may be
employed to evaluate expertise that analyzes explicit (e.g., public profile data and group
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:23

memberships) as well as implicit information (e.g., textual posts) to extract user interestsand fields of expertise [5]. In [6], the authors formalized this concept and introduced theconcept of expertise level. They assumed an expert system in place which crawls a social
network graph to identify and extract the field of expertise of various participants. Moreformally, the expertise level is given as E = η (RE ∩ P E )/η (RE ), where set RE denotes the set of
required expertise, PE denotes the set of participant’s expertise, and η(A) is the numberof elements in a given set A ∈ {RE, PE}. This approach may be helpful when the system
requires the execution of a very specific task that requires prior expertise, and in general
may increase the overall trustworthiness. However, the drawbacks are that this approach
does not take into account malicious behavior, and that expert recruitment systems may
be impractical in large-scale mobile crowdsensing systems.
• Use of Outlier Detection and Similarity. Some existing works [79, 205] have proposed the useof outlier detection techniques [48] to obtain trustworthiness evidence. This approach isrooted in the idea that in a typical mobile crowdsensing application, the system often has
access to multiple data that characterize the same physical phenomenon but are sent bydifferent participants. Given such data redundancy, performing outlier detection may help
inferring the trustworthiness of participants. Outlier detection determines the likelihood of adata coming from an untrusted source by measuring its distance to a common value (e.g.,
average fuel price); the smaller the distance, the more trustworthy is the device. A drawbackof this approach is that it considers only the information in the current observation period
while ignoring the historical information.
Outlier detection can be broadly classified as either model-based or consensus-based techniques [73]. A model-based approach requires a priori knowledge of the underlying physicalprocess in which the application is interested, which may be difficult to acquire in mostcontexts. On the other hand, consensus-based techniques work on detecting inconsistencies
among data, and use the deviations from a common consensus to identify outliers [161]. Thisapproach may work in most cases, however it does not take into account collusion-based
attacks by malicious users [7, 180, 191].
By following the same idea, the concept of similarity has also been leveraged recently[140, 192]. In particular, for two sensing reports i and i ′, the similarity score S (i, i ′ ) between
the two reports is a quantity between −1 and 1 where −1 means completely conflicting witheach other and 1 means exactly the same. In [192], the trust score takes into account thesimilarity score, by considering a similarity factor ∆i to information item i belonging to acollection Ci of reports as follows:P
i,i ′ ∈C i ,i=i ′ S (i,i ′ )
∆i =
(15)
|Ci | − 1
where |Ci | is the number of reports in the collection. Each report in a collection will beassigned with a similarity factor. A negative similarity factor means there are more conflicts
in the collection while a positive similarity factor means there are more supports.
• Ground-truth data. The use of limited ground-truth data to acquire trustworthiness evidence
has been employed in prior work to ease the impact of malicious participants. In [141, 144],
we have proposed to utilize a limited number of mobile trusted participants (MTPs) to help
build reputation scores in a secure manner, and thus ‘bootstrap’ the trust in the system[27]. Specifically, MTPs are participants hired by the sensing application to periodicallygenerate reliable reports that reflect the actual status of the event being monitored aroundtheir locations. A similar methodology is being successfully used in the National Map CorpsACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:24

F. Restuccia et al.
project [112] developed by the U.S. Department of Geographical Survey (USGS), where USGS
employees are recruited to validate the crowdsourced data, such as the exact location ofschools and cemeteries. Furthermore, in the Crowd Sourcing Rangeland Conditions project[82], Kenyan pastoralists are recruited by researchers to validate the sensed data regarding
local vegetation conditions. A similar concept has been explored in [131], where the authors
use anchor nodes (i.e., participants completely trustworthy) to improve reputation scores.
We point out some advantages of using MTPs to collect trustworthiness evidence: (i) the
approach does not assume any particular mobility pattern, speed or trajectory of the MTPs;
and (ii) conversely from endorsement- or spatio-temporal location-based approaches, MTPs
guarantee trustworthiness through the evaluation of the accuracy of the data provided by
the participant. This gives the approach resiliency to GPS-spoofing-based attacks, since
the trustworthiness of sensed data will be guaranteed regardless of the accuracy of theparticipants’ locations. In addition, the use of MTPs may help prevent collusion-based attacks
by users. The downside of MTPs is that they also inevitably represent an additional cost forthe mobile crowdsensing system, as MTPs need to be recruited.
The use of low-level sensor data has also been proposed to collect evidence of trustworthiness.
In particular, GPS, accelerometer, gyroscope, compass, microphone, etc. can be used to verify
whether the participant’s voluntary high-level input is consistent with the observations fromlow-level sensor data. For example, let us suppose a participant submits a report indicatinga heavy traffic jam. This report is not going to be trustworthy if the device accelerometershows that it is in high-speed motion. Therefore, this approach looks for a consistency
between the participant’s voluntary input with the involuntary opportunistic input. Ad-hoc
connectivity of smartphone devices through WiFi and Bluetooth may also be used to validate
the smartphone’s position and avoid malicious behavior, as proposed in [165, 180, 193]. The
disadvantage of this approach is that an adversary may still forge low-level sensor data,
unless modules such as the ones in [156] are used to verify sensor data.

6.3 Computation of Trust Scores
There exist been several approaches in the literature as regards to how to compute trust scoresfor mobile crowdsensing participants. In the following, we summarize the most relevant ones anddiscuss advantages and disadvantages.
6.3.1 Gompertz’s Function Model. Gompert’z function [62] is a mathematical model for a timeseries, where growth is slowest at the start and end of a time period. The right-hand or future value
asymptote of the function is approached much more gradually by the curve than the left-hand or
lower valued asymptote, in contrast to the simple logistic function in which both asymptotes areapproached by the curve symmetrically. It is a special case of the generalized logistic function. Thec ·tequation is defined as G (t; a, b, c) = a · e b ·e , where a, b and c are the distribution’s parameters.
Figure 5 shows the Gompert’z function with different b and c values. It is traditionally used tomodel the population growth in a confined space, as birth rates first increase and then slow as
resource limits are reached [25, 212].
Huang et al. proposed in [78, 79] a trust framework exemplified in the context of a urban noisemonitoring system, where it is assumed the adversary compromises the hardware to generatespurious sensor readings. Every participating device’s sensing report is labeled as cooperative or
non cooperative based on the output of an outlier detection algorithm. The proposed reputation
system works over such a binary evidence. The system updates the trust scores every time periodof duration T , where each period is labeled as k. An epoch contains a number of time slots denotedby t. Sensor readings from any device i, over a duration T is denoted as X i,k = {x i,t , · · · , x i,t +T −1 }.
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:25

Fig. 5. Gompertz’s function with different b and c parameters.

A “watchdog” module takes in X i,k as input and generates a rating level denoted by 0 < pi,k < 1,which is an average calculated based on a distance similarity measure between the advertised
x i,t and the consensus r t . All pi,k are initialized to n1 , and are iteratively updated in every k-th
epoch, such that all past ratings of device i over past epochs and current epoch is denoted as′pi,k ′ , ∀k = 1, · · · , k. Based on this information, the reputation of the device i is calculated by using
the Gompertz’s function, which converts past cooperation levels into a cumulative score Ri,k . The′
input to the Gompertz’s function pi,k is given as′

pi,k =kX′

λ (k −k ) pi,k

(16)

′

k =1

where 0 < λ ≥ 1 is known as an aging parameter that controls how quickly past observations
are forgotten. The rationale is that, if the cooperation levels are consistently high, so must be
the reputation score. The reputation score is at first low and then it increases non-linearly if theparticipant is cooperating effectively.
The advantage of using the Gompertz’s function is the ease of implementation and its efficacy inmodeling trust, as it effectively captures dynamic behavior of the participants. However, as pointed
out in [144], the approach does not take into account the possibility that a substantial number ofparticipants may be malicious.
6.3.2 Jøsang’s Belief Model. In a previous work [144], we proposed a trust model based on
Jøsang’s belief model [87] that combines the reliability and participation level of users to produce
an aggregate trust score, which is then used to decide (i) the incentive or pay-off level of theparticipant, and (ii) whether or not accept his/her sensing reports. Mobile trusted participants(MTPs) are used to collect evidence of trustworthiness. More formally, at every time step j and forevery user i, the contribution is compared with that of an MTP. A ’match’ event occurs if the MTP
input agrees with the user i’s contribution. Likewise, a ‘mismatch’ event occurs if they do not agree.
And an ‘uncertain’ event occurs if the MTP is not available at that time j. Jøsang’s opinions areused to quantify both reliability and trustworthiness levels, which are updated reflecting a matchor a mismatch with the MTP’s contribution (if any) at a particular time step.
The final combined measure of the reliability and trustworthiness opinions (respectively ωi, j,rand ωi, j,t ) is given by the conjunction operator defined as follows:
br · bt
dr + dt − dr · dt
ωi, j,r t =  br · ut + bt · ur + ur · ut (17)

ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:26

F. Restuccia et al.

The reputation r ij of user i and time j is computed using the expectation operator as discussed
in Section 6.1. Similarly, in [141] the author models the probability of deeming a sensing reportreliable in the case it has not been validated by an MTP as follows:
P{A | V } = P{V } · P{F } +1· P{V }
2(18)

where P{F } is the probability that the participant will submit unreliable information. This formula
is inspired from Jøsang’s trust model and can be explained as follows. The first part, P{V } · P{F },represents the “degree of belief” in the participant; it is higher when the user is validated most of
the time (P{V } close to 1) and the reports are reliable. The second part, 1/2 · P{V }, represents the“degree of uncertainty” in the users; it is higher when most of the reports have not been validated.
Note that, as P{V } increases, the value of P{A | V } approximates to P{F }. Also, if P{V } = 0 (i.e., no
MTPs are present), the system deems as reliable every report with probability 1/2 (coin tossing), as
there is no reason to be more inclined to accept or reject the report if no information is available.
6.3.3 Beta Distribution Model. Some existing work has leveraged the Beta distribution [89] tocompute trust scores [56, 138]. Usually the Beta distribution is suitable when the evidence state
space is binary. However, evidence with a ternary state space can be alternatively modeled bycoarsening the evidence into a binary space [88], approximating the third state space into partially
positive and partially negative ratings/feedbacks/outcomes.
More formally, the Beta distribution suggests that if r is the number of positive and s is thenumber of negative outcomes from a set of N = r + s number of interactions with an entity, then
the trust of that entity over this action can be modeled as a Beta distribution with parametersspecified by φ = r + 1 and κ = s + 1, such that the posterior probability density function given as:f (p|φ, κ) =Γ(φ + κ)· p φ−1 (1 − p) κ−1Γ(φ) · Γ(κ)(19)

A Bayesian estimation of parameters shows that the mean of the probability density function in
Equation (19) can accurately model trust metrics given by
E=

φ
r +1
=
φ +κ r +s +2(20)

As demonstrated in [79], the Beta trust scoring takes a less aggressive approach in penalizing
users that contribute corrupted data. On the other hand, the period over which a participant
may contribute corrupted data may potentially be short-lived (particularly, when this happens
unintentionally). Furthermore, the frequency of occurrence of such events may also be high (e.g., auser may frequently place her phone in the bag while it is collecting noise pollution samples). In
such cases, the Beta scoring may not be the most effective way to capture such dynamic behavior.
6.3.4 Dirichlet Model. This trust model is used when the state space of the trustworthinessevidence is not binary but instead follows a multinomial distribution [138, 196]. The multinomial
distribution is the generalization of the binomial distribution when k > 2 potential outcomes are
possible, where each trial results in one of the k outcomes from a set of N possible trials. Therefore,
the count of observations may fit very well with this model. Specifically, observations data D for any
participant can be treated as a multinomial distribution, provided the probabilities of occurrenceof each outcome are mathematically represented as p(D|X ). Here, D is a vector representing theobserved data, such that D = {d 1 , d 2 , dk }. Similarly, X represents the probability vector denoting
the probability of occurrence of each outcome such that X = {x 1 , x 2 , x 3 }. In general these arerepresented as D = {di |1 ≤ i ≤ k} and X = {x i |1 ≤ i ≤ k}.
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:27

However, initially X is unknown and hence the problem is estimation of parameters in X given
evidence or data D. The probability density function (PDF) of a Dirichlet distribution returns the
subjective probabilities x i (also known as degree of belief) for Ki number of competing events
given each possible outcome i has been observed di − 1 times. Given that observation count di − 1 is
known from the gathered evidence, we can calculate the probabilities of each event. More formally,⃗ is defined as
the Dirichlet probability density function with variables x⃗ and parameters d,PkkΓ(di ) Yf (⃗x |d⃗) = Qk i=1
x idi −1 ,
(21)
Γ(d
)
i
i=1
i=1
The relation between observation parameter di and actually observed outcomes r i is that r i +C ·ai =Pdi , where ki=1 ai = 1 and C > 0, ai > 0 such that zero occurrence of an outcome preserves thecondition that di > 0.
Obtaining the trust score is equivalent to computing the expectation of the Dirichlet distribution,
which is computed asdiE(x i |d⃗) = Pk
(22)
i=1 d iwhere di is known as the total evidence count for event i. The degrees of belief associated with theoutcomes are expressed as the mean of each outcome.
6.3.5 Fuzzy Logic Model. Prior work [3–6] has suggested to leverage fuzzy logic [42] to compute
trust scores based on the collected trustworthiness evidence. In such work, it is supposed thatevery participant is part of an existing online social network platform. The system works in a
publish/subscribe manner where a requester publishes a sensing task. A set of contributors fromher social network graph known as participants are responsible to report their observations related
to the sensing task. Any requester and contributor pair form a friendship relation. Every requested
task is assumed to have a deadline d associated with it.
The authors of [3–6] assume the trustworthiness level of a contributing user is affected by thefollowing five factors that are a combination of personal and social factors. The personal factors
include expertise (E), timeliness (T), location (L), friendship duration (F), and interaction time gap(I). The proposed trust frameworks leverage fuzzy logic to calculate a comprehensive trust rating foreach contribution, referred to as the Trust of Contribution (ToC). The inputs to the fuzzy inference
system are the crisp values of the Quality of Contribution (QoC) and Trust of Participant (ToP).
The QoC is a value between 0 and 1 and reflects the quality of the sensing report (i.e., accuracy,
precision, timeliness) and is determined by external outlier detection algorithms. On the other
hand, the ToP is derived as a weighted sum of the five personal factors, where the weights control
the extent to which each of these factors affect the trustworthiness.
The trust frameworks derives the ToC as a function of the ToP and QoC by using a fuzzy logic
inference system. The steps involved in such computation are described as follows.
(1) Fuzzifier. Since the ToP and QoC are crisp values (i.e., a scalar value and not fuzzy sets),the fuzzifier converts these values into a linguistic variable. In other words, it determines
the degree to which these inputs belong to each of the corresponding fuzzy sets. The fuzzy
sets for QoC, ToP are defined as {Very Low, Low, Medium, High}, while the ToC fuzzy
set is {Very Low, Low, Medium, High, Very High}. For any set X , a membership functionµ (X ) : X → [0, 1] quantifies the grade of membership of an element x ∈ X to the fuzzy set X .
The value of 0 means that x is not a member of the fuzzy set and 1 means that x is fully amember of the fuzzy set. The values between 0 and 1 characterize fuzzy members, which
belong to the fuzzy set only partially.
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:28
Paper
Amintoosi et al. [6]Amintoosi and Kanhere [5]Huang et al. [78, 79]
Luo et al. [106]Ren et al. [140]
Wang et al. [188, 189]

Wang et al. [191]Yu et al. [205]Restuccia [141, 144]

F. Restuccia et al.
Summary of Contributions
Framework for identification and recruitment of well-suited participants in largescale online social networks with unknown
network topology and complex friendship relations.
Application-agnostic reputation frameworkwith utilization of PageRank algorithm as thebasis of trust.
Proposes for the first time the use of Gompertz’s function for trust scoring.
Introduces a social concept called nepotism by linking mobile users into a social
network with endorsement relations, andproposing the overlaying of this networkwith investment-like economic implications.
Proposes a participant selection scheme tochoose the well-suited participants for thesensing task under a fixed task budget.
Derives a trust model based on information similarity and path difference, present amechanism to detect and reduce the effect ofcollusion attacks.
Discusses a trust model based on information similarity and path difference, present amechanism to detect and reduce the effect ofcollusion attacks.
Introduced the accumulated reputation
model (ARM), where trust is computed andaccumulated based on sensing data.
Proposed trust framework based on Mobile
Trusted Participants (MTPs).

Evidence Collection
Spatio-temporal Location,
dorsement, ExpertiseEn-

Trust Scoring
Fuzzy logic

Spatio-temporal Location, Personal Endorsement, Expertise

Fuzzy logic

Outlier Detection

Gompertz’s
tion
Customized

Endorsement

Spatio-temporal Location,
dorsement, Similarity

Func-

En-

Customized

Spatio-temporal Location, Similarity

Customized

Spatio-temporal Location, Similarity

Customized

Outlier Detection

Gompertz’s
tion

Outlier Detection

GroundTruth,
Jøsang’s model

Func-

Table 5. Summary of Trust Frameworks for Mobile Crowdsensing(2) Inference Engine. The role of inference engine is to convert fuzzy inputs (QoC and ToP) to thefuzzy output (ToC) by leveraging If-Then type fuzzy rules. To define the output zone, themax-min composition is used [94].
(3) Defuzzifier. A defuzzifier converts the ToC fuzzy value to a crisp value in the range of [0, 1].
The Centre of Gravity (COG) [172] defuzzification method is perhaps the most commonlyused and popular defuzzification technique with the advantage of quick and highly accuratecomputations.
6.4 How to Improve Existing WorkIn this section, we have surveyed relevant existing work towards the development of effective
and efficient trust frameworks for mobile crowdsensing. Table 5 summarizes the most relevant
contributions discussed in this section. Trust frameworks have garnered significant attention fromthe research community, given they possess many desirable characteristics that make them an
effective choice for implementing QoI in mobile crowdsensing. Among them, (i) their ease of
implementation; and (ii) their flexibility and capability to adapt to dynamic behavior of participants.
Despite the advantages provided by the trust framework, several research challenges still linger
and need to be addressed by future work, which are enumerated below.
(1) The approach of contextual information for trustworthiness evidence suffers from many
disadvantages. Most importantly, malicious participants can tamper with their smart devices
or use apps that can forge contextual information, rendering the trust assessment processuseless [193]. Additional research must be devoted on finding the right scenarios where it isappropriate to use contextual information to assess trustworthiness.
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:29(2) The impact of different participants’ behaviors on the trust scores of participants is still
unclear. In particular, there is a need to mathematically model the participants’ behavior
and predict the optimum parameters that may not only make existing trust frameworks
100% robust against carefully planned attacks, but also stimulate good behavior among users.
Although there exists some research work in this direction [38, 167, 169, 207], the literature
still lacks a comprehensive investigation on these topics.
(3) Most of existing work have not investigated the scalability and computational complexityof known trust frameworks. This aspect is fundamental given the large scale of mobile
crowdsensing systems. Rigorous theoretical analysis, as well as implementation on parallel
frameworks such as MapReduce [35], are necessary to address the scalability issue and ensurethat proposed trust frameworks are implementable in real-world scenarios.
7

OPEN RESEARCH CHALLENGES AND DIRECTIONSIn this section, we propose novel research challenges pertaining to QoI in mobile crowdsensing.
7.0.1 Assessing Trustworthiness of Complex Information. We point out that most of existing works
have considered binary (i.e., true/false) or multinomial (i.e., belonging to a set of values) information.
However, the very purpose of mobile crowdsensing is to allow collection of information that iscomplex. By complex, we mean information that is (i) multimedial, for example audio and video; and(ii) qualitative, which expresses an opinion regarding a particular event under consideration, forexample, traffic/weather status over a particular area. Solving this issue is paramount to obtaining
meaningful information from each sensing report and ultimately, assessing its QoI, but at the same,
it is extremely challenging.
Multimedia input includes audio/video recording and pictures [10, 136], and are the most challenging to process in terms of QoI. Regardless, we need novel algorithms to (i) compute QoI on
multimedia information; and (ii) compare QoI levels between different multimedia inputs if we want
to apply existing truth discovery and trust mechanisms on this type of data. Although there exist
some work in the context of sensor networks [41, 74], to the best of our knowledge the problem of
QoI in multimedia-based mobile crowdsensing remains substantially unexplored. Among existing
work, in [141, 144]we proposed trust frameworks for traffic monitoring applications. In particular,
the use of MTPs could be fundamental to assess trustworthiness of participants in opinion-based
systems. However, it is yet to be explored how to model and manage opinions. One possible
direction could be to embed mechanisms similar to Jøsang’s trust or fuzzy logic to model opinions.
7.0.2 Gamification for QoI Improvement. The concept of gamification [38] has been proposed inmany contexts as a viable and effective technique to increase QoI by persuading the participants
with a natural (i.e., based on joy or entertainment) incentive type, instead of a remunerative one[91, 121, 169]. For example, the leaderboard-based system used by Waze [52] has proven to beeffective in motivating the users to contribute reliable data. On the other hand, recent studies [66]cast doubts on gamification, as the authors argue that gamification effects are greatly dependent
on the context in which gamification is being implemented, as well as on the users using it.
It is still uncertain whether these mechanisms may work effectively in mobile crowdsensing. In
particular, we need to understand and validate the conditions by which gamification may result ina good strategy, and possibly, embed existing trust/truth discovery framework in the context ofgamification. Also, gamification is currently based on heuristics, and we lack modeling languages for
gamification [70] which may be able to capture the complex interactions in mobile crowdsensing.
7.0.3 Game-theoretical Trust Frameworks. As one possible future research direction, we envision
the development of game-theoretical trust frameworks based on behavioral game theory (BGT) [21].
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:30

F. Restuccia et al.

Specifically, the concept of bounded rationality [159, 160] may be used to analyze less-than-rational
behavior in participants. Below are some examples of why participants may not be perfectly
rational:• Participants may lack the ability to understand the game, for example, because of incomplete
information at their disposal. The information may also be asymmetrical, which means, someusers know more than others about the game being played.
• Participants may not be able to play the optimum strategy if the interaction between the user
and the system is repeated. Also, the equilibrium played by the participants may be calculated
by considering a different time-horizon, and therefore, mismatch the ones calculated by thesystem.
• In realistic contexts, the participating users might not know the details of the game been
played. Therefore, they need to learn over time the strategy that maximizes the payoff. In
such context, users might have different learning strategies. For example, some might use
a reinforcement approach [11] (i.e., adapting the strategy as a function of current and past
payoffs), while some might use anticipatory learning [22] to reason more thoughtfully on
what the other players are doing and the game being played.
Therefore, it becomes necessary to embed these and other procedural aspects while modeling
trust frameworks. We need to identify restrictions on the space of strategies and then optimize
based on those restrictions. One way to incorporate procedural aspects, and therefore, uncertainty
in the equilibria that will be played by users, is to study the Quantal Response Equilibria (QRE) of
the game, a concept first proposed in [113]. QRE provide a statistical framework to analyze games
that incorporate realistic limitations to rational choice modeling of games. In particular, QRE allow
every strategy to be played with non-zero probability, and thus any strategy is possible (though
not necessarily reasonable). By far, the most common specification for QRE is the logit equilibrium(LQRE) [113], where the player’s strategies are chosen according to the probability distributionP
exp{λ · a−i ∈A−i P{a −i } · ui (ai , a −i )}P{ai } = P
P′a i′ ∈Ai exp{λ · a −i ∈A−i P{a i } · u i (a i , a −i )}(23)

The λ parameter is defined as the “rationality” parameter: the choice of action becomes purely
random as λ → 0, whereas the action with higher expected payoff is chosen as λ → ∞. Analogously
to Nash equilibria, QRE can be calculated for normal-form games [113] and extensive-form games[114]. Recent advances in the field have shown that the computation of QRE can be done bymaximum likelihood estimation of the model’s parameters, as explained in [31], or by techniquessuch as the homotopy method, developed in [149]. The value of λ is usually chosen according toexperimental data; however, such data does not exist yet to formulate QRE for games in the context
of mobile crowdsensing.
Additional research is therefore needed to model the λ parameter and therefore, obtain more
realistic equilibria. Although it is true in general that irrationality might be dependent on the mobile
crowdsensing application being considered, we believe that large-scale data collection campaigns
with sufficient number of participants could help solve this issue.
8

CONCLUSIONS

Mobile crowdsensing has become one of the most promising paradigms for social sensing, as
it allows to dramatically reduce infrastructure costs and obtain detailed information about thephenomenon being monitored. On the other hand, it brings into the picture novel and exciting multidisciplinary research challenges, most of them are yet to be understood and solved. In particular,
given the strong presence of human beings in the sensing loop, understanding in details the human
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:31

dynamics and predictable behavior in mobile crowdsensing is fundamental to guarantee quality of
information (QoI).
In this paper, we have surveyed the topic of QoI in mobile crowdsensing. After proposing
a comprehensive framework that defines the concept of QoI in mobile crowdsensing, we havedefined and analyzed in details the research challenges in enforcing and estimating the QoI, as
well as discussing the already existing related work on the topic. By analyzing the limitations andweaknesses of existing work, we have provided a roadmap of possible directions of novel future
research in the field.
The research challenges defined in this paper are just the tip of the iceberg. These and manyother challenges, such as energy consumption, security and privacy, efficiency, and so on, are still
to be tackled. Nevertheless, we expect that in the future, mobile crowdsensing applications will
become deeply in our daily lives, and will help build a society in which sensing will be ultimately
ubiquitous and protagonist in every aspect of human life.
ACKNOWLEDGEMENT
We would like to thank the anonymous reviewers for their valuable comments, which helped us
improve the quality of the paper. This material is based upon work supported by the National
Science Foundation under Grant No. CNS-1545037, CNS-1545050, DGE-1433659 and IIS-1404673.
Any opinions, findings, and conclusions or recommendations expressed in this material are thoseof the authors and do not necessarily reflect the views of the National Science Foundation.
REFERENCES[1] Tarek Abdelzaher, Yaw Anokwa, Péter Boda, Jeff Burke, Deborah Estrin, Leonidas Guibas, Aman Kansal, Samuel
Madden, and Jim Reich. 2007. Mobiscopes for Human Spaces. IEEE Pervasive Computing 6, 2 (April 2007), 20–29. DOI:
http://dx.doi.org/10.1109/MPRV.2007.38[2] Amro Al-Akkad, Leonardo Ramirez, Alexander Boden, Dave Randall, and Andreas Zimmermann. 2014. Help Beacons:
Design and Evaluation of an Ad-hoc Lightweight s.O.S. System for Smartphones. In Proceedings of the SIGCHI Conferenceon Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 1485–1494. DOI:http://dx.doi.org/10.
1145/2556288.2557002[3] Haleh Amintoosi and Salil S. Kanhere. 2013a. A Trust-Based Recruitment Framework for Multi-hop Social Participatory
Sensing. In 2013 IEEE International Conference on Distributed Computing in Sensor Systems. 266–273. DOI:http://dx.doi.org/10.1109/DCOSS.2013.29[4] Haleh Amintoosi and Salil S. Kanhere. 2013b. A Trust Framework for Social Participatory Sensing Systems. Springer
Berlin Heidelberg, Berlin, Heidelberg, 237–249. DOI:http://dx.doi.org/10.1007/978-3-642-40238-8_20[5] Haleh Amintoosi and Salil S. Kanhere. 2014. A Reputation Framework for Social Participatory Sensing Systems. Mobile
Networks and Applications 19, 1 (2014), 88–100. DOI:http://dx.doi.org/10.1007/s11036-013-0455-x[6] Haleh Amintoosi, Salil S. Kanhere, and Mohammad Allahbakhsh. 2015. Trust-based privacy-aware participant
selection in social participatory sensing. Journal of Information Security and Applications 20 (2015), 11 – 25. DOI:
http://dx.doi.org/10.1016/j.jisa.2014.10.003 Security, Privacy and Trust in Future Networks and Mobile Computing.
[7] Kelsey D. Atherton. 2014. Israeli students spoof Waze with fake traffic jam, http://www.popsci.com/article/gadgets/israeli-students-spoof-waze-app-fake-traffic-jam. Popular Science (2014).
[8] James E. Bailey and Sammy W. Pearson. 1983. Development of a tool for measuring and analyzing computer user
satisfaction. Management science 29, 5 (1983), 530–545.
[9] Fenye Bao, Ing-Ray Chen, MoonJeong Chang, and Jin-Hee Cho. 2012. Hierarchical Trust Management for WirelessSensor Networks and its Applications to Trust-Based Routing and Intrusion Detection. IEEE Transactions on Network
and Service Management 9, 2 (June 2012), 169–183. DOI:http://dx.doi.org/10.1109/TCOMM.2012.031912.110179[10] Xuan Bao and Romit Roy Choudhury. 2010. MoVi: Mobile Phone Based Video Highlights via Collaborative Sensing. In
Proceedings of the 2010 ACM International Conference on Mobile Systems, Applications, and Services (MobiSys ’10). ACM,
New York, NY, USA, 357–370. DOI:http://dx.doi.org/10.1145/1814433.1814468[11] Andrew G Barto. 1998. Reinforcement learning: An introduction. MIT press.
[12] Suvadip Batabyal and Parama Bhaumik. 2015. Mobility Models, Traces and Impact of Mobility on Opportunistic
Routing Algorithms: A Survey. IEEE Communications Surveys Tutorials 17, 3 (thirdquarter 2015), 1679–1707. DOI:
http://dx.doi.org/10.1109/COMST.2015.2419819
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:32

F. Restuccia et al.

[13] Peter Benson. 2009. ISO 8000 Data Quality: The fundamentals, part 1. Real-World Decision Support (RWDS) Journal 3, 4(2009).
[14] José M Bernardo and Adrian FM Smith. 2001. Bayesian theory. (2001).
[15] Dimitri P Bertsekas. 1999. Nonlinear Programming. Athena scientific Belmont.
[16] Kamal K. Bharadwaj and Mohammad Yahya H. Al-Shamri. 2009. Fuzzy computational models for trust and reputation
systems. Electronic Commerce Research and Applications 8, 1 (2009), 37 – 47. DOI:http://dx.doi.org/https://doi.org/10.
1016/j.elerap.2008.08.001[17] Chatschik Bisdikian, Lance M. Kaplan, and Mani B. Srivastava. 2013. On the Quality and Value of Information in
Sensor Networks. ACM Transactions on Sensor Networks 9, 4, Article 48 (July 2013), 26 pages. DOI:http://dx.doi.org/10.
1145/2489253.2489265[18] Alessandro Bozzon, Marco Brambilla, Stefano Ceri, Matteo Silvestri, and Giuliano Vesci. 2013. Choosing the rightcrowd: expert finding in social networks. In Proceedings of the 16th International Conference on Extending Database
Technology. ACM, 637–648.
[19] Sergey Brin and Lawrence Page. 2012. Reprint of: The anatomy of a large-scale hypertextual web search engine.
Computer networks 56, 18 (2012), 3825–3833.
[20] Jeff Burke, Deborah Estrin, Mark Hansen, Andrew Parker, Nithya Ramanathan, Sasank Reddy, and Mani B. Srivastava.
2006. Participatory sensing. In Proceedings of the 2006 Workshop on World-Sensor-Web: Mobile Device Centric Sensor
Networks and Applications (WSW ’06). 117–134.
[21] Colin F. Camerer. 2003. Behavioural studies of strategic thinking in games. Trends in cognitive sciences 7, 5 (2003),
225–231.
[22] Colin F. Camerer, Teck-Hua Ho, and Juin-Kuan Chong. 2002. Sophisticated experience-weighted attraction learningand strategic teaching in repeated games. Journal of Economic Theory 104, 1 (2002), 137–188.
[23] Andrew T. Campbell, Shane B. Eisenman, Nicholas D. Lane, Emiliano Miluzzo, and Ronald A. Peterson. 2006. Peoplecentric Urban Sensing. In Proceedings of the 2006 ACM Annual International Workshop on Wireless Internet (WICON ’06).
ACM, New York, NY, USA, Article 18. DOI:http://dx.doi.org/10.1145/1234161.1234179[24] Andrew T. Campbell, Shane B. Eisenman, Nicholas D. Lane, Emiliano Miluzzo, Ronald A. Peterson, Hong Lu, Xiao
Zheng, Mirco Musolesi, Kristóf Fodor, and Gahng-Seop Ahn. 2008. The Rise of People-Centric Sensing. IEEE Internet
Computing 12, 4 (July 2008), 12–21. DOI:http://dx.doi.org/10.1109/MIC.2008.90[25] F. Caravelli, L. Sindoni, F. Caccioli, and C. Ududec. 2016. Optimal growth trajectories with finite carrying capacity. 94,
2 (Aug 2016), 022315. DOI:http://dx.doi.org/10.1103/PhysRevE.94.022315[26] Javier Carbo, Jose M Molina, and Jorge Davila. 2003. Trust management through fuzzy reputation. International
Journal of Cooperative Information Systems 12, 01 (2003), 135–155.
[27] Christiano Castelfranchi and Rino Falcone. 2010. Trust theory: A socio-cognitive and computational model. Vol. 18. John
Wiley & Sons.
[28] Kevin Chan, Kelvin Marcus, Lisa Scott, and Rommie Hardy. 2015. Quality of information approach to improving
source selection in tactical networks. In Information Fusion (Fusion), 2015 18th International Conference on. 566–573.
[29] Huihui Chen, Bin Guo, Zhiwen Yu, and Qi Han. 2016. Toward real-time and cooperative mobile visual sensing andsharing. In IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications. 1–9.
DOI:http://dx.doi.org/10.1109/INFOCOM.2016.7524481[30] Xiao Chen, Min Liu, Yaqin Zhou, Zhongcheng Li, Shuang Chen, and Xiangnan He. 2017. A Truthful Incentive
Mechanism for Online Recruitment in Mobile Crowd Sensing System. Sensors 17, 1 (2017).
[31] Syngjoo Choi, Douglas Gale, and Shachar Kariv. 2012. Social learning in networks: a Quantal Response Equilibrium
analysis of experimental data. Review of Economic Design 16, 2-3 (2012), 135–157. DOI:http://dx.doi.org/10.1007/
s10058-012-0122-x[32] Radu-Ioan Ciobanu, Radu-Corneliu Marin, Ciprian Dobre, and Valentin Cristea. 2017. Trust and reputation managementfor opportunistic dissemination. Pervasive and Mobile Computing 36 (2017), 44 – 56. DOI:http://dx.doi.org/https://doi.org/10.1016/j.pmcj.2016.09.016 Special Issue on Pervasive Social Computing.
[33] Henry Corrigan-Gibbs, Nakull Gupta, Curtis Northcutt, Edward Cutrell, and William Thies. 2015. Deterring Cheatingin Online Environments. ACM Transactions on Computer-Human Interaction 22, 6, Article 28 (Sept. 2015), 23 pages.
DOI:http://dx.doi.org/10.1145/2810239[34] Paulo C. G. Costa, Kathryn B. Laskey, Erik Blasch, and Anne-Laure Jousselme. 2012. Towards unbiased evaluation of
uncertainty reasoning: The URREF ontology. In Information Fusion (FUSION), 2012 15th International Conference on.
2301–2308.
[35] Jeffrey Dean and Sanjay Ghemawat. 2008. MapReduce: simplified data processing on large clusters. Commun. ACM 51,
1 (2008), 107–113.
[36] Ofer Dekel and Ohad Shamir. 2009. Good Learners for Evil Teachers. In Proceedings of the 26th Annual International
Conference on Machine Learning (ICML ’09). ACM, New York, NY, USA, 233–240. DOI:http://dx.doi.org/10.1145/1553374.
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:33

1553404[37] A. P. Dempster, N. M. Laird, and D. B. Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm.
Journal of the Royal Statistical Society, Series B 39, 1 (1977), 1–38.
[38] Sebastian Deterding, Dan Dixon, Rilla Khaled, and Lennart Nacke. 2011. From Game Design Elements to Gamefulness:
Defining "Gamification". In Proceedings of the 2011 ACM International Academic MindTrek Conference: Envisioning Future
Media Environments (MindTrek ’11). ACM, New York, NY, USA, 9–15. DOI:http://dx.doi.org/10.1145/2181037.2181040[39] Ellie D’Hondt, Matthias Stevens, and An Jacobs. 2013. Participatory noise mapping works! An evaluation of participatory
sensing as an alternative to standard techniques for environmental monitoring. Pervasive and Mobile Computing 9, 5(2013), 681 – 694. DOI:http://dx.doi.org/10.1016/j.pmcj.2012.09.002 Special issue on Pervasive Urban Applications.
[40] eBay. 2017. The eBay Online Marketplace. (2017). http://www.ebay.com[41] James Edwards, Ahmed Bahjat, Yurong Jiang, Trevor Cook, and Thomas F. La Porta. 2014. Quality of information-aware
mobile applications. Pervasive and Mobile Computing 11 (2014), 216 – 228. DOI:http://dx.doi.org/https://doi.org/10.
1016/j.pmcj.2013.05.003[42] S. A. Ehikioya. 1999. A characterization of information quality using fuzzy logic. In Fuzzy Information Processing
Society, 1999. NAFIPS. 18th International Conference of the North American. 635–639. DOI:http://dx.doi.org/10.1109/
NAFIPS.1999.781771[43] Samina Ehsan and Bechir Hamdaoui. 2012. A Survey on Energy-Efficient Routing Techniques with QoS Assurances
for Wireless Multimedia Sensor Networks. IEEE Communications Surveys Tutorials 14, 2 (Second 2012), 265–278. DOI:
http://dx.doi.org/10.1109/SURV.2011.020211.00058[44] Larry P. English. 2009. Information Quality Applied: Best Practices for Improving Business Information, Processes andSystems. Wiley Publishing.
[45] Ericsson. 2016. Ericsson Mobility Report. (2016). Retrieved October 16, 2016 from https://www.ericsson.com/res/docs/
2016/ericsson-mobility-report-2016.pdf[46] Rasool Fakoor, Mayank Raj, Azade Nazi, Mario Di Francesco, and Sajal K. Das. 2012. An Integrated Cloud-based
Framework for Mobile Phone Sensing. In Proceedings of the First Edition of the MCC Workshop on Mobile Cloud
Computing (MCC ’12). ACM, New York, NY, USA, 47–52. DOI:http://dx.doi.org/10.1145/2342509.2342520[47] Meng Fang and Xingquan Zhu. 2014. Active learning with uncertain labeling knowledge. Pattern Recognition Letters
43 (2014), 98 – 108. DOI:http://dx.doi.org/10.1016/j.patrec.2013.10.011[48] Peter Filzmoser. 2016. Identification of multivariate outliers: a performance study. Austrian Journal of Statistics 34, 2(2016), 127–138.
[49] Internet Engineering Task Force. 1998. RFC 2386 - A Framework for QoS-based Routing in the Internet. (1998).
Retrieved October 16, 2016 from https://tools.ietf.org/html/rfc2386[50] Benoit Frenay and Michel Verleysen. 2014. Classification in the Presence of Label Noise: A Survey. IEEE Transactions on
Neural Networks and Learning Systems 25, 5 (May 2014), 845–869. DOI:http://dx.doi.org/10.1109/TNNLS.2013.2292894[51] Sebastian Frischbier, Peter Pietzuch, and Alejandro Buchmann. 2014. Managing Expectations: Runtime Negotiation ofInformation Quality Requirements in Event-Based Systems. Springer Berlin Heidelberg, Berlin, Heidelberg, 199–213.
DOI:http://dx.doi.org/10.1007/978-3-662-45391-9_14[52] Roy Furchgott. 2010. New York Times. Filling in Map Gaps With Waze Games. (2010). Retrieved December 4, 2014
from http://wheels.blogs.nytimes.com/2010/05/06/filling-in-the-map-gaps-with-waze-games/[53] Ujwal Gadiraju, Gianluca Demartini, Ricardo Kawase, and Stefan Dietze. 2015a. Human Beyond the Machine: Challenges
and Opportunities of Microtask Crowdsourcing. IEEE Intelligent Systems 30, 4 (July 2015), 81–85. DOI:http://dx.doi.
org/10.1109/MIS.2015.66[54] Ujwal Gadiraju, Ricardo Kawase, Stefan Dietze, and Gianluca Demartini. 2015b. Understanding Malicious Behavior in
Crowdsourcing Platforms: The Case of Online Surveys. In Proceedings of the 33rd Annual ACM Conference on Human
Factors in Computing Systems (CHI ’15). ACM, New York, NY, USA, 1631–1640. DOI:http://dx.doi.org/10.1145/2702123.
2702443[55] Alban Galland, Serge Abiteboul, Amélie Marian, and Pierre Senellart. 2010. Corroborating Information from Disagreeing
Views. In Proceedings of the Third ACM International Conference on Web Search and Data Mining (WSDM ’10). ACM,
New York, NY, USA, 131–140. DOI:http://dx.doi.org/10.1145/1718487.1718504[56] Saurabh Ganeriwal, Laura K Balzano, and Mani B Srivastava. 2008. Reputation-based framework for high integrity
sensor networks. ACM Transactions on Sensor Networks (TOSN) 4, 3 (2008), 15.
[57] Raghu K. Ganti, Fan Ye, and Hui Lei. 2011. Mobile crowdsensing: current state and future challenges. IEEE Communications Magazine 49, 11 (November 2011), 32–39. DOI:http://dx.doi.org/10.1109/MCOM.2011.6069707[58] Hui Gao, Chi Harold Liu, Wendong Wang, Jianxin Zhao, Zheng Song, Xin Su, Jon Crowcroft, and Kin K. Leung. 2015. A
Survey of Incentive Mechanisms for Participatory Sensing. IEEE Communications Surveys Tutorials 17, 2 (Secondquarter2015), 918–943. DOI:http://dx.doi.org/10.1109/COMST.2014.2387836[59] Peixin Gao, Hui Miao, John S. Baras, and Jennifer Golbeck. 2016. STAR: Semiring Trust Inference for Trust-Aware
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:34

F. Restuccia et al.

Social Recommenders. In Proceedings of the 10th ACM Conference on Recommender Systems (RecSys ’16). ACM, New
York, NY, USA, 301–308. DOI:http://dx.doi.org/10.1145/2959100.2959148[60] Shravan Gaonkar, Jack Li, Romit Roy Choudhury, Landon Cox, and Al Schmidt. 2008. Micro-Blog: Sharing andQuerying Content Through Mobile Phones and Social Participation. In Proceedings of the 2008 ACM International
Conference on Mobile Systems, Applications, and Services (MobiSys ’08). ACM, New York, NY, USA, 174–186. DOI:
http://dx.doi.org/10.1145/1378600.1378620[61] Lincy M George and KR Remesh Babu. 2016. A mobile crowd sensing based task assignment in Internet of Things. In
Emerging Technological Trends (ICETT), International Conference on. IEEE, 1–4.
[62] Benjamin Gompertz. 1825. On the Nature of the Function Expressive of the Law of Human Mortality, and on a New
Mode of Determining the Value of Life Contingencies. Philosophical Transactions of the Royal Society of London 115(1825), 513–583. http://www.jstor.org/stable/107756[63] Tyrone Grandison and Morris Sloman. 2003. Trust management tools for internet applications. In International
Conference on Trust Management. Springer, 91–107.
[64] Domenico De Guglielmo, Francesco Restuccia, Giuseppe Anastasi, Marco Conti, and Sajal K. Das. 2016. Accurate andEfficient Modeling of 802.15.4 Unslotted CSMA/CA through Event Chains Computation. IEEE Transactions on Mobile
Computing 15, 12 (Dec 2016), 2954–2968. DOI:http://dx.doi.org/10.1109/TMC.2016.2528248[65] Bin Guo, Zhu Wang, Zhiwen Yu, Yu Wang, Neil Y. Yen, Runhe Huang, and Xingshe Zhou. 2015. Mobile Crowd Sensing
and Computing: The Review of an Emerging Human-Powered Sensing Paradigm. Comput. Surveys 48, 1, Article 7(Aug. 2015), 31 pages. DOI:http://dx.doi.org/10.1145/2794400[66] Juho Hamari, Jonna Koivisto, and Harri Sarsa. 2014. Does Gamification Work? – A Literature Review of Empirical
Studies on Gamification. In Proceedings of the 2014 Hawaii International Conference on System Sciences (HICSS ’14). IEEE
Computer Society, Washington, DC, USA, 3025–3034. DOI:http://dx.doi.org/10.1109/HICSS.2014.377[67] Richard HR Harper. 2014. Trust, computing, and society. Cambridge University Press.
[68] Daojing He, Chun Chen, Sammy Chan, Jiajun Bu, and Athanasios V. Vasilakos. 2012. ReTrust: Attack-Resistantand Lightweight Trust Management for Medical Sensor Networks. IEEE Transactions on Information Technology inBiomedicine 16, 4 (July 2012), 623–632. DOI:http://dx.doi.org/10.1109/TITB.2012.2194788[69] Jaymi Heimbuch. 2016. Turn Your iPhone Into An Environmental Sensor. (2016). Retrieved October 16, 2016 fromhttp://www.treehugger.com/gadgets/turn-your-iphone-into-environmental-sensor.html[70] Philippe Herzig, Kay Jugel, Christof Momm, Michael Ameling, and Alexander Schill. 2013. GaML - A Modeling Language
for Gamification. In Proceedings of the 2013 IEEE/ACM International Conference on Utility and Cloud Computing (UCC’13). 494–499. DOI:http://dx.doi.org/10.1109/UCC.2013.96[71] Kashmir Hill. 2016. Fusion: If you use Waze, hackers can stalk you. (2016). Retrieved October 16, 2016 fromhttp://fusion.net/story/293157/waze-hack/[72] Jun-Won Ho, Matthew K. Wright, and Sajal K. Das. 2012. ZoneTrust: Fast Zone-Based Node Compromise Detection
and Revocation in Wireless Sensor Networks Using Sequential Hypothesis Testing. IEEE Transactions on Dependable
and Secure Computing 9, 4 (2012), 494–511. DOI:http://dx.doi.org/10.1109/TDSC.2011.65[73] Victoria Hodge and Jim Austin. 2004. A survey of outlier detection methodologies. Artificial intelligence review 22, 2(2004), 85–126.
[74] M. Anwar Hossain, Pradeep K. Atrey, and Abdulmotaleb El Saddik. 2011. Modeling and Assessing Quality of Informationin Multisensor Multimedia Monitoring Systems. ACM Trans. Multimedia Comput. Commun. Appl. 7, 1, Article 3 (Feb.
2011), 30 pages. DOI:http://dx.doi.org/10.1145/1870121.1870124[75] Baoqi Huang, Guodong Qi, Xiaokun Yang, Long Zhao, and Han Zou. 2016. Exploiting Cyclic Features of Walking
for Pedestrian Dead Reckoning with Unconstrained Smartphones. In Proceedings of the 2016 ACM International
Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’16). ACM, New York, NY, USA, 374–385. DOI:
http://dx.doi.org/10.1145/2971648.2971742[76] Chao Huang and Dong Wang. 2015. Spatial-Temporal Aware Truth Finding in Big Data Social Sensing Applications. In
Trustcom/BigDataSE/ISPA, 2015 IEEE, Vol. 2. 72–79. DOI:http://dx.doi.org/10.1109/Trustcom.2015.564[77] Chao Huang and Dong Wang. 2016. Topic-aware social sensing with arbitrary source dependency graphs. In Proceedingsof the 15th International Conference on Information Processing in Sensor Networks. IEEE Press, 7.
[78] Kuan Lun Huang, Salil S. Kanhere, and Wen Hu. 2010. Are You Contributing Trustworthy Data?: The Case for a
Reputation System in Participatory Sensing. In Proceedings of the 2010 ACM International Conference on Modeling,
Analysis, and Simulation of Wireless and Mobile Systems (MSWIM ’10). ACM, New York, NY, USA, 14–22. DOI:
http://dx.doi.org/10.1145/1868521.1868526[79] Kuan Lun Huang, Salil S. Kanhere, and Wen Hu. 2014. On the need for a reputation system in mobile phone based
sensing. Ad Hoc Networks 12 (2014), 130 – 149. DOI:http://dx.doi.org/https://doi.org/10.1016/j.adhoc.2011.12.002[80] Simon Jackman. 2009. Bayesian analysis for the social sciences. Vol. 846. John Wiley & Sons.
[81] Luis G. Jaimes, Idalides J. Vergara-Laurens, and Andrew. Raij. 2015. A Survey of Incentive Techniques for Mobile Crowd
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:35

Sensing. IEEE Internet of Things Journal 2, 5 (Oct 2015), 370–380. DOI:http://dx.doi.org/10.1109/JIOT.2015.2409151[82] Nathan Jensen. 2015. From pastoralists to Mechanical Turks: Using the crowd to validate crowdsourced data. http://tinyurl.com/zwbhk3w. (2015).
[83] Jinfang Jiang, Guangjie Han, Feng Wang, Lei Shu, and Mohsen Guizani. 2015. An Efficient Distributed Trust Model for
Wireless Sensor Networks. IEEE Transactions on Parallel and Distributed Systems 26, 5 (May 2015), 1228–1237. DOI:
http://dx.doi.org/10.1109/TPDS.2014.2320505[84] Jinfang Jiang, Guangjie Han, Chunsheng Zhu, Sammy Chan, and Joel J. P. C. Rodrigues. 2017. A Trust Cloud Modelfor Underwater Wireless Sensor Networks. IEEE Communications Magazine 55, 3 (March 2017), 110–116. DOI:
http://dx.doi.org/10.1109/MCOM.2017.1600502CM[85] Haiming Jin, Lu Su, and Klara Nahrstedt. 2017. CENTURION: Incentivizing Multi-Requester Mobile Crowd Sensing.
arXiv preprint arXiv:1701.01533 (2017).
[86] Haiming Jin, Lu Su, Houping Xiao, and Klara Nahrstedt. 2016. INCEPTION: Incentivizing Privacy-preserving Data
Aggregation for Mobile Crowd Sensing Systems. In Proceedings of the 17th ACM International Symposium on Mobile Ad
Hoc Networking and Computing (MobiHoc ’16). ACM, New York, NY, USA, 341–350. DOI:http://dx.doi.org/10.1145/
2942358.2942375[87] Audun Jøsang. 1999. An Algebra for Assessing Trust in Certification Chains.. In NDSS, Vol. 99. 80.
[88] Audun Jøsang and Zied Elouedi. 2007. Interpreting belief functions as Dirichlet distributions. In European Conference
on Symbolic and Quantitative Approaches to Reasoning and Uncertainty. Springer, 393–404.
[89] Audun Josang and Roslan Ismail. 2002. The beta reputation system. In Proceedings of the 15th bled electronic commerce
conference, Vol. 5. 2502–2511.
[90] Audun Jøsang, Roslan Ismail, and Colin Boyd. 2007. A survey of trust and reputation systems for online service
provision. Decision support systems 43, 2 (2007), 618–644.
[91] Ryoma Kawajiri, Masamichi Shimosaka, and Hisashi Kahima. 2014. Steered Crowdsensing: Incentive Design Towards
Quality-oriented Place-centric Crowdsensing. In Proceedings of the 2014 ACM International Joint Conference on Pervasive
and Ubiquitous Computing (UbiComp ’14). ACM, New York, NY, USA, 691–701. DOI:http://dx.doi.org/10.1145/2632048.
2636064[92] Wazir Z. Khan, Yang Xiang, Mohammed Y. Aalsalem, and Quratulain Arshad. 2013. Mobile Phone Sensing Systems: A
Survey. IEEE Communications Surveys and Tutorials 15, 1 (First 2013), 402–427. DOI:http://dx.doi.org/10.1109/SURV.
2012.031412.00077[93] Yeolib Kim and Robert A. Peterson. 2017. A Meta-analysis of Online Trust Relationships in E-commerce. Journal ofInteractive Marketing 38 (2017), 44 – 54. DOI:http://dx.doi.org/https://doi.org/10.1016/j.intmar.2017.01.001[94] George Klir and Bo Yuan. 1995. Fuzzy sets and fuzzy logic. Vol. 4. Prentice hall New Jersey.
[95] Shirlee-ann Knight. 2008. User perceptions of information quality in World Wide Web information retrieval behaviour.
Ph.D. Dissertation. Edith Cowan University.
[96] Koteswararao Kondepu, Francesco Restuccia, Giuseppe Anastasi, and Marco Conti. 2012. A hybrid and flexible
discovery algorithm for wireless sensor networks with mobile elements. In 2012 IEEE Symposium on Computers andCommunications (ISCC). 000295–000300. DOI:http://dx.doi.org/10.1109/ISCC.2012.6249311[97] Andreas Krause, Eric Horvitz, Aman Kansal, and Feng Zhao. 2008. Toward Community Sensing. In Proceedings ofthe 2008 ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN ’08). 481–492. DOI:
http://dx.doi.org/10.1109/IPSN.2008.37[98] Nicholas D. Lane, Emiliano Miluzzo, Hong Lu, Daniel Peebles, Tanzeem Choudhury, and Andrew T. Campbell.
2010. A survey of mobile phone sensing. IEEE Communications Magazine 48, 9 (Sept 2010), 140–150. DOI:http://dx.doi.org/10.1109/MCOM.2010.5560598[99] Na Li and Sajal K. Das. 2013. A trust-based framework for data forwarding in opportunistic networks. Ad Hoc Networks
11, 4 (2013), 1497 – 1509. DOI:http://dx.doi.org/https://doi.org/10.1016/j.adhoc.2011.01.018 1. System and Theoretical
Issues in Designing and Implementing Scalable and Sustainable Wireless Sensor Networks2. Wireless Communications
and Networking in Challenged Environments.
[100] Xiaoyong Li, Feng Zhou, and Junping Du. 2013. LDTS: A Lightweight and Dependable Trust System for Clustered
Wireless Sensor Networks. IEEE Transactions on Information Forensics and Security 8, 6 (June 2013), 924–935. DOI:
http://dx.doi.org/10.1109/TIFS.2013.2240299[101] Yaliang Li, Jing Gao, Chuishi Meng, Qi Li, Lu Su, Bo Zhao, Wei Fan, and Jiawei Han. 2016a. A Survey on Truth
Discovery. SIGKDD Explor. Newsl. 17, 2 (Feb. 2016), 1–16. DOI:http://dx.doi.org/10.1145/2897350.2897352[102] Yaliang Li, Jing Gao, Chuishi Meng, Qi Li, Lu Su, Bo Zhao, Wei Fan, and Jiawei Han. 2016b. A Survey on Truth
Discovery. SIGKDD Exploration Newsletter 17, 2 (Feb. 2016), 1–16. DOI:http://dx.doi.org/10.1145/2897350.2897352[103] Shuyi Lin, Wenxing Hong, Dingding Wang, and Tao Li. 2017. A survey on expert finding techniques. Journal ofIntelligent Information Systems (2017), 1–25.
[104] Yan Liu, Bin Guo, Yang Wang, Wenle Wu, Zhiwen Yu, and Daqing Zhang. 2016. TaskMe: Multi-task Allocation in
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:36

F. Restuccia et al.

Mobile Crowd Sensing. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous
Computing (UbiComp ’16). ACM, New York, NY, USA, 403–414. DOI:http://dx.doi.org/10.1145/2971648.2971709[105] Tie Luo, Salil S. Kanhere, and Hwee-Pink Tan. 2014a. SEW-ing a Simple Endorsement Web to Incentivize Trustworthy
Participatory Sensing. In Proceedings of the 2014 Annual IEEE Communications Society Conference on Sensor, Mesh and
Ad Hoc Communications and Networks (SECON ’14).
[106] Tie Luo, Salil S. Kanhere, and Hwee-Pink Tan. 2014b. SEW-ing a Simple Endorsement Web to incentivize trustworthy
participatory sensing. In 2014 Eleventh Annual IEEE International Conference on Sensing, Communication, and Networking(SECON). 636–644. DOI:http://dx.doi.org/10.1109/SAHCN.2014.6990404[107] Fenglong Ma, Yaliang Li, Qi Li, Minghui Qiu, Jing Gao, Shi Zhi, Lu Su, Bo Zhao, Heng Ji, and Jiawei Han. 2015.
FaitCrowd: Fine Grained Truth Discovery for Crowdsourced Data Aggregation. In Proceedings of the 21th ACM SIGKDDInternational Conference on Knowledge Discovery and Data Mining (KDD ’15). ACM, New York, NY, USA, 745–754. DOI:
http://dx.doi.org/10.1145/2783258.2783314[108] Stephen Paul Marsh. 1994. Formalising trust as a computational concept. (1994).
[109] Jermaine Marshall, Munira Syed, and Dong Wang. 2016. Hardness-Aware Truth Discovery in Social Sensing Applications. In 2016 International Conference on Distributed Computing in Sensor Systems (DCOSS). 143–152. DOI:
http://dx.doi.org/10.1109/DCOSS.2016.9[110] E. Masazade, M. Fardad, and P. K. Varshney. 2012. Sparsity-Promoting Extended Kalman Filtering for Target Tracking
in Wireless Sensor Networks. IEEE Signal Processing Letters 19, 12 (Dec 2012), 845–848. DOI:http://dx.doi.org/10.1109/
LSP.2012.2220350[111] Mini Mathew and Ning Weng. 2014. Quality of Information and Energy Efficiency Optimization for Sensor Networks
via Adaptive Sensing and Transmitting. IEEE Sensors Journal 14, 2 (Feb 2014), 341–348. DOI:http://dx.doi.org/10.1109/
JSEN.2013.2282774[112] Elizabeth A. McCartney, Kari J. Craun, Erin Korris, David A. Brostuen, and Laurence R. Moore. 2015. Crowdsourcing
The National Map. Cartography and Geographic Information Science 42, sup1 (2015), 54–57. DOI:http://dx.doi.org/10.
1080/15230406.2015.1059187[113] Richard D. McKelvey and Thomas R. Palfrey. 1995. Quantal response equilibria for normal form games. Games andEconomic Behavior 10, 1 (1995), 6–38.
[114] Richard D. McKelvey and Thomas R. Palfrey. 1998. Quantal response equilibria for extensive form games. Experimental
Economics 1, 1 (1998), 9–41. DOI:http://dx.doi.org/10.1007/BF01426213[115] Chuishi Meng, Wenjun Jiang, Yaliang Li, Jing Gao, Lu Su, Hu Ding, and Yun Cheng. 2015. Truth Discovery on Crowd
Sensing of Correlated Entities. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems(SenSys ’15). ACM, New York, NY, USA, 169–182. DOI:http://dx.doi.org/10.1145/2809695.2809715[116] Chuishi Meng, Houping Xiao, Lu Su, and Yun Cheng. 2016. Tackling the Redundancy and Sparsity in Crowd Sensing
Applications. In Proceedings of the 14th ACM Conference on Embedded Network Sensor Systems CD-ROM. ACM, 150–163.
[117] Chenglin Miao, Wenjun Jiang, Lu Su, Yaliang Li, Suxin Guo, Zhan Qin, Houping Xiao, Jing Gao, and Kui Ren.
2015. Cloud-Enabled Privacy-Preserving Truth Discovery in Crowd Sensing Systems. In Proceedings of the 13th
ACM Conference on Embedded Networked Sensor Systems (SenSys ’15). ACM, New York, NY, USA, 183–196. DOI:
http://dx.doi.org/10.1145/2809695.2809719[118] Hayam Mousa, Sonia Ben Mokhtar, Omar Hasan, Osama Younes, Mohiy Hadhoud, and Lionel Brunie. 2015. Trust
management and reputation systems in mobile participatory sensing applications: A survey. Computer Networks 90(2015), 49 – 73. DOI:http://dx.doi.org/10.1016/j.comnet.2015.07.011 Crowdsourcing.
[119] Eric J. Msechu and Georgios B. Giannakis. 2012. Sensor-Centric Data Reduction for Estimation With WSNs via
Censoring and Quantization. IEEE Transactions on Signal Processing 60, 1 (Jan 2012), 400–414. DOI:http://dx.doi.org/
10.1109/TSP.2011.2171686[120] Lik Mui, Mojdeh Mohtashemi, and Ari Halberstadt. 2002. A computational model of trust and reputation. In System
Sciences, 2002. HICSS. Proceedings of the 35th Annual Hawaii International Conference on. IEEE, 2431–2439.
[121] Tridib Mukherjee, Deepthi Chander, Anirban Mondal, Koutsuv Dasgupta, Amit Kumar, and Ashwin Venkat. 2014.
CityZen: A Cost-Effective City Management System with Incentive-Driven Resident Engagement. In Proceedings of the2014 IEEE International Conference on Mobile Data Management (MDM ’14), Vol. 1. 289–296. DOI:http://dx.doi.org/10.
1109/MDM.2014.41[122] Sarfraz Nawaz and Cecilia Mascolo. 2014. Mining Users’ Significant Driving Routes with Low-power Sensors. In
Proceedings of the 2014 ACM Conference on Embedded Network Sensor Systems (SenSys ’14). ACM, New York, NY, USA,
236–250. DOI:http://dx.doi.org/10.1145/2668332.2668348[123] R. Olfati-Saber. 2007. Distributed Kalman filtering for sensor networks. In 2007 46th IEEE Conference on Decision andControl. 5492–5498. DOI:http://dx.doi.org/10.1109/CDC.2007.4434303[124] Tiago Oliveira, Matilde Alhinho, Paulo Rita, and Gurpreet Dhillon. 2017. Modelling and testing consumer trust
dimensions in e-commerce. Computers in Human Behavior 71 (2017), 153 – 164. DOI:http://dx.doi.org/https://doi.org/
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:37

10.1016/j.chb.2017.01.050[125] International Standards Organization. 2015. ISO 9000:2015 - Quality management. (2015). Retrieved October 16, 2016
from http://www.iso.org/iso/home/standards/management-standards/iso_9000.htm[126] Robin W. Ouyang, Lance M. Kaplan, Alice Toniolo, Mani Srivastava, and Timothy J. Norman. 2016a. Aggregating
Crowdsourced Quantitative Claims: Additive and Multiplicative Models. IEEE Transactions on Knowledge and Data
Engineering 28, 7 (July 2016), 1621–1634. DOI:http://dx.doi.org/10.1109/TKDE.2016.2535383[127] Robin W. Ouyang, Mani Srivastava, Alice Toniolo, and Timothy J. Norman. 2016b. Truth Discovery in Crowdsourced
Detection of Spatial Events. IEEE Transactions on Knowledge and Data Engineering 28, 4 (April 2016), 1047–1060. DOI:
http://dx.doi.org/10.1109/TKDE.2015.2504928[128] Jeff Pasternack and Dan Roth. 2010. Knowing What to Believe (when You Already Know Something). In Proceedings ofthe 23rd International Conference on Computational Linguistics (COLING ’10). Association for Computational Linguistics,
Stroudsburg, PA, USA, 877–885. http://dl.acm.org/citation.cfm?id=1873781.1873880[129] Jeff Pasternack and Dan Roth. 2011. Generalized Fact-finding. In Proceedings of the 20th International Conference
Companion on World Wide Web (WWW ’11). ACM, New York, NY, USA, 99–100. DOI:http://dx.doi.org/10.1145/1963192.
1963243[130] Silvana S. Pereira, Roberto Lòpez-Valcarce, and Alba Pagès-Zamora. 2013. A Diffusion-Based EM Algorithm forDistributed Estimation in Unreliable Sensor Networks. IEEE Signal Processing Letters 20, 6 (June 2013), 595–598. DOI:
http://dx.doi.org/10.1109/LSP.2013.2260329[131] Maryam Pouryazdan, Burak Kantarci, Tolga Soyata, Luca Foschini, and Houbing Song. 2017. Quantifying User
Reputation Scores, Data Trustworthiness, and User Incentives in Mobile Crowd-Sensing. IEEE Access 5 (2017), 1382–
1397. DOI:http://dx.doi.org/10.1109/ACCESS.2017.2660461[132] Guo-Jun Qi, Charu C. Aggarwal, Jiawei Han, and Thomas Huang. 2013. Mining Collective Intelligence in Diverse
Groups. In Proceedings of the 22nd International Conference on World Wide Web (WWW ’13). ACM, New York, NY, USA,1041–1052. DOI:http://dx.doi.org/10.1145/2488388.2488479[133] Fudong Qiu, Fan Wu, and Guihai Chen. 2015. Privacy and Quality Preserving Multimedia Data Aggregation
for Participatory Sensing Systems. IEEE Transactions on Mobile Computing 14, 6 (June 2015), 1287–1300. DOI:
http://dx.doi.org/10.1109/TMC.2014.2352253[134] Kiran K. Rachuri, Mirco Musolesi, Cecilia Mascolo, Peter J. Rentfrow, Chris Longworth, and Andrius Aucinas. 2010.
EmotionSense: A Mobile Phones Based Adaptive Platform for Experimental Social Psychology Research. In Proceedingsof the 2010 ACM International Conference on Ubiquitous Computing (Ubicomp ’10). ACM, New York, NY, USA, 281–290.
DOI:http://dx.doi.org/10.1145/1864349.1864393[135] Sarvapali D Ramchurn, Nicholas R Jennings, Carles Sierra, and Lluis Godo. 2004. Devising a trust model for multi-agent
interactions using confidence and reputation. Applied Artificial Intelligence 18, 9-10 (2004), 833–852.
[136] Rajib Rana, Chun Tung Chou, Nirupama Bulusu, Salil Kanhere, and Wen Hu. 2014. Ear-Phone: A context-aware noise
mapping using smart phones. Pervasive and Mobile Computing 0 (2014), –. DOI:http://dx.doi.org/10.1016/j.pmcj.2014.
02.001[137] Heena Rathore, Venkataramana Badarla, and Supratim Shit. 2016. Consensus-Aware Sociopsychological Trust
Model for Wireless Sensor Networks. ACM Trans. Sen. Netw. 12, 3, Article 21 (July 2016), 27 pages. DOI:http://dx.doi.org/10.1145/2903721[138] Sasank Reddy, Deborah Estrin, and Mani Srivastava. 2010. Recruitment Framework for Participatory Sensing Data
Collections. Springer Berlin Heidelberg, Berlin, Heidelberg, 138–155. DOI:http://dx.doi.org/10.1007/978-3-642-12654-3_
9[139] Sasank Reddy, Andrew Parker, Josh Hyman, Jeff Burke, Deborah Estrin, and Mark Hansen. 2007. Image Browsing,Processing, and Clustering for Participatory Sensing: Lessons from a DietSense Prototype. In Proceedings of the2007 ACM Workshop on Embedded Networked Sensors (EmNets ’07). ACM, New York, NY, USA, 13–17. DOI:http://dx.doi.org/10.1145/1278972.1278975[140] Ju Ren, Yaoxue Zhang, Kuan Zhang, and Xuemin (Sherman) Shen. 2015. SACRM: Social Aware Crowdsourcing with
Reputation Management in mobile sensing. Computer Communications 65 (2015), 55 – 65. DOI:http://dx.doi.org/https://doi.org/10.1016/j.comcom.2015.01.022 Mobile Ubiquitous Sensing from Social Network Viewpoint.
[141] Francesco Restuccia. 2016. Mechanisms for improving information quality in smartphone crowdsensing systems. Ph.D.
Dissertation. Missouri University of Science and Technology.
[142] Francesco Restuccia, Giuseppe Anastasi, Marco Conti, and Sajal K. Das. 2012. Performance analysis of a hierarchicaldiscovery protocol for WSNs with Mobile Elements. In 2012 IEEE International Symposium on a World of Wireless,
Mobile and Multimedia Networks (WoWMoM). 1–9. DOI:http://dx.doi.org/10.1109/WoWMoM.2012.6263708[143] Francesco Restuccia, Giuseppe Anastasi, Marco Conti, and Sajal K. Das. 2014. Analysis and Optimization of a Protocol
for Mobile Element Discovery in Sensor Networks. IEEE Transactions on Mobile Computing 13, 9 (Sept 2014), 1942–1954.
DOI:http://dx.doi.org/10.1109/TMC.2013.88
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:38

F. Restuccia et al.

[144] Francesco Restuccia and Sajal K. Das. 2014. FIDES: A trust-based framework for secure user incentivization inparticipatory sensing. In Proceeding of IEEE International Symposium on a World of Wireless, Mobile and Multimedia
Networks 2014. 1–10. DOI:http://dx.doi.org/10.1109/WoWMoM.2014.6918972[145] Francesco Restuccia and Sajal K. Das. 2015. Lifetime optimization with QoS of sensor networks with uncontrollable
mobile sinks. In 2015 IEEE 16th International Symposium on A World of Wireless, Mobile and Multimedia Networks(WoWMoM). 1–9. DOI:http://dx.doi.org/10.1109/WoWMoM.2015.7158130[146] Francesco Restuccia and Sajal K. Das. 2016. Optimizing the Lifetime of Sensor Networks with Uncontrollable MobileSinks and QoS Constraints. ACM Trans. Sen. Netw. 12, 1, Article 2 (March 2016), 31 pages. DOI:http://dx.doi.org/10.
1145/2873059[147] Francesco Restuccia, Sajal K. Das, and Jamie Payton. 2016a. Incentive Mechanisms for Participatory Sensing: Survey
and Research Challenges. ACM Trans. Sen. Netw. 12, 2, Article 13 (April 2016), 40 pages. DOI:http://dx.doi.org/10.1145/
2888398[148] Francesco Restuccia, Srinivas C. Thandu, Sriram Chellappan, and Sajal K. Das. 2016b. RescuePal: A smartphone-based
system to discover people in emergency scenarios. In 2016 IEEE 17th International Symposium on A World of Wireless,
Mobile and Multimedia Networks (WoWMoM). 1–6. DOI:http://dx.doi.org/10.1109/WoWMoM.2016.7523566[149] Jiang Rong, Tao Qin, and Bo An. 2015. Computing Quantal Response Equilibrium for Sponsored Search Auctions.
In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems (AAMAS ’15).
International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1803–1804. http://dl.acm.
org/citation.cfm?id=2772879.2773445[150] Matthew Roth. 2013. Wikimedia Foundation sends cease and desist letter to WikiPR. (2013). Retrieved October 16,
2016 from https://blog.wikimedia.org/2013/11/19/wikimedia-foundation-sends-cease-and-desist-letter-to-wikipr/[151] Sankardas Roy, Mauro Conti, Sanjeev Setia, and Sushil Jajodia. 2014. Secure Data Aggregation in Wireless Sensor
Networks: Filtering out the Attacker’s Impact. IEEE Transactions on Information Forensics and Security 9, 4 (April 2014),
681–694. DOI:http://dx.doi.org/10.1109/TIFS.2014.2307197[152] Jordi Sabater-Mir and Mario Paolucci. 2007. On representation and aggregation of social evaluations in computational
trust and reputation models. International Journal of Approximate Reasoning 46, 3 (2007), 458–483.
[153] Mahmood Salehi, Azzedine Boukerche, Amir Darehshoorzadeh, and Abdelhamid Mammeri. 2016. Towards a novel
trust-based opportunistic routing protocol for wireless networks. Wireless Networks 22, 3 (2016), 927–943. DOI:
http://dx.doi.org/10.1007/s11276-015-1010-4[154] G. Enrico Santagati and Tommaso Melodia. 2015. U-Wear: Software-Defined Ultrasonic Networking for Wearable
Devices. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys’15). ACM, New York, NY, USA, 241–256. DOI:http://dx.doi.org/10.1145/2742647.2742655[155] Fatemeh Saremi, Omid Fatemieh, Hossein Ahmadi, Hongyan Wang, Tarek Abdelzaher, Raghu Ganti, Hengchang Liu,
Shaohan Hu, Shen Li, and Lu Su. 2016. Experiences with GreenGPS – Fuel-Efficient Navigation Using Participatory
Sensing. IEEE Transactions on Mobile Computing 15, 3 (March 2016), 672–689. DOI:http://dx.doi.org/10.1109/TMC.
2015.2421939[156] Stefan Saroiu and Alec Wolman. 2010. I Am a Sensor, and I Approve This Message. In Proceedings of the Eleventh
Workshop on Mobile Computing Systems &#38; Applications (HotMobile ’10). ACM, New York, NY, USA, 37–42. DOI:
http://dx.doi.org/10.1145/1734583.1734593[157] Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get Another Label? Improving Data Quality andData Mining Using Multiple, Noisy Labelers. In Proceedings of the 14th ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining (KDD ’08). ACM, New York, NY, USA, 614–622. DOI:http://dx.doi.org/10.1145/
1401890.1401965[158] Xiaohong Sheng and Yu-Hen Hu. 2005. Maximum likelihood multiple-source localization using acoustic energy
measurements with wireless sensor networks. IEEE Transactions on Signal Processing 53, 1 (Jan 2005), 44–53. DOI:
http://dx.doi.org/10.1109/TSP.2004.838930[159] Herbert A Simon. 1957. Models of man; social and rational. (1957).
[160] Herbert Alexander Simon. 1982. Models of bounded rationality: Empirically grounded economic reason. Vol. 3. MITpress.
[161] Tilo Strutz. 2016. Data fitting and uncertainty: A practical introduction to weighted least squares and beyond. Springer
Vieweg.
[162] Lu Su, Qi Li, Shaohan Hu, Shiguang Wang, Jing Gao, Hengchang Liu, Tarek F. Abdelzaher, Jiawei Han, Xue Liu, Yan
Gao, and Lance Kaplan. 2014. Generalized Decision Aggregation in Distributed Sensing Systems. In Real-Time Systems
Symposium (RTSS), 2014 IEEE. 1–10. DOI:http://dx.doi.org/10.1109/RTSS.2014.40[163] Yan Sun, Hong Luo, and Sajal K. Das. 2012. A Trust-Based Framework for Fault-Tolerant Data Aggregation in WirelessMultimedia Sensor Networks. IEEE Transactions on Dependable and Secure Computing 9, 6 (Nov 2012), 785–797. DOI:
http://dx.doi.org/10.1109/TDSC.2012.68
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:39[164] N. Suri, G. Benincasa, R. Lenzi, M. Tortonesi, C. Stefanelli, and L. Sadler. 2015. Exploring value-of-information-based
approaches to support effective communications in tactical networks. IEEE Communications Magazine 53, 10 (October2015), 39–45. DOI:http://dx.doi.org/10.1109/MCOM.2015.7295461[165] Manoop Talasila, Reza Curtmola, and Cristian Borcea. 2015. Collaborative Bluetooth-based location authentication
on smart phones. Pervasive and Mobile Computing 17, Part A (2015), 43 – 62. DOI:http://dx.doi.org/10.1016/j.pmcj.
2014.02.004[166] Alexey Tarasov, Sarah Jane Delany, and Brian Mac Namee. 2014. Dynamic estimation of worker reliability incrowdsourcing for regression tasks: Making it work. Expert Systems with Applications 41, 14 (2014), 6190 – 6210. DOI:
http://dx.doi.org/10.1016/j.eswa.2014.04.012[167] Chen-Khong Tham and Tie Luo. 2015. Quality of Contributed Service and Market Equilibrium for Participatory Sensing.
IEEE Transactions on Mobile Computing 14, 4 (April 2015), 829–842. DOI:http://dx.doi.org/10.1109/TMC.2014.2330302[168] Thermodo. 2016. Thermodo temperature sensor. (2016). Retrieved October 16, 2016 from http://www.thermodo.com[169] Yoshitaka Ueyama, Morihiko Tamai, Yutaka Arakawa, and Keiichi Yasumoto. 2014. Gamification-based incentive
mechanism for participatory sensing. In Proceedings of the 2014 IEEE International Conference on Pervasive Computing andCommunications Workshops (PERCOM Workshops ’14). 98–103. DOI:http://dx.doi.org/10.1109/PerComW.2014.6815172[170] International Telecommunication Unit. 2008. ITU-T Rec. E.800. (2008). Retrieved October 16, 2016 from http://www.itu.int/rec/T-REC-E.800-200809-I[171] R. Annie Uthra and S. V. Kasmir Raja. 2012. QoS Routing in Wireless Sensor Networks – a Survey. Comput. Surveys
45, 1, Article 9 (Dec. 2012), 12 pages. DOI:http://dx.doi.org/10.1145/2379776.2379785[172] Werner Van Leekwijck and Etienne E Kerre. 1999. Defuzzification: criteria and classification. Fuzzy sets and systems
108, 2 (1999), 159–178.
[173] Tom Vanderbilt. 2016. Waze: The App That Changed Driving. (2016). Retrieved October 16, 2016 from http://www.mensjournal.com/gear/cars/waze-the-app-that-changed-driving-20160208[174] Dong Wang, Tarek Abdelzaher, Hossein Ahmadi, Jeff Pasternack, Dan Roth, Manish Gupta, Jiawei Han, Omid
Fatemieh, Hieu Le, and Charu C. Aggarwal. 2011. On Bayesian interpretation of fact-finding in information networks.
In 14th International Conference on Information Fusion. 1–8.
[175] Dong Wang, Tarek Abdelzaher, and Lance Kaplan. 2015. Social Sensing: Building Reliable Systems on Unreliable Data
(1st ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
[176] D. Wang, T. Abdelzaher, L. Kaplan, R. Ganti, S. Hu, and H. Liu. 2013. Exploitation of Physical Constraints for Reliable
Social Sensing. In Real-Time Systems Symposium (RTSS), 2013 IEEE 34th. 212–223. DOI:http://dx.doi.org/10.1109/RTSS.
2013.29[177] Dong Wang, Md Tanvir Amin, Shen Li, Tarek Abdelzaher, Lance Kaplan, Siyu Gu, Chenji Pan, Hengchang Liu,
Charu C. Aggarwal, Raghu Ganti, Xinlei Wang, Prasant Mohapatra, Boleslaw Szymanski, and Hieu Le. 2014. Using
Humans As Sensors: An Estimation-theoretic Perspective. In Proceedings of the 13th International Symposium on
Information Processing in Sensor Networks (IPSN ’14). IEEE Press, Piscataway, NJ, USA, 35–46. http://dl.acm.org/citation.
cfm?id=2602339.2602344[178] Dong Wang, Lance Kaplan, Hieu Le, and Tarek Abdelzaher. 2012. On Truth Discovery in Social Sensing: A Maximum
Likelihood Estimation Approach. In Proceedings of the 11th International Conference on Information Processing in Sensor
Networks (IPSN ’12). ACM, New York, NY, USA, 233–244. DOI:http://dx.doi.org/10.1145/2185677.2185737[179] Gang Wang, Sarita Schoenebeck, Haitao Zheng, and Ben Zhao. 2016b. ”Will Check-in for Badges”: Understanding Biasand Misbehavior on Location-Based Social Networks. (2016). http://www.aaai.org/ocs/index.php/ICWSM/ICWSM16/
paper/view/13027[180] Gang Wang, Bolun Wang, Tianyi Wang, Ana Nika, Haitao Zheng, and Ben Y. Zhao. 2016. Defending Against Sybil
Devices in Crowdsourced Mapping Services. In Proceedings of the 14th Annual International Conference on Mobile
Systems, Applications, and Services (MobiSys ’16). ACM, New York, NY, USA, 179–191. DOI:http://dx.doi.org/10.1145/
2906388.2906420[181] G Alan Wang, Jian Jiao, Alan S Abrahams, Weiguo Fan, and Zhongju Zhang. 2013. ExpertRank: A topic-aware expert
finding algorithm for online knowledge communities. Decision Support Systems 54, 3 (2013), 1442–1451.
[182] Jiangtao Wang, Yasha Wang, Daqing Zhang, Feng Wang, Yuanduo He, and Liantao Ma. 2017. PSAllocator: Multi-Task
Allocation for Participatory Sensing with Sensing Capability Constraints. In Proceedings of the 2017 ACM Conference
on Computer Supported Cooperative Work and Social Computing (CSCW ’17). ACM, New York, NY, USA, 1139–1151.
DOI:http://dx.doi.org/10.1145/2998181.2998193[183] Jiangtao Wang, Yasha Wang, Daqing Zhang, Leye Wang, Haoyi Xiong, Abdelsalam Helal, Yuanduo He, and Feng
Wang. 2016. Fine-Grained Multitask Allocation for Participatory Sensing With a Shared Budget. IEEE Internet of Things
Journal 3, 6 (2016), 1395–1405.
[184] Jianwei Wang, Yuping Zhao, and Dong Wang. 2007. A Novel Fast Anti-Collision Algorithm for RFID Systems.
In 2007 International Conference on Wireless Communications, Networking and Mobile Computing. 2044–2047. DOI:ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

1:40

F. Restuccia et al.

http://dx.doi.org/10.1109/WICOM.2007.511[185] Richard Y. Wang and Diane M. Strong. 1996. Beyond Accuracy: What Data Quality Means to Data Consumers. Journalof Management Information Systems 12, 4 (1996), 5–33. DOI:http://dx.doi.org/10.1080/07421222.1996.11518099[186] Shiguang Wang, Lu Su, Shen Li, Shaohan Hu, Tanvir Amin, Hongwei Wang, Shuochao Yao, Lance Kaplan, and Tarek
Abdelzaher. 2015. Scalable Social Sensing of Interdependent Phenomena. In Proceedings of the 14th International
Conference on Information Processing in Sensor Networks (IPSN ’15). ACM, New York, NY, USA, 202–213. DOI:http://dx.doi.org/10.1145/2737095.2737114[187] Shiguang Wang, Dong Wang, Lu Su, Lance Kaplan, and Tarek F. Abdelzaher. 2014. Towards Cyber-Physical Systems
in Social Spaces: The Data Reliability Challenge. In Real-Time Systems Symposium (RTSS), 2014 IEEE. 74–85. DOI:
http://dx.doi.org/10.1109/RTSS.2014.19[188] Xinlei Wang, Wei Cheng, Prasant Mohapatra, and Tarek Abdelzaher. 2013. ARTSense: Anonymous reputation andtrust in participatory sensing. In Proceedings of the 2013 IEEE International Conference of Computer Communications(INFOCOM ’13). 2517–2525. DOI:http://dx.doi.org/10.1109/INFCOM.2013.6567058[189] Xinlei Wang, Wei Cheng, Prasant Mohapatra, and Tarek Abdelzaher. 2014. Enabling Reputation and Trust in
Privacy-Preserving Mobile Sensing. IEEE Transactions on Mobile Computing 13, 12 (Dec 2014), 2777–2790. DOI:
http://dx.doi.org/10.1109/TMC.2013.150[190] Xingbo Wang, Minyue Fu, and Huanshui Zhang. 2012. Target Tracking in Wireless Sensor Networks Based on the
Combination of KF and MLE Using Distance Measurements. IEEE Transactions on Mobile Computing 11, 4 (April 2012),
567–576. DOI:http://dx.doi.org/10.1109/TMC.2011.59[191] Xinlei Wang, Kannan Govindan, and Prasant Mohapatra. 2011a. Collusion-resilient quality of information evaluation
based on information provenance. In Proceedings of the 2011 Annual IEEE Communications Society Conference on Sensor,
Mesh and Ad Hoc Communications and Networks (SECON ’11). 395–403. DOI:http://dx.doi.org/10.1109/SAHCN.2011.
5984923[192] Xinlei Wang, Kannan Govindan, and Prasant Mohapatra. 2011b. Collusion-resilient quality of information evaluation
based on information provenance. In 2011 8th Annual IEEE Communications Society Conference on Sensor, Mesh and Ad
Hoc Communications and Networks. 395–403. DOI:http://dx.doi.org/10.1109/SAHCN.2011.5984923[193] Xinlei Wang, Amit Pande, Jindan Zhu, and Prasant Mohapatra. 2016a. STAMP: Enabling Privacy-Preserving Location
Proofs for Mobile Users. IEEE/ACM Transactions on Networking 24, 6 (December 2016), 3276–3289. DOI:http://dx.doi.
org/10.1109/TNET.2016.2515119[194] Waze. 2016. The Waze Traffic Monitoring Application by Google. (2016). Retrieved October 16, 2016 from http://www.waze.com[195] Merriam Webster. 2017. Definition of Trust. (2017). Retrieved Februrary 16, 2017 from https://www.merriam-webster.
com/dictionary/trust[196] Qikun Xiang, Jie Zhang, Ido Nevat, and Pengfei Zhang. 2017. A Trust-based Mixture of Gaussian Processes Model for
Robust Participatory Sensing. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems(AAMAS ’17). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1760–1762.
http://dl.acm.org/citation.cfm?id=3091282.3091430[197] Fei Xiong, Yun Liu, and Junjun Cheng. 2017. Modeling and predicting opinion formation with trust propagation in
online social networks. Communications in Nonlinear Science and Numerical Simulation 44 (2017), 513 – 524. DOI:
http://dx.doi.org/https://doi.org/10.1016/j.cnsns.2016.09.015[198] Lin Yao, Yanmao Man, Zhong Huang, Jing Deng, and Xin Wang. 2016d. Secure Routing Based on Social Similarity
in Opportunistic Networks. IEEE Transactions on Wireless Communications 15, 1 (Jan 2016), 594–605. DOI:http://dx.doi.org/10.1109/TWC.2015.2476466[199] Shuochao Yao, Md Tanvir Amin, Lu Su, Shaohan Hu, Shen Li, Shiguang Wang, Yiran Zhao, Tarek Abdelzaher, Lance
Kaplan, Charu Aggarwal, and others. 2016b. Recursive ground truth estimator for social data streams. In Information
Processing in Sensor Networks (IPSN), 2016 15th ACM/IEEE International Conference on. IEEE, 1–12.
[200] S. Yao, M. T. Amin, L. Su, S. Hu, S. Li, S. Wang, Y. Zhao, T. Abdelzaher, L. Kaplan, C. Aggarwal, and A. Yener.
2016a. Recursive Ground Truth Estimator for Social Data Streams. In 2016 15th ACM/IEEE International Conference on
Information Processing in Sensor Networks (IPSN). 1–12. DOI:http://dx.doi.org/10.1109/IPSN.2016.7460719[201] Shuochao Yao, Shaohan Hu, Shen Li, Yiran Zhao, Lu Su, Lance Kaplan, Aylin Yener, and Tarek Abdelzaher. 2016c. On
Source Dependency Models for Reliable Social Sensing: Algorithms and Fundamental Error Bounds. In Distributed
Computing Systems (ICDCS), 2016 IEEE 36th International Conference on. IEEE, 467–476.
[202] Xiaoxin Yin, Jiawei Han, and Philip S. Yu. 2008. Truth Discovery with Multiple Conflicting Information Providers on
the Web. IEEE Transactions on Knowledge and Data Engineering 20, 6 (June 2008), 796–808. DOI:http://dx.doi.org/10.
1109/TKDE.2007.190745[203] Mohamed Younis, Izzet F. Senturk, Kemal Akkaya, Sookyoung Lee, and Fatih Senel. 2014. Topology management
techniques for tolerating node failures in wireless sensor networks: A survey. Computer Networks 58 (2014), 254 – 283.
ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.

QoI in Mobile Crowdsensing Survey

1:41

DOI:http://dx.doi.org/10.1016/j.comnet.2013.08.021[204] Dian Yu, Hongzhao Huang, Taylor Cassidy, Heng Ji, Chi Wang, Shi Zhi, Jiawei Han, Clare R Voss, and Malik MagdonIsmail. 2014a. The Wisdom of Minority: Unsupervised Slot Filling Validation based on Multi-dimensional Truth-Finding.
In COLING. 1567–1578.
[205] Ruiyun Yu, Rui Liu, Xingwei Wang, and Jiannong Cao. 2014b. Improving Data Quality with an Accumulated Reputation
Model in Participatory Sensing Systems. Sensors 14, 3 (2014), 5573–5594. DOI:http://dx.doi.org/10.3390/s140305573[206] Alexandros Zenonos, Sebastian Stein, and Nicholas R. Jennings. 2015. Coordinating Measurements for Air Pollution
Monitoring in Participatory Sensing Settings. In Proceedings of the 2015 International Conference on Autonomous Agentsand Multiagent Systems (AAMAS ’15). International Foundation for Autonomous Agents and Multiagent Systems,
Richland, SC, 493–501. http://dl.acm.org/citation.cfm?id=2772879.2772943[207] Daqing Zhang, Haoyi Xiong, Leye Wang, and Guanling Chen. 2014. CrowdRecruiter: Selecting Participants for
Piggyback Crowdsensing Under Probabilistic Coverage Constraint. In Proceedings of the 2014 ACM International
Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’14). ACM, New York, NY, USA, 703–714. DOI:
http://dx.doi.org/10.1145/2632048.2632059[208] Maotiang Zhang, Panlong Yang, Chang Tian, Shaojie Tang, Xiaofeng Gao, Baowei Wang, and Fu Xiao. 2016. QualityAware Sensing Coverage in Budget-Constrained Mobile Crowdsensing Networks. IEEE Transactions on VehicularTechnology 65, 9 (Sept 2016), 7698–7707. DOI:http://dx.doi.org/10.1109/TVT.2015.2490679[209] Bo Zhao, Benjamin I. P. Rubinstein, Jim Gemmell, and Jiawei Han. 2012. A Bayesian Approach to Discovering Truth
from Conflicting Sources for Data Integration. Proceedings of the VLDB Endowement 5, 6 (Feb. 2012), 550–561. DOI:
http://dx.doi.org/10.14778/2168651.2168656[210] Pengfei Zhou, Yuanqing Zheng, and Mo Li. 2014. How Long to Wait? Predicting Bus Arrival Time With Mobile
Phone Based Participatory Sensing. IEEE Transactions on Mobile Computing 13, 6 (June 2014), 1228–1241. DOI:
http://dx.doi.org/10.1109/TMC.2013.136[211] Tongqing Zhou, Zhiping Cai, Kui Wu, Yueyue Chen, and Ming Xu. 2017. FIDC: A framework for improving
data credibility in mobile crowdsensing. Computer Networks 120 (2017), 157 – 169. DOI:http://dx.doi.org/https://doi.org/10.1016/j.comnet.2017.04.015[212] MH Zwietering, Il Jongenburger, FM Rombouts, and K Van’t Riet. 1990. Modeling of the bacterial growth curve.
Applied and environmental microbiology 56, 6 (1990), 1875–1881.

Received June 2017; revised August 2017; accepted XXX

ACM Transactions on Sensor Networks, Vol. 1, No. 1, Article 1. Publication date: August 2017.