UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

CrowdSignals: A Call to Crowdfund theCommunity’s Largest Mobile Dataset
Evan Welbourne

Abstract

CrowdSignals Campaign

Researchers from diverse backgrounds critically dependon mobile datasets. From training and testing activity
recognition models, to verifying hypotheses in social
science, mobile data is indispensable. Unfortunately,
mobile data collection requires significant time andbudget for infrastructure as well as subject recruiting,screening, training, legal agreements, equipment, andcompensation. We estimate up to 70% of the resourcesin a study may be spent on data collection. Moreover,this massive investment can combine with institutional,legal, and political issues to create a disincentive tosharing with the community. In this paper, we propose
andjustify acrowdfunded
andcrowdsourced
methodology for longitudinal mobile data collection thatcuts researcher costs by orders of magnitude, removes
barriers to data sharing, and boosts data value for all
stakeholders. We also present CrowdSignals, a first
instantiation which will generate the largest labeledmobile dataset available to the community.

San Francisco, CA 94117 USA
evan@crowdsignalscampaign.com
Emmanuel Munguia Tapia
CrowdSignals Campaign
San Francisco, CA 94117 USA
emmanuel@crowdsignalscampaign.com

Figure 1: Diverse Sponsors of
the CrowdSignals Campaign
build funding and surveys inexchange for rich mobile datafrom 100s of smartphones.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and thatcopies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, topost on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Request permissions from
Permissions@acm.org.
UbiComp '14, September 13 - 17 2014, Seattle, WA, USA
Copyright 2014 ACM 978-1-4503-3047-3/14/09…$15.00.
http://dx.doi.org/10.1145/2638728.2641309

873

Introduction
With a global penetration rate over 96% [5], a majorityof people own mobile phones. In developed urban areas
where the rate is 100%, everyone owns a smartphone
and other devices - if not, they will before 2020 [8].
These devices capture our shared thoughts and feelings
along with rich streams of sensor, social, and system

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

Contributions
Low-Cost:
crowdfunding
amortizes expenses over a bigcommunity, cutting costs byorders of magnitude.
Sharing: crowdfunding meansthat all interested parties areengaged from the start, sodatasharing
isacorecomponent of the campaign
from its inception.
Scale: the crowd generates alarger pool of funds than theindividual, enabling larger scaledata collection.
Customization:sponsors
specify surveys with which
subjects
label
thedata,customizing
andaddingsignificantly more value.
Ethical: by following bestpractices for informed consent,useragreements,privacy,security, and data sharing wematch the standards held by
most institutional IRBs.
In addition, we present andexplain
theCrowdSignals
Campaign
asaconcrete
example of this approach.

data. Data on our activities, environment, mobility
patterns, preferences, interactions, social life, andsoftware usage are all latent in this mobile data. Assuch, mobile data can be used to recognize, track, andreact to practically any phenomena in our proximity, be
it physical, psychological, or sociological.
Consequently, mobile data is integral to research and ina variety of fields. The most valuable data are collectedlongitudinally from many subjects in naturalistic
settings because it produces more robust results. The
value of a data set increases with each logged datatype, and it increases dramatically more when subjects
label their data with their current activity, feeling,opinion, place, social situation, or surroundings. Thelabels can later be used by machine learning algorithms
to find interesting correlations or to classify events

Campaign
MIT Reality

100

Mining [2]Students

Dartmouth

9000+

CenceMe [10]PeopleNokia LDCC [11]Mobile Data Collections
Several longitudinal mobile data collection campaigns
since 2000 are summarized in Table 1 with respect toscale, date, duration, and estimated cost. Eachaccelerated research in important ways and represents
a different combination of methodology, cost, andexperimental control. In this section we present andcontrast several key aspects of these campaigns.

874< 200students

MIT Social fMRI

< 200[1]students

UB PhoneLab[14]200
students

SMU LiveLabs

30K

[15]

peopleSamsung CS

Unfortunately, data is scarce and mobile data collection
is expensive, time consuming, and difficult. In thispaper, we propose and justify a crowdfunded andcrowdsourced methodology that cuts costs by orders ofmagnitude, facilitates sharing, and increases value forall stakeholders. We also introduce the CrowdSignals
campaign (see Figure 1), the first instantiation of thisapproach which will create the largest labeled mobile
dataset available to the community.

Subjects[17]63peopleLengthCost(mo.)(US $)

2004

9$110K+

2008

N/A

$75K+

2009

24

$1.2M+

2011

9$150K+

2012

12-48

$1.6M+

2013

12+

$20M+

2013

3

$15K

Date

Table 1. Mobile data collection campaigns since 2000.

Methodologies
Three key methodologies are: local administration (LA),app store-based (AS), and crowdsourced (CS). Most
common is LA in which researchers painstakingly
recruit, on-board, and manage subjects in person,compensating them with smartphones and a mobile
data plan. The MIT, Nokia, UB, and SMU campaigns areexamples of this approach. CenceMe used AS in which
an app is distributed to 1000s of users via an app store.
In the CS approach [17], remote subjects are rapidly
recruited, managed, and paid with a crowdsourcing
service, but they install a data collection app on theirown phone. Our methodology builds on this withcrowdfunding.

WORKSHOP: HASCA

Collected DataLocation and Radios:Bluetooth, GPS, GSM, WLAN
(30-60 sec, every 10 min)Sensors: Accelerometer,
Ambient Temperature, Gravity,
Gyroscope, Light, Magnetic
Field, Microphone, Orientation,
Pressure, Proximity, Humidity,
Rotation (10 sec, every 5 min)Social: Calls, Contacts, SMS;
sensitive content is hashed(every 5 hours)
System and Networking:Battery, Connections, Network
Traffic (every 5 min)User Interaction: App
Installs, App Launch/Close,
Browser Logs, Phone Settings,
Configuration (event-triggered)
Subject Feedback:Lockscreen Questions(every screen unlock)
ESM Questionnaires(1-2 times per day)
Entry/Exit Surveys(beginning and end of study)
Figure 2: List of collected datatypes along with the sampling
window size (if applicable) andsampling intervals.

Local
Admin

AppStore

Crowdsource

SoftwareHardware

Admin

Comp

$2000 x$300-$600$2000 x$50-$100 x

4 months xx# months x# months x# developers # subjects # assistants# subjects$2000 x8 months x

N/A

$N/A

N/A

$500 x$20-$50 x

N/A# months x# months x# assistants# subjects# developers$2000 x4 months x# developersTable 2. Estimated US$ cost for methodologies, including:
Software Development (grad students), Hardware (phones),
Compensation (cash, data plan), and Admin (grad students).

Cost Breakdown
Campaigns incur costs in several ways (see Table 2).
First is developer salary for the app and server. This is
more for the high-fidelity AS app. LA also incurs thecost of subject smartphones. A third cost is admin
staff; AS avoids this and CS minimizes it. The last costis payment, which includes data plan for LA studies andis eliminated in AS. Estimates in Table 2 assume gradstudents are developers and admins. Since many data
analyses take 3-4 weeks, mobile data collection may
consume upwards of 70% of a research budget.
Experimental Control
Methodologies differ in terms of control. AS releases an
app into the wild to see what data comes back. In
contrast, LA emphasizes face-to-face recruiting andadmin with complex control (e.g., interventions). CS
studies such as CrowdSignals lay between: adminsinteract with remote subjects, but surveys and simple
protocol changes can be used to enhance control.

875

Crowdfunding
In crowdfunding, a group of sponsors pay to fund a
project in exchange for some reward. Here we discuss
the anatomy of a crowdfunding campaign for mobile
data collection using CrowdSignals as an example.
Sponsors
Sponsors come from any field that uses large mobile
datasets (e.g., geography, health, sociology, ubicomp).
For example, a professor may sponsor because sheneeds a large sensor data set from diverse subjects,
labeled with activities like driving a car or riding a bus.
The sponsors split the total cost and because no single
institution bears the total expense, there are fewer
financial and political barriers to sharing the data.
Funding
Crowdfunding campaigns set a funding goal that willallow the organizers to execute the project. For mobile
data collection this should cover all costs, scaled to thesize and duration of the study. The early CrowdSignals
goal is $50K for 250 subjects over 4 months, with $30K
devoted to compensation and for development andadmin. Based on early success, CrowdSignals may set
stretch goals that significantly increase the number ofsubjects and the duration of the study.
Rewards
Crowdfunding offers rewards based on sponsorship. We
reward sponsors with data. However, richness of data
and allowable usages vary by sponsorship level.
Tentatively, CrowdSignals sponsors receive anonymouslogs (e.g., apps, calls, SMS), mid-level sponsors also
receive sensor data and surveys, higher levels receive
all data including more sensitive logs (e.g., location)and may specify custom surveys (see next Section).

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

System Architecture
We use an extended version of the architecture
presented by Welbourne et al. [17]. This includes a
crowdsourcing service, an Android data collection app,and a cloud server. The section describes theCrowdSignals architecture as an example.
Crowdsourcing
Recruiting, management, and payment are conductedon crowdsourcing platforms Elance [3] and ODesk [12].
We will also recruit external subjects (e.g., throughonline or physical ads) into the platform where they canbe more easily managed. Subjects will interact with ateam of trained study administrators during the study.
(a)Data Collection App
The Android app has a background service, configurable
surveys, and basic controls. The service launches onboot, collecting the data in Figure 2. All sensitive data
(e.g., SMS content) is hashed to protect subjects. The
service securely uploads encrypted, compressed datawhen the subject’s phone connects to WLAN.

(b)
Figure2:(a)Lockscreen

feedback for frequent, lightweightlabels. (b) Open Data Kit Survey
for configurable, more in-depth
ESM questionnaires.

Sponsors may solicit ground truth from subjects with asurvey framework that offers lockscreen surveys andexperience sampling method (ESM) [7] surveys (seeFigure 2). The lockscreen solicits frequent, lightweight
feedback with a multiple choice question (e.g., “How doyou feel?”) every time subjects unlock their phone about 19 times per day on average [16]. ESM
questionnaires use the Open Data Kit (ODK) [13] forconfigurable surveys and participatory sensing (e.g.,audio, video, barcodes). Top sponsors may specify
custom ESM questionnaires using ODK’s JSON survey
specifications; lockscreen surveys are specified as acombination of text and image files.

876

Basic controls allow subjects to start and stop all datacollection. Subjects may also review the amount of dataand survey responses they have uploaded. Finally,
subjects may explicitly upload their data whenconnected to WLAN using an “Upload Now” button.
Cloud Server
The cloud server includes an HTTP server that accepts
secure uploads from the clients, and batch scripts that
post-process received data. The scripts ensure thefidelity of the data and track how much each subject
has uploaded. This allows us to pro-actively contact
subjects when problems arise and to pro-rate paymentbased on how much data each subject has contributed.

Study Protocol
Figure 3 shows the end-to-end flow of a crowdfunded,
crowdsourced data collection campaign which we
describe in more detail below.
Funding and Requirements Gathering
Funding is organized using a crowdfunding platform[6,9]. During the funding period, input from sponsors
regarding surveys and target subject demographics
must be collected and reviewed as well.
Recruiting and On-Boarding
Study administrators begin recruiting and on-boarding
by inviting candidates to apply on the crowdsourcing
platform – this may include candidates that are notalready on the platform. Admins then screen applicants
and walk them through informed consent, after which
subjects receive instructions on how to install and use
the app. Finally, subjects complete entry surveys
collecting information on demographics, personality,preferences, or other information useful to sponsors.

WORKSHOP: HASCA

Data Collection and Management
Once subjects are uploading, admins start monitoringcollected data with server scripts and solving any
problems that arise (e.g., bugs, missing subject data).
During this phase subjects may opt-out or may leave
the study because they can no longer participate fortechnical reasons (e.g., switched to iPhone). Subjects
may also forfeit their pay to delete all data at any time.
Otherwise subjects are paid at the end of the study
according to the amount of data they have contributed.

[2] Eagle, N. and Pentland, A. Reality mining: sensing
complex social systems. PUC, Vol. 10 Issue 4, pp.
255-268 (2006)

Post-Processing and Sharing
Finally, data is post-processed and shared to sponsors.
Watermarks are applied to track any leaks. Sponsors
sign a data sharing agreement on ethical use and bestpractices for storage and access. This includes clausessuch as: no contacting subjects directly or reverse
engineering anonymized data, and no onward transfer.
Finally, the dataset is posted online and sponsors arenotified so that they can download their copies.

[8] Khan, S. and Marzec, E., Tech Trends 2014:
Wearables. Deloitte University Press,
http://www.dupress.com/articles/2014-techtrends-wearables[4] Funf | Open Sensing Framework http://funf.org/[5] ICT. The World in 2014, ICT Facts and Figures. ICT
http://www.itu.int/en/ITU-D/Statistics/Pages/facts/[6] IndieGogo. https://www.indiegogo.com/[7] Intille, S. S., et al., Eliciting user preferences using
image-based experience sampling and reflection,
ACM CHI '02, pp. 738-739 (2002).

[9] Kickstarter. https://www.kickstarter.com/[10] Miluzzo, E., et al. Sensing meets mobile social
networks: the design, implementation andevaluation of the CenceMe application. ACM SenSys(2008).
[11] Nokia Lausanne Data Collection Campaign,
https://research.nokia.com/page/11367

Conclusion
Mobile data is central to research, but it is expensive
and there are many barriers to use. We presented acrowdfunded, crowdsourced methodology that cuts costby orders of magnitude and reduces barriers to sharing.
We also introduced an extensible survey framework forsponsor customization. As an example, we presented
CrowdSignals, which if funded, will produce the largest
labeled, longitudinal mobile dataset. We invite
participation through both critique and sponsorship.

References
Figure 3: Study protocol[3] Elance. http://www.elance.com[1] Aharony, N. et al. Social fMRI: Investigating andshaping social mechanisms in the real world.
Pervasive and Mobile Computing 7 643-659 (2011)877[12] ODesk. http://www.odesk.com[13] Open Data Kit. http://www.opendatakit.org[14] Phone Lab: A Programmable Smartphone Testbed
http://www.phone-lab.org/[15] LiveLabs, http://centres.smu.edu.sg/livelabs/[16] Vaish, R., et al. Twitch Crowdsourcing: Crowd
Contributions in Short Bursts of Time. CHI’14(2014)[17] Welbourne, E., et al. Crowdsourced Mobile Data
Collection: Lessons Learned from a New Study
Methodology. HotMobile’14 (2014)