KYOTO UNIVERSITY

Crowdsourcing for Big Data Analytics

Hisashi Kashima (Kyoto University)
Satoshi Oyama (Hokkaido University)
Yukino Baba (Kyoto University)
DEPARTMENT OF INTELLIGENCE SCIENCEAND TECHNOLOGY

1KYOTO UNIVERSITYCrowdsourcing for data analytics
Part I: Crowdsourcing for data analytics (Hisashi Kashima)
Part II: Crowdsourcing for datafication (Satoshi Oyama)
Part III: Crowdsourcing for analysis (Yukino Baba)
Part IV: Future direction (Yukino Baba)

Tutorial slides are found at

http://goo.gl/Amif93
2

KYOTO UNIVERSITY

Part I: Crowdsourcing for data analytics

3

KYOTO UNIVERSITY

Big challenge in big data analytics:
Manpower bottleneck Automatic data analysis techniques (e.g. machine learning) are often
considered as main components of data analytics
 Data analysis is heavily labor intensive– Manual processing dominates a large portion of data analysis process– 1990s-2000s: introduction of data mining techniques and data analysis
process standards (e.g., CRISP-DM)
data analysis processbusiness
understanding

4

datacollection/
computerization

data cleansing/curationmodeling/visualization

evaluation/interpretation

KYOTO UNIVERSITYBig shortage of data scientists:
Implies labor intensity of data analysis
 “By 2015, 4.4 million IT jobs globally will be created to supportbig data”, but “only one-third of the IT jobs will be filled”- Peter Sondergaard (Senior VP at Gartner)
 “Data Scientist: The Sexiest Job of the 21st Century”- Thomas H. Davenport and D.J. Patil, Harvard Business Review
 These statements imply the labor intensity of data analysis

5KYOTO UNIVERSITY

Rise of crowdsourcing:
On-demand access to massive on-line labor Crowdsourcing: Outsourcing human-intelligence tasks to a large
group of unspecified people via Internet⇔ "outsourcing" offers tasks to specified contractors
 Advantage of crowdsourcing:–Volume: Access to a massive amount of human intellects–Quality: "wisdom of crowds"

Howe, J. Crowdsourcing. 2004

6

KYOTO UNIVERSITY

Two types of crowdsourcing:
Explicit crowdsourcing and implicit crowdsourcing
1. Explicit crowdsourcing: directly asks for contributions–Marketplace (e.g., Amazon Mechanical Turk)–Volunteer (e.g., citizen science such as Galaxy Zoo)
2. Implicit crowdsourcing:embeds tasks into other forms in order to motivate
worker participations–Unavoidable tasks (e.g., reCAPTCHA)–Games with purposes (e.g., ESP game)

7

KYOTO UNIVERSITYCrowdsourcing marketplace:
Online labor markets for explicit crowdsourcing
 Crowdsourcing marketplaces:
Web services for connecting requesters with workers– Advantages for requesters: Access to on-demand workforce– Advantages for workers: New work style unbound by time or place Emergence of online crowd-labor marketplaces– Mechanical Turk, oDesk, Clickworker, …
crowd workers
crowdsourcingmarketplacerequester
postingtasks/ payments

8task
execution

KYOTO UNIVERSITY

Amazon Mechanical Turk:World-biggest microtask crowdsourcing marketplace
: Since established in 2005, a de-facto standardof crowdsourcing marketplace in computer science– 0.5 million workers from 190 countries (in 2011)– can access human intelligence on-demand Automatic creation of tasks through API
Extraction of purchase
information form a receipt

96-cent reward

KYOTO UNIVERSITY

Task types in crowdsourcing marketplaces:
Simple, structured tasks to complex, unstructured tasks
 Very easy tasks:
Image labeling, entity resolution, format checking… Microtasks:
Output format: Yes/Nomore complex

 Easy tasks: Restaurant reviews, web service tests, …

Popular incomputerscience

Output format: Multiple-choice or short sentences

 Tasks requiring some expertise: Logo design, report writing, …
Output format: Images or sentences

 Complex tasks requiring high level of expertise:
Web/software developments, professional work, …
Output format: System, codes, documents
10KYOTO UNIVERSITY

reCAPTCHA:
A ”Turing test” distinguishing humans and machines
 System shows two text images to a user and asks to readboth

 Character recognition is difficult for computers but easy forhumans, only human can pass the test
I CAN’T READ
IT …

11I CAN!

KYOTO UNIVERSITY

Implicit crowdsourcing of reCAPTCHA:
Use system logs for document digitization
 The system knows the correct answer of only one of the twowords

 The other is a word that OCR systems fail to recognize– Users are forced to help document digitization

CORRECTANSWER:UNKNOWN

12CORRECTANSWER:“Donovan”“Pleasure” and“Donovan”

KYOTO UNIVERSITY

Trend in academic research:
Exponential growth of crowdsourcing research 2005: Amazon Mechanical Turk was launched 2006: “crowdsourcing” and ”human computation” were advocated 2013: 1st conference on human computation and crowdsourcing(HCOMP)#papers related to crowdsourcing
14000
12000
10000
8000

exponentialgrowth6000
4000
2000
0

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Counted using Google Scholar

13

KYOTO UNIVERSITYCrowdsourcing in computer science:
Popular use in NLP, CV, HCI, DB, IR, ML/DM, …
 NLP: text understanding, annotation, language translation

 CV: image understanding, annotation, detection
 DB/IR: data generation/integration, search, evaluation
 ML/DM: data collection/annotation
Microtasks are automaticallyissued for SQL execution

14

Franklin et al. CrowdDB: Answering Queries with Crowdsourcing.
VLDB 2011.

KYOTO UNIVERSITYExample of crowdsourcing in DB research:
Data generation and comparison
 CrowdDB uses crowdsourcing for

1. Data generation
2. Data comparison:– Identity test of two data instances– Sorting data instances

15

Franklin et al. CrowdDB: Answering Queries with Crowdsourcing.
VLDB 2011.

KYOTO UNIVERSITYCrowdsourcing for data analytics:
Crowdsourced execution of data analysis process Use the power of crowds with various knowledge/skills to execute thelabor-intensive data analysis processneeds businessknowledge

needs some domain
knowledge/skills

needs to be close
to data sourcesbusiness
understanding

datacollection/
computerization

data cleansing/curationneeds data
analysis skills
modeling/visualizationneeds domain
knowledge
evaluation/interpretation

process executionby crowds

16

KYOTO UNIVERSITYCrowdsourcing usages in data analysis:
Datafication and analysis
 Data analysis process is divided into two parts:
1. Datafication: data collection, data cleansing, annotation, …

2. Analysis: data modeling, visualization, evaluation, …
datafication
datacollection/
computerization

17

data cleansing/curationanalysismodeling/visualization

evaluation/interpretation

KYOTO UNIVERSITYCrowdsourcing for datafication:
Microtasks for data collection, annotation, and cleansing
 Microtasks for data annotation and cleaning–Suited for simple human-intelligence tasks:• Image labeling, speech recognition, …
–Small payment for a small piece of work
• Large-scale datasets can be processed with reasonablecosts

18

KYOTO UNIVERSITYCrowdsourcing for analysis:
Competitions, peer reviewing, …
 Competitions for data modeling–Suited for creative/complex tasks:• Design, software development, analysis, …
–Winner-takes-all payment• A few high-quality results can be obtained Peer reviewing for assessing results

19

KYOTO UNIVERSITY

Technical issues:
Incentive design and quality control
 Incentive design–Crowdsourcing workers are not necessarily well-motivated–Gamification and reward optimization
 Quality control–Crowd-workers have different skills and motivations,and sometimes they are malicious–Quality of crowdsourcing results is uneven–Statistical quality control methods

20

KYOTO UNIVERSITY

Following parts…:
Crowdsourced datafication and analysis
Part III:Part II:
Crowdsourcing for datafication Crowdsourcing for analysis

datafication
datacollection/
computerization

21

data cleansing/curationanalysismodeling/visualization

evaluation/interpretation

KYOTO UNIVERSITY