A Framework and Tool for Collaborative Extraction of Reliable Information

A Framework and Tool for Collaborative
Extraction of Reliable Information
Graham Neubig1, Shinsuke Mori2,Masahiro Mizukami1
1

Nara Institute of Science and Technology
2
Kyoto University1

A Framework and Tool for Collaborative Extraction of Reliable Information

Background2

A Framework and Tool for Collaborative Extraction of Reliable Information

What is Information Extraction?
●Find useful information from large amounts of noise
Info about
Hobbies

Information
Source(e.g. Internet)

Word-of-mouth
Information

Info. about
Events3

A Framework and Tool for Collaborative Extraction of Reliable Information

Information Extraction in Times of Crisis●Noise is particularly prevalent in times of crisis
Provision of ANPI_NLP
Safety Info. Project[Neubig+ 11]

InformationSource(e.g. Internet)#99japanRequests for
Safety Info Project[Aida+ 13]

Evacuation Shelters/
Rescue Supplies
4

A Framework and Tool for Collaborative Extraction of Reliable Information

Necessities for Crisis-time
Information Extraction●Speed●●Necessary to provide information ASAP to those in need

Absolute Reliability
Provision of mistaken information could be deadly● In general, info will likely require confirmation beforeconsumption●●

Difficult to Predict Needs●●●

Wildfire → Wind, Earthquake → Diapers, Radiation

Many volunteers! [Starbird+10, Neubig+11]
Challenge: How do we let volunteers work efficiently as
5possible to provide reliable information quickly?

A Framework and Tool for Collaborative Extraction of Reliable Information

This Work●We propose a method for efficient extraction of reliable
information:
Use machine learning (relevance feedback) to decide
which examples to show to annotators● Web-based collaborative interface to allow multiple
annotators to work on a single task●●Evaluation on data from Twitter●Toolkit freely available open source
webigator:
http://www.phontron.com/webigator
6

A Framework and Tool for Collaborative Extraction of Reliable Information

Information Extraction Framework

7

A Framework and Tool for Collaborative Extraction of Reliable Information

Information Extraction Task
They are distributing water
They really need to open moreevacuation areas in Sendai!
at Ishinomaki High School today.
I was able to fill up my car at
the gas station at XXX.

Got to the evacuation center,but I'm almost out of battery!

Information filtering: Remove documents with noactionable information● Information extraction: Identify which terms fill slots(e.g. status, location)●●For Twitter, documents are small but numerous, sofiltering is a challenge
8A Framework and Tool for Collaborative Extraction of Reliable Information

Information Filtering as Classification●

Binary classification of “useful or not?”●●Define features, use machine learning to learn weights

Notable for large proportion of negative examples

Pos.

Normal Classification
ooox
ooxoxx xx

Filteringooxx xxxxxxx x xxxx

Neg.

xxxxx
9

A Framework and Tool for Collaborative Extraction of Reliable Information

Constructing a Classifier
Requires Lots of Data
Little Data

Lots of Dataooooooooxxxxx xooooxxxxx x
Bold = Lots of Data

10

A Framework and Tool for Collaborative Extraction of Reliable Information

Active Learning●

Way to create a good classifier efficiently●

Choose examples to annotate based on predictions
Positiveooooooxxxx xx

Negative11

A Framework and Tool for Collaborative Extraction of Reliable Information

Active Learning●Way to create a good classifier efficiently●

Choose examples to annotate based on predictions
ooooooxxxxx x

12

A Framework and Tool for Collaborative Extraction of Reliable Information

Active Learning●

Way to create a good classifier efficiently●

Choose examples to annotate based on predictions
ooooooxxxxx x

13

A Framework and Tool for Collaborative Extraction of Reliable Information

Active Learning●

Way to create a good classifier efficiently●

Choose examples to annotate based on predictions
ooooooxxxxx x

14A Framework and Tool for Collaborative Extraction of Reliable Information

Active Learning●Way to create a good classifier efficiently●Choose examples to annotate based on predictions
ooooooxxxxx x

15

A Framework and Tool for Collaborative Extraction of Reliable Information

Problems with Unbalanced Data
●In information extraction, almost everything is negativeoxoxx x xxxxx x xxxxxxxxx
16

A Framework and Tool for Collaborative Extraction of Reliable Information

Problems with Unbalanced Data
●In information extraction, almost everything is negativeoxoxx x xxxxx x xxxxxxxxx
17

A Framework and Tool for Collaborative Extraction of Reliable Information

Problems with Unbalanced Data●

In information extraction, almost everything is negativeoxoxx x xxxxx x xxxxxxxxx
18

A Framework and Tool for Collaborative Extraction of Reliable Information

Problems with Unbalanced Data●

In information extraction, almost everything is negativeoxoxx x xxxxx x xxxxxxxxx
19

A Framework and Tool for Collaborative Extraction of Reliable Information

Problems with Unbalanced Data
●In information extraction, almost everything is negativeoxoxx x xxxxx x xxxxxxxxx
20

A Framework and Tool for Collaborative Extraction of Reliable Information

Our Simple Fix●

Small change to example selection criterion
Standard: Select low confidence examples
Proposed: Select examples with high probability ofbeing positive●

Effective when final human check is necessary●

Labeling a positive example =finding a highly reliable piece of information21

A Framework and Tool for Collaborative Extraction of Reliable Information

Our Simple Fix●Finds many positive examples quicklyoxoxx xxxxxxx x xxxxxxxx●

Using these positive examples, learn characteristics
that help pick out more

22

A Framework and Tool for Collaborative Extraction of Reliable Information

Scaling Up23

A Framework and Tool for Collaborative Extraction of Reliable Information

Too Much Data!
●●e.g. Twitter after the Great East Japan Earthquake =peak of 1237 tweets/second
Problems with:
Viewing even the high scoring tweets with one person● Rescoring every tweet after each round of learning●24

A Framework and Tool for Collaborative Extraction of Reliable Information

Collaborative Web-based Interface●Allow multiple annotators to cooperate
WorkersWeb
UI

Display
Text

Web
UI

Text
Retrieval/Scoring
ServerInternet

Web
UI
SubmitLabel
Information
List
25

A Framework and Tool for Collaborative Extraction of Reliable Information

Web Interface

26

A Framework and Tool for Collaborative Extraction of Reliable Information

Efficiency Improvements
1) Simple keyword search filter
Type
Evacuation/Supplies
Safety Info Request
Safety Info Provision

Keywords
evacuation area, water supplies, food supplies
contact, cannot, waiting
contact, safe2) Rescoring policy
Maintain a sorted list of highly scored examples● When retrieving next example:●Choose the example highest in the cache, rescore● After rescoring, still better than second best, return● Otherwise, return to beginning●27

A Framework and Tool for Collaborative Extraction of Reliable Information

Experiments

28

A Framework and Tool for Collaborative Extraction of Reliable Information

Evaluation●

Compared Methods:
Keyword search● Proposed learning-based method●●Target:179M tweets week after Great East Japan Earthquake● Three types of info: evacuation/rescue supplies, safety
info request, safety info provision●●Evaluation measure:Amount of reliable information extracted in 30 mins.
● Use shared Google Doc as repository for information●29

A Framework and Tool for Collaborative Extraction of Reliable Information

Effect of Learning●

Experiments with one
annotator for three tasks

Information Extracted Filtering Accuracy
1Rescue Supplies/Evacuation
Areas80706050403020●●Observable increase inamount of information
extracted and accuracy

10Some tasks easier thanothers

40005101520250510152025051015202580

51015202530

51015202530

510152025 3030

Safety Info. Request

7060
50
30
2010080

0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
030 01
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
030 01
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
030 0Safety Info. Provision

7060
50
4030
20
100w/ Learning

w/o Learning

A Framework and Tool for Collaborative Extraction of Reliable Information

Effect of Collaboration
Experiments with 1-3 users using same interface

Pieces of Information●

1401201008060402001 User
2 Users
3 Users051015202530

Time (Minutes)●As expected, increasing users = increasing efficiency 31

A Framework and Tool for Collaborative Extraction of Reliable Information

Conclusion●

A method for information filtering that focuses onpositive examples●

More effective than simple keyword search●

Remaining challenges:
Identification/clustering of duplicates● Application to identification of slots as well
●webigator:
http://www.phontron.com/webigator32