Assessing the Quality of Wikipedia Editors
through Crowdsourcing
Yu Suzuki and Satoshi Nakamura
Nara Institute of Science and Technology
8916-5 Takayama, Ikoma, Nara 6300192, Japan{ysuzuki, s-nakamura}@is.naist.jp
ABSTRACTIn this paper, we propose a Wikipedia editor-quality assessment method. Here, we define quality of editors as an
approval rate for texts contributed by Wikipedia editors.
When an editor adds a text and many users approve of thetext, the editor is assessed as high quality.
Methods based on peer review is the major approach[10],[3],[4] used to detect vandals. In these methods, the quality of an editor is calculated using the edit histories of articles. We assume that low-quality text will be quickly deleted
by other editors, whereas high-quality text will remain unchanged for a long time.
Many peer-review methods, however, do not consider themeaning of the text. For example, if the sentence “Wikipedia
has good quality articles.” is changed to “Wikipedia does nothave good quality articles.”, the meaning is completely changed,
but if the former sentence is changed to “Wikipedia has finequality articles.”, the meaning is not changed. In both cases,
several terms are added and deleted, and we cannot decide
whether the meaning is actually changed by only considering the quantity of terms changed. Proposed methods basedon peer review that rely on systems capturing the additionand deletion of terms are therefore limited.
Automatic detection of changes in sentence meaning ishard, but humans can easily detect these changes. We believe that crowdsourcing techniques can be used to detectchanges in sentence meanings that cannot be captured bycurrent natural language processing techniques. This approach should enable us to accurately capture the purposeof edits, and thus improve the accuracy of quality assessment.
In this paper, we therefore propose a method for improving the accuracy of quality assessment for Wikipedia editors.
The contributions of this paper are:In this paper, we propose a method for assessing the quality of Wikipedia editors. By eﬀectively determining whether
the text meaning persists over time, we can determine theactual contribution by editors. This is used in this paper
to detect vandal. However, the meaning of text does notalways change if a term in the text is added or removed.
Therefore, we cannot capture the changes of text meaning
automatically, so we cannot detect whether the meaning oftext survives or not. To solve this problem, we use crowdsourcing to manually detect changes of text meaning. In
our experiment, we confirmed that our proposed method
improves the accuracy of detecting vandals by about 5%.

Keywords
Wikipedia, quality, crowdsourcing, vandalism

1. INTRODUCTION
Wikipedia1 is one of the most successful encyclopedias on
the Internet. Unlike strictly controlled Web-based encyclopedias such as Nupedia2 or Citizendium3 , anyone can freely
edit any article and these edits are immediately reflectedin the final version of the articles. Many benign editors
submit good-quality articles, but many vandals attempt todamage articles. These vandals are identified by readers
and administrators, and then are tagged as “blocked users”.
As a result, these vandals are prohibited from editing any
Wikipedia article. As of September 15, 2015, there wereabout eleven thousand active editors, 4 including about two
thousand blocked editors. Therefore, the ability to assess thequality of Wikipedia editors has become very important[8].
1• We use crowdsourcing to manually detect changes oftext meaning.

https://www.wikipedia.org
http://nupedia.wikia.com/wiki/Category:Nupedia (revived pages)3
http://en.citizendium.org/wiki/
4
https://en.wikipedia.org/wiki/Wikipedia:Size_of_
Wikipedia
2• We calculate the quality of Wikipedia editors using thesurvival time of the text meaning.

2.

RELATED WORK

Much research has been done on implicit features regarding user decisions which a system can predict from a user’s
behavior. When a system uses these features, users do notneed to input an evaluation of items. Our proposed method
uses this approach. However, how can a user’s evaluations
be predicted from their behavior?
Adler et al. [1, 2, 3] and Wilkinson et al. [11] propose
a method for calculating quality values from edit histories.

Copyright is held by the International World Wide Web Conference Committee (IW3C2). IW3C2 reserves the right to provide a hyperlink to theauthor’s site if the Material is used in electronic media.
WWW’16 Companion, April 11–15, 2016, Montréal, Québec, Canada.
ACM 978-1-4503-4144-8/16/04.
http://dx.doi.org/10.1145/2872518.2891113 .

1001

Article a1
v1

v2

v3

v4

versions

ea

eb

vi

…

v5

vi+1
si,1

Wikipedia
Edit History

editor eavi

1. Extraction of Wikipedia edit history

2. Estimation of positive/negative ratingsfrom editor’s edits

editor eb

positive rating

si+1,1si,2

si+1,2si,3si+1,3eaeb
negative rating

sentence pairs

Positive ratings?

p1 ={si,1, si+1,1}

si,1

si+1,1

Information
is deleted

Negative ratinge?

p2 ={si,2, si+1,3}

si,2

si+1,3

Informationis addedp2 ={si,3, si+1,2}

si,3

si+1,2vi
Crowdsourcing

eb

Automatic
Classifier

Crowdsourcing

si,3 does notcorrespondto si+1,2

ed

Positive

3.Generation of editor’s reputation graphea

Negative

1.0

Figure 2: Estimating an editor’s peer review (detailsof Step 2)

ec

0.5

0.3

0.6

1. Extract the text diﬀerence between two versions fromthe edit history

4. Assessment of editor’s quality

Figure 1: Proposed method

2. Estimate each editor’s rating based on the text diﬀerences

This method is based on the survival ratios of texts. Hu etal. [6] also propose a method for calculating article quality using editor quality, which is similar to our proposed
method. This method focuses on unchanged content, andthey assume that that an editor considers a text to be goodtext if the editor does not change that text. However, thismethod does not consider the original editors. Therefore,
for an article which has only one version – i.e., the text of
the article has not been edited by other editors – we cannot
calculate text quality values using existing methods. In our
method, we do consider editors. Therefore, if the editor ofa new text edits other texts, and these edited texts are left
unchanged or deleted by other editors, we can calculate thequality of the new text.
In these research, edit distance is generally used to detectthe diﬀerences of two versions. However, if the positions ofsentences are changed, or if two sentences are merged into
one sentence, edit distance cannot capture actual diﬀerence.
WikiWho [5] is proposed to solve this issue. However, thismethod does not always detect reverted texts. Moreover, if
the terms in the sentence are dramatically changed but themeaning of the sentences are the same, WikiWho treat these
two sentences as diﬀerent sentences. In our method, we use
crowdsourcing to solve this problem.

3. Improve the identification of each editor’s rating byother editors using crowdsourcing
4. Predict the ratings of each editor’s edits provided byother editors
In this section, we explain these four steps. In particular,
we explain Step 2 in detail in Section 3.2.

3.1

Extraction of text differences

First, we input an edit history of Wikipedia articles to our
proposed system. In the edit history, all versions of articles
are recorded. Each version includes a snapshot of text, aname of an editor, and a timestamp from when the version
is created. Here, we define that an article a has a series ofversions V = {v1 , v2 , · · · , vN }, where vi is the i-th edited
version.
We then extract which part of the text is edited between
an old version and a new version. In our system, we extract
a set of sentence pairs P = {p1 , p2 , · · · , pM }, where pt is asentence pair which includes two sentences soldand snew
.
j
j
old
new
sj is a sentence in an old version, and sj
is a sentence
which may correspond to soldin a new version.
j
To generate sentence pairs P from the series of versions
A, we first make all combinations of the versions, and generate version pairs V = {(v1 , v2 ), (v1 , v3 ), · · · , (vN −1 , vN )}.
To reduce the calculation time, we generate pairs (vi , vj ),where 0 < i − j ≤ α. We then split the text of vi and
vj into sentences using a period as a delimiter. As a result, we get a list of sentence vi = {si,1 , si,2 , · · · , si,l(vi ) }and vj = {sj,1 , sj,2 , · · · , sj,l(vj ) }, where l(vi ) is the numberof sentences in vi . In this process, we remove Wiki-stylesymbols like “[” and “{” from the sentences. Moreover, weremove sentences where the ratios of symbols and numbers
are more than 50% of the sentence, because these sentences
are typically parts of tables.
Next, we calculate which sentences in an old version correspond to sentences in a new version. We use a vector spacemodel to measure the similarity of sentences. We divide

3. PROPOSED METHOD
Our proposed method consists of the following steps (Figure. 1):
Step 1. Extract all versions of articles in a Wikipedia edit history file
Step 2. Estimate positive/negative ratings from an editor’s edits
Step 3. Generate a reputation graph for an editor
Step 4. Assess the quality of the editor
Step 2 is described in more detail below (Figure. 2):1002

sentences of vi and vj into terms using word segmentation
tools, such as POS taggers or morphological analysis tools.
We then represent the sentence si,k as a term vector t(si,k )as follows:t(si,k ) = [f (t1 , si,k ), f (t2 , si,k ), · · · f (tK , si,k )]vides the sentence pairs in Pm to crowdsourcing workers andthen aggregates the responses of the workers.
First, the system provides a sentence pair and the following two questions to the workers via a Web interface (Figure
3):
Q1: From the old sentence to the new sentence, how has
the content been modified? (Multiple-answer question)(1)

where ti is a separate term, and f (ti , si,k ) is a tf/idf valueof ti in si,k . When we calculate an idf value of ti , we use
an article as a document unit. Therefore, if a term occurs
multiple times in one article, we set the document frequencyof the term to 1. Using cosine similarity as sim(si,k , sj,m ) =si,k ·sj,m, we find the sentence sj,m which is the most sim|si,k ||sj,m |ilar to si,k in vj . If sim(si,k , sj,m ) = 1, si,k and sj,m are thesame, so we should add the sentence pair of si,k and sj,mto P . If sim(si,k , sj,m ) is not 1, but exceeds the thresholdβ, the sentence should be categorized as partially changed.
We then put the pair of sentences pt = {si,k , sj,m } into P .
If sim(si,k , sj,m ) is lower than β, si,k does not correspondto sj,m , so we do not add this sentence pair.

1. The new sentence has more information than the oldsentence.
2. The old sentence has more information than the new
sentence.
3. The meanings of the old and new sentences are slightly
diﬀerent.
4. The old sentence does not correspond to the new sentence.

3.2 Assignment of Six Types of Label

5. The old sentence does not make sense.

Next, we assign six labels – “EQUAL,”“ADD,”“DELETE”,“ADD+DELETE”, “NO CORRESPONDENCE”, and “NOT
new
MAKE SENSE” – to sentence pairs in pt = {sold} ∈ P.
j , sjold
new“EQUAL” means that sj and sj
have the same meaning.
If sold
and sneware written using diﬀerent terms and difjjferent grammatical structures yet have the same meaning,
the label should be “EQUAL.” “ADD” means that a new
sentence contains all of the information from an old sentence and some added information. “DELETE” means thata new sentence contains only part of the information froman old sentence. “ADD+DELETE” means that a new sentence contains part of the information from an old sentence
and adds some information. We assign this label if an old
sentence is partially changed, but the old and new sentences
have some of the same information. “NO CORRESPONDENCE” means that an old sentence and a new sentence
have completely diﬀerent meanings. “NOT MAKE SENSE”
means that either an old sentence or a new sentence does
not make sense.
We assign these six labels to the sentence pairs in P . As
we stated in the Introduction, this task is diﬃcult to process
automatically for all sentence pairs. However, it would be
expensive to process this task for all sentence pairs by crowdsourcing because there are many sentence pairs. To reducethe cost and increase the accuracy of assigning labels, wecategorize the sentence pairs into two groups: sentence pairs
which should be processed by crowdsourcing, and sentencepairs which should be processed by the huristic rules.
When we browse the sentence pairs, it is diﬃcult to labelpairs by the huristic rules if an editor both adds and deletes
terms to and from an old sentence to make a new sentence.
Labeling is easier if the edits between an old sentence anda new sentence are only additions or deletions of terms, but
not both. Therefore, we categorize a set of sentence pairs P
into two groups Pm and Pa , where Pm is a set of sentencepairs containing edits of both addition and deletion, and Pa
is a set of the other sentence pairs.

6. The new sentence does not make sense.
When some information is deleted and other informationis added, we expect that the workers will choose both (1)and (2). When both the old and new sentences do not makesense, the workers should choose both (5) and (6).
Q2: From the old sentence to the new sentence, how has
the readability been modified? (Single-answer question)
1. Improved. The editor has corrected some misspelledwords or grammatical errors in the old sentence.
2. Unchanged.
3. Worsened. The editor has created some misspelledwords or grammatical errors in the old sentence.
4. The old sentence does not correspond to the new sentence, or the old sentence or new sentence does notmake sense.
Q1 is about the modification of sentence meanings, andQ2 is about the modification of vocabulary and grammatical errors. We use these two questions because we have todeal in diﬀerent ways with two kinds of modification – themodification of content and that of readability. If we use
a question like, “How has the old sentence been changed?”,the workers may not distinguish these two types of modification. We only observe the diﬀerences in sentence meanings,
not the diﬀerences in readability. Therefore, although we
ask Q2, the question about readability, the Q2 responses
are ignored.
We set the condition that at least two workers must assign
labels for each sentence pair. If more than half of the workers select the same options for a sentence pair, we assign
labels to the sentence pair using the rules described at Table 1. However, if workers select diﬀerent options from eachother, we add workers. If more than 10 workers are assigned
to one sentence pair, and no option is selected by more thanhalf of the workers, we assign the label “NO CORRESPONDENCE” to the sentence pair.

3.2.1 Labeling of Edits through Crowdsourcing
The goal of this task is to assign one of the six labels
to each sentence pair in Pm . To accomplish this, we haveconstructed a web-based system for crowdsourcing that pro-1003

Sentences for compare

Diﬀerence of sentence

Q1: Are information added or deleted? (Multiple selection)
Please select “add” and “delete” if information is both addedand deleted
add
deleteunchanged (including correction of grammatical errors)
not corresponded
sentence #1 does not make sense
sentence #2 does not make sense
Q2: How the readability changed? (Single selection)
Please select “improve” or “worsened” if misspell or grammatical errors are corrected.
improved
worsened
unchanged
sentence #1 or #2 does not make sense, or #1 and #2 do not corresponded with each otherFigure 3: A system interface for crowdsourcing workers

3.2.2 Automatic Labeling of Edits

where rp (p(si , sj )) = 1 means that if si is changed to sj ,part of the information in si is deleted. n(si ) and n(sj )are the amounts of information in si and sj , respectively.
Therefore, if a sentence pair is assigned a label of “DELETE”
or “ADD+DELETE” by the manual or automatic labelingdescribed in section 3.2, rp (p(si , sj )) is set to 1. Otherwise,rp (p(si , sj )) is set to 0.
Next, we define r(ea → eb , vi ), a rating from ea to eb atversion vi , as follows:{
1 if ea deletes eb ’s information at vi
r(ea → eb , vi ) =0 else(3)

The goal of this task is to assign labels to sentence pairs
in Pa . In Pa , there are three types of sentence pair: 1) all
terms in an old sentence are included in a new sentence andterms are added in the new sentence, 2) all terms in a new
sentence are included in an old sentence and some terms
from the old sentence are deleted in the new sentence, and
3) an old sentence and a new sentence are the same. We
automatically assign the “ADD” label to sentence pairs oftype 1), the “DELETE” label to those of type 2), and the“EQUAL” label to those of type 3).

3.3 Editor’s Reputation
From the sentence pairs with labels assigned, we set the
ratings of editors. We assume that editor ea ∈ E gives
positive ratings to eb ∈ E if a text of ea is not deleted by eb ,and ea gives negative ratings to eb if a text of ea is deleted
by eb . Using this assumption, we assign editor’s ratings byaggregating the labels of sentence pairs.
First, we set rp (p(si , sj )) as follows:{
1 if n(si ) ⊃ n(sj )rp (p(si , sj )) =(2)
0 elseSelection of Q1
1. and 2.
1.
2.
3.
4.
5. and 6.
5.
6.

where r(ea → eb , vi ) = 1 means that ea deletes eb ’s information in version vi more than once. To calculate this equation,
we collect sentence pairs where the old sentence is edited byea and the new sentence is edited by eb .

3.4

Assessment of an Editor’s Quality

Finally, we calculate a quality score q(ea ) for editor ea asfollows:
(∑
)
∑
vi ∈V
ek ∈E r(ek → ea , vi )q(ea ) =1 −(4)
|E(ea )|
where |E(ea )| is the number of all sentence pairs with ea
as an editor of the edited sentence. If ea adds a version vi
and the added information is deleted by other editors, thevalue of r(ek → ea , vi ) increases; thus, the value of q(ea )decreases.

label
ADD+DELETE
ADD
DELETEEQUAL
NO CORRESPONDENCE
NOT MAKE SENSENOT MAKE SENSENOT MAKE SENSE

4.

EXPERIMENTAL EVALUATION

We experimentally evaluated the accuracy of assessing editor quality through the method presented in this paper. In
our experiment, we measured how our method can extract
low quality editors, and calculate recall and precision ratio.

Table 1: Rules for assessing labels1004

We used a baseline method as our proposed method without using crowdsourcing. In the baseline method, when we
processed Step 2 described in section 3.2.1, we used only automatically labeled sentence pairs to Pm , and we did not use
the labels by crowdsourcing. We assigned a label “DELETE”
to sentence pairs if more than one term was deleted. Otherwise, we assigned a label “NOT DELETE” to sentence pairs.
In section 3.3, we only considered sentence pairs labeled“DELETE” or “ADD+DELETE”; we did not use sentencepairs labeled “NOT DELETE.”

0.1

baseline
proposed

Precision

0.08
0.06
0.04
0.02

4.1 Experimental Setup
We used the edit history data of Japanese Wikipedia asof May 12, 2015. In this data, there were 1,523,561 articles,
3,016,675 editors, and 45,207,644 versions. If we assessed
quality values for all editors, though, the crowdsourcing cost
would be too high. Therefore, we selected target articles
from four categories: “Sports,”“Islam,”“Bird,” and “Hawaii”.
The articles in these categories are maintained by active usergroups, so we expected the articles to be well maintained.
The numbers of articles and editors are shown in Table 2.
To calculate the recall and precision ratio, we need toprepare a correct answer set. As far as we know, there isno good-quality editor list for Wikipedia. If we were to
manually create a list of good-quality editors, it would bevery hard for us to create appropriate, unbiased editor sets.
Therefore, we instead used the blocked user list provided by
Wikipedia5 to identify low-quality users. As shown in Table
2, the number of blocked editors for the target articles weused was 1, 601.
In the step described in section 3.1, we set a threshold
β = 0.7, because this value proved to be the most accurate
value when we did our preliminary experiment.

0
0

0.2

0.4
0.6
Recall0.8

1Figure 4: Recall-Precision curveand no label was selected by more than 50% of the workers,
the sentence pair was labeled “UNKNOWN.”
To collect the evaluation results from workers using crowdsourcing, we constructed a Web-based system using Ruby on
Rails 4.2 and Oracle Database Server 12c. Using this system, we showed five sentence pairs to each of the workers
and then the workers input the evaluation results throughthe system. When a worker had assessed 100 sentence pairs,
we paid 50 JPY (about 0.5 USD) to the worker; however, if
a worker assessed fewer than 100 sentence pairs, we did not
pay anything. For most workers, one assessment took about
30 seconds per sentence pair. As a result, we paid 15, 000
JPY to collect 24, 884 evaluations. We collected these assessments over a period of three weeks.

4.2 Crowdsourcing

4.3

In our experiment, we used crowdsourcing to assign labels
to sentence pairs. There are many crowdsourcing platforms
such as Amazon Mechanical Turk6 and CrowdFlower7 . We
choose to use Crowdworks8 , one of the major crowdsourcing
platforms in Japan, because the target articles were written
in Japanese and the workers would have to read, understand,and assign labels for sentence pairs written in Japanese.
The crowdsourcing statistics are shown in Table 3. To
ensure the accuracy of labeling through crowdsourcing, eachsentence pair was evaluated by more than two workers. When
two workers for one sentence pair assigned diﬀerent labels,
our system assigned one more worker to the sentence pair.
If more than five workers were assigned to one sentence pair,Figure 4 shows a recall-precision curve demonstrating thatour proposed method can calculate more accurate editor
quality values than the baseline method. At any recall ratio, the precision ratio of our proposed method was about
5% greater than that of the baseline system. At the lowest
recall ratio, in particular, the precision ratio of our proposed
method was 5% while that of the baseline method was 3%.
However, several sentences were not appropriately labeled
by workers. There seem to have been two reasons for this:some workers did not consistently assign appropriate labels,
and for several sentence pairs it was very diﬃcult to assign
appropriate labels. To solve the first issue, we should measure the accuracy of each worker’s assessment.
The second issue is a serious problem. For several articles, readers required some subject knowledge to understand the articles. Therefore, if workers lacked the required
knowledge, they were likely to misjudge in their evaluations.
Moreover, if changes in the sentences were complex, the5
https://en.wikipedia.org/wiki/Category:Blocked_
Wikipedia_users
6
https://www.mturk.com/mturk/
7
http://www.crowdflower.com
8
https://crowdworks.jp/

target articles
articleseditors
blocked editors
sentence pairs

Experimental Results and Discussion

crowdsourcing
workers
evaluations
sentence pairs
cost

#
4,412
78,3401,601
759,190

Table 2: Experimental Setup.

#
227
24,8848,040
0.5 JPY/evaluation(≈ 0.005 USD)

Table 3: Crowdsourcing Statistics

1005

workers could have been confused when selecting options.
For example, if a large amount of information is deleted anda small amount is added, a worker may be uncertain as towhether to ignore the small part added. To solve this issue,
we should use an unsupervised method, such as a majority
vote, because we cannot create an answer set for all cases.

5. CONCLUSION
In this paper, we have proposed a method for assessing
the quality of editors through crowdsourcing techniques. In
current assessment methods based on the peer review of
Wikipedia editors, text that survives multiple edits is assessed as good-quality text. However, these methods do notconsider the changes in sentence meanings, because it is diﬃcult to automatically capture these changes. In our method,
we use crowdsourcing techniques to solve this problem. As
a result, the precision ratio increases by about 5%.
In this work, we only aimed at detecting vandals. Therefore, we only considered the negative ratings of peer reviews.
However, we also have positive ratings such as ”ADD” and”EQUAL” available. By using both positive and negative
ratings, we will be able to identify good-quality editors.
Moreover, we should be able to identify many types of vandal, which we cannot do through our current method. With
our method, we can identify vandals who delete informationfrom Wikipedia, but we cannot identify vandals who write
grammatically bad sentences or who enter many misspelledterms. In Q2 from section 3.2.1, though, we ask crowdsourcing workers about grammatical errors and misspelled terms.
Therefore, we will be able to also identify these vandals andgood editors using crowdsourcing techniques.
Regarding our future work, there are many vandals changing the Wikipedia content, and there are also many vandals
among crowdsourcing workers. If there are too many badquality workers, and these workers assess large quantities
of sentence pairs, the assessment accuracy will deteriorate.
Methods for assessing the quality of crowdsourcing workers
have been developed, such as those of Raykar et al.[9] andIpeirotis et al. [7]. Applying such techniques should improve
the accuracy of our proposed assessment.
In our experiments, we used the Japanese Wikipedia dataset.
Therefore, we should confirm this method to the other language versions of Wikipedia datasets.

6. ACKNOWLEDGEMENTS
This work was supported by JSPS KAKENHI Grant Number 23700113, and NAIST Bigdata Project.

7. REFERENCES[1] B. Adler and L. de Alfaro. A content-driven
reputation system for the Wikipedia. In Proceedings ofthe 16th international conference on World Wide Web(WWW ’07), pages 261–270, 2007.
[2] B. T. Adler, K. Chatterjee, L. de Alfaro, M. Faella,
I. Pye, and V. Raman. Assigning trust to wikipedia
content. In Proceedings of the 4th International
Symposium on Wikis, WikiSym ’08, pages 26:1–26:12,
New York, NY, USA, 2008. ACM.
[3] B. T. Adler, L. de Alfaro, I. Pye, and V. Raman.
Measuring author contributions to the wikipedia. In
Proceedings of the 4th International Symposium on

1006

Wikis, WikiSym ’08, pages 15:1–15:10, New York, NY,
USA, 2008. ACM.
[4] L. De Alfaro, A. Kulshreshtha, I. Pye, and B. T.
Adler. Reputation systems for open collaboration.
Commun. ACM, 54(8):81–87, Aug. 2011.
[5] F. Flöck and M. Acosta. Wikiwho: Precise andeﬃcient attribution of authorship of revisioned
content. In Proceedings of the 23rd International
Conference on World Wide Web, WWW ’14, pages
843–854, New York, NY, USA, 2014. ACM.
[6] M. Hu, E. Lim, A. Sun, H. W. Lauw, and B. Vuong.
Measuring Article Quality in Wikipedia: Models and
Evaluation. In Proceedings of ACM International
Conference on Information and Knowledge
Management (CIKM 2007), pages 243–252, 2007.
[7] P. G. Ipeirotis, F. Provost, and J. Wang. Quality
management on amazon mechanical turk. In
Proceedings of the ACM SIGKDD Workshop on
Human Computation, HCOMP ’10, pages 64–67, New
York, NY, USA, 2010. ACM.
[8] S. Kumar, F. Spezzano, and V. Subrahmanian. Vews:
A wikipedia vandal early warning system. In
Proceedings of the 21th ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining,
KDD ’15, pages 607–616, New York, NY, USA, 2015.
ACM.
[9] V. C. Raykar and S. Yu. An entropic score to rank
annotators for crowdsourced labeling tasks. In
Proceedings of the 2011 Third National Conference on
Computer Vision, Pattern Recognition, Image
Processing and Graphics, NCVPRIPG ’11, pages
29–32, Washington, DC, USA, 2011. IEEE Computer
Society.
[10] Y. Suzuki and M. Yoshikawa. Assessing quality scoreof wikipedia article using mutual evaluation of editors
and texts. In Proceedings of the 22nd ACM
international conference on Conference on
information and knowledge management, CIKM ’13,
pages 1727–1732, New York, NY, USA, 2013. ACM.
[11] D. M. Wilkinson and B. A. Huberman. Cooperationand quality in wikipedia. In Proceedings of the 2007
international symposium on Wikis (WikiSym ’07),
pages 157–164. ACM, 2007.