Externalities in Knowledge Production:
Evidence from a Randomized Field Experiment†∗
Marit Hinnosaar‡

Toomas Hinnosaar§

Michael E. Kummer¶

Olga Slivko‖
August 18, 2021

Abstract
Are there positive or negative externalities in knowledge production? We analyze
whether current contributions to knowledge production increase or decrease thefuture growth of knowledge. To assess this, we use a randomized field experiment that
added content to some pages in Wikipedia while leaving similar pages unchanged. We
compare subsequent content growth over the next four years between the treatment
and control groups. Our estimates allow us to rule out effects on four-year growth ofcontent length larger than twelve percent. We can also rule out effects on four-year
growth of content quality larger than four points, which is less than one-fifth of thesize of the treatment itself. The treatment increased editing activity in the first two
years, but most of these edits only modified the text added by the treatment. Our
results have implications for information seeding and incentivizing contributions.
They imply that additional content may inspire future contributions in the shortand medium-term but do not generate large externalities in the long term.

JEL: L17, L86, C93
Keywords: knowledge accumulation, user-generated content, Wikipedia, public goods
provision, field experiment∗

Data and code for this article are available in Hinnosaar et al. (2021a), Harvard Dataverse, doi:
10.7910/DVN/T4VFCX.
†
We are grateful to John Duffy, two anonymous referees, Yan Chen, Chris Forman, Willa Friedman,
Shane Greenstein, David Hugh-Jones, Tobias Kretschmer, Giovanni Mastrobuoni, Ignacio Monzón, Juan
Morales, Abhishek Nagaraj, Stefan Penczynski, Imke Reimers, Stephan Seiler, Ananya Sen, Matthias
Sutter, and Michael Zhang for valuable comments. We would also like to thank the seminar participantsat the University of Strasbourg, the Collegio Carlo Alberto, George Mason University, ParisTech 2019,
Digital Economy Workshop 2019 (Católica Lisbon), ZEW Conference on the Economics of ICT, theAdvances with Field Experiments 2019 Conference (University of Chicago), SED 2021 for valuable input.
‡
University of Nottingham, Collegio Carlo Alberto, and CEPR, marit.hinnosaar@gmail.com.
§
University of Nottingham and CEPR, toomas@hinnosaar.net.
¶
University of East Anglia, Georgia Institute of Technology, and ZEW, M.Kummer@uea.ac.uk.
‖
Rotterdam School of Management, Erasmus University, slivko@rsm.nl.

1

1

Introduction

Knowledge is a key input to many economic activities and a driver of economic growth(Romer, 1990; Grossman and Helpman, 1993; Jones, 1995). An increasing share ofknowledge is created in the form of user-generated content: consumer feedback systems,
discussion boards, Q&A sites, open-source software, social networks, and online information
repositories, such as Wikipedia. Online knowledge repositories have the potential torevolutionize how society aggregates and transmits knowledge. This potential stems fromtheir ability to combine and aggregate the input of many individuals independent of timeand location, and the generated content can be retrieved at a low cost.1
A key component to the success of user-generated content repositories is a sufficient
flow of content contributions by users. Hence, understanding the drivers of contributions
to user-generated content has been an important question in economics and management
for the past two decades (Lerner and Tirole, 2003). While traditional motives of public
goods provision play a vital role in user-generated content, recent literature has identified anovel driver in the form of a feedback loop—a dynamic by which small initial contributionsof content inspire ever more follow-on content contributions by other users (Aaltonen and
Seiler, 2016; Kane and Ransbotham, 2016; Zhu et al., 2020).
In this paper, we investigate whether there are positive or negative externalities inuser-generated content production. Understanding and quantifying such externalities has
important policy implications. If content generation has positive externalities on futurecontent generation, then information seeding and paid contributions may have a highreturn on investment in terms of added stimulated growth (Aaltonen and Seiler, 2016).2
On the other hand, if content generation has negative externalities, then such policies may
backfire and be ineffective or even lead to worse eventual outcomes (Nagaraj, 2019).
Due to the reflection problem (Manski, 1993), externalities in content generation
are difficult to identify. An externality occurs when a contribution by a user motivates
other users to contribute (positive externality) or prevents further contributions (negative
externality). Yet, correlation in contributions does not necessarily attest to an externality.
A positive correlation may arise when two users are exposed to the same external shock,such as a news article or a research finding. Similarly, a non-causal negative correlation
may be caused by processes with periodic updates, such as elections or periodically updated
1
Traditional channels of personal knowledge transmission require a double-coincidence demand andsupply of knowledge. The “knowledge-seeker” and the “knowledge-holder” have to meet in person or atleast at the same time. The elimination of such double-coincidences has been modeled to understandthe advantage of monetary over barter-economies. Kiyotaki and Wright (1989). These features givesuch systems a drastic competitive advantage that may affect the education sector and other traditional
channels of knowledge transmission. The sector of encyclopedic knowledge is one of the most salient
examples of the new technology’s potential.
2
Nagaraj (2019) describes how such policies have been used by Wikipedia (seeding articles on morethan 30,000 US cities from US Census Bureau data), OpenStreetMap (US Census maps), and Reddit(fake user accounts).

2

statistics. To identify the causal effect, shocks to content growth and contributions must be
independent over time. We estimate the causal impact of additional content on subsequent
contributions using a randomized field experiment in Wikipedia. Randomization ensuresthat the addition of content is exogenous in terms of future content generation. As arguedby the literature analyzing social interactions (including Manski (1993) in general and more
specifically Aaltonen and Seiler (2016) and Zhu et al. (2020)), randomized experiments
are the best way to cleanly identify causal relationships in such interactions.
The exogenous variation in our data comes from a randomized field experiment, which
was conducted in 2014. The experiment added relevant content to randomly chosen
Wikipedia pages while leaving similar pages unchanged. The treatment added about two
paragraphs (approximately 2,000 characters) and one picture to each page in the treatmentgroup. The pages were about mid-sized Spanish cities in different language editions of
Wikipedia.3
For our analysis, we collect data from multiple sources. To measure the impact on thequantity of content and editing activity, we use a dataset of Wikipedia editing histories,
which includes all versions of Wikipedia pages in the treatment and the control groups. To
measure the effect on the quality of content, we use two approaches. First, we developed aquality rating scheme, and for each page, we obtained quality ratings by two independent
raters who are fluent in the corresponding language. Second, we use text analysis tocompare the content across different language editions of Wikipedia and measure similarity
to the corresponding pages in the Spanish Wikipedia. We use the Spanish Wikipedia
as a benchmark because it provides the most detailed coverage of Spanish cities among
all language editions of Wikipedia. Our dataset and the experimental setting allow us
to analyze both short-term and long-term effects, up to four years after the experiment.
Our main outcomes of interest are the quantity and quality of content. To study editing
activity, we also analyze the number of unique editors, the number of edits, and the amountof content added and deleted.
We find that the pages that were improved by adding about 2,000 characters were still
only longer by about the same amount, even four years after treatment. Similarly, while
the added content increased the quality, the difference four years later was about the sameas immediately after the treatment. Our estimates allow us to rule out effects on four-year
growth of content length larger than twelve percent. We can also rule out the effects on
four-year growth of content quality larger than four points, which is less than one-fifth of
the size of the treatment itself. We do find some short-term impact. In particular, we findthat the editing activity increased in the first two years after the treatment. In this initialstage, the treatment increased the number of Wikipedia users editing the treated pages
and increased the number of edits. However, there was no such impact in the third and3

A comprehensive description of the experiment is provided in Hinnosaar et al. (2021b), who studied
the impact of this treatment on real-world outcomes.

3fourth years post-experiment. Moreover, even in the first years, most of the additional
edits only modified the text added by the treatment.
While our study benefits from clean identification, it faces two important limitations.
First, our test has low power to measure small effects on content growth. Second, ourresults might not generalize to other platforms or content in other stages of development.
In our setting, the content is relatively mature but still incomplete. It is plausible that theresults would be different if we had studied either new pages or almost complete pages.
Our main finding has a clear policy implication—at least in settings similar to the one
studied here, investments in information seeding and incentivizing additional content contributions may temporarily increase user participation, but do not have a large cumulative
effect on content growth. Therefore information seeding and incentivizing contributions
are mainly a matter of direct cost-benefit analysis: they pay off if and only if the costs ofcreating the content are lower than the value of the new content.
Our paper contributes to the literature that studies externalities in user-generated
content production. The closest to our work are Aaltonen and Seiler (2016), Kane andRansbotham (2016), Nagaraj (2019), and Zhu et al. (2020). Aaltonen and Seiler (2016) and
Kane and Ransbotham (2016) used detailed observational data from Wikipedia. Nagaraj(2019) used a natural experiment on OpenStreetMap, a Wikipedia-style digital map-making
community, which started with better seeding information in some regions compared toothers for quasi-random reasons. Zhu et al. (2020) studied a natural experiment that
motivated students to contribute to Wikipedia. The papers arrived at contradicting
conclusions, which warrant further investigation regarding this issue. Our paper is thefirst to study the question using a randomized field experiment, which allows a clean
identification of the underlying externalities, especially when analyzing the long-term
impact.
More generally, the paper belongs to the literature that analyzes what drives contributions of user-generated content, which represents a salient and highly relevant digital public
good.4 Among the main drivers of content contributions, the studies addressed the role ofpersonal gain (Shah, 2006), social comparison (Chen et al., 2010), group size (Zhang andZhu, 2011), networks (Fershtman and Gandal, 2011; Ransbotham et al., 2012), attention
spillovers (Kummer, 2020), symbolic awards (Gallus, 2017), performance feedback (Huang
et al., 2018), monetary rewards vs social motives (Sun et al., 2017), contributor diversity(Ren et al., 2015), and economic conditions, such as unemployment (Kummer et al., 2019)and migration (Slivko, 2018). Social motives have been shown to affect public good
provision (Goldstein et al., 2008; Lacetera and Macis, 2010; Ayres et al., 2013). Specifically,in the case of digital public goods, examples include ranking movies on MovieLens (Chen
4

Other studies on Wikipedia have analyzed biases in Wikipedia’s content (Greenstein and Zhu, 2012,2018; Hinnosaar, 2019) and the impact of Wikipedia on market outcomes (Xu and Zhang, 2013; Hinnosaar
et al., 2021b) and science (Thompson and Hanley, 2018).

4

et al., 2010), editing articles on Wikipedia (Zhang and Zhu, 2011; Algan et al., 2013), andendorsing messages of Facebook users (Egebark and Ekström, 2017).5 In our setting, social
externalities are rather implicit, as individuals contributing content are not connected byany direct social ties, and the interactions with the other members of the community canoccur only in the process of contributing knowledge to the Wikipedia articles. Our paper
extends the literature by using variation from a randomized field experiment to measure
the impact of additional content on future content generation.
The structure of the paper is as follows. In the next section, we describe the experimentand provide some background on Wikipedia editing. Section 3 describes the data. Section 4
presents the results of the impact of the treatment on the subsequent growth in content
quantity and quality and on editing activity. Section 5 interprets our results in a simple
theoretical framework. Section 6 discusses the connection between our findings and related
literature. It also discusses the implications and generalizability of our findings. Section 7
concludes.

2

Experiment and background

2.1

Experiment

The field experiment added content (text and photos) to randomly chosen Wikipedia
pages. The sample consisted of 240 Wikipedia pages. Specifically, it consisted of the pagesof 60 Spanish cities in the French, German, Italian, and Dutch editions of Wikipedia.
The cities were all medium-sized, excluding the largest like Madrid and Barcelona, andalso excluding the smaller cities. The Wikipedia pages in these languages were relatively
short—up to 24,000 characters in each of these four languages.
Each city and each language edition of Wikipedia was treated equally. For each city,
its page was assigned to the treatment group in two randomly chosen languages. In
each language edition of Wikipedia, 30 randomly chosen city pages were assigned to thetreatment group. Specifically, to obtain balance in the treatment and control groups, therandomization was stratified.6 The 60 cities were divided into ten equal-sized groups.
Within each group, each city was randomly assigned to one of six treatment arms. The six
treatment arms were as follows: treat the city page in one of the six possible language
pairs (French & German; French & Italian; French & Dutch; German & Italian; German& Dutch; Italian & Dutch). This resulted in a design where the number of pages whichwere treated equaled the number of those that remained in the control group.
The Wikipedia pages were treated mid-August, 2014. The treatment added about
5

More generally, the literature suggests strong effects of social influence on individual choices related
to savings (Duflo and Saez, 2002, 2003), education (Hanushek et al., 2003; De Giorgi et al., 2010),
entertainment (Salganik et al., 2006), etc.
6
For further details of the randomization, see Hinnosaar et al. (2021b).

5

2,000 characters of text and photos to each page in the treatment group. The added text
and photos were mostly obtained from the corresponding Spanish and English language
Wikipedia pages. Because all the pages were about Spanish cities, the Spanish Wikipedia
typically contained more information than the other language versions. The English
language version of the page, typically, was also more detailed than in the languages inthe experiment. Hence, there was information available in Spanish and English pages thatwas missing from the other language editions of Wikipedia. The treatment translated thattext and added it to the corresponding pages in the treatment group.
The treatment of the pages in Dutch Wikipedia was not successful. While in French,
German, and Italian Wikipedia, the added text and photos survived well over time, all the
additions to Dutch Wikipedia were deleted within 24 hours (by a single editor). Wikipedia
allows anyone to edit. It also means that anyone can delete or undo the latest changes
by reverting to a previous version of the page. This happened in the Dutch version of
Wikipedia, where 24 hours after the treatment, all the pages looked as if they had neverbeen treated. Therefore, we exclude Dutch pages from our main analysis and restrict
attention to the 180 pages in French, German, and Italian. Robustness analysis showsthat our results do not change if Dutch pages are included in the analysis.

2.2Power analysis

We acknowledge that the sample size and power of our tests are rather small. We analyze
this in appendix A, which presents power analysis for one of our main outcome variables(page length) and the main editing activity variable (number of users). As estimation
results in section 4 show, we would expect the power for other measures to be similar.
To provide some context, let us discuss the expected effect sizes, given the results fromprevious literature. The main measure studied in most previous works is the impact ofadded content on the number of future contributors. Aaltonen and Seiler (2016) estimatethat adding 10,000 characters of content leads to 0.204 additional users per week, whichcorresponds to about 0.18 additional users per month for 2,000 characters as added by thetreatment in our case. Zhu et al. (2020) estimates are similar: their median treatment size
was 3,180 characters and estimated increase of 2 unique users per six-month period, which
implies 0.21 new users per month per 2,000 added characters.7
The literature provides less guidance for the long-term effect size for the quantity andquality of the content. Aaltonen and Seiler (2016) use their estimates for the impact on
the number of users to simulate a possible effect on the content quantity and find 45%
growth in content. The only other paper to estimate this effect is Nagaraj (2019), who
7

Kane and Ransbotham (2016) provide some evidence that the effect could be larger for less developed
content. They find that in the case of less developed articles, 1%-increase in length implies 0.03–0.04 moremonthly contributors. In our case, the treatment was on average 23% of the page length, which would
imply 0.7–0.9 more users.

6

finds a long-term effect of negative 10%.
Figures A.1a and A.1b in appendix A describe the power analysis for page length.
Figure A.1a shows that when the Dutch pages are included in the sample, as originallyintended, then if the true treatment effect is 10% increase in page length over 4 years, wewould reject the null hypothesis of no effect at 10%-significance level with 76% probability
and at 5%-significance level with 65% probability. The minimum detectable effect size
is about 12%.8 If we exclude the Dutch pages (figure A.1b), we lose some power, but
the minimum detectable effect is still around 13%. Indeed, our study is underpowered
to detect small long-term effects, but there should be no difficulties detecting even halfof the effect-size suggested by Aaltonen and Seiler (2016). Figures A.1c and A.1d showthat our experiment has relatively more power to detect meaningful effects on editing
activity. Even if we exclude the Dutch pages (figure A.1d), the minimum detectable effectsize is 0.11 users, and we should certainly be able to detect the effect sizes suggested in
the literature. Section 4 describes the ex-post minimum detectable effect sizes (with ourrealized data), which turn out to be similar to those described here (calculated based on
the pre-experiment data).

2.3

Background on Wikipedia and its editing

Wikipedia exists in 309 languages, and the different language editions are not identical.
The differences across languages made the experimental design possible. As we show insection 3.5, Spanish Wikipedia contained much more information about Spanish cities thanthe pages in our sample. This imbalance allowed the treatment to translate informationfrom Spanish Wikipedia to the target languages.
Why don’t Wikipedia editors translate the content between languages themselves?
First, it requires language skills. Many people are monolingual (Eurobarometer, 2012).
English language skills would not be enough because the pages in English Wikipedia
are also rather incomplete (as we show in section 3.5). The language skills required totranslate between Spanish, French, German, and Italian are not common. Table C.1 showsthat in France, Germany, and Italy, less than 10% of the population can read Spanish, andin Spain, less than 10% of the population can read the languages in our sample. Second,
perhaps equally importantly, Wikipedia is written by volunteers whose motivation depends
largely on how fun the editing process is (Nov, 2007). Presumably, the task of translating
information is somewhat mundane compared to other possible uses of time. Third, notethat for the majority of the 309 language editions of Wikipedia, automatic translationis still not good enough. Furthermore, even when automatic translation is technicallypossible, to use it, it would require that Wikipedians agree on which language version is
the superior one.
8

The minimum detectable effect size is calculated at 5%-significance level and 80% power.

7

Let us briefly describe how Wikipedia editing works on these pages. While large editsthat add an entire section or paragraph definitely exist, they do not make up the majority
of Wikipedia edits. Most Wikipedia edits are small, fixing typos, grammar, and formatting,rearranging text without modifying content. Figure C.1a shows that 75% of the edits topages in our sample (before treatment) add less than 100 characters (about one sentence).
42% of edits are marked by the editor as minor edits, which means not modifying content.
Only about 4% of edits add more than 1000 characters (about one paragraph).
Where do the small edits come from? We hypothesize that some of these small edits
occur when editors keep track of pages that they are interested in. On Wikipedia, any
registered user can sign up to be notified when a certain page has been modified. We findsome evidence supporting this on the corresponding city pages in English and Spanish
Wikipedia. Figure C.1b shows that almost half of these pages have more than 30 editorsthat get notified of all changes. Unfortunately, if there are less than 30 watchers for a page,
Wikipedia does not report the exact number. Hence, we just know that only 8 pages in our
sample have at least 30 persons signed up for notifications. Nevertheless, the comparison
with the pages in English and Spanish Wikipedia suggests that for the remaining pages,
there should be some watchers as well.

3

Data

We combine multiple sources of data. To measure the impact on the quantity of content
and editing activity, we use a dataset of Wikipedia editing histories. An editing history
contains the full text of each revision9 of each page starting from the creation of the page
until the beginning of September 2018.
To measure the effect on the quality of content, we use two approaches. First, wedeveloped a quality rating scheme, and for each page, we obtained quality ratings by
two independent raters who are fluent in the corresponding language. They rated threerevisions of each page in our sample (before treatment, after treatment, and four years
after treatment) comparing it to the English Wikipedia as a benchmark. Second, we use
text analysis to compare the content across different language versions of Wikipedia andmeasure similarity to the corresponding pages in the Spanish Wikipedia.
Our sample consists of the 180 pages in the experiment, which are the pages of 60
cities in French, German, and Italian Wikipedia. In the following subsections, we describe
the construction of the dataset and variables used in the analysis.
9

A revision (or an edit) is a version of a Wikipedia article saved at a specific moment by a particularuser. All revisions with the corresponding metadata, including full text, user, and timestamp, are preservedby Wikipedia and publicly available.

8

3.1

Page length0

Average page length
4000
8000
12000

One of our main outcome variables is the page length after the experiment. We measure
page length in characters, including spaces and wiki markup commands.
Figure 1 presents average page length in the treatment and control groups. Until
the experiment in August 2014, the average page length in the control and treatmentgroups was rather similar.10 The experiment added significant length to the pages inthe treatment group. After the experiment, the difference has been relatively stable. An
exception is a sharp increase in the mean of the treatment group in August 2016. Thisjump comes from the efforts of a single editor who worked hard to improve one page in
French Wikipedia—the page of the city of Cordoba.11 Appendix C presents the same
figure, first without French Cordoba (figure C.2a) and second, with the logarithm of page
length (figure C.2b), both of which show no evidence of an increase in the treatment groupaverage in 2016.

2004m1

2007m1

2010m1
2013m1MonthsTreatment
Treatment - Control

2016m1
Control

Figure 1: Average page length in the treatment and control groups
Notes: The number of observations is 90 in the control and 90 in the treatment groups. The experimentmonth (August 2014) is marked by the dashed vertical line.

Similar dynamics can be seen when looking at the changes separately by language(figures 2a to 2c). As expected, the placebo test with the Dutch pages (figure 2d) showsthat the assignment to the treatment group had no impact. Page length is one possible
output measure of knowledge production in Wikipedia. Similar dynamics as in figure 1 canalso be seen in figure C.3 in appendix C, which presents alternative measures of content:
images and plain text (that is, html elements removed from the parsed text).
10

The drop in both the treatment and control groups in early 2013 comes from technical changes
in Wikipedia: Addbot removed about 2,000 characters from each page with an explanation similar to“Migrating 77 interwiki links, now provided by Wikidata”.
11
By August 2016, the page of Cordoba in French Wikipedia was relatively long, with 19,426 characters(at the time 93% of the pages in our sample were shorter than that). During August 2016, this user
increased the page length to 100,702 characters, which is almost twice the length of the longest page at
the time (57,076 characters). Our conclusions do not change if we exclude this page.

9

15000
Average page length
5000
10000

Average page length
5000
10000 15000

002004m1

2007m1

2010m1
2013m1MonthsTreatment
Treatment - Control

2016m1

2004m1

Control

2007m1

2010m1
2013m1MonthsTreatment
Treatment - Control

Control(b) German

00

Average page length
2000
4000

Average page length
2000
4000
6000

6000

(a) French

2016m1

2004m1

2007m1

2010m1
2013m1MonthsTreatment
Treatment - Control

2016m1

2004m1

Control

2007m1

2010m1
2013m1MonthsTreatment
Treatment - Control(c) Italian

2016m1
Control(d) Placebo: Dutch

Figure 2: Average page length in the treatment and control groups, by language
Notes: On each figure, the number of observations is 30 in the control and 30 in the treatment groups.
The experiment month (August 2014) is marked by the dashed vertical line.

To make the treatment and the control groups comparable, we subtracted the length oftext added by the treatment from the length of pages in the treatment group. Moreover,as the distribution of page lengths is relatively skewed, we use the logarithm of page lengthin our estimations.

3.2

Quality ratings

To assess the changes in the quality of Wikipedia articles, we hired six research assistants
to rate the quality of articles. Each article in French, German, and Italian Wikipedia in
our sample was evaluated by two raters, who were fluent in the respective language aswell in English.
We asked the raters to evaluate the quality of three versions of 60 Wikipedia articles inour sample. Version A was the latest version before August 1st, 2014 (i.e., pre-treatment),
version B the latest version before September 1st, 2014 (i.e., post-treatment), and version
C the latest version before September 1st, 2018 (i.e., four years after the treatment). As a
10benchmark, we used articles in English-language Wikipedia (as of September 1st, 2018).12
Each version of each article was rated in five dimensions on a scale where 0 is the lowest
possible rating and 100 means equivalence to the benchmark page from English Wikipedia(for detailed instructions, see figure C.4 in the Appendix). The five dimensions were thefollowing. (1) Completeness: the article comprehensively covers all relevant aspects ofthe city (compared to the article in English). (2) Well-written: the prose is clear, concise,and spelling and grammar are correct. (3) Illustrated: the article includes photos that are
relevant to the topic and have suitable captions (compared to the article in English). (4)Interesting: the article makes the city seem like an exciting place to visit (compared tothe article in English). (5) Overall: Overall, the article is a high-quality reference source(compared to the article in English).
For example, a score of 100 in the completeness dimension means that the article covers
the relevant aspects of the city as comprehensively as the corresponding page in English.
It may occur if the pages cover the same topics in the same level of detail, or if they coverdifferent topics, but the missing parts “balance out” in the eyes of the raters. A score of
50 would mean that the page is half as good and 200 that the page is twice as good as thebenchmark page from English Wikipedia.
Figure 3 presents the change in quality during the treatment (August 2014) andwithin four years after treatment (September 2014–September 2018). It shows that in thetreatment group, as expected, there was a large increase in quality during the treatment.
The increase takes place in overall quality and in all measured dimensions except wellwritten. Indeed, during the treatment month, pages in the treatment group become slightly
less well-written. This is expected because, in the experiment, the treatment text was
written outside of Wikipedia and then copied to Wikipedia. During the same time inAugust 2014, the changes in the control group within August 2014 are negligible. Within
the following four years, overall quality increases in both treatment and control groups,
and interestingly these increases are much smaller than the treatment itself. However, thefour-year increase in the treatment group is of a similar size as that in the control group.
To further help to understand the context of the quality measures, figures C.5a–C.5b in
the Appendix show that the quality measures are correlated with page length.13

3.3

Measures of editing activity

To better understand the process of content creation, we also study editing activity. To
construct the measures of editing activity, we start with 30,601 edits (revisions) from 180
12

As we show below, the articles about Spanish cities in English-language Wikipedia are sometimes
quite incomplete, so ideally we would have preferred to use Spanish Wikipedia as a comparison. Because
the combination of necessary language skills is not common, it would have been prohibitively costly.
13
Similar correlations between quantity and quality of content have been found previously, for example
by Chen et al. (2019)

11

20

40

Sep 2014 - 201815

Sep 2014 - 2018
Control

Aug 2014

Sep 2014 - 2018
Control

Treatment(d) InterestingSep 2014 - 2018
Control

Treatment(c) Illustrated

4

Δ Similarity to Spanish

10

6

8

20
15

0

0

-5

0

5

Δ Well-written

20
10

Δ Interesting

Aug 2014

Treatment(b) Completeness30(a) Overall quality

10

Δ Illustrated

Aug 2014

Treatment

2

Control

0

0

5

10

20

Δ Completeness

30

30
20
10

Δ Overall quality0

Aug 2014

Aug 2014

Sep 2014 - 2018
Control

Treatment(e) Well-written

Aug 2014

Sep 2014 - 2018
Control

Treatment(f) Similarity to Spanish

Figure 3: Change in quality during the treatment month (August 2014) and within four
years after treatment (from September 2014 until September 2018), separately for thecontrol and treatment groups.
Notes: The number of observations is 90 in the control and 90 in the treatment groups.

Wikipedia pages. This set includes all the edits except those generated as part of thetreatment in the experiment. Following Aaltonen and Seiler (2016), we restrict the sample
of edits in the following ways. First, we exclude edits by bots (about 30% of edits); these
are non-human user accounts that generate automated edits. Specifically, we define bots as
users whose username occurs in the list of bots (in the English, French, German, or Dutch
Wikipedias) or whose username includes “bot”. Second, we exclude reverts, which are editsthat restore any previous version of the same page (about 7% of remaining edits). Third,
we exclude vandalism (about 0.8% of remaining edits). We use the following criteria toclassify an edit as vandalism: (a) an edit that only deletes text from the previous revision,and (b) the revision immediately after vandalism reverts the article back to a past revision.
Then we are left with 19,586 productive edits generated by human users.
To analyze the impact of treatment on editing activity, we construct three types ofmonthly measures that characterize how many people edited the pages, how many times
they edited, and how much they edited. The first measure is the number of unique users
editing a page per month. We define a unique user by the username for registered users andby IP address for anonymous users. The second measure is the number of edits per month.
To avoid double-counting of micro-edits,14 we first aggregate edits to the day-user-page
14
Many Wikipedia editors save many revisions to the same page in a short period of time, for example,
generating a new revision after each sentence they write. This is partly motivated by the fact that someone
else might edit the page at the same time.

12

level and then sum these up to month-page level. The third measure is edit distance—thenumber of characters an edit added plus the number of characters it deleted compared tothe previous version of the page.15 We aggregate the edit distance measure to monthly
level. Figure C.6 in appendix C describes the average editing activity in the treatment
and control groups over time.
In addition to the aggregate measures of editing activity, we separate edits that directly
modify the treatment text and those that modify other parts of the page. We classify edits
into these two categories using a method similar to Hinnosaar et al. (2021b). For eachpage in the treatment group, we use the diff algorithm between the revision before andafter the treatment to determine treatment text—the exact text added by the treatment.
For each revision post-treatment, using the diff algorithm between the treatment text andthis revision, we check whether the revision deletes any part of the treatment text. If therevision doesn’t delete anything from the treatment text, we classify the revision as one
that edited other parts of the page.

3.4

Similarity to Spanish Wikipedia

To complement the human-rated page quality data, specifically page completeness, we also
evaluate the changes in quality by computing the similarity with Spanish Wikipedia. We
use the articles in the Spanish language (as of September 1st, 2018) as a benchmark as
they provide the most detailed coverage of Spanish cities among all language versions of
Wikipedia. Therefore, if the coverage of topics in another language becomes more similar
to the corresponding article in the Spanish language, this can be interpreted as increasing
completeness of the article.
As a measure of similarity, we use the Tversky index (Tversky, 1977). Formally, if
the set of terms mentioned in the article of the target language is A and the set of terms
mentioned in the Spanish article is B, then the Tversky index is computed as
S(A, B) =|A ∩ B|,|A ∩ B| + α|A \ B| + β|B \ A|(1)

where α ≥ 0 and β ≥ 0 are parameters. If α = β = 1, the index becomes equivalent to the
Jaccard similarity index, and if α = β = 0.5, it simplifies to the Sørensen-Dice similarity
index.16 More generally, a larger α puts a bigger weight on A (interpreted as a variant)
15

Edit distance is widely used in computational linguistics and computer science to measure the similarityof strings. It is a generic term that allows any weights of insert, delete, and substitution operations.
Common variants put weight 1 to addition and deletions, and weight substitutions either by 1 (called
Levenshtein distance) or by 2 (the measure we use). For each edit, we calculate the edit distance using
PHP FineDiff class at the granularity level of a character.
16
Both indexes are used to measure the similarity between two documents. The Jaccard index is also
known as the Intersection over Union and sometimes called the Tanimoto similarity. Another similarity
measure used in earlier works in economics (Thompson and Hanley, 2018; Chen et al., 2019) is cosine
similarity. It would be preferable if we want to capture the similarity of two pages not only in terms of13and β a bigger weight on B (interpreted as a prototype). As our aim is to use the Tversky
index as a completeness measure (how complete is A in comparison to B), we set α = 0and β = 1, which simplifies the similarity measure to S(A, B) = |A∩B|.
|B|
Before comparing the articles in the treatment languages with their Spanish counterparts, we need to extract the set of terms mentioned in each article. For this, we firsttranslate all articles into English using Yandex Translate API.17 Then, we remove all
so-called stop words, i.e., words that do not reflect the specifics of the content.18 We thenuse the Porter stemming algorithm (Porter, 1980) to remove the endings from words in
English so that only root words remain. Finally, we drop all remaining strings that areshorter than three characters or contain non-alphabetic characters. This process gives us
a set of terms for each article that we use in computing the Tversky index.19
Figure 3f presents the change in similarity during the treatment and within four
years after treatment. As with quality, we see a large increase in the treatment group in
similarity during treatment and much smaller increases within the following four years
in both treatment and control groups. Figures C.5c–C.5d in the appendix show thatsimilarity to Spanish is correlated with page length and also with completeness (compared
to English).

3.5

Summary statistics

Comparison of pages in treatment and control groups. Table 1 presents thecomparison of pre-treatment page length, quality, and editing activity in the treatmentgroup versus the control group. The table shows that there were no significant differences
between the two groups before the treatment. Figures C.8–C.9 in the appendix present
the kernel density estimates of full distributions separately for the treatment and control
groups for all the variables in table 1. We conclude that while there are differences between
the groups, by and large, the randomization was rather successful.
Are the pages in our sample already rather complete? To answer the question,ideally, we would like to use Wikipedia’s own quality ratings. Unfortunately, the pages inour sample have not been rated. Each language edition of Wikipedia has its own quality
rating system, but only in the English Wikipedia is the system widely used. Therefore, we
assess the completeness of the pages in two steps. First, we determine the quality of thecorresponding city pages in the English Wikipedia. Then we compare the pages in our
sample to the English pages.
content covered but the language and tone. As we aim to capture completeness of the page compared to abenchmark, we found the Tversky index most appropriate for the task.
17
For more information, see https://tech.yandex.com/translate/.
18
For example, pronouns (“it”, “their”) and prepositions (“on”, “before”). The full list is in figure C.7.
19
Terms such as “america”, “archipelago”, “area”, “arona”; about 250–500 terms for each article.

14

Table 1: Comparison of pre-treatment characteristics in the treatment versus the control
group.
Control Treatment t-test Wilcoxon test Obs.
group
group p-value
p-value
mean
mean
(1)
(2)
(3)
(4)
(5)
Log. length before treatment
8.586
8.611
0.842
0.655
180
Quality rating before treatment
72.70070.917
0.810
0.602
180
Quality: complete before treatment
71.65671.706
0.996
0.758
180
Quality: interesing before treatment
75.80072.917
0.777
0.561
180
Quality: well-written before treatment
92.86793.122
0.858
0.388
180
Quality: illustrated before treatment
72.37273.772
0.854
0.441
180
Similarity to Spanish before treatment
17.48817.251
0.885
0.820
180
Aver. # of users before treatment
0.378
0.333
0.321
0.285
180
Aver. # of edits before treatment
0.396
0.351
0.341
0.364
180
Aver. edit dist. before treatment
79.01376.786
0.882
0.738
180
Aver. capped edit dist. before treatment 42.511
38.238
0.470
0.583
180
Notes: Column 1 and 2 present the means of pre-treatment values of variables, separately for the control
and the treatment group. Column 3 presents the p-value of the t-test for whether the difference between
the control and treatment groups is significantly different from zero. Column 4 presents the p-value of thecorresponding Wilcoxon rank-sum test. Column 5 presents the number of observations used in each test.

Figure C.10 in the appendix presents the distribution of the quality ratings in English
Wikipedia of the 60 city pages that correspond to the pages in our sample. These city
pages have been assigned the lowest possible grades (Stub, Start, and class C) or have notbeen rated at all (9 cities).20 The highest rated pages, class C, according to the Wikipedia
rating scale, still miss important content. Hence, we conclude that the 60 city pages are
all low-quality articles in the English Wikipedia.21
Having seen that all the English pages still have room for improvement, how do thepages in the sample compare to the English pages? Table 2 compares our pages to thepages in English in 3 dimensions: relative length, (calculated) similarity, and (human
rated) completeness. Panel A column 1 shows that after treatment the median page inthe treatment group is about 75% of the relative length, about 33% in terms of similarity,
and almost 100% in terms of completeness compared to the corresponding page in English.
Recall here that we saw above that the pages in English were far from complete. Panel A,
column 2 shows that, as expected, after treatment the median page in the control group isrelatively shorter and of relatively lower quality. But before the treatment (panel B), themedian pages in the treatment and control groups are similar.
20

The three lowest grades in the Wikipedia content assessment are: Stub—a very basic description of
the topic; Start—developing but still quite incomplete; Class C—substantial but is still missing important
content or contains much irrelevant material. Source: https://en.wikipedia.org/wiki/Wikipedia:
Content_assessment.
21
Note that these articles, on average, are also not very important according to the English Wikipedia
article importance scheme. The scheme uses ratings from Low, Mid, High, to Top. Only 7 of the pages are
rated as High importance, 15 as Mid, and 9 as Low importance, 29 of the articles have not been assigned
an importance rating, which probably also implies that those articles are not highly important.

15

Table 2: Completeness of the median page compared to English and Spanish Wikipedia

Relative length
Similarity
Completeness
Relative length
Similarity
CompletenessCompared to English Compared to Spanish
Treatment Control Treatment Control
(1)
(2)
(3)
(4)
Panel A: After treatment (2014 September)
75.6
53.4
50.2
35.2
33.1
27.8
24.1
17.5
98.4
72.4
.
.
Panel B: Before treatment (2014 August)
57.9
53.2
37.7
35.1
26.2
27.7
17.3
17.5
71.7
71.7
.
.

Notes: Each cell presents the median from 90 pages either in the treatment group (columns 1 and 3) orcontrol group (columns 2 and 4).

Columns 3–4 in table 2 present the comparison with the pages in the Spanish Wikipedia.
Compared to Spanish, the pages in the sample are even shorter and less similar. After thetreatment, the median page in the treatment group is still only 50% of the length of thecorresponding city page in Spanish. While it is not clear that all that material should be
included in the French, German, and Italian Wikipedia, at least we can say that there isadditional material that was important enough to be included in the Spanish Wikipedia.
Also note that the treatment added only material on topics relevant for tourists, suchas the city’s main sights and culture. Hence, while the treatment made the pages more
complete on these topics, on other topics that are typically covered on each city page, suchas demographics, economy, education, and government, there is probably still room for
improvement.

4

Results

We start by estimating the impact of the treatment on growth after treatment, both inlength and quality. We are looking at growth during four years after treatment. After that,
we go into more details in two ways. First, we study the effects on different dimensions ofquality. Then we analyze the short- and long-term effects using a difference-in-differences
estimator.
To estimate the effect of the treatment on growth after treatment, we compare thegrowth of pages (indexed by i) in the treatment and control group controlling for city andlanguage fixed effects:∆yi = β0 + β1 T reatmentGroupi + LanguageF Ei + CityF Ei + εi(2)

The coefficient of interest is β1 on T reatmentGroupi , which is an indicator variable thattakes value one if the page was assigned to the treatment group and zero if it was assigned
16

to the control group. The outcome variable ∆yi = y2018September − y2014September measuresthe change in outcome from September 2014 to September 2018. Specifically, the outcome
variables are the change in the logarithm of page length, change in the overall quality
rating, and the change in the similarity to the corresponding Spanish Wikipedia article.22
Table 3 presents the estimates of the effect of treatment on subsequent growth overfour years post-treatment.23 The point estimates are 7% of the standard deviation forlength and -3% for quality.24 But the estimates are imprecise. The 95%-confidence interval
for length is from -7% to +12%, and the ex-post minimum detectable effect size is 13%.
Similarly, the 95%-confidence interval for the growth in quality is from -4 to +3 points
and the ex-post minimum detectable effect size is 5 points. Note that these bounds are no
more than one third of the size of the treatment itself. The estimates for the similarity
index are analogous to the quality rating.
Table 3: The long-term effect of treatment on the growth in page length and quality.
Dependent variable: y2018Sep − y2014Sep .

Treatment group
Language FE
City FE
Mean dep. var.
SD dep. var.
Adj. R-squared
Observations

Change in page length or quality (y2018Sep − y2014Sep )∆Log. page length∆Quality rating∆Similarity to Spanish
(1)
(2)
(3)
0.026
-0.375-0.590(0.048)(1.777)(0.624)
Yes
Yes
Yes
Yes
Yes
Yes
0.190
6.589
2.032
0.353
11.958
4.409
0.259
0.116
0.199
180
180
180

Notes: Each column presents estimates from a separate cross-section regression of 180 Wikipedia pages.
The dependent variable ∆yi = y2018September − y2014September is the change in logarithm of page length(column 1), change in the overall quality rating (column 2), and the change in the similarity to thecorresponding Spanish Wikipedia article (column 3). All regressions include language fixed effects andcity fixed effects. Standard errors are reported in parentheses.

Table 4 presents the results of the treatment on the growth in different dimensions ofquality. In all four dimensions of quality, measuring how complete, interesting, illustrated,
and well-written the page is, the estimates are not statistically significant. We can rejectthat the effect of treatment on subsequent growth is more than one fourth the size of22

To make the pages comparable, we subtract the length of text added by the treatment from thelength of pages in the treatment group after treatment (both in 2014 and in 2018). We do that because
the outcome variable measures the percentage change, and hence, without subtracting the length of thetreatment text, the treatment group would have a higher base when calculating the percentage, then thesame increase in characters would give a smaller percentage for the treatment group.
23
Table C.2 in the appendix shows that the results are robust to alternative control variables. Table C.3
shows that the results are also robust when including the Dutch pages either in the control group or
estimating intention-to-treat.
24
For expositional clarity, we interpret the coefficient in column 1 as measuring a percentage change inlength. This is a logarithmic approximation that performs well when changes are small which is the casein our sample.

17

the treatment itself for the completeness and interesting quality dimensions. The largest
point estimate is for well-written—the effect of treatment on subsequent growth in thisdimension is about 5 points but the estimates are imprecise. This is consistent with what
we would expect, given that the treatment itself reduced the quality in this dimension by
3 points.
Table 4: The long-term effect of treatment on the growth in different dimensions of pagequality. Dependent variable: y2018Sep − y2014Sep .

Treatment group
Language FE
City FE
Mean dep. var.
SD dep. var.
Adj. R-squared
Observations

Change in page quality (y2018Sep − y2014Sep )∆Complete∆Interesting∆Illustrated∆Well-written
(1)
(2)
(3)
(4)
-0.967-0.150-2.0374.658(2.254)(1.851)(3.752)(5.980)
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
6.939
5.700
12.186
4.875
15.212
12.20127.28037.879
0.122
0.080
0.243
0.003
180
180
180
180

Notes: Each column presents estimates from a separate cross-section regression of 180 Wikipedia pages.
The dependent variable ∆yi = y2018September − y2014September is the change in the following dimensionsof page quality: complete (column 1), interesting (column 2), illustrated (column 3), and well-written(column 4). All regressions include language fixed effects and city fixed effects. Standard errors arereported in parentheses.

Next, we analyze both the short- and long-term effects. We estimate the following
difference-in-differences regression using page-level monthly panel data:
yit =

X

βs · 1[Y eart = s] · T reatmentGroupi + M onthF Et + LanguageCityF Ei + εit (3)swhere the sum over s is taken over the years {−3, −2, −1, 1, 2, 3, 4} and Y eart measures
years since the experiment. The year of the experiment is the baseline, which is why s = 0
is excluded. The regression includes fixed effects for each page i (language and city pair)and for each time period t. The coefficients of interest are the β-s on T reatmentGroupi
and year dummy interactions. All the year and treatment group interactions, including forpre-treatment years, are presented graphically in figure C.11 in the appendix. Figure C.11
shows that there is no evidence of differential pre-treatment trends.
Table 5 presents the estimates of short- and long-term effects from regression (3). In
column 1, the outcome variable is the logarithm of page length minus treatment text.25
The estimates of the short- and long-term impacts of treatment on page length are allinsignificant at ten percent level. The estimate of the long-term (four-year) effect on page
length is similar to that in table 3.
25

To make the pages comparable, we subtract the length of text added by the treatment from the lengthof pages in the treatment group after treatment. Hence, the estimates should be interpreted as the effectof treatment on page length after removing the mechanical increase created by the treatment.

18

Table 5: The short- and long-term effects of treatment on subsequent page length andediting activity.
Log. length # users # edits # edits
Edit
Cappedexcludingexcluding distanceedit
treatment
treatment
distance
(1)
(2)
(3)
(4)
(5)
(6)
Treatment × year 1
0.045
0.112*** 0.118**
0.006-13.609 12.970(0.032)(0.043) (0.049) (0.045) (40.176) (9.250)
Treatment × year 2
0.051
0.109** 0.128**
0.067
107.260 17.967*(0.039)(0.047) (0.056) (0.055) (101.633) (10.414)
Treatment × year 3
0.049
0.008
0.001
-0.046-37.161-6.849(0.047)(0.043) (0.049) (0.049) (30.389) (9.412)
Treatment × year 4
0.020
0.012
0.013
-0.041
2.626-3.498(0.056)(0.041) (0.046) (0.044) (74.799) (8.922)
Month FE
Yes
Yes
Yes
Yes
Yes
Yes
Treatment ×{ year-1,-2,-3 }
Yes
Yes
Yes
Yes
Yes
Yes
Language-City FE
Yes
Yes
Yes
Yes
Yes
Yes
Mean dep. var.
8.651
0.329
0.347
0.329
89.49136.062SD dep. var.
0.813
0.694
0.759
0.740
1120.108 131.787
Observations
17100
17100
17100
17100
17100
17100
Notes: A unit of observation is a page-month pair. The dependent variable is the logarithm of page
length minus treatment text (column 1) or a monthly number of users (columns 2), edits (columns 3),edits not editing treatment text (column 4), edit distance (column 5), or capped edit distance (column 6).
Treatment × year 1 is an indicator variable that takes value one during the first year post-treatment if thepage belongs to the treatment group and zero otherwise; and similarly for the other years. All regressions
include page fixed effects and month fixed effects. All the year and treatment group interactions, including
for pre-treatment years, are presented graphically in figure C.11 in the appendix. The sample is a balanced
sample from September 2010 to August 2018, excluding the treatment month of August 2014. Standard
errors, reported in parentheses, are clustered by page (180 pages). *** Indicates significance at the 1
percent level, ** at a 5 percent level, * at a 10 percent level.

In columns 2 and 3, the outcome variables are the number of users (people editing
the page) and edits per month. The treatment increased the number of users and thenumber of edits during the first two years after the experiment. Specifically, the treatment
increased the monthly number of users editing the page by about 0.11 users (column 2)
and the monthly number of edits by 0.12–0.13 edits (column 3). However, these increases
are only short-lived. In the third and fourth year, for both measures, the estimates oftreatment effect are insignificant at ten percent level, and the point estimates are almost
ten times smaller in magnitude. In the fourth year, we can reject an effect of the same
size found in the first years (0.11 users).26
What do these editors and edits do in the first two years post-treatment if it has
surprisingly little effect on page length? A natural explanation could be that the additional
edits simply polish the text added by the treatment. To study this, we re-calculated
the number of edits per month while excluding edits that directly edited the text addedby the treatment. The estimates using this outcome variable are presented in column
4. The results show that when excluding the edits that directly affect the text added by26

For the long-term (four-year) effect, the ex-post minimum detectable effect size is 0.11 users.

19the treatment, then the treatment effect is much smaller. We conclude that most of theshort-term increase in editing comes from editing the content added by the treatment.
In column 5, the outcome variable is the number of characters added plus deleted.
The treatment had no statistically significant (at ten percent level) effect on the measure.
Coefficients vary in sign over the years. The largest point estimate is in the second year:
the (statistically insignificant) coefficient estimate implies the treatment effect of about
100 characters per month.
Because the distribution of the number of characters added plus deleted has a longtail, in column 6, we use an alternative measure calculated from individual capped edits.
The individual edits are capped from above at the 90th percentile. The 90th percentile
equals about 500 characters and is about 10 times larger than the median edit. In thisway, the capped edit distance measure gives smaller weight to long edits. Estimates incolumn 6 show that in the second year after the experiment, the treatment increases the
capped change in characters by about 18 characters per month. Provided that the averageword length across languages in the experiment is about 10.4 characters, our treatment
increased the edit distance by about two words.27 This is in line with the findings fromcolumns 3–4, which showed that most of the increase in editing comes from editing thecontent added by the treatment. In later years, the point estimates of the treatment effectare even smaller, and the estimates are statistically insignificant.
Finally, we analyze whether the effect of the treatment is heterogeneous, varying by
subgroups of the pages. Tables C.4 and C.5 in the appendix re-estimate the regressions
in table 3, allowing the effect of treatment to vary by the quality, completeness, relative
length (compared to the pages in Spanish Wikipedia), and the age of the page. While all
the estimates are statistically insignificant, the pattern in the point estimates seems tosuggest that the treatment effect was larger on lower quality and less complete pages.
Multiple hypothesis testing. We run many tests, and with a large number of tests,
we could easily get false positives, i.e., simply by chance, some estimates could turn out
to be statistically significant. That is, the probability of incorrectly rejecting at least
one null hypothesis is greater than the probability of incorrectly rejecting each individual
hypothesis test. To address the concern, we adjust for multiple hypothesis testing byadjusting for the family-wise error rate (the probability of incorrectly rejecting at least
one null hypothesis belonging to the same family of hypothesis).
We prefer not to assign all the tested hypotheses to the same family because we viewthat some of our outcomes are more important than others. Specifically, we are mainlyinterested in long-term outcomes and on the impact on length and quality, not on the
inputs like various measures of editing. Therefore, to adjust the family-wise error rate,
we group our hypotheses into the following families: (1) long-term effects on length and27

Source: http://www.ravi.io/language-word-lengths.

20

quality, (2) short-term effects on editing behavior, (3) long-term effects on editing behavior.
Specifically, we use Westfall and Young (1993) multiple hypothesis p-value adjustment as
implemented by Jones et al. (2019), employing 10,000 bootstrap draws.
Tables C.6–C.8 in the appendix present the adjusted p-values. As expected, theadjusted p-values are much larger, and therefore our conclusions regarding the long-term
effects are unchanged. Using the stricter p-values, the short-term effects on editing activity
become statistically insignificant. Therefore, a conservative approach would be to interpretthe findings of the short-term effects simply as suggestive. On the other hand, the stricter
p-values might be viewed as too conservative, considering that the literature oftentimes
does not correct for multiple hypothesis testing, and the number of our tests has increasedin part to show robustness. For this reason, in our preferred estimates, we use unadjusted
p-values.

5

Theoretical framework

To provide a framework for interpreting our results and unify findings from the related
literature, we introduce a simple theoretical model of the private provision of public goods.
We postpone the technical details of the model to appendix B and present its implications
by way of figure 4 here. The theoretical framework involves a sequence of players making
public good contributions. Depending on the current state of the public good (denoted by
X), each contribution can have positive or negative externalities for future contributions.
We first show that if there are no externalities, the rate of contributions is on average
constant over time and therefore growth path of the public good value X is linear, as
illustrated in figure 4a.28 This means that a jump in contributions (such as experimental
treatment) would lead to a parallel shift in the growth path and the long-term outcomes are
shifted by about the same magnitude as the initial treatment. With negative externality,
the rate of contributions decreases with the state and therefore growth path is decreasing
and long-term effect of a jump in contributions is smaller than the short-term effect(figure 4b). Conversely, with positive externality, growth path is increasing over time andlong-term effect of a jump in contributions is bigger than the short-term effect (figure 4c).
Finally, as we argue below, perhaps the most realistic scenario for the externalities is an
U-shaped externality function—initially the externality is positive and eventually negative.
This would leave to S-shaped growth path (figure 4d). In this case the comparison ofshort-term and long-term impacts depends on time horizon as well as the moment oftreatment.
Let us now discuss how our empirical results can be interpreted within this theoretical
28

Without externalities, our model is similar to the model in Chen et al. (2019), with two differences: thepayoffs in their model depend on social impact (i.e., number of viewers) and participation is endogenous.
A crucial difference in our model is the inclusion of externalities.

21

X

X

t

t

(a) No externalities(b) Negative externality
X

X

t

t

(c) Positive externality(d) Inverted-U-shaped externality

Figure 4: The expected growth of the state under different assumptions about externalities
Notes: Time (or number of contributors) on the horizontal axis and state of the public good on the verticalaxis. For more details, see appendix B.

framework. The first question to ask is which channels lead to either positive or negative
externalities? We can highlight several channels leading to positive externalities. Attention:some Wikipedia editors sign up as watchers for a page and get notifications for each time
the page is edited. The reason they do it is to maintain the quality of the page, andtherefore some of these notifications must lead to future edits. Moreover, contributions
increase the number of page views, mostly by increasing visibility in search rankings(Hinnosaar et al., 2021b). Some of these additional eyeballs are likely to convert tocontributions (Kane and Ransbotham, 2016; Kummer, 2020; Zhu et al., 2020). Learningand inspiration: existing content provides contributors new information about a topic
and gives them ideas for contributions (Olivera et al., 2008). Social motives: additional
content signals potential interest in the community (Zhang and Zhu, 2011; Chen et al.,2019). All these channels would lead to positive externalities, i.e., more contributions by
earlier editors either make it easier to contribute or raise their benefits from contributions.
Without loss of generality, this can be modeled as a reduction of the marginal cost ofcontributions, as we did in the previous section. These channels likely have a declining
impact, i.e., during the early stages of the page development, we would expect the impacts
to be larger than for mature pages, which already have a large amount of content.
On the other hand, there are also channels that lead to negative externalities: Crowding
out: if users add the content that adds most value at the least possible cost (“low hanging
fruits”), then new content raises the cost of future contributions. Freeriding: new content

22

tells potential editors that someone is already taking care of the edits of this particular
page, and they may choose to point their attention elsewhere. Increasing complexity: aWikipedia page should be a coherent reference source; the more content there is, the more
possibilities there are for combinations on how the material could be organized and moreparts that need to have the same style and structure. All these channels lead to negative
externalities, i.e., more contributions by earlier editors either make it more difficult tocontribute or reduce their benefits from contributions. Without loss of generality, this canbe modeled as an increasing marginal cost of contribution. It is likely that these channels
for negative externalities become more prevalent as content matures and converges tocompleteness.
Combining these observations, in different settings, the total effect may lean either
towards positive or negative externalities, depending on which channels are more active.
It is likely that for relatively new and incomplete pages, channels inducing positive
externalities are more prevalent. As the page matures, channels for positive externalities
become less important, and channels for negative externalities more important. This would
imply an inverted-U-shaped externality that we saw in the previous section.
The remaining question is how large are these externalities, which is the empiricalquestion that our analysis addresses. Our empirical results show that a treatment ∆has approximately the same effect ∆ on the outcomes four years later. This finding is
consistent with two possibilities: either there are no externalities (figure 4a), or there areboth positive and negative externalities in the same magnitude (figure 4d). Both cases
are plausible, as the treatment contributed to relatively mature pages that were still far
from being complete. The results about treatment heterogeneity (tables C.4 and C.5)
provide some evidence in favor of the second possibility as they seem to indicate that thetreatment had a more positive effect on lower quality and less complete pages.

6

Discussion

6.1

Comparison with related literature

Aaltonen and Seiler (2016) and Kane and Ransbotham (2016) study the impact of added
content on the short-term editing activity. The estimates from Aaltonen and Seiler (2016)
imply that additional content of 2,000 characters leads to about 0.18 additional monthly
users. Zhu et al. (2020) estimate the impact over six months and find an effect of similar
magnitude, about 0.21 additional monthly contributors per 2,000 added characters. Ourestimates largely confirm the findings from the literature. The magnitude of our estimates isslightly smaller (about 0.11 additional users per month), the difference might be explainedby the fact that the other papers are measuring more immediate effects, while we measure
the impact over a year or two.
23

However, we find that there is no large long-term impact on editing activity or content
growth. This finding is related to the results in Kane and Ransbotham (2016), who findthat although additional content may bring additional contributions to pages that are
relatively incomplete, this effect may disappear once the pages become complete. Our
results on treatment heterogeneity provide some evidence towards this, although the effects
are not statistically significant.
Such differences in editing activity are consistent with a simple explanation that inearlier stages of the content life-cycle, the channels leading to positive externalities are
more active, while the channels leading to negative externalities become more dominant
in the long-term. In particular, as some editors have signed up as watchers who makesure that added content is up to the quality standards (see section 2.3), we would expectthat immediately after content is added there is increased editing activity focusing on theadded content. Our analysis of the short-term editing activity supports this conjecture.29
On the other hand, Nagaraj (2019) studies the impact of better-quality seeding dataon quality outcomes ten years later. He finds a negative effect of about 10%, which isa large difference from the conclusion of Aaltonen and Seiler (2016), whose simulations
implied that the positive effect on short-term editing activity could lead to about 45%
better output. Our results about the impact of added content on the long-term growth incontent quantity and quality provide some support in favor of the first number, but we do
not have enough power to measure small effects.
The finding from Nagaraj (2019) is partially consistent with the implications of ourtheoretical framework. Assuming that a time horizon of ten years is sufficient for thecontent to be close to completeness, we would expect that early differences in content
would disappear in the long-term. In other words, in the long term, we would expect thenegative externality to dominate. Explaining the negative impact on outcomes requires an
explanation beyond our model and analysis. The key mechanism proposed by Nagaraj(2019) to explain the negative externality is the “ownership effect”, which plays a lessprominent role in Wikipedia. Nagaraj (2019) suggests that contributors who added
particular bridges or streets on the user-generated map may feel more responsible forkeeping these objects updated over time. Therefore, the treatment of adding more seeding
information may backfire by not allowing the ownership of objects to arise naturally. Allother papers mentioned here, including our work, focus on textual content in Wikipedia,
where ownership is less clear, and we would thus expect the negative effect of adding
content to be less prominent.
29

Note that there are other, more subtle differences in the research environments that may affect theoutcomes. For example, in our setting, the added content comes from an outside source. Instead, in thesettings of Kane and Ransbotham (2016), Aaltonen and Seiler (2016), and Zhu et al. (2020), the added
content is created by the community.

24

6.2

Generalizability

A word of caution is in order here. The results from Wikipedia might not generalize
to other user-generated content platforms. As an example, a relevant difference among
the platforms is the magnitude of the contributor’s personal benefit. In Wikipedia, thepersonal benefit from contributing is likely to be smaller than in open-maps or open-source
software. For example, a user of open-maps could directly benefit from correcting a mistakeon a map, while an error in Wikipedia is unlikely to have any personal consequences.
Another example is the “ownership effect” proposed by Nagaraj (2019), which seems to bea key driver in the Wikipedia-style mapping service, where each object (street, building,etc.) explicitly states who has edited this object. As Wikipedia does not display who
wrote each part of the page, the ownership assignment is less clear. Therefore, we would
expect this channel for negative externality to be less active.
Moreover, our results also might not generalize to settings in other stages in content
development. Our theoretical model highlighted that the externalities might depend on
the existing amount of available content, additional content in early stages being more
beneficial than later. Our empirical results about the heterogeneity provide some suggestive
evidence that this might be the case. However, the experiment was not designed to analyze
this heterogeneity and is underpowered to do that. We would encourage future researchthat aims at uncovering how the existing content (or more generally the lifecycle of thecontent development) affects the externalities in content creation.

7

Conclusions

In this paper, we studied the impact of added content on the subsequent long-term growthof content. We identify the causal effect using exogenous variation from a randomized
field experiment in Wikipedia where the treatment added content to randomly chosen
Wikipedia pages. Our estimates allow us to rule out effects on four-year growth of content
length larger than twelve percent. We can also rule out the effects on four-year growthof content quality larger than four points, which is less than one-fifth of the size of thetreatment itself. We find that the treatment increased subsequent content generation in
the first two years. Specifically, it increased the number of edits and editors. However, theamount of content these users added was on average only a couple of words, and most of
their edits modified the content added by the treatment.
Our findings have a clear policy implication—in settings like the one studied here,
information seeding and motivating content creation is not necessarily enough to generate a large increase in future content generation. However, these policies are also notcounterproductive as they can stimulate additional edits, and we can rule out a large
discouragement effect on future contributions. Therefore, it is mostly a matter of direct25

cost-benefit analysis whether such policies pay off.
Our results may not generalize to settings where other channels leading to positive or
negative externalities are more prominent. For example, it is possible that in the earlystages of content development, information seeding may be beneficial. On the other hand,
it is also possible that in situations where individual contributions are well-identified, theseeding policies may backfire.

References
Aaltonen, A. and S. Seiler (2016): “Cumulative Growth in User-Generated Content
Production: Evidence from Wikipedia,” Management Science, 62, 2054–2069.
Algan, Y., Y. Benkler, M. Fuster Morell, and J. Hergueux (2013): “Cooperation in a Peer Production Economy Experimental Evidence From Wikipedia,”
manuscript.
Ayres, I., S. Raseman, and A. Shih (2013): “Evidence From Two Large Field
Experiments That Peer Comparison Feedback Can Reduce Residential Energy Usage,”
Journal of Law, Economics, and Organization, 29, 992–1022.
Chen, Y., R. Farzan, R. E. Kraut, I. YeckehZaare, and A. F. Zhang (2019):“Motivating Contributions to Public Information Goods : A Personalized Field Experiment on Wikipedia,” manuscript.
Chen, Y., F. M. Harper, J. Konstan, and S. X. Li (2010): “Social Comparisons andContributions to Online Communities: A Field Experiment on Movielens,” American
Economic Review, 100, 1358–98.
De Giorgi, G., M. Pellizzari, and S. Redaelli (2010): “Identification of Social
Interactions Through Partially Overlapping Peer Groups,” American Economic Journal:
Applied Economics, 2, 241–75.
Duflo, E. and E. Saez (2002): “Participation and Investment Decisions in a Retirement
Plan: The Influence of Colleagues’ Choices,” Journal of Public Economics, 85, 121–148.
——— (2003): “The Role of Information and Social Interactions in Retirement Plan
Decisions: Evidence From a Randomized Experiment,” Quarterly Journal of Economics,
118, 815–842.
Egebark, J. and M. Ekström (2017): “Liking What Others "Like": Using Facebook
to Identify Determinants of Conformity,” Experimental Economics, 1–22.

26

Eurobarometer (2012): “Europeans and Their Languages,” Special Report 386, European Commission.
Fershtman, C. and N. Gandal (2011): “Direct and Indirect Knowledge Spillovers:
the "Social Network" of Open-Source Projects,” RAND Journal of Economics, 42, 70–91.
Gallus, J. (2017): “Fostering Public Good Contributions with Symbolic Awards: A
Large-Scale Natural Field Experiment at Wikipedia,” Management Science, 63, 3999–
4015.
Goldstein, N. J., R. B. Cialdini, and V. Griskevicius (2008): “A Room Witha Viewpoint: Using Social Norms to Motivate Environmental Conservation in Hotels,”
Journal of Consumer Research, 35, 472–482.
Greenstein, S. and F. Zhu (2012): “Is Wikipedia Biased?” American Economic
Review: Papers and Proceedings, 343–348.
——— (2018): “Do Experts or Crowd-based Models Produce More Bias? Evidence from
Encyclopedia Britannica and Wikipedia,” MIS Quarterly, 42, 945–959.
Grossman, G. M. and E. Helpman (1993): Innovation and Growth in the Global
Economy, MIT press.
Hanushek, E. A., J. F. Kain, J. M. Markman, and S. G. Rivkin (2003): “Does Peer
Ability Affect Student Achievement?” Journal of Applied Econometrics, 18, 527–544.
Hinnosaar, M. (2019): “Gender Inequality in New Media: Evidence from Wikipedia,”
Journal of Economic Behavior & Organization, 163, 262–276.
Hinnosaar, M., T. Hinnosaar, M. Kummer, and O. Slivko (2021a): “Replication
Data for: "Externalities in Knowledge Production: Evidence from a Randomized Field
Experiment",” Harvard Dataverse, DOI: 10.7910/DVN/T4VFCX.
——— (2021b): “Wikipedia Matters,” Journal of Economics & Management Strategy.
Hinnosaar, T. (2018): “Optimal Sequential Contests,” manuscript.
Huang, N., G. Burtch, B. Gu, Y. Hong, C. Liang, K. Wang, D. Fu, andB. Yang (2018): “Motivating User Generated Content with Performance Feedback:Evidence from Randomized Field Experiments,” Management Science, forthcoming.
Jones, C. I. (1995): “R&D-Based Models of Economic Growth,” Journal of Political
Economy, 103, 759–784.

27

Jones, D., D. Molitor, and J. Reif (2019): “What Do Workplace Wellness Programs
Do? Evidence From the Illinois Workplace Wellness Study,” Quarterly Journal of
Economics, 134, 1747–1791.
Kane, G. C. and S. Ransbotham (2016): “Content as Community Regulator: The
Recursive Relationship Between Consumption and Contribution in Open Collaboration
Communities,” Organization Science, 27, 1258–1274.
Kiyotaki, N. and R. Wright (1989): “On Money as a Medium of Exchange,” Journalof Political Economy, 97, 927–954.
Kummer, M. (2020): “Attention in the Peer Production of User Generated Content Evidence from 93 Pseudo-Experiments on Wikipedia,” manuscript.
Kummer, M., O. Slivko, and X. Zhang (2019): “Unemployment and Digital Public
Goods Contribution,” Information Systems Research, forthcoming.
Lacetera, N. and M. Macis (2010): “Social Image Concerns and Prosocial Behavior:
Field Evidence From a Nonlinear Incentive Scheme,” Journal of Economic Behavior &Organization, 76, 225–237.
Lerner, J. and J. Tirole (2003): “Some Simple Economics of Open Source,” Journalof Industrial Economics, 50, 197–234.
Manski, C. F. (1993): “Identification of Endogenous Social Effects: The Reflection
Problem,” Review of Economic Studies, 60, 531–542.
Nagaraj, A. (2019): “Information Seeding and Knowledge Production in Online Communities: Evidence from OpenStreetMap,” manuscript.
Nov, O. (2007): “What Motivates Wikipedians?” Communications of the ACM, 50,
60–64.
Olivera, F., P. S. Goodman, and S. S.-L. Tan (2008): “Contribution Behaviors inDistributed Environments,” MIS Quarterly, 32, 23–42.
Porter, M. F. (1980): “An Algorithm for Suffix Stripping,” Program, 14, 130–137.
Ransbotham, S., G. C. Kane, and N. H. Lurie (2012): “Network Characteristics andthe Value of Collaborative User-Generated Content,” Marketing Science, 31, 387–405.
Ren, Y., J. Chen, and J. Riedl (2015): “The Impact and Evolution of Group Diversity
in Online Open Collaboration,” Management Science, 62, 1668–1686.
Romer, P. M. (1990): “Endogenous Technological Change,” Journal of Political Economy,
98, S71–S102.
28

Salganik, M. J., P. S. Dodds, and D. J. Watts (2006): “Experimental Study ofInequality and Unpredictability in an Artificial Cultural Market,” Science, 311, 854–856.
Shah, S. K. (2006): “Motivation, Governance, and the Viability of Hybrid Forms in
Open Source Software Development,” Management Science, 52, 1000–1014.
Slivko, O. (2018): “Online "Brain Gain": Do Immigrants Return Knowledge Home?”manuscript.
Sun, Y., X. Dong, and S. McIntyre (2017): “Motivation of User-Generated Content:
Social Connectedness Moderates the Effects of Monetary Rewards,” Marketing Science,
36, 329–337.
Thompson, N. and D. Hanley (2018): “Science Is Shaped by Wikipedia: Evidence
From a Randomized Control Trial,” manuscript.
Tversky, A. (1977): “Features of Similarity,” Psychological Review, 84, 327–352.
Westfall, P. H. and S. S. Young (1993): Resampling-Based Multiple Testing:
Examples and Methods for p-Value Adjustment, vol. 279, John Wiley & Sons.
Xu, S. X. and X. Zhang (2013): “Impact of Wikipedia on Market Information Environment: Evidence on Management Disclosure and Investor Reaction,” MIS Quarterly,
37, 1043–1068.
Zhang, X. and F. Zhu (2011): “Group Size and Incentives to Contribute: A Natural
Experiment at Chinese Wikipedia,” American Economic Review, 101, 1601–1615.
Zhu, K., D. Walker, and L. Muchnik (2020): “Content Growth and Attention
Contagion in Information Networks: A Natural Experiment on Wikipedia,” Information
Systems Research, forthcoming.

29

A

Online Appendix: Power analysis

We conduct the power analysis for one of our main outcome variable, page length, and themain editing activity variable, number of users, using data from years 2010–2014 before theexperiment and Monte Carlo simulations with 10,000 samples. We draw random samples(with replacement) of size 60 of cities. That gives us the samples of pages of size either
240 (with Dutch pages) or 180 (without Dutch pages). We randomize according to therandomization protocol described in section 2.1 and estimate the following regressions.
In the case of page length, we compare the growth of pages (indexed by i) in thetreatment and control group controlling for language fixed effects:∆logLengthi = β0 + β1 T reatmentGroupi + LanguageF Ei + εi

(4)The outcome variable is the change in the logarithm of page length from 2010 to 2014
(that is growth during four years before treatment).
In the case of the number of users, we compare the average number of uses in thetreatment and control group, controlling for past number of users and language fixed
effects:
U sers2014,i = β0 + β1 T reatmentGroupi + β2 U sers2011,i + LanguageF Ei + εi

(5)The outcome variable U sers2014,i is the yearly average of the number of monthly users
from August 2013 to July 2014, and we calculate U sers2011,i analogously. Note that since
this cross-section regression includes the lagged outcome variable U sers2011,i , we expect
the estimates to be similar to a difference-in-differences estimator.
Figure A.1 presents the relationship between power and the true effect size at 5% and10%-significance level. For both page length and the number of users, it shows powerwith and without the Dutch pages, hence with the sample size of either 240 or 180 pages.
Figure A.1a shows that when the Netherlands is included in the sample, as originallyintended, then if the true treatment effect is 10% increase in page length over four years, wewould reject the null hypothesis of no effect at 10%-significance level with 76% probability
and at 5%-significance level with 65% probability. The minimum detectable effect size
is about 12%. If we exclude the Netherlands (figure A.1b), we lose some power, but theminimum detectable effect is still around 13%. Figure A.1d shows that even if we exclude
the Netherlands, the minimum detectable effect size is 0.11 users, and we should certainly
be able to detect the effect sizes suggested in the literature.
To summarize, our study is underpowered to detect small long-term effects on page
length, but we can detect even half of the effect-size suggested by Aaltonen and Seiler(2016). On the number of users, our experiment has relatively more power, being able todetect the effect sizes suggested in the literature.
A1

1
0

.2

Power
.4
.6

.8

1
.8
Power
.4
.6
.2
0

5

10

15

20

5

10

Effect size (%)α= 0.115

20

Effect size (%)α= 0.05

α= 0.1

.8
Power
.4
.6
.2
0

0

.2

Power
.4
.6

.8

1

(b) Page length (without Dutch)

1(a) Page length (with Dutch)

α= 0.05

.05

.1
.15
Effect size (monthly number of users)
α= 0.1.2.05

α= 0.05.1.15
Effect size (monthly number of users)
α= 0.1(c) Number of users (with Dutch).2

α= 0.05(d) Number of users (without Dutch)

Figure A.1: Power analysis for the effect on the page length and the number of users
Notes: Calculated using data from years 2010–2014 before the experiment and Monte Carlo simulations
with 10,000 samples.

A2

B

Online Appendix: Theoretical framework

The model focuses on provision of a single public good (a Wikipedia page) by a sequenceof agents (editors). The initial state (value) of the public good is X0 ≥ 0. Time is discrete
and in each period t ∈ N, one agent t arrives, gets an i.i.d. draw of parameters (αt , βt , γt ),observes the current state Xt−1 , chooses a contribution xt ≥ 0, which increases the state
to Xt = Xt−1 + xt , and then leaves the model.30 Agent t gets payoff
ut (xt , Xt−1 ) = vt (Xt ) + wt (xt ) − ct (xt , Xt−1 ),(6)where vt (Xt ) = αt Xt = α(Xt−1 + xt ) is the value of the public good, wt (xt ) = βt xt is theprivate benefit of the agent’s own contribution (for example warm-glow), ct (xt , Xt−1 ) =γt xt · x2t − µ(Xt−1 ) is the cost of contribution xt , and µ(Xt−1 ) is the externality.31
Let us first consider a benchmark with no externalities, i.e., µ(Xt−1 ) = 0. Then thenot
optimal contribution of agent t that maximizes (6) is x∗t = max 0, αtγ+β. Note that int
this case, the contributions are independent of the current state and therefore the expectednot
growth rate of contributions is a constant that equals E max 0, αtγ+β. This implies thattan exogenous contribution ∆ (such as provided by the experiment) has the same effect at
any period (parallel shift). This is illustrated by figure 4a.
The externality function µ(Xt−1 ) enters the cost function so that the marginal cost is∂ct= γt (xt − µ(Xt−1 )) .
∂xt

(7)

The marginal cost depends linearly on contribution xt and additively on externality.
Therefore, if the externality function µ(Xt−1 ) is constant, the state Xt−1 does not affectmarginal cost. If the externality function µ(Xt−1 ) is decreasing, there is a negative
externality, i.e., the marginal cost increases with the state. For example, this may occur
with free-riding: the better the current state of the public good, the fewer reasons there areto contribute more. On the other hand, if the externality function µ(Xt−1 ) is increasing,
there is a positive externality, i.e., the marginal cost decreases with the state. For example,
earlier contributions may give later contributors ideas (inspire) on how to contribute.
Perhaps the most realistic case is an inverted-U-shaped µ(Xt−1 ) function. Initially, when
the state is low, it is quite costly to contribute, when the state increases, it becomes easier,but eventually, as the state becomes very high, it becomes again more and more costly to30

Extending the model to stochastic arrivals would not change the qualitative results.
For simplicity we assume only cost externalities. A significant simplification of the analysis is that thebenefit from the public good only depends on the current state. If agents’ benefit would depend on the
expected eventual state of the public good, they would have to take into account how their contributions
affect the future contributions. This would be a much more complicated sequential game, which undersome conditions can be solved using the inverted best-response approach introduced in Hinnosaar (2018).
Qualitative implications would remain the same without these simplifications.
31

A3

find something to contribute.
The optimal contribution of agent t depends on parameters and the externality:(x∗t (Xt−1 ) = max)

α t + βt+ µ(Xt−1 )
0,
γt

(8)

When the externality is negative, the optimal contribution x∗t becomes smaller as thestate Xt−1 increases (it becomes more costly to contribute). Therefore, the equilibrium
growth rate of the state decreases over time. An exogenous addition ∆ (for example, anexperimental treatment) has a long-term effect smaller than ∆. Figure 4b illustrates thiscase. On the other hand, if the externality is positive, it becomes easier to contribute.
Then, the equilibrium contributions and the growth rate of the state are increasing overtime. In this case, an exogenous addition ∆ has a long-term effect larger than ∆. See
figure 4c for an illustration.
Finally, when the externality function is inverted-U-shaped, then the growth rateis initially increasing as contributions become easier. However, over time, as the state
converges to its upper limit, the contributions become more costly, and therefore thegrowth slows down. An early treatment ∆ leads to fast growth by reducing the costs,
whereas a later treatment ∆ (of the same amount) has a reduced impact as it slows down
the growth. Figure 4d illustrates this case.

A4

C

Online Appendix: Additional figures and tables
EnglishSpanish<30 30-49 50-99 100+

<30 30-49 50-99 100+100

Sample74.5

80

Add less than 100 char.

Minor edit

41.8

0

20

40
Percent

60
40

55.2

20

Both add and delete

Percent

4.2

60

80

(a) Edits by type (as a percentage)0

Add more than 1000 char.

<30

30-49

(b) Distribution of pages by the number of watchers

Online Appendix Figure C.1: Edit types and editors watching watching each page
Notes: Figure C.1a presents the edits by type as a percentage out of 100 calculated using the edits in thesample of 180 pages pre-treatment. Figure C.1b presents the distribution of the number of watchers inthe sample, in the English Wikipedia, and in the Spanish Wikipedia as measured in January 2020.

A5

Average of log. page length
2
4
6
8
10

12000

0

Average page length
4000
8000
0

2004m1

2007m1

2010m1
2013m1MonthsTreatment
Treatment - Control

2016m1

2004m1

Control

2007m1

2010m1
2013m1MonthsTreatment
Treatment - Control(a) Page length without Cordoba in French

2016m1
Control(b) Logarithm of page length

Online Appendix Figure C.2: Robustness: Page length00Average number of images
5
10
15
20

Average length of plain text
4000
8000
12000

Notes: The number of observations used to calculate the average is 90 in the control group and 89(Figure C.2a) or 90 (Figure C.2b) in the treatment group. The experiment month (August 2014) is marked
by dashed vertical line.

2004m1

2007m1

2010m1
2013m1MonthsTreatment
Treatment - Control

2016m1

2004m1

Control

2007m1

2010m1
2013m1MonthsTreatment
Treatment - Control(a) Images

2016m1
Control(b) Plain text

Online Appendix Figure C.3: Other output measures
Notes: The number of observations is 90 in the control and 90 in the treatment groups. The experimentmonth (August 2014) is marked by dashed vertical line. Plain text is obtained by removing html elementsfrom the parsed text.

A6

Online Appendix Figure C.4: A screenshot with instructions given to research assistants
who rated the quality of articles as described in section 3.2

A7

10
7

8

Log. length

910
9Log. length
8
7

1

2

3

4

5

1

Quality

2

3

4

5

Completeness (compared to English)

excludes outside values

excludes outside values(b) Completeness versus log. length

0

7

8

Log. length

9

Similarity to Spanish
20
40

10

60

(a) Quality versus log. length

1

2

3

4

5

1

Similarity to Spanish
excludes outside values(c) Similarity to Spanish versus log. length

2

3

4

5

Completeness (compared to English)
excludes outside values(d) Completeness versus similarity to Spanish

Online Appendix Figure C.5: Quality, completeness (compared to English), similarity(compared to Spanish), and log. length
Notes: We group the characteristic on the horizontal axes into quintiles. For each quantile, the graphpresents the median (as a horizontal line) and the interval from the 25th to the 75th percentile (as thebox) of the variable on the vertical axes. All measures are from pre-treatment (in August 2014.

A8

1
Average number of users
.2
.4
.6
.8
0

2004m1

2007m1

2010m1
2013m1MonthsTreatment

2016m1

Control0

Average number of edits
.2
.4
.6
.8
1

(a) Number of users

2004m1

2007m1

2010m1
2013m1MonthsTreatment

2016m1

Control0

Average edit distance
500
1000

1500

(b) Number of edits

2004m1

2007m1

2010m1
2013m1MonthsTreatment

2016m1

Control(c) Edit distance

Online Appendix Figure C.6: Average input measures in the treatment and control groups
per month
Notes: The number of observations is 90 in the control and 90 in the treatment groups. The experimentmonth (August 2014) is marked by dashed vertical line.

A9

’ourselves’, ’hers’, ’between’, ’yourself’, ’but’, ’again’, ’there’, ’about’, ’once’,’during’, ’out’, ’very’, ’having’, ’with’, ’they’, ’own’, ’an’, ’be’, ’some’, ’for’, ’do’,’its’, ’yours’, ’such’, ’into’, ’of’, ’most’, ’itself’, ’other’, ’off’, ’is’, ’s’, ’am’, ’or’,’who’, ’as’, ’from’, ’him’, ’each’, ’the’, ’themselves’, ’until’, ’below’, ’are’, ’we’,’these’, ’your’, ’his’, ’through’, ’don’, ’nor’, ’me’, ’were’, ’her’, ’more’, ’himself’,’this’, ’down’, ’should’, ’our’, ’their’, ’while’, ’above’, ’both’, ’up’, ’to’, ’ours’,’had’, ’she’, ’all’, ’no’, ’when’, ’at’, ’any’, ’before’, ’them’, ’same’, ’and’, ’been’,’have’, ’in’, ’will’, ’on’, ’does’, ’yourselves’, ’then’, ’that’, ’because’, ’what’, ’over’,’why’, ’so’, ’can’, ’did’, ’not’, ’now’, ’under’, ’he’, ’you’, ’herself’, ’has’, ’just’,’where’, ’too’, ’only’, ’myself’, ’which’, ’those’, ’i’, ’after’, ’few’, ’whom’, ’t’,’being’, ’if’, ’theirs’, ’my’, ’against’, ’a’, ’by’, ’doing’, ’it’, ’how’, ’further’, ’was’,’here’, ’than’Online Appendix Figure C.7: List of stop words used to clean the text for computing the
Tversky similarity measure in section 3.4

A10

Kernel density estimate

0

.1

.2

Density.3

Density
.005
.01

.4

.5

.015

Kernel density estimate

7

8

9Log. length before treatment

10

11

0

100

200
300
Quality rating before treatment

Treatment
Control

400

500

Treatment
Controlkernel = epanechnikov, bandwidth = 0.3190

kernel = epanechnikov, bandwidth = 12.2066(a) Log. length(b) Overall quality
Kernel density estimate

0

0

.002

.002

Density.004 .006

Density.004 .006

.008

.008

.01

.01

Kernel density estimate

0

200
400
600
Quality: complete before treatment

800

0

100
200
300
Quality: illustrated before treatment

Treatment
Control

Treatment
Controlkernel = epanechnikov, bandwidth = 14.6479

kernel = epanechnikov, bandwidth = 14.6479(c) Completeness(d) Illustrated

Kernel density estimate

Kernel density estimate

0

0

.02

Density.04

Density
.005
.01

.06

.08

.015

400

0

200
400
600
Quality: interesing before treatment

800

60

80
100
120
Quality: well-written before treatment

Treatment
Control

140

Treatment
Controlkernel = epanechnikov, bandwidth = 13.4273

kernel = epanechnikov, bandwidth = 2.7126(e) Interesting(f) Well-written

Online Appendix Figure C.8: Distributions of page length and quality before treatment,
separately by treatment and control groups
Notes: Kernel density estimates of the pre-treatment distributions, separately for the control and treatmentgroups.

A11

Kernel density estimate

0

0

.01

.5

Density1

Density
.02
.03

1.5

.04

2

.05

Kernel density estimate

0

20
40
60
Similarity to Spanish before treatment

80

0

.5
1
1.5
Aver. \# of users before treatment

Treatment
Control2

Treatment
Controlkernel = epanechnikov, bandwidth = 3.1610

kernel = epanechnikov, bandwidth = 0.0904(a) Similarity to Spanish(b) Average number of users
Kernel density estimate

0

0

.5

.002

Density1

Density
.004
.006

1.5

.008

2

Kernel density estimate

0

.5
1
1.5
Aver. \# of edits before treatment

2

0

200
400
Aver. edit dist. before treatment

Treatment
Control

600

Treatment
Controlkernel = epanechnikov, bandwidth = 0.0954

kernel = epanechnikov, bandwidth = 29.6023(c) Average number of edits(d) Average edit distance0

Density
.005
.01

.015

Kernel density estimate

0

50
100
150
200
Aver. capped edit dist. before treatment

250

Treatment
Controlkernel = epanechnikov, bandwidth = 12.1162(e) Average capped edit distance

Online Appendix Figure C.9: Distributions of page similarity to Spanish and editing
activity before treatment, separately by treatment and control groups
Notes: Kernel density estimates of the pre-treatment distributions, separately for the control and treatmentgroups.

A12

Featured article
Featured list
Class A
Good article
Class BClass CStart
Stub
Not rated
010

20
Number of articles

30

Online Appendix Figure C.10: Quality of the 60 city pages (corresponding to our sample)
in the English Wikipedia

A13

.2

.15

.1

.1
.05

0

0

-.1

-.05
-.1

-3

-1
1
Years from treatment

3

-3

3

(b) # Users

-.2

-.1

-.1

0

0

.1

.1

.2

.2

(a) Log. length (minus treatment text)-1
1
Years from treatment

-3

-1
1
Years from treatment

3

-3

3

(d) # Edits excluding treatment

-20

-100

0

0

100

20

200

40

300

(c) # Edits-1
1
Years from treatment

-3

-1
1
Years from treatment

3-3(e) Edit distance-1
1Years from treatment

3(f) Capped edit distance

Online Appendix Figure C.11: Panel
Notes: Point estimates and 90% confidence intervals from regressions in table 5 of the treatment groupand years since treatment interactions. A unit of observation is a page-month pair. Regressions include
page fixed effects and month fixed effects. The sample is a balanced sample from September 2010 toAugust 2018, excluding the treatment month August 2014.

A14

Online Appendix Table C.1: Foreign language reading skills, % of population
FrenchGermans
Italians
Reading English
Reading French
Reading German
Reading Italian

Reading Spanish Reading English
9
32
2
33
4
26
Spanish reading other languages
15
7
1
2

Source: Eurobarometer (2012)

A15

Online Appendix Table C.2: Robustness of the effect of treatment on subsequent growthin page length and quality, alternative controls. Dependent variable: y2018Sep − y2014Sep .
Change in page length or quality (y2018Sep − y2014Sep )∆Log. page length∆Quality rating∆Similarity to Spanish
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
Treatment group 0.012 0.012 0.012 -0.867 -0.867 -0.867 -0.478 -0.478 -0.478(0.053) (0.053) (0.048) (1.786) (1.802) (1.683) (0.658) (0.657) (0.611)
Group FE
No
Yes
No
No
Yes
No
No
Yes
No
Language FE
No
No
Yes
No
No
Yes
No
No
Yes
Mean dep. var.
0.190 0.190 0.190 6.589 6.589 6.589 2.032 2.032 2.032
SD dep. var.
0.353 0.353 0.353 11.958 11.958 11.958 4.409 4.409 4.409
Observations
180
180
180
180
180
180
180
180
180
Notes: Each column presents estimates from a separate cross-section regression of 180 Wikipedia pages.
The dependent variable is the change in logarithm of page length (columns 1-3), change in the overall
quality rating (columns 4-6), and the change in the similarity to the corresponding Spanish Wikipedia
article (columns 7-9). Regressions in columns 1, 4, and 7 don’t include any controls besides the indicator
for the treatment group, regressions in columns 2, 5, and 8 include group dummies, and regressions incolumns 3, 6, and 9 include language fixed effects. Standard errors are reported in parentheses.

Online Appendix Table C.3: Robustness of the effect of treatment on subsequent growthin page length and quality, Dutch pages included. Dependent variable: y2018Sep − y2014Sep .

Treatment group
Language FE
City FE
Mean dep. var.
SD dep. var.
Adj. R-squared
Observations

Change in page log. length (y2018Sep − y2014Sep )Dutch in control gr.
Intention to Treat
(1)
(2)
0.021
0.004
(0.043)(0.036)
Yes
Yes
Yes
Yes
0.156
0.156
0.319
0.319
0.248
0.247
240
240

Notes: Each column presents estimates from a separate cross-section regression of 240 Wikipedia pages.
The dependent variable is the change in logarithm of page length. All regressions include language fixed
effects and city fixed effects. In column 1, all Dutch pages are assigned to the control group. Column 2
presents the intention-to-treat estimate. Standard errors are reported in parentheses.

A16Online Appendix Table C.4: The effect of treatment on subsequent growth in page lengthand quality, heterogeneity by page quality. Dependent variable: y2018Sep − y2014Sep .

Treatment group
Treatment group× Below medianBelow medianLanguage FE
City FE
Mean dep. var.
SD dep. var.
Adj. R-squared
Observations
Treatment group
Treatment group× Below medianBelow medianLanguage FE
City FE
Mean dep. var.
SD dep. var.
Adj. R-squared
Observations

Change in page length or quality (y2018Sep − y2014Sep )∆Log. page length ∆Quality rating ∆Similarity to Spanish
(1)
(2)
(3)
Panel A: Heterogeneity by page quality-0.034-1.564-0.901(0.073)(2.743)(0.975)
0.118
2.340
0.612
(0.111)(4.129)(1.467)
0.079
3.084
0.260
(0.084)(3.125)(1.111)
Yes
Yes
Yes
Yes
Yes
Yes
0.190
6.589
2.032
0.353
11.958
4.409
0.282
0.128
0.189
180
180
180
Panel B: Heterogeneity by page completeness-0.002-1.310-0.768(0.072)(2.732)(0.969)
0.052
1.788
0.342
(0.110)(4.131)(1.465)0.144*
3.153
0.557(0.082)(3.094)(1.097)
Yes
Yes
Yes
Yes
Yes
Yes
0.190
6.589
2.032
0.353
11.958
4.409
0.294
0.125
0.191
180
180
180

Notes: Each column presents estimates from a separate cross-section regression of 180 Wikipedia pages.
The dependent variable is the change in logarithm of page length (column 1), change in the overall quality
rating (column 2), and the change in the similarity to the corresponding Spanish Wikipedia article (column
3). All regressions include language fixed effects and city fixed effects. An indicator for treatment group isinteracted with an indicator whether either page pre-treatment overall quality (panel A) or completeness(panel B) is below the median. Standard errors are reported in parentheses.

A17Online Appendix Table C.5: The effect of treatment on subsequent growth in page
length and quality, heterogeneity by page length and page age. Dependent variable:y2018Sep − y2014Sep .

Treatment group
Treatment group× Below medianBelow medianLanguage FE
City FE
Mean dep. var.
SD dep. var.
Adj. R-squared
Observations
Treatment group
Treatment group× Below medianBelow medianLanguage FE
City FE
Mean dep. var.
SD dep. var.
Adj. R-squared
Observations

Change in page length or quality (y2018Sep − y2014Sep )∆Log. page length ∆Quality rating ∆Similarity to Spanish
(1)
(2)
(3)
Panel A: Heterogeneity by page relative length (to Spanish)
0.043-0.608-1.157(0.070)(2.599)(0.914)
0.000
1.557
1.427
(0.099)(3.655)(1.285)0.221**
6.928*
1.644(0.099)(3.680)(1.294)
Yes
Yes
Yes
Yes
Yes
Yes
0.190
6.589
2.032
0.353
11.958
4.409
0.289
0.148
0.225
180
180
180
Panel B: Heterogeneity by page age
0.053
0.676
-0.158(0.062)(2.302)(0.807)-0.084-3.234-1.328(0.121)(4.459)(1.564)
0.038
1.011
0.458
(0.103)(3.816)(1.339)
Yes
Yes
Yes
Yes
Yes
Yes
0.190
6.589
2.032
0.353
11.958
4.409
0.250
0.105
0.190
180
180
180

Notes: Each column presents estimates from a separate cross-section regression of 180 Wikipedia pages.
The dependent variable is the change in logarithm of page length (column 1), change in the overall quality
rating (column 2), and the change in the similarity to the corresponding Spanish Wikipedia article (column
3). All regressions include language fixed effects and city fixed effects. An indicator for treatment group isinteracted with an indicator whether either page pre-treatment length (panel A) or age (panel B) is below
the median. Standard errors are reported in parentheses.

A18

Online Appendix Table C.6: The effect of treatment on subsequent growth in page lengthand quality, p-values adjusted for multiple hypothesis testing
Outcome variable∆ log. length∆ quality∆ similarity∆ complete∆ interesting∆ illustrated∆ well-written

Coef.
(1)
0.026
-0.375-0.590-0.967-0.150-2.037
4.658

SE
(2)
0.048
1.777
0.624
2.254
1.851
3.752
5.980

Unadj. p-value
(3)
0.586
0.833
0.346
0.669
0.936
0.588
0.438

Adj. p-value
(4)
0.974
0.974
0.952
0.974
0.974
0.974
0.970

Notes: Westfall and Young (1993) multiple hypothesis p-value adjustment as implemented by Jones et al.
(2019) employing 10,000 bootstrap draws.

Online Appendix Table C.7: Short-term effects of treatment on subsequent editing activity,
p-values adjusted for multiple hypothesis testing
Outcome variable# users: year 1# users: year 2# edits: year 1# edits: year 2# edits excl. treatment: year 1# edits excl. treatment: year 2
Edit distance: year 1
Edit distance: year 2
Capped edit distance: year 1
Capped edit distance: year 2

Coef.
(1)
0.122
0.119
0.138
0.140
0.022
0.072
12.806100.284
17.28220.512

St.er.
(2)
0.030
0.035
0.032
0.043
0.030
0.043
31.301105.570
5.609
8.654

Unadj. p-value
(3)
0.000
0.001
0.000
0.002
0.462
0.099
0.683
0.344
0.003
0.019

Adj. p-value
(4)
0.310
0.385
0.294
0.406
0.873
0.668
0.873
0.873
0.420
0.511

Notes: Westfall and Young (1993) multiple hypothesis p-value adjustment as implemented by Jones et al.
(2019) employing 10,000 bootstrap draws.

A19

Online Appendix Table C.8: Long-term effects of treatment on subsequent editing activity,
p-values adjusted for multiple hypothesis testing
Outcome variable# users: year 3# users: year 4# edits: year 3# edits: year 4# edits excl. treatment: year 3# edits excl. treatment: year 4
Edit distance: year 3
Edit distance: year 4
Capped edit distance: year 3
Capped edit distance: year 4

Coef.
(1)
0.014
0.024
0.011
0.029
-0.035-0.026-13.534
1.296-4.1580.485

St.er.
(2)
0.028
0.033
0.032
0.036
0.031
0.033
21.69279.2655.294
5.343

Unadj. p-value
(3)
0.621
0.480
0.726
0.424
0.269
0.443
0.534
0.987
0.434
0.928

Adj. p-value
(4)
0.983
0.982
0.988
0.979
0.910
0.982
0.983
0.996
0.982
0.996

Notes: Westfall and Young (1993) multiple hypothesis p-value adjustment as implemented by Jones et al.
(2019) employing 10,000 bootstrap draws.

A20