Pairing Wikipedia Articles Across Languages
Pierre Nugues
Lund University
Department of Computer Science
Lund, Sweden
Pierre.Nugues@cs.lth.se

Marcus Klang
Lund University
Department of Computer Science
Lund, Sweden
Marcus.Klang@cs.lth.se

Abstract
Wikipedia has become a reference knowledge source for scores of NLP applications. One of
its invaluable features lies in its multilingual nature, where articles on a same entity or concept
can have from one to more than 200 different versions. The interlinking of language versionsin Wikipedia has undergone a major renewal with the advent of Wikidata, a unified scheme toidentify entities and their properties using unique numbers. However, as the interlinking is still
manually carried out by thousands of editors across the globe, errors may creep in the assignmentof entities. In this paper, we describe an optimization technique to match automatically language
versions of articles, and hence entities, that is only based on bags of words and anchors. We
created a dataset of all the articles on persons we extracted from Wikipedia in six languages:
English, French, German, Russian, Spanish, and Swedish. We report a correct match of at least
94.3% on each pair.

1Introduction

Wikipedia has become a major reference knowledge source for applications such as IBM Watson (Ferrucci, 2012) or Google’s knowledge graph (Singhal, 2012). Wikipedia is available in more than 280
languages such as Indonesian, Amharic, Nahuatl, or Hindi and its content and coverage are continuously
growing thanks to millions of contributors.
One of the major steps to organize the linguistic diversity of Wikipedia has been the creation of Wikidata: A centralized repository of entities and concepts identified by unique numbers. Before Wikidata,an editor creating an article, say on Madagascar in Spanish, had to link it manually to already existingversions of the same entity. Now, Wikidata stores links to all the versions in a centralized repository(Fig. 1) and adding a new language is carried out through this repository and a unique number: Q1019
in the case of Madagascar. In addition to associating unique identifiers to entities, Wikidata uses a set ofabout 2,500 properties, as of June 1, 2016, to describe them. One of these properties is instance of, P31,
that enables the editors to define an ontology. Madagascar, for example, is an instance of a sovereignstate, an island, an island nation, a country, and a member state of the United Nations (Fig. 1).
Although Wikidata has simplified the linking process, it is still a manual operation that is not errorfree and articles may be incorrectly linked across the languages. In this paper, we report and evaluate
a technique that, given a human entity, automatically identifies the set of articles describing this entity
across six Wikipedia languages: English, French, German, Russian, Spanish, and Swedish. We report acorrect match of at least 94.7% for each pair selected among the six versions.

2Previous Work

Comparable corpora, like the language versions of Wikipedia, have been used extensively as resources
to extract word translations or parallel sentences. Rapp et al. (2012) for instance, used Wikipedia articles in nine languages to identify word translations through keywords and a word alignment algorithm.
Schamoni et al. (2014) proposed to use links to retrieve Wikipedia articles in English similar to an article
This work is licensed under a Creative Commons Attribution 4.0 International License.
creativecommons.org/licenses/by/4.0/

License details: http://72
Proceedings of the Open Knowledge Base and Question Answering (OKBQA) Workshop,
pages 72–76, Osaka, Japan, December 11 2016.

Figure 1: Left part: The first language versions of Madagascar in Wikidata. The languages appear
in alphabetic order out of 195. Right part: Membership of Madagascar to ontology classes using theinstance of property; three classes are listed out of fivein German. Domínguez García et al. (2012) extracted hyponymy relations from the Wikipedia categorysystem across languages. Chiao and Zweigenbaum (2002) used a set of documents in French and English collected with medical taxonomy terms to produce translational equivalents. Sproat et al. (2006)
used English and Chinese stories from the Xinhua News agency to identify named entity transliterations.
Finally, Smith et al. (2010) improved translation performance using sets of parallel sentences that they
extracted from Wikipedia.
Although these works carry out some kind of matching across languages, we could not find references
on a systematic attempt to pair descriptions of an entity in multiple language versions. To the best of ourknowledge, we are the first to propose and evaluate a method in this field.

3Collecting the Corpus

We used six dumps of Wikipedia in English, French, German, Russian, Spanish, and Swedish1 fromwhich we extracted all the persons. We carried out this extraction using the instance of property, where
we collected all the articles that had a direct link to the node denoting a human entity (Q5 in Wikidata).
Table 1, left part, shows the counts of articles on persons broken down per language.
From this person data set, we extracted all the articles having the six language versions. Again, weused the Wikidata identifier to determine the available language versions of an entity. We obtained a totalof 1,938,861 unique persons having at least one version in one of the six languages and where 39,636
had versions in the six languages (Table 1, right part).
Language
endefrCount
1,257,604
579,656446,308

Languageruessv

Count
297,202262,538
178,894Versions
654Count
39,63642,98665,725

Versions
32
1Count115,692
284,658
1,390,164

Table 1: Left part: Counts of articles on persons per language version. Right part: Number of versions
for the articles on persons

4Method

To implement the comparison method, we restricted the articles to their first paragraphs that we represented as bags of words and entity identifiers (Wikidata Q-number). For a given pair of languages, wethen determined the best pair of paragraphs using the cosine similarity.
The articles on persons in Wikipedia show a similar structure, where the first paragraph starts with thename of the person and reports a few basic facts such as the dates and places of birth and death using
1Retrieved in May and September 2015.

73

numbers and proper nouns. In languages using the Gregorian calendar, the numbers are often the sameacross the versions and many proper nouns also have identical forms and spellings.
For instance, the first paragraph on Shakespeare in the French Wikipedia,
William Shakespeare, né probablement le 26 avril 1564 à Stratford-upon-Avon et mort le 3mai (23 avril) 1616 dans la même ville, [...] à représenter les aspects de la nature humaine.
shares seven words with the corresponding paragraph in English:
William Shakespeare (/"SeIkspI@r/; 26 April 1564 (baptised) – 23 April 1616) was an English
poet, playwright, and actor, [...] and the “Bard of Avon” [...] than those of any other playwright.
while it does not share a single word with Napoléon’s one:
Napoléon Bonaparte (/n@"poUli@n, -"poUlj@n/; French: [napOleÕ bOnapaKt], born Napoleone diBuonaparte; 15 August 1769 – 5 May 1821) was a French military and political leader [...]
Napoleon implemented foundational liberal reforms in France and throughout Europe. [...]We represented each article as a bag of words of its first paragraph with the vector space model (Saltonet al., 1974). In addition to the words, we used resolved anchors from the first paragraph with their Qnumbers as terms. We extracted all the unique tokens and Q-numbers from all the documents from which
we excluded 250 stop words that we defined as the words that occur in the highest number of documents
Nacross the four versions. We used this variant of T F · IDF : weight(term) = tf (term, d) · log df (term,D),
where tf is the term frequency in the current document d (a paragraph); df is the document frequency,that is the number of paragraphs D that contain this term; N is the total number of documents; in ourcase, the total number of paragraphs.
Given a pair of languages, the two sets of articles in their respective languages and their association
form a weighted bipartite graph, where the comparison (matching) step can be formulated as a linear
assignment problem (Jacobi, 1865; Kuhn, 1955; Jonker and Volgenant, 1987). The worst case of computing the weight matrix involves O(N 2 ) operations, which for our dataset corresponds to 1.57 billion
operations, while the assignment problem has a O(N 3 ) worst case complexity. This figure is still in therealm of feasibility, but could quickly get worse with more categories. Fortunately, our comparisons involve pairs that typically share few terms, and most often none at all. In this case, their cosine similarity
is 0. In our data set, 95% of the matrix elements are zero, which makes the computation of an optimal
solution tractable using a sparse linear assignment algorithm.

5

Exploratory Analysis

Taking advantage of the sparsity, we conducted an exploratory analysis and a preliminary evaluation.
We compared paragraphs that shared at least one term and we implemented a simplified assignment
algorithm that reduced drastically the number of operations. Given a pair of languages, the source andthe target, we compared each document from the source with all the target documents that shared at least
one term with it and we assigned the target document that had the maximal similarity. This simplified
assignment corresponds to the initial cover in Kuhn (1955).
We applied the comparison algorithm to the set of articles. For all the language pairs, the number of
misclassified articles is less than 17.0%, a surprisingly low figure for such a simple method. Table 2
shows the results, where the most confused language pair is French–English and the less one, Swedish–German.
We also examined the influence of the cosine similarity on the method using the precision and recallscores. We applied a cutoff to this similarity to validate a pair that we varied between 0 and 1. Figure 2
shows the recall and precision on the Swedish–French pair with respect to this cutoff. All the otherpairs show a similar pattern. A cutoff of 0 always selects the highest cosine similarity whatever its value,
while a 1 will request the paragraphs to have exactly the same words. We can see that a very high recall isreached without cutoff, while the precision is moderately improved by it. A perfect precision is reachedwhen the cosine similarity is greater than 0.76, while a high cutoff discards all the pairs.
74

1.0

recallprecision
f1

0.8

deen
es
fr
ru
sv

f1

0.6
0.4
0.2
0.0
0.0

0.2

0.4

cutoff

0.6

0.8

1.0

Figure 2: Recall and precision with respect to thecutoff for the Swedish–English pair

deen
es
fr
ru
sv

de–
2.0
2.0
1.7
3.1
0.9

en
2.0
–
3.1
3.2
5.0
2.6

es
1.9
3.2
–
2.6
4.8
2.4

de–
11.8
6.0
5.3
7.1
5.0

en
15.5
–
14.5
17.0
14.8
16.4

es
8.6
13.6–
9.9
11.4
10.0

fr
6.6
14.8
8.4
–
10.3
8.7

ru
9.7
14.9
12.8
12.7
–
13.4

sv
6.3
13.9
8.4
8.7
10.7
–

Table 2: Percentages of misclassified pairs using
the naïve method. First column: source language;
first row: target language
fr
1.7
3.2
2.6
–
4.6
2.4

ru
3.1
5.0
4.8
4.7
–
4.9

sv
1.0
2.6
2.4
2.4
5.7
–

avg
1.9
3.2
3.0
2.9
4.7
2.7

Table 3: Percentages of misclassified pairs using the linear assignment algorithm.

6Linear Assignment

Finally, we applied the sparse linear assignment algorithm by Jonker and Volgenant (1987) to all the pairs
in our data set using a modified version of the Fiji library from Schindelin et al. (2012). For each pair, weexcluded the documents that shared no word and we applied the algorithm to the resulting matrix. In spiteof the matrix sizes (nearly 40, 000 × 40, 000) and the algorithm complexity, the run time to compute anassignment matrix takes less than two hours on Intel Xeon desktop computer. Table 3 shows the results,
where the most confused language pair is Russian–Swedish and the less one, Swedish–German.

7Discussion and Conclusion

The matching method we proposed shows it could reach a high accuracy with a standard bag-of-word
technique, including words and entity identifiers, even with the simplified assignment method. This surprisingly high accuracy is due to the similar structure adopted by most articles on persons in their firstparagraph. The proper nouns and the dates this paragraph contains proved to be sufficiently discriminative to have error rates less than 5.7% across the languages.
We applied this method to languages having the highest number of views per hour on Wikipedia2 ,English and Spanish, as well as French and Swedish, that show no or little proper noun inflection, andhence where the proper nouns are identical across the versions. Nonetheless, the errors we obtained withRussian, if larger, are still comparable to those we got with the other languages: The pair (en, fr) shows
an error of 3.2%, while (en, ru) is of 5.0%, for instance. Such results can be explained by the Q-numbers
appearing in the bag-of-word vectors that are shared by the languages of a pair, whatever the script ormorphology. They suggest this method is applicable to nonLatin scripts or to languages with a richer
inflection. We believe this technique paves the way for an automatic matching of comparable textual
resources across languages as well as interactive tools to support the creation and linking of new articles.

Acknowledgments
This research was supported by Vetenskapsrådet, the Swedish research council, under the Det digitaliserade samhället program.
2

http://stats.wikimedia.org/75

References
Yun-Chuang Chiao and Pierre Zweigenbaum. 2002. Looking for candidate translational equivalents in specialized, comparable corpora. In Proceedings of the 19th International Conference on Computational Linguistics Volume 2, COLING ’02, pages 1–5.
Renato Domínguez García, Sebastian Schmidt, Christoph Rensing, and Ralf Steinmetz, 2012. ComputationalLinguistics and Intelligent Text Processing: 13th International Conference, CICLing 2012, New Delhi, India,March 11-17, 2012, Proceedings, Part I, chapter Automatic Taxonomy Extraction in Different Languages Using
Wikipedia and Minimal Language-Specific Information, pages 42–53. Springer Berlin Heidelberg, Berlin,Heidelberg.
David Angelo Ferrucci. 2012. Introduction to “This is Watson”. IBM Journal of Research and Development,
56(3.4):1:1 –1:15, May-June.
Carl Jacobi. 1865. De investigando ordine systematis aequationum differentialum vulgarium cujuscunque. Journal für die reine und angewandte Mathematik, 64:297–320.
Roy Jonker and Anton Volgenant. 1987. A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing, 38(4):325–340.
Harold W. Kuhn. 1955. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly,
2:83–97.
Reinhard Rapp, Serge Sharoff, and Bogdan Babych. 2012. Identifying word translations from comparable documents without a seed lexicon. In Proceedings of the Eighth International Conference on Language Resourcesand Evaluation (LREC-2012), pages 460–466, Istanbul, Turkey, May.
Gerard Salton, A. Wong, and C. S. Yang. 1974. A vector space model for automatic indexing. Technical Report
TR74-218, Department of Computer Science, Cornell University, Ithaca, New York.
Shigehiko Schamoni, Felix Hieber, Artem Sokolov, and Stefan Riezler. 2014. Learning translational andknowledge-based similarities from relevance rankings for cross-language retrieval. In Proceedings of the 52nd
Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 488–494,Baltimore, Maryland, June. Association for Computational Linguistics.
Johannes Schindelin, Ignacio Arganda-Carreras, Erwin Frise, Verena Kaynig, Mark Longair, Tobias Pietzsch,
Stephan Preibisch, Curtis Rueden, Stephan Saalfeld, Benjamin Schmid, Jean-Yves Tinevez, Daniel JamesWhite, Volker Hartenstein, Kevin Eliceiri, Pavel Tomancak, and Albert Cardona. 2012. Fiji: an open-source
platform for biological-image analysis. Nature Methods, 9(7):676–682, 07.
Amit Singhal. 2012. Introducing the knowledge graph: things, not strings. Official Google Blog. http://googleblog.blogspot.com/2012/05/introducing-knowledge-graph-things-not.
html, May.
Jason R. Smith, Chris Quirk, and Kristina Toutanova. 2010. Extracting parallel sentences from comparable
corpora using document level alignment. In Human Language Technologies: The 2010 Annual Conference ofthe North American Chapter of the Association for Computational Linguistics, HLT ’10, pages 403–411.
Richard Sproat, Tao Tao, and ChengXiang Zhai. 2006. Named entity transliteration with comparable corpora. InProceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of theAssociation for Computational Linguistics, pages 73–80, Sydney, Australia, July.

76