DEPTH: An Enhanced Depth Metric for Wikipedia Corpora Quality
Year: 2023

Abstract:
Wikipedia articles are a common source of training data for Natural Language Processing (NLP) research, especially as a source for corpora in languages other than English. However, research has shown that not all Wikipedia editions are produced organically by native speakers, and there are substantial levels of automation and translation activities in the Wikipedia project that could negatively impact the degree to which they truly represent the language and the culture of native speakers. To encourage transparency in the Wikipedia project, Wikimedia Foundation introduced the depth metric as an indication of the degree of collaboration or how frequently users edit a Wikipedia edition’s articles. While a promising start, this depth metric suffers from a few serious problems, like a lack of adequate handling of inflation of edits metric and a lack of full utilization of users-related metrics. In this paper, we propose the DEPTH metric, provide its mathematical definitions, and describe how it reflects a better representation of the depth of human collaborativeness. We also quantify the bot activities in Wikipedia and offer a bot-free depth metric after the removal of the bot-created articles and the bot-made edits on the Wikipedia articles.

heading: 1 Introduction
text: The Wikipedia project is a free online encyclopedia that aims to enable and involve people all over the globe in creating and disseminating knowledge. Wikipedia articles, i.e., content pages of Wikipedia, are also a common source of training data for Natural Language Processing (NLP) research, especially as a source for corpora in languages other than English. In particular, Wikipedia articles are used to train many Large Language Models (LLMs), such as ELMo (Embeddings from Language Models), which has been trained on the English Wikipedia and news crawl data (Peters et al., 2018); BERT (Bidirectional Encoder Representations from Transformers) has been trained on books with a crawl
The arrows in the 3rd and 4th columns indicate the changes in the rankings of editions when depth and DEPTH+ are compared to the global rank, and the arrows in the 5th column indicate the changes in rankings when DEPTH+ and depth are compared head-to-head.
of the English Wikipedia articles (Devlin et al., 2018; Petroni et al., 2019); GPT-3 (Generative Pre-trained Transformer) has also been trained on five large datasets including the English Wikipedia (Brown et al., 2020); LaMDA (Language Model for Dialogue Applications) and PaLM (Pathways Language Model) were trained on a huge mixed dataset that includes Wikipedia articles, news articles, source code, and social media conversations (Thoppilan et al., 2022; Chowdhery et al., 2022); and LLaMA (Large Language Model Meta AI) was also pre-trained on the multilingual articles of Wikipedia from June to August 2022, covering 20 languages with a percentage of 4.5% of its overall training dataset size (Touvron et al., 2023).
Wikipedia corpora (editions) exist for more than 300 of the over 7,000 languages spoken worldwide.
1The global rank of Wikipedia editions is calculated using the total number of articles (content pages) (Wikipedia, 2023a). See Appendix A for the full list.
175
the depth values with “– –” for languages when the number of articles < 100,000, and the depth metric value > 300.
However, these corpora vary substantially in size and quality, and the Wikipedia project provides a rich set of metadata and metrics to help users compare the different corpora. Table 1 includes examples of some of these metrics across ten languages, including the number of articles, the number of non-articles (e.g., user pages, redirects, images, project pages, templates, and support pages), the total number of pages (articles and non-articles), the total number of edits, the number of users, the number of active users, and the number of admins. The difference between users and active users is that users refer to the number of user accounts regardless of current activity, whereas active users refer to registered users who have made at least one edit in the last thirty days (Wikipedia, 2023a).
In this paper, we will use the 320 open Wikipedia corpora available today, as listed in the appendices. We will not include the 13 closed Wikipedia editions (Afar, Northern Luri, Marshallese, Ndonga, Choctaw, Kwanyama, Herero, Hiri Motu, Kanuri, Muscogee, Sichuan Yi, Akan, and Nauruan). Closed editions are read-only, meaning registered users can no longer edit any content pages (Wikipedia, 2023a; Wikimedia Commons, 2023; Wikimedia Meta-Wiki, 2023). Since articles in closed editions can no longer be edited, the active users metric drops to zero because it only counts users active in the last 30 days2. The last three columns of Table 1 contain filtered depth metric (as the Wikipedia project does it), unfiltered depth metric (as we used to sort the table), and the new DEPTH+ metric we are proposing in this paper. The current general formula of the depth metric used by Wikipedia is defined as the following:
Depth = Edits ·NonArticles Articles2 · ( 1− ArticlesTotal ) (1)
2We would love to see the Wikimedia Foundation, in its Wikipedia project, maintain and report a count of the number of users who have ever made an edit in corpora (edition) rather than only reporting on the last 30 days. Such a metric would continue to be relevant even for closed editions.
The Wikimedia Foundation introduced the depth metric as an indicator of Wikipedia’s collaborative quality to show how frequently a Wikipedia edition’s articles are edited or updated by the users and is intended to indicate the depth of collaboration among contributors to corpora. The first variant of depth metric was added to the Wikipedia project in 2006, using only the first factor, the total number of edits divided by the number of articles. After that, the Wikipedia project added an additional factor of non-articles divided by articles. In 2007, the depth metric was again updated to add the third factor, the stub ratio, or one minus the articles divided by the total pages (Wikimedia Foundation, 2023c).
In this paper, we aim to explore the limitations of the depth metric and propose a new enhanced depth metric, DEPTH+, to address these limitations. Figure 1 previews a comparison of Wikipedia’s unfiltered depth metric and our DEPTH+ metric for the top ten Wikipedia editions based on global rank (i.e., the total number of articles).
We observe that not all Wikipedia editions are produced organically by native speakers, and a substantial level of automation and translation is often used, which can negatively affect the integrity and trustworthiness of these articles. For example, Alshahrani et al. (2022) studied the Arabic Wikipedia editions (Modern Standard Arabic, Egyptian Arabic, and Moroccan Arabic) and found that more than one million articles have been shallowly translated from English using either direct translation or template-based translation (by one registered user) in the Egyptian Arabic Wikipedia edition. Unsurprisingly, some of these top ten Wikipedia editions, in Table 1, are mostly botgenerated, auto-translated, or even small enough not to be considered a common Wikipedia edition (Wikipedia, 2023a; Wikimedia Foundation, 2023a). We found that in the Vietnamese and Serbo-Croatian Wikipedia editions more than 58% and 55% of their articles are bot-created, respec-
tively (Wikipedia, 2023a; Wikimedia Foundation, 2019, 2023d). While automation and translation activities are not always problematic, we argue that metrics like the depth that do not distinguish between organic content generated by native speakers and bot-generated content can be a misleading indicator of the collaboration and richness in a dataset.
Section 2 examines the current depth metric used in Wikipedia, rewrites its mathematical representations, and underscores its limitations. In Section 3, the paper quantifies the bot activities within the Wikipedia project. Section 4 introduces a new metric called DEPTH+, presents its mathematical definitions, and highlights its features. We shed light on the limitations of our work in Section 5. Lastly, Sections 6 and 7 briefly discuss related work, provide a concise conclusion, and offer a few future research ideas.

heading: 2 Depth Metric
text: The Wikipedia depth metric is currently reported in two places: List of Wikipedias (Wikipedia, 2023a) and Wikipedia Article Depth (Wikimedia Foundation, 2023c). Notably, the Wikipedia project filters the calculations of this depth metric and reports depth values only for the Wikipedia editions with more than 100,000 articles. If a Wikipedia edition has a depth value > 300 and the total number of articles < 100,000, then the depth metric value is arbitrarily replaced by “– –”. This has the side effect of placing the English Wikipedia edition at the top of Wikipedia’s ranking by depth metric. To better understand how the depth metric behaves, we manually calculate and report unfiltered depth metric values of all Wikipedia editions.
Returning to Table 1, the set of languages displayed shows the top ten Wikipedia editions ordered by the depth metric without filtering. We can see that most of the listed Wikipedia editions are small corpora. It is notable that English, the largest and oldest of the editions, is widely believed to have the most collaborative editing, but it only comes in third. Notably, only half of these ten editions (English, Ripuarian, SerboCroatian, Vietnamese, and Bihari) would remain after Wikipedia’s filtering. The other half would have been given high depth values without filtering using ad-hoc limits, suggesting that the current depth metric may not truly reflect the collaborative quality of corpora. To expand on Table 1, we plotted the highest 50 Wikipedia editions ordered by
the depth metric values in Figure 2. Once again, most Wikipedia editions in the highest ranks are counterintuitively small or uncommon languages, while large corpora, such as French (fr), Spanish (es), and Italian (it), all widely believed to have substantial collaborative editing, appear late in the ranking. Overall, this observation motivated our quest for an improved depth metric that would not require ad-hoc filtering.
In the following subsections, we discuss the formulas of the depth metric, rewrite its mathematical representations, and explain some of its limitations.

heading: 2.1 Formulas of Depth Metric
text: The Wikimedia Foundation, in its Wikipedia project, introduces two mathematical formulas for the depth metric that are written in high-level quantitative terms (Wikimedia Foundation, 2023c). In this work, we rewrite these mathematical definitions of the depth metric in detailed formal mathematical representations.
We have already seen one formula for the depth metric in Equation 1. That version emphasizes the three factors added by the Wikipedia project over time. After some simple algebraic transformations, there is an alternate version, Equation 2. It may not be immediately obvious that Equation 2 is equivalent to Equation 1, but for reference, we have provided the full derivation of Equation 2 in Supplementary Section 8.
Depth = EditsTotal · ( NonArticles Articles )2 (2)
Let Wi represent all Wikipedia editions where i = {1, 2, 3, . . . , 320} (As noted earlier, we are not including the 13 closed editions). Let the total number of edits ofWi be EWi where e = {1, 2, 3, . . . , n}, let the total number of articles of Wi be AWi where a = {1, 2, 3, . . . , n}, let the total number of non-articles ofWi be RWi where r = {1, 2, 3, . . . , n}, and lastly, let the total number of pages of Wi be TWi where TWi = AWi +RWi . Therefore, our rewrite, using the mathematical representations, of the general mathematical definition of the depth metric ofWi is described as follows:
DWi = EWi · RWi A2Wi · ( 1− AWiTWi ) (3)
3We changed a few Wikipedia language codes for the sake of data visualization in some figures and tables, such as: . Tarantino: (roa-tara) → (tar). . Aromanian: (roa-rup) → (roa). . Southern Min: (zh-min-nan) → (zhm). . Classical Chinese: (zh-classical) → (zhc).
Wikipedia since it is the largest Wikipedia edition. We can see that most languages in the highest ranks are either small or uncommon. See Appendix B for the full list.
The rewrite of the simplified mathematical definition of the depth metric of Wi is also described using the mathematical representations as follows:
DWi = EWi TWi · (RWi AWi )2 (4)

heading: 2.2 Problems of Depth Metric
text: Having presented the detailed formulas for the depth metric, in this section, we now discuss its key limitations.

heading: 2.2.1 Depth Metric is Bot-influenced
text: The current depth metric is misleading because it measures the total activity on the Wikipedia project, which includes bot and automation activities, instead of solely measuring the human activities, interactions, and collaborations on the project. While not all automated activities are problematic, they provide a misleading sense of the level of collaboration which is one of the stated functions of the depth metric. As an example, the bot-made edits undoubtedly maximize the measurements of the edits metric, causing incorrect calculations of the depth metric. For instance, we found that in the Serbo-Croatian and Inuktitut Wikipedia editions more than 41% and 39% of the total edits on their articles are bot-made, respectively (Wikipedia, 2023a; Wikimedia Foundation, 2019, 2023d).
Furthermore, the current depth metric considers the non-articles in Equations 1 and 2, mostly user pages, redirects, project pages, templates, and discussion pages that are not directly correlated to human activities on Wikipedia articles. Although the users or admins could discuss the contents of articles on their pages (forums), these discussions are not included in the content pages and are not counted toward human activities on those pages.

heading: 2.2.2 Depth Metric is Easy-inflatable
text: The depth metric uses the edits metric as one of the fundamental metrics on which the depth measurements rely. Yet, editing wars in the Wikipedia project inflate this metric of edits, causing inaccurate measurements of the depth metric, even though editing wars are a normal part of Wikipedia’s life that is sometimes hard to control (Wikimedia Foundation, 2023b). As an example of the editing wars, in late July 2022, the Wikipedia project locked the English Wikipedia page about the “recession” and set restrictions on who could edit this page. The freeze was set after a lot of editors made a series of revisions to the definition of “recession” (National Public Radio (NPR), 2022).

heading: 2.2.3 Depth Metric Misses User Activity
text: The depth metric only utilizes a few already calculated metrics by the Wikipedia project, such as articles, non-articles, total pages, and edits, but it does not take advantage of any other metrics related to users of any type, like users, admins, and active users. These user-related metrics already exist and have been calculated by the Wikipedia project for almost all editions (Wikipedia, 2023a). We believe utilizing more metrics could give us insights into the collaborative quality of the Wikipedia editions.

heading: 3 Quantification of Bot Activities
text: The Wikimedia Foundation, in its Wikipedia project, permits users or editors to use bots (software programs) to automate repetitive and everyday tasks in many Wikipedia editions (Wikipedia, 2023d, 2022). The only advantage of Wikipedia bots is to make edits rapidly, yet they can disrupt the Wikipedia project if they are incorrectly designed or operated without approval. For these
reasons, Wikipedia bot policy has been developed and enforced (Wikipedia, 2023c). However, these Wikipedia bots in the past years noticeably are not used only to commit edits but also to create articles on the Wikipedia project, which often produces unrepresentative, inorganic content that does not echo the complex structure of the human languages, does not express the views of the native speakers of those languages, and does not represent the cultural richness and historical heritage of those languages and their people (Alshahrani et al., 2022). As an example of Wikipedia bots, the “Lsjbot” bot is responsible for creating more than 6 million articles (99.61%) in the Cebuano Wikipedia edition, one million articles (90%) in the Waray Wikipedia edition, and one million articles (68%) in the Swedish Wikipedia edition (Popular Science, 2014; Wikimedia Foundation, 2019; Wikipedia, 2023b).
We discuss the quantification and clear labeling of bot-generated Wikipedia articles and bot-made edits on these articles in different Wikipedia editions. If bot-generated content was clearly labeled, it could be included where helpful or ignored when it is not. For instance, if an NLP task involves measuring the opinions or biases of native speakers, including content that has been translated from another language is likely to reflect the opinions or biases of the authors of the original text from which it was translated.

heading: 3.1 Bot-generated Articles
text: To quantify the bot-generated articles in all Wikipedia editions, we used the online Wikimedia Statistics4 service (https://stats.wikimedia.org) to collect the total number of bot-created articles. Specifically, we collected the statistics of the new content pages (articles) that are created by both group-bots (logged-in registered users who are part of a bot group) and name-bots (logged-in registered users whose name contains ‘bot’) (Wikimedia Foundation, 2023d). Next, we summed these totals of the bot-generated articles for each Wikipedia edition and subtracted them from the already calculated metrics: articles and total pages by the Wikipedia project to ultimately have a bot-free depth metric.
Table 2 shows the top ten Wikipedia editions that have the most bot-created articles in the Wikipedia project, ordered by the percentage of how much
4We took a data snapshot of all Wikipedia editions’ statistics on the 31st of March, 2023, using the online Wikimedia Statistics service (Wikimedia Foundation, 2023d).
most bot-created articles, ordered by the percentage of how much bot automation each Wikipedia edition has. We highlighted the Cebuano Wikipedia edition since it comes second in the global rank and has the highest number of bot-generated articles (content pages). See Appendix C for the full list.
bot automation each Wikipedia edition has. We can see that the Cebuano Wikipedia edition—the second Wikipedia edition in the globe rank in terms of the total number of articles has 99.61% of its total number of articles are bot-generated.

heading: 3.2 Bot-made Edits on Articles
text: With the same aim as above, we want to quantify and eliminate the bot-made edits on Wikipedia articles in all Wikipedia editions. We used the online Wikimedia Statistics service to collect the total number of bot-made edits on articles (content pages). Particularly, we collected the statistics of the made edits on the articles that were done by both group-bots and name-bots (Wikimedia Foundation, 2023d). After that, we summed these totals of the bot-made edits for each Wikipedia edition and subtracted them from the existing edits metric by the Wikipedia project to eventually have a bot-free depth metric.
Table 3 shows the top ten Wikipedia editions with the most bot-made edits on their articles in the Wikipedia project, ordered by the percentage of bot automation each Wikipedia edition has. It is clear the Cebuano Wikipedia edition—the second Wikipedia edition in the globe rank in terms of the total number of articles has 94.05% of its total number of edits on its articles (content pages) are bot-made edits.
4 DEPTH+ Metric
The depth metric is a useful indicator of Wikipedia’s collaborative quality, which reflects how frequently a Wikipedia edition’s articles are edited or updated by users (Wikimedia Foundation, 2023c). However, we believe the depth metric must be enhanced to solve some of the limitations spotlighted in this study.
most bot-made edits on their articles, ordered by the percentage of how much bot automation each Wikipedia edition has. We highlighted the Cebuano Wikipedia edition since it comes second in the global rank and has the highest bot-made edits on its articles (content pages). See Appendix D for the full list.
In the following subsections, we revise the original depth definitions after quantifying and removing bot activities, propose the DEPTH+ metric as an enhanced depth metric for Wikipedia corpora quality, mathematically define its definitions, and highlight its key features.

heading: 4.1 Revision of Depth Definitions
text: To better reflect true collaborative activities in the DEPTH+ metric, we will first remove the botcreated Wikipedia articles and the bot-made edits on the Wikipedia articles from the depth metric. We revisit the mathematical definitions of the depth metric and redefine the related metrics: edits, articles, and total pages accordingly.
Let all Wikipedia editions beWi, let the total number of edits ofWi be EWi , let the total number of bot-made edits ofWi be EbWi where eb = {1, 2, 3, . . . , n}, let the total number of articles of Wi be AWi , let the total number of bot-created articles of Wi beAbWi where ab = {1, 2, 3, . . . , n}, let the total number of non-articles ofWi be RWi , and lastly, let the total number of pages ofWi be TWi . Therefore, the updated mathematical definitions of these metrics: edits, articles, and total pages of Wi using the mathematical representations after removing the bot activities are defined as follows:
EWi = EWi − EbWi (5)
AWi = AWi −AbWi (6)
TWi = (AWi−AbWi) +RWi (7)
4.2 Formulas of DEPTH+ Metric
We understand that (NonArticlesArticles ) from Equations 1 and 2 are to emphasize that the article count of a Wikipedia edition is just the tip of the ice-
berg, and other metrics, such as user pages, project pages, and discussion pages, are crucial indicators of “Wikipedianness” and the ( EditsArticles) from Equations 1 and 2 are also to emphasize that some Wikipedia editions might only include some copied and pasted articles or articles written by only one single registered user (which does not necessarily mean they are biased, but surely means they are not collaboratively edited, i.e., “Wikipedian”) (Wikimedia Foundation, 2023b).
However, we propose a few significant additions to the depth metric’s formulas. We first add a few available user-related metrics, like users, admins, and active users, to the DEPTH+ metric and call them the “editors” metric. The difference between users and active users is that users refer to the number of user accounts regardless of current activity, whereas active users refer to registered users who have made at least one edit in the last thirty days (Wikipedia, 2023a). We add the active users over the users to normalize the measurements of the DEPTH+ metric and add the admins as a constraint that gives the large Wikipedia editions higher priority, assuming that the larger the Wikipedia edition, the greater the number of admins.
The formula of the “editors” metric is defined as:
Editors = Admins · ActiveUsersUsers (8) Secondly, we propose a few meaningful modifications to the depth metric’s formulas, where we eliminate the square power of the depth simplified equation (in bold), Equation 2, (NonArticlesArticles )
2, because the square power will double the depth metric measurements, and we prefer to keep the DEPTH+
metric values relatively small. We also eliminate the subtraction part of the stub ratio (in bold) from Equation 1, (1–ArticlesTotal ), because it was added to decrease the results of the stub ratio in 2007 (Wikimedia Foundation, 2023a), but now, it is irrelevant since we added the active users over the users to normalize the measurements of the DEPTH+ metric and added the admins metric as a constraint to give large Wikipedia editions higher priority.
The DEPTH+ metric is finally defined by combining the above modifications on Equations 1 and 2 with Equation 8 of the “editors” metric and inserting the revised mathematical definitions of metrics: edits, articles, and total pages from Equations 5, 6, and 7 to exclude the bot activities, as the following:
DEPTH+ = Editors · Edits·NonArticles Articles2 ·ArticlesTotal (9)
We highlighted English Wikipedia since it is the largest Wikipedia edition. We can see that most languages in the highest ranks are either large or common Wikipedia editions. See Appendix E for the full list.
The DEPTH+ metric can rearrange to a simplified equivalent formula as the following:
DEPTH+ = Editors · EditsTotal · NonArticlesArticles (10)
Let all Wikipedia editions beWi where i = {1, 2, 3, . . . , 320} for the 320 open editions, let the total number of admins ofWi beMWi where m = {1, 2, 3, . . . , n}, let the total number of active users of Wi be VWi where v = {1, 2, 3, . . . , n}, let the total number of users ofWi be UWi where u = {1, 2, 3, . . . , n}, and lastly, let the “editors” ofWi be OWi .
Therefore, the mathematical definition of the “editors” metric ofWi using the mathematical representations is described as the following:
OWi =MWi · VWi UWi
(11)
Let the total number of edits ofWi be EWi where e = {1, 2, 3, . . . , n} (Equation 5), let the total number of articles of Wi be AWi where a = {1, 2, 3, . . . , n} (Equation 6), let the total number of non-articles ofWi beRWi where r = {1, 2, 3, . . . , n}, and let the total number of pages ofWi be TWi where TWi = (AWi – AbWi) +RWi (Equation 7).
Therefore, the general mathematical definition of the DEPTH+ metric ofWi using the mathematical representations is described as the following:
D+W i = OWi · EWi · RWi A2Wi · AWiTWi (12)
Lastly, the simplified mathematical definition of the DEPTH+ metric ofWi using the mathematical representations is described as the following:
D+W i = OWi · EWi TWi · RWiAWi (13)
4.3 Features of DEPTH+ Metric
The DEPTH+ metric overcomes some of the drawbacks of the depth metric, employs Wikipedia’s users-related metrics, and offers bot-free Wikipedia editions. Revisiting Figure 1, we see that the changes in the global rank for the top ten languages (editions) regarding the number of articles on the Wikipedia project when both metrics (depth and DEPTH+) are applied, illustrating that the DEPTH+
metric successfully prioritizes the large and most common Wikipedia editions.
Figure 3 shows the highest 50 Wikipedia editions ordered by the DEPTH+ metric values after eliminating all bot activities (bot-generated articles and bot-made edits). Unlike the depth metric, we no longer use a somewhat arbitrary filtering step to disadvantage lower-resource languages. It makes sense that older, larger editions like English may have richer collaboration and depth, but using a filtering step to remove small languages does not seem fair. Small languages could have rich collaboration and depth as well. With the DEPTH+ metric, we see that the English Wikipedia edition is at the top of the rank without filtering, followed by very large editions like German (de), French (fr), Italian (it), and Japanese (ja), but smaller languages still have the potential to score high on the DEPTH+
ranking. For example, the Greenlandic Wikipedia edition was filtered in the depth metric, but with the DEPTH+ metric, it is now among the top 50 Wikipedia editions. The DEPTH+ metric successfully removes the bot-generated Wikipedia editions from the top of the rankings.
The original depth metric did not include any user-related metrics offered by the Wikipedia project, only focusing on the edits activities of the
removing the bot-activity, we can see that Wikipedia editions like Pali (pi), Cebuano (ceb), Bishnupriya (bpy), and Vietnamese (vi) still have unintuitively high depth values.
different types of pages (articles and non-articles) and neglecting the activity of the different types of users (users, admins, and active users) who contributed to these edits in the first place. The DEPTH+ metric introduced the “editors” metric (see Equation 8), which utilizes these metrics: admins, users, and active users and actively puts the users at the heart of the DEPTH+ metric.
We quantified the bot activities of creating articles (content pages) and the bot activities of editing those articles. We also successfully integrated our quantifications of the bot activities into the DEPTH+ metric. We also found that the DEPTH+
metric is more robust than the original depth metric when we remove all bot activities and apply the two metrics. The DEPTH+ metric returns mostly identical calculations when we include or remove all bot activities from the metric, whereas the original depth metric returns completely different questionable calculations, as shown in Figure 4.

heading: 5 Limitations
text: The DEPTH+ metric resolves the problem of botgenerated Wikipedia editions that have many botcreated articles and bot-made edits on their articles. Yet, the DEPTH+ metric does not fix the problem of automatically translated Wikipedia editions in the Wikipedia project that their articles have been largely translated by poor direct translation or shallow template-based translation. The quantifications of these automatically translated Wikipedia editions in the Wikipedia project cannot be carried out as systematically as the bot-generated Wikipedia editions, and examining each Wikipedia edition separately is the only way to accomplish such quantification. Another limitation of the DEPTH+ met-
ric is depending on the active users metric, which dynamically decreases the DEPTH+ metric values if there are no editing activities on the articles in the last thirty days. We preferred to use the total unique users who made at least one edit but do not have that figure, so we are approximating it with the already calculated active users metric by the Wikipedia project.

heading: 6 Related Work
text: Due to the widespread use of Wikipedia articles as training corpora for many NLP toolchains, especially for low-resource languages, many researchers have addressed the importance of transparency in the Wikipedia project, encouraged the transparency values in the project, and proposed improvements on accountability and social transparency through visualizations. For example, Suh et al. (2008) presented a social dynamic analysis tool called “WikiDashboard” to improve the social transparency and accountability of Wikipedia articles. This tool aims to enhance the interpretation, communication, and trustworthiness of Wikipedia articles by visualizing the social dynamics and editing patterns of every article and editor in the Wikipedia project.
Biuk-Aghai et al. (2014) also studied the visualization of large-scale human collaboration on the Wikipedia project, analyzed the co-authoring across the entire Wikipedia editions in various languages (English, German, Chinese, Swedish, and Danish), and found it to follow a geometric distribution in all the investigated language editions. To better understand the geometric distribution of co-author counts across different topics on the Wikipedia project, they aggregated Wikipedia
content by category and visualized it in a form resembling a geographic map. These geographically looking map visualizations show significant differences in co-author counts across different topics in all the visualized Wikipedia language editions.
At the intersection of transparency and underrepresentation in the Wikipedia project, Wali et al. (2020) discussed the available Wikipedia corpora for eight languages: English, Chinese, Arabic, Urdu, Farsi, French, Spanish, and Wolof. They closely examined the typical NLP pipeline and highlighted that significant limitations persist even when a language is technically supported, hindering full participation. They specifically compared the number of language speakers to the number of articles in the respective Wikipedia edition, using the “Articles/1000 Speakers” metric. Despite the dedicated efforts of numerous Wikipedia contributors who have invested substantially in compiling a vast multilingual dataset, not all language speakers have equal opportunities to contribute to the Wikipedia project.

heading: 7 Conclusion and Future Work
text: We have discussed Wikipedia’s current depth metric in detail, rewritten its mathematical representations, and underlined the limitations of its representation of the depth of collaboration in Wikipedia corpora. We also quantified the bot activities in the Wikipedia project and excluded the bot-created articles and the bot-made edits on Wikipedia articles. We lastly proposed the DEPTH+ metric, defined its formal definitions, and highlighted its features, including a better representation of the depth of collaborativeness, a user-centered depth metric, and bot-free Wikipedia editions after the removal of the bot-generated articles and the bot-made edits on those Wikipedia editions’ articles.
We hypothesize that a metric that is a better measure of authentic human collaborativeness will be a better measure of the degree to which corpora authentically represents the language and the culture of native speakers. One key aspect of our future work is to find ways to test this hypothesis. Specifically, we aim to examine the performance and societal implications of training LLMs on unrepresentative and inorganic corpora, particularly on the bot-generated Wikipedia articles.
Reproducibility
Data collection, implementation of the DEPTH+ metric, and an expanded technical report can be found on GitHub at https://github.com/SaiedAlshahrani/DEPTHplus.

heading: Future Work
text: One key aspect of our future work is to find ways to test this hypothesis. Specifically, we aim to examine the performance and societal implications of training LLMs on unrepresentative and inorganic corpora, particularly on the bot-generated Wikipedia articles. Reproducibility
Data collection, implementation of the DEPTH+ metric, and an expanded technical report can be found on GitHub at https://github. com/SaiedAlshahrani/DEPTHplus