nyuuzyou/wikis huggingface dataset:
Data Platform/Analyze data

__NOTOC__
This page outlines the tools and systems available for analyzing private Wikimedia data. For public data, see meta:Research:Data.
== Key terms ==
;Analytics clients
: Analytics clients (also called "stat boxes") are servers in the production cluster that enable you to access private data and resources for statistical computation. They're called "analytics clients" because they act as clients accessing data from other databases.
;Analytics cluster
: The "analytics cluster" is a catch-all term for compute resources and services running inside of the Analytics VLAN, which itself is inside of WMF production network. Individual systems within the analytics cluster include Hadoop, and related components that run the Data Lake. 
; Data Lake
: The Data Lake is a large, analytics-oriented repository of Wikimedia data.
;Hadoop
: A collection of services for batch processing of large data. See Hadoop.
;HDFS
: A file system for the Hadoop framework, which WMF uses to store files of various formats the Data Lake. 
;Hive
:A system that projects structure onto flat data (text or binary) in HDFS and allows this data to be queried using an SQL-like syntax.
== Get access to internal data ==
Private data lives in same server cluster that runs Wikimedia's production websites. Often, this means you need production access to access it.
== Query and analyze data ==
After you have access to internal data and systems, you can start exploring and querying data in the Data Lake. 
== Use internal versions of public resources ==
You can access some popular public data sources more quickly and efficiently by using these internal data platform tools or datasets.
For a full overview of the types of data available internally and publicly, see Discover data.
== Next steps ==
To learn about how to publish and share your analyses through dashboards, visualizations, and more, see Transform and publish data.
Category:Data platform
Category:Landing page
Data Platform/Data Lake/Content

This page lists Data Lakes datasets that contain content, such as wikitext or wikibase JSON:
* mediawiki_wikitext_current
* mediawiki_wikitext_history
* wikidata_entity
* wikidata_item_page_link
We also have a new generation of datasets that are updated more frequently:
* wmf_dumps.mediawiki_content_history_v1
For more information, including publicly-available content datasets, see Data_Platform/Discover_data#Content_data.
Category:Content data
Category:Data platform
Data Platform/Data Lake/Content/Wikidata item page link

The wmf.wikidata_item_page_link table (available on Hive) contains links between a wikidata item and its related wikipedia pages in various languages. It is generated by joining wmf.wikidata_entity site-links to reconstructed localized-namespaced-page-titles from wmf.mediawiki_page_history and wmf_raw.mediawiki_project_namespace_map.
The dataset is fully regenerated every week, following the wmf.wikidata_entity dataset snapshot release.
WARNING: The wmf.mediawiki_page_history dataset being released every month, there is a divergence between sitelinks available and pages available as the month pass. This means links correctness is better when using snapshots early in the month than later in the month.
== Current Schema ==
$ hive --database wmf
hive (wmf)> describe wikidata_item_page_link;
OK
col_name	data_type	comment
item_id 	string 	The wikidata item_id (Q32753077 for instance)
wiki_db 	string 	The db project of the page the wikidata item links to
page_id 	bigint 	The id of the page the wikidata item links to
page_title 	string 	The title of the page the wikidata item links to
page_namespace 	int 	The namespace of the page the wikidata item links to
page_title_localized_namespace	string 	The title with localized namespace header of the page the wikidata item links to
snapshot 	string 	Versioning information to keep multiple datasets (YYYY-MM-DD for regular weekly imports)
	 	 
# Partition Information	 	 
# col_name 	data_type 	comment 	 
snapshot 	string 	Versioning information to keep multiple datasets (YYYY-MM-DD for regular weekly imports)
Notice the snapshot field. It is a Hive partition, an explicit mapping to weekly import in HDFS. You must include this partition predicate in the where clause of your queries (even if it is just snapshot > '0'). Partitions allow you to reduce the amount of data that Hive must parse and process before it returns you results. For example, if are only interested in the 2020-01-20 snaphsot, you should add where snapshot = '2020-01-20'. This will instruct Hive to only process data for partitions that match that partition predicate. You may use partition fields as you would any normal field, even though the field values are not actually stored in the data files.
And entry (like, say, Q1) that has a presence in a number of wikis will have one entry per wiki on this table.
== Sample queries ==
SELECT
 wiki_db,
 page_id,
 COUNT(1) as items_numbers
FROM wmf.wikidata_item_page_link
WHERE snapshot='2020-02-10'
GROUP BY wiki_db, page_id
ORDER BY items_numbers DESC
LIMIT 10;
== Changes and known problems since 2020-02 ==
Date from	Task	Details
2020-02 	 	Table is created with first automated snapshots.
== See also==
* The code that generates the dataset:
** https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/wikidata/WikidataPageItemLink.scala
Category:Content data
Category:Data platform
Data Platform/Data Lake/Data Issues

== All Subpages of ==
== Annotating Superset Dashboards ==
We recommend the following approaches for excluding or annotating data that contains known data quality issues:
* Use date filters to exclude data from analysis for the affected time period
* For time series visualizations:
** Visually block out the period of the data loss and add annotation with the problem summary and from and to dates. For example:[[File:Wikistats Pageviews with Data Loss Period Visually Blocked.png|alt=Wikistats pageviews time series graph with data loss period visually blocked|none|thumb|600x600px|Between June 2021 and January 2022, pageview data was underreported due to caching nodes in the US data centers that had stopped collecting traffic data. For more details see the /2021-06-04 Traffic Data Loss report on Wikitech. Time series graph from Wikistats.]]
** Use overlays to annotate the data. For users of Superset an annotation layer can be created and reused. For example, for the /2021-06-04 Traffic Data Loss, an annotation layer is available called “Pageview Data Loss June 2021-January 2022”:
** For point in time issues, use a data point annotation.
* When it is not feasible to remove data from an existing report or dashboard, add an annotation or footnote describing the impact of the data issue.
Data Platform/Data Lake/Data Issues/2023-01-08 Webrequest Data Loss

Date issue detected: 	2023-01-08
Date resolved: 	2023-01-09
Start of issue: 	2023-01-08
Status: 	Resolved
Phabricator Ticket(s):	https://phabricator.wikimedia.org/T328354
https://phabricator.wikimedia.org/T326658
== Summary ==
On the 8TH of January 2023, some caching nodes failed to send some webrequest traffic resulting in loss of data between the hours of 0800 till 1400. We estimate this resulted in a 2.83% loss of webrequest-text data leading to an underreporting of pageviews and all webrequest related data.
== Description ==
The caching nodes in the Eqsin datacenter failed to collect some traffic data between the hours of 8:00 and 14:00 on the 8th of January 2023. This webrequest-text traffic data was affected during this time period. The table below contains details data loss during the affected time period.
Time 	Estimated Data loss
2023-01-08 08:00	2.84%
2023-01-08 09:00	2.66%
2023-01-08 10:00	2.55%
2023-01-08 11:00	2.916%
2023-01-08 12:00	3.09%
2023-01-08 13:00	3.08%
2023-01-08 13:00	2.717%
In summary, on 2023/01/8 between the 8th and 14th hours, we lost an average of 2.83% of webrequest-text data (from eqsin datacenter only).
== Recommendations ==
SInce we are unable to recover the lost data, we recommend going ahead to process the data received during the affected time period.We recommend excluding the affected data for the time period (2023/01/08 8:00 through 2023/01/08 14:00) from the analysis report.
== Root Cause ==
The Eqsin datacenter experienced some network issues on the 8th of January 2023.
== Affected Datasets ==
* Web Requests (webrequest)
* Pageviews ( pageview_hourly, pageview_actor)
* Projectview hourly
* Mediacounts
* Uniques Devices
* Browser general
* Mediawiki API request
* Mobile apps sessions and uniques metrics
* Interlanguage navigation
== Followup Steps ==
No followup steps since the root cause is beyond the data engineering team’s control.
== Reproducing the Issues ==
Data loss can be verified using this script:
sudo -u analytics kerberos-run-command analytics spark2-sql --master yarn -S \
 --jars /usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core.jar \
 -f /srv/deployment/analytics/refinery/hive/webrequest/check_dataloss_false_positives.sparksql \
 -d table_name=TABLE \
 -d webrequest_source=SOURCE \
 -d year=YEAR \
 -d month=MONTH \
 -d day=DAY \
 -d hour=HOUR
If the output of this query contains rows that have the false_positive field to false, there is real data-loss
Category:Data platform
Data Platform/Data Lake/Edits/Geoeditors/Public

This is a public version of the Geoeditors Monthly dataset. It reports the number of active editors per country per month for a set number of countries.
== Data Format ==
The data is released monthly as a flat file at our Dumps site. The file will have the following columns:
* wiki db: the code name for the wiki, "enwiki" for English Wikipedia, at this time the dataset is available just for Wikipedias
* country: the name of the country with editors of this wiki
* activity level: how many edits this group of editors has made in the past month (either 5 to 99 or more than 100)
* lower bound: at least this many editors in this group
* upper bound: at most this many editors in this group
Note: The final count of editors is an aggregate of both registered and anonymous editors. It may happen that an editor edits as both registered and anonymous in the same month. If so, that editor is going to be counted twice.
Note: Bots are identified as well as possible and filtered out, so this data does not include any bot activity we can identify.
== Privacy ==
Since this data has many privacy concerns this public release applies the following changes to make the data reveal less while providing value for a public audience:
=== Country Protection List ===
=== No exact counts ===
To add a small amount of imprecision to the data, instead of saying, for example, there are 5 editors editing Estonian Wikipedia from Romania, we say there are between 1 and 10. This does not dramatically improve the privacy of the dataset, but it adds a small amount of uncertainty if someone is trying to guess the country of an editor. The amount of uncertainty does not depend on the bucket size but rather in the number of countries for which there are editors for a given project.
=== Only active wikis ===
We are only releasing data for wikis with at least 3 active editors on any given month. That's three distinct editors making 5 or more edits in a month. Past research indicates that any less activity than that can't support the healthy collaboration and exchange of ideas essential to wikis.
== Risk Assessment ==
Category:Editors data
Category:Data platform
Data Platform/Data Lake/Edits/MediaWiki history dumps/FAQ

== How are rows ordered? ==
This dataset is centered around edit activity. Each row with an event_entity == 'revision' represents an edit to a page, but carries with it information that can make the dataset confusing. As you get familiar with it, remember that each row is an edit, and every column stores additional context about that edit. So, for example, a user's edit count goes up when they make an edit. The fields that start with event_user store context about the user making the edit. So the event_user_revision_count field stores this user's updated edit count, incremented by one since their previous edit. If you need page-centric measures, look at the page_ fields. For example, the page_revision_count field will be incremented by one by this edit. To get the total number of edits on a single page, look for the row with the greatest event_timestamp for that page, and read the page_revision_count.
== How do I get the number of unique editors of a given page? ==
None of the fields will help you to do this now, but we're always looking to improve the usefulness of this dataset, you're welcome to submit a request like this on our Phabricator board.
== Doing certain kinds of analysis on this data is very expensive, any plans to make it queryable? ==
We're trying to find ways to publish this in a queryable form, because we understand it makes some questions easy to answer while others remain expensive.
Category:Edits data
Category:Data platform
Data Platform/Data Lake/Edits/Mediawiki project namespace map

The wmf_raw.mediawiki_project_namespace_map table (available on Hive) contains project and namespace data for every project referenced in the Wikimedia sitematrix. It is generated by querying the sitematrix API for the project-list, and then for each project querying the project API for its site-info data.
The dataset is fully regenerated on the first of every month.
TODO: Merge this dataset within the canonical_data hive database.
== Current Schema ==
$ hive --database wmf_raw
hive (wmf)> describe mediawiki_project_namespace_map;
OK
col_name	data_type	comment
hostname 	string 	Canonical URL for the project, for example ja.wikipedia.org
dbname 	string 	Database name for the project, for example jawiki
namespace 	int 	for example 0, 100, etc.
namespace_canonical_name	string 	the english prefix if exists, otherwise the localized prefix
namespace_localized_name	string 	the localized prefix
namespace_is_content	int 	Whether this namespace is a content namespace
snapshot 	string 	Versioning information to keep multiple datasets (YYYY-MM for regular labs imports)
	 	 
# Partition Information	 	 
# col_name 	data_type 	comment 
snapshot 	string 	Versioning information to keep multiple datasets (YYYY-MM for regular labs imports)
Notice the snapshot field. It is a Hive partition, an explicit mapping to weekly import in HDFS. You must include this partition predicate in the where clause of your queries (even if it is just snapshot > '0'). Partitions allow you to reduce the amount of data that Hive must parse and process before it returns you results. For example, if are only interested in the 2020-01-20 snaphsot, you should add where snapshot = '2020-01-20'. This will instruct Hive to only process data for partitions that match that partition predicate. You may use partition fields as you would any normal field, even though the field values are not actually stored in the data files.
== Changes and known problems since 2019-08 ==
Date from	Task	Details
2017-?? 	?? 	Table is created with first automated snapshots (doc created on 2020-02)
== See also==
* The code that generates the dataset:
** https://github.com/wikimedia/analytics-refinery/blob/master/bin/download-project-namespace-map
Category:Edits data
Category:Data platform
Data Platform/Data Lake/Edits/Metrics

This table stores daily and monthly metrics computed over the denormalized mediawiki history dataset. It is partitioned by wiki_db and metric name to facilitate using its data outside of Hive, namely for display in Dashiki.
==Schema==
col_name	data_type	comment
dt 	string 	The date of this measurement, as YYYY-MM-DD
value 	bigint 	The measurement 
snapshot 	string 	Versioning information to keep multiple datasets (YYYY-MM for regular labs imports)
metric 	string 	The metric being computed to measure
wiki_db 	string 	The wiki this measurement pertains to
	 	 
# Partition Information	 	 
# col_name 	data_type 	comment 
	 	 
snapshot 	string 	Versioning information to keep multiple datasets (YYYY-MM for regular labs imports)
metric 	string 	The metric being computed to measure
wiki_db 	string 	The wiki this measurement pertains to
As May 2023, possible values for metric include:
 1. daily_edits
 2. daily_edits_by_anonymous_users
 3. daily_edits_by_bot_users
 4. daily_edits_by_registered_users
 5. daily_unique_anonymous_editors
 6. daily_unique_bot_editors
 7. daily_unique_editors
 8. daily_unique_page_creators
 9. daily_unique_registered_editors
 10. monthly_new_editors
 11. monthly_new_registered_users
 12. monthly_surviving_new_editors
== Definition ==
The Hive queries that generate these metrics are in wikimedia/analytics-refinery/oozie/mediawiki/history/metrics.
Category:Metrics
Category:Data platform
Data Platform/Data Lake/Edits/Structured data/Commons entity

The structured_data.commons_entity table (available on Hive) is a conversion of the commonswiki structured-data entities JSON dumps in parquet. In wikibase (the underlying data-model of structured-data), entities information is stored in JSON, leading to dumps in that format. A conversion is needed for a better data-format in parquet, as JSON makes extensive use of maps (objects with a variable name), which are less easy to use in Parquet when the data-model is actually well-defined.
New full dumps are generated and copied to the analytics hadoop cluster every week, and then converted to Parquet and added as a new snapshot in the hive table.
== Current Schema[edit | edit source] ==
 $ hive --database structured_data
 
 hive (wmf)> describe commons_entity;
 
 TODO: Add description when table will be created.
 
Notice the snapshot field. It is a Hive partition, an explicit mapping to weekly import in HDFS. You must include this partition predicate in the where clause of your queries (even if it is just snapshot > '0'). Partitions allow you to reduce the amount of data that Hive must parse and process before it returns you results. For example, if are only interested in the 2020-01-20 snaphsot, you should add where snapshot = '2020-01-20'. This will instruct Hive to only process data for partitions that match that partition predicate. You may use partition fields as you would any normal field, even though the field values are not actually stored in the data files.
== Sample queries[edit | edit source] ==
 TODO: Add query example
== Changes and known problems since 2020-02[edit | edit source] ==
Date from	Task	Details
2021-12 	 	Table is created and first imports and conversions are automated.
== See also[edit | edit source] ==
* The code that imports and converts:
** TODO: Add link to import code
** https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/structureddata/jsonparse/JsonDumpConverter.scala
Category:Data platform
Data Platform/Data Lake/Events

== Accessing Event Data ==
As of September 2020, you have a choice of three engines that can run SQL queries against the Data Lake: Presto, Hive, and Spark. If you're not sure which to choose, Hive is good to start with. All three engines can be used from the Analytics clients.
* WMF maintains 2 separate git schema repositories:
** schemas/event/primary
** schemas/event/secondary
* Schemas and schema status for the old EventLogging system are documented at Research:Schemas
* Event vs event_sanitizied*
** The eventdatabase stores original (unsanitized) events within a 90 day retention period, while event_sanitized database is an archive of sanitized events, beyond the 90 day retention period.
** Sanitized event data is processed per WMF’s Privacy Policy and Data Retention Guidelines.
Category:Data platform
Data Platform/Data Lake/Traffic

Traffic refers to pageviews to the pages of a wiki project. This page links to detailed information about traffic datasets in the Data Lake.
Most of the datasets below are updated at hourly granularity, meaning that you'll get an hour of new data every hour, with between 2 and 3 hours delay (for the hour to be finished, and the data to be computed).
== Datasets ==
=== Hive tables ===
These datasets are available as Hive tables and can be queried using one of the available SQL engines, or accessed directly through HDFS. 
Dataset Name 	Description
webrequest hive table
- See also a separate list of Hive tables derived from webrequest	The webrequest stream contains data on all the hits to Wikimedia's servers. This includes requests for page HTML, images, CSS, and Javascript, as well as requests to the API.
pageview_actor hive table 	The wmf.pageview_actor table is a smaller version of webrequest table with fewer columns.
pageview_hourly hive table 	The wmf.pageview_hourly table contains 'pre-aggregated' webrequest data, filtered to keep only pageviews, and aggregated over a predefined set of dimensions.
projectview_hourly hive table	The wmf.projectview_hourly table is 'pre-aggregated' webrequest data at the project level. It is different from the wmf.pageview_hourly dataset in that it involves less dimensions and is therefore smaller in data size (and faster to query).
unique devices 	This dataset gives you how many distinct devices visit our projects
browser general 	This dataset gives you pageview statistics broken down by user-agent related dimensions like OS family, OS major, browser family, browser major
mediawiki_api_request 	The mediawiki_api_request table provides the log of api requests to MediaWiki
mobile apps session metrics 	Contains aggregate stats about pageview sessions on the Android and iOS Wikipedia mobile apps
mobile apps uniques 	Counts how many different Android and iOS Wikipedia mobile apps installs accessed Wikimedia sites during the given day or month
inter language 	Traffic between different languages on the same project family
virtualpageview_hourly 	Provides data about page previews on desktop Wikipedia
*
=== Dumps ===
These datasets are made available as files, updated at regular intervals.
* Pageviews and Projectviews dumps [To be updated]
* Compressed pageviews dumps [To be updated]
* mediacounts
* Wikipedia clickstream
== Deprecated or Obsolete Datasets ==
The following datasets are no longer in use, but the pages are kept to document history:
* pagecounts-raw
* pagecounts-all-sites
* Search requests
* Search requests clicks
== Access ==
All data in the Data Lake is private by default. For this, reference Data_Platform/Data access. Some of the data above is public in other systems (see Analytics main page)
== History ==
Some partial information about the evolution of publishing analytics data at WMF is recorded here in a timeline.
Category:Traffic data
Category:Data platform