nyuuzyou/wikis huggingface dataset:
Data Platform/Data Lake/Traffic/Browser general

wmf.browser_general is an intermediate table that makes it easier and faster to get pageview statistics broken down by user-agent related dimensions like OS family, OS major, browser family, browser major, etc. In addition to ad-hoc queries, it is also used to generate the Browser Reports.
=== Current Schema ===
$ hive --database wmf
hive (wmf)> describe browser_general;
OK
col_name data_type comment
access_method string (desktop|mobile web|mobile app)
os_family string OS family: Windows, Android, etc.
os_major string OS major version: 8, 10, etc.
browser_family string Browser family: Chrome, Safari, etc.
browser_major string Browser major version: 47, 11, etc.
view_count bigint Number of pageviews.
year int Unpadded year of request.
month int Unpadded month of request.
day int Unpadded day of request.
 
# Partition Information 
# col_name data_type comment 
 
year int Unpadded year of request.
month int Unpadded month of request.
day int Unpadded day of request.
Notice the year, month and day fields. These are Hive partitions, and are explicit mappings to daily files in HDFS. You must include at least one partition predicate in the where clause of your queries (even if it is just year > 0). Partitions allow you to reduce the amount of data that Hive must parse and process before it returns your results. You may use partition fields as you would any normal field, even though the field values are not actually stored in the data files.
=== Loading of anonymized data ===
wmf.browser_general is loaded through this oozie job. It is a basic query that extracts data from wmf.pageview_hourly and wmf.projectview_hourly, with a threshold-based anonymization applied to it. For privacy and readability reasons, any rows that initially represent less than K% of the total view count are grouped into a single row, where all columns (os_family, os_major, browser_family, browser_major) are assigned to the "Other" constant (see job config for exact K and constant values).
=== Example query ===
SELECT
 browser_family,
 browser_major,
 SUM(view_count) as view_count
FROM wmf.browser_general
WHERE
 access_method IN ('desktop', 'mobile web') AND
 CONCAT(year, '-', LPAD(month, 2, '0'), '-', LPAD(day, 2, '0')) >= '2016-01-01' AND
 CONCAT(year, '-', LPAD(month, 2, '0'), '-', LPAD(day, 2, '0')) < '2016-02-01'
GROUP BY
 browser_family,
 browser_major
ORDER BY view_count DESC
LIMIT 1000;
=== Changes and known problems since 2016-03-21 ===
Date from	Date until	Task	Details
beginning	now 	 	Inconsistent data in #all-sites-by-os-and-browser for IE7
... 	... 	... 	...
Category:Traffic data
Category:Data platform
Data Platform/Data Lake/Traffic/Interlanguage

= Inter-language Datasets =
This page describes datasets that count web requests which stay within a project family but change language.
== Interlanguage Navigation ==
The wmf.interlanguage_navigation table in Hive measures the number of webrequests, for example, coming from de.wikipedia.org and going to ro.wikipedia.org. The schema for this table is below, and in this example, the fields would have had these values:
* project_family: wikipedia
* previous_project: de
* current_project: en
* navigation_count: the number of webrequests that made this jump on this specific date
* date: 2017-11-01
 `project_family` string COMMENT 'The project family to aggregate on',
 `previous_project` string COMMENT 'The project (language) found in the referers of this group of requests',
 `current_project` string COMMENT 'The project (language) of this group of requests',
 `navigation_count` bigint COMMENT 'The number of times a user navigated from the previous to the current project',
 `date` string COMMENT 'The date in YYYY-MM-DD format'
==See also==
*The code that generates it
Category:Traffic data
Category:Data platform
Data Platform/Data Lake/Traffic/Pagecounts-ez

This dataset is described on its dumps download page.
This dataset is a compressed format of the best pageview data that the Wikimedia Foundation had at any point in its historyː
* From 2007 to December 2015, it compressed the pagecounts-raw dataset, which is now deprecated (providing pageviews per project from December 2007 on, and pageviews per article from late 2011 on)
* From Dec 2015 to Present day, it compresses the pageviews dataset
More information about each of those datasets can be found on their pages.
==One hour skewing issue==
The data on this dataset, when compared to the canonical Pageviews API, is skewed one hour to the left. This means that on Pagecounts-EZ reports as midnight the pagecount value that in reality corresponds to 11PM the previous day:
Dataset 	12am	1am	2am	3am	4am	5am	6am	7am	8am	9am	10am	11am	12pm
Pageview API 	23 	234	43 	345	64 	12 	534	654	43 	645	98 	65 	75
Pagecounts EZ	89 	23 	234	43 	345	64 	12 	534	654	43 	645 	98 	65
==See also==
*Erik Zachte's 2011 announcement of this dataset (source for the description on the dumps download page, link is broken there): ,
*Phabricator task to make per-article data available in pagecounts-ez back to 2008 (instead of 2011)
*m:Learning patterns/Tips for reading project codes from pageviews data files
Category:Traffic data
Category:Data platform
Data Platform/Data Lake/Traffic/Pageview hourly/Fingerprinting Over Time

In a plan to secure our user data, the pageview_hourly dataset needs to be sanitized in such a way that it does not allow to track user path. See this page for a broad view on the pageview_hourly sanitization project.
This page go through as fingerprinting analysis over time, using webrequest and pageview_hourly datasets.
=== Data Preparation ===
 CREATE TABLE hashes1 STORED AS PARQUET AS
 SELECT
 hash(
 pageview_info['project'],
 pageview_info['language_variant'],
 access_method,
 x_analytics_map['zero'],
 agent_type,
 geocoded_data['continent'],
 geocoded_data['country_code'],
 geocoded_data['country'],
 geocoded_data['subdivision'],
 geocoded_data['city'],
 user_agent_map) as h,
 COUNT(DISTINCT ip) as dip,
 COUNT(DISTINCT pageview_info['page_title']) AS dpv,
 COUNT(1) as c FROM wmf.webrequest
 WHERE webrequest_source = 'text'
 AND year = 2016
 AND month = 1
 AND day = 11
 AND agent_type = 'user'
 GROUP BY
 hash(
 pageview_info['project'],
 pageview_info['language_variant'],
 access_method,
 x_analytics_map['zero'],
 agent_type,
 geocoded_data['continent'],
 geocoded_data['country_code'],
 geocoded_data['country'],
 geocoded_data['subdivision'],
 geocoded_data['city'],
 user_agent_map);
 WITH pv_hashes1 AS
 (
 SELECT
 year, month, day,
 hash(project, language_variant, access_method, zero_carrier, agent_type,
 continent, country_code, country, subdivision, city, user_agent_map) as pvh,
 SUM(view_count) as vc
 FROM wmf.pageview_hourly
 WHERE ((year = 2015 AND month > 5) OR (year = 2016 AND month = 1 AND day < 12))
 AND agent_type = 'user'
 GROUP BY
 year,
 month,
 day,
 hash(project, language_variant, access_method, zero_carrier, agent_type,
 continent, country_code, country, subdivision, city, user_agent_map)
 )
 SELECT
 year, month, day,
 SUM(CASE WHEN dip = 1 THEN 1 ELSE 0 END) as dip1_n,
 SUM(CASE WHEN dip = 2 THEN 1 ELSE 0 END) as dip2_n,
 SUM(CASE WHEN dip = 3 THEN 1 ELSE 0 END) as dip3_n,
 SUM(CASE WHEN dip = 4 THEN 1 ELSE 0 END) as dip4_n,
 SUM(CASE WHEN dip = 5 THEN 1 ELSE 0 END) as dip5_n,
 SUM(CASE WHEN dip > 5 THEN 1 ELSE 0 END) as dipPlus_n,
 SUM(CASE WHEN dip = 1 THEN pvh1.vc ELSE 0 END) as dip1_vc,
 SUM(CASE WHEN dip = 2 THEN pvh1.vc ELSE 0 END) as dip2_vc,
 SUM(CASE WHEN dip = 3 THEN pvh1.vc ELSE 0 END) as dip3_vc,
 SUM(CASE WHEN dip = 4 THEN pvh1.vc ELSE 0 END) as dip4_vc,
 SUM(CASE WHEN dip = 5 THEN pvh1.vc ELSE 0 END) as dip5_vc,
 SUM(CASE WHEN dip > 5 THEN pvh1.vc ELSE 0 END) as dipPlus_vc
 FROM pv_hashes1 pvh1
 INNER JOIN pv_san.hashes1 h1
 ON (pvh1.pvh = h1.h)
 GROUP BY
 year, month, day
 ORDER BY year, month, day
 LIMIT 1000;
=== Results ===
As shown below, the decrease of number of hashes match is almost linear. Nonetheless the remaining number of hashes from December still existing in June is very substantial (over a million distinct hashes), particularly with hashes involving only one IP, the most dangerous bucket in term of browsing pattern reconstruction.
Category:Pageviews
Category:Data platform
Data Platform/Data Lake/Traffic/Webrequest/Tagging

== Webrequest Tagging ==
We are in the process of adding a “tags” column to webrequest. This tag column is an array that can hold values like: “portal”, “wikidata", "pageview".
The pageview refinement process will be enhanced with a tagging step, in which some requests (pageviews or not) will be marked with one of many tags. 
Once tagging phase is completed a second process will read the tag column. A small number of tags will be used for splitting 
the webrequest dataset in smaller datasets using hive dynamic partitioning. Many of our regular data-generation jobs read every record in webrequest 
when they actually need only a portion of it. Splitting the data into pre-filtered datasets will optimize our jobs, 
as they would be able to read just pertinent data. 
Not all tags will be used for partitioning, just a smaller set, other tags might be short lived and used to more efficiently select records from webrequest table.
=== Usage of tags column in SQL ===
The tags column is an array<string>, a hive complex type. Selects to get elements can look like:
 Select tags from webrequest where year=2017 and month=09 and day=09 and hour=09 limit 5;
This might return something like:
 ["wikidata-query","sparql"]
 ["portal"]
 ["wikidata-query","sparql"]
 []
 []
Also:
 Select tags[0]
Will return 1st element of array.
== Code example ==
See how tags are implemented to "tag" wdqs 
https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-core/src/main/java/org/wikimedia/analytics/refinery/core/webrequest/tag/WDQSTagger.java
Category:Traffic data
Category:Data platform
Data Platform/Data Lake/Traffic/mobile apps session metrics

The tables mobile_apps_session_metrics and mobile_apps_session_metrics_by_os (available on Hive) contain aggregate stats about pageview sessions on the Android and iOS Wikipedia mobile apps, updated weekly. 
* mobile_apps_session_metrics_by_os calculates them for a timespan of 7 days each, for iOS and Android separately. Deprecated as of June 2023 (T329310) 
* mobile_apps_session_metrics has numbers for overlapping 30-day timespans, without distinguishing between the two apps. Deprecated as of 2018.
'''
A session is defined as a sequence of pageviews from the same app ID that does not exceed 30 minutes of inactivity.
In both tables, each row provides the minimum, maximum, and four percentiles (10th, 50th, 90th, 99th) for one of the following three metrics, as well as an overall count:
* Sessions per user (i.e. app UUID)
* Pageviews per session 
* Session length (the time between the first and last event. Not reported for sessions consisting of only one pageview)
== Schema ==
> DESCRIBE wmf.mobile_apps_session_metrics_by_os;
col_name	data_type	comment
year	int	Unpadded year of report run date
month	int	Unpadded month of report run date
day	int	Unpadded day of report run date
date_range	string	Period for which report was run
type	string	Type of session metric
os_family	string	OS family breakdown
count	int	Value of count for given metric
min	int	Min value for given metric
max	int	Max value for given metric
p_1	string	1st Percentile for given metric
p_50	string	50th Percentile for given metric
p_90	string	90th Percentile for given metric
p_99	string	99th Percentile for given metric
Notes:
* For each metric, the count variable refers to the length of the series that was used to calculate the quantiles. So for sessions per user, the count is the number of users; for pageviews per session, the count is the number of sessions; for session length, the count is the number of sessions that have more than one pageview (see the code).
==Caveats==
* The field name p_1 and its description are inaccurate - it contains the 10th percentile, not the 1st. 
* The stats only cover users who have opted-in to usage data (iOS) / have not opted out (Android). 
* On Jan 11, 2018, we found a bug (T184768) in the quantiles function which would result in imprecise ranges of quantiles. 
* Session length is calculated as the difference between the first and last pageview timestamp in a session. This means 1) only sessions with at least two pageviews are counted and 2) the recorded session length should be less than or equal to the actual session length on the client side. 
== See also ==
* Scala code that calculates the data
* phab:T86535 2015 task about the introduction of these metrics, with an outline of the calculation method
* phab:T117615 2015/16 task about the creation of the by_os variant (providing the data separately for iOS and Android, and for weekly instead of monthly timespans)
Category:Metrics
Category:Traffic data
Category:Data platform
Data Platform/Data Lake/Traffic/mobile apps uniques

The tables mobile_apps_uniques_daily and mobile_apps_uniques_monthly(available in the wmf database on Hive) count how many different Android and iOS Wikipedia mobile apps installs accessed Wikimedia sites during the given day or month, determined as the number of app uuids appearing in the webrequest table (via the X-Analytics header or formerly the query part of the URL). 
== Schemas ==
 > DESCRIBE wmf.mobile_apps_uniques_daily;
 col_name	data_type	comment
 year	int	Unpadded year of request
 month	int	Unpadded month of request
 day	int	Unpadded day of request
 platform	string	Mobile platform from user agent parsing
 unique_count	bigint	Distinct uuid count
 > DESCRIBE wmf.mobile_apps_uniques_monthly;
 platform	string	Mobile platform from user agent parsing
 unique_count	bigint	Distinct uuid count
 col_name	data_type	comment
 year	int	Unpadded year of request
 month	int	Unpadded month of request
 platform	string	Mobile platform from user agent parsing
 unique_count	bigint	Distinct uuid count
== Caveats ==
* The stats only cover users who have opted-in to sending usage data (iOS, since March 2016) / have not opted out (Android).
* Android numbers were inflated by 10% or more from around September 2016 to March 2017 due to a bug (T163403).
* The counting method assumes that all installs not detected as iOS/iPhone in the user agent are Android apps.
== See also ==
* Underlying queries: daily / monthly
* 2014/15 tasks for the creation of these tables: T76534, T88308
* A per-country version of the daily active users was previously available at tbayer.mobile_apps_uniques_by_country. The query used to create it is documented at https://phabricator.wikimedia.org/T180651#3833399.
Category:Traffic data
Category:Data platform
Data Platform/Data Lake/Traffic/referrer daily/Dashboard

The referrer_daily dataset has many facets and ways in which someone might want to aggregate or split the data. Turnilo is currently the best solution at Wikimedia for visualizing data with these properties, but our Turnilo instance is designed for private datasets so a public instance needed to be created in order to share this dataset more broadly. Some technical details are given below and the dashboard can be found at: https://wiki-search-referrals.wmcloud.org
== Turnilo instance ==
The turnilo dashboard is hosted on a Cloud VPS instance by the Wikimedia Research team. The code for setting up the instance can be found here: https://github.com/wikimedia/research-api-endpoint-template/tree/turnilo-druid
== Data backend ==
Turnilo depends on a Druid database backend to scale up effectively. Initially as TSV-backend was used but this quickly became too slow as the size of the dataset grew.
== Updating ==
Until a more streamlined workflow is developed, updates are handled via a string of daily scripts run via crontab:
* Export new TSV with yesterday's data from HDFS
* Reformat data to match Turnilo's expected format and append to single flat-file (see below)
* Update flat-file on Analytics server
* Download new file on Turnilo instance and restart dashboard with updated data backend
import argparse
import csv
from datetime import datetime
import json
import os
def main():
 parser = argparse.ArgumentParser()
 parser.add_argument("--input_tsv_dir", help="TSV with referral data")
 parser.add_argument("--output_tsv", help="TSV for Turnilo")
 parser.add_argument("--append", action="store_true", help="Append to end of JSON")
 parser.add_argument("--header", action="store_true", help="TSV file has a header with fieldnames.")
 args = parser.parse_args()
 tsvs = sorted([fn for fn in os.listdir(args.input_tsv_dir) if fn.endswith('.tsv')])
 input_header = ['country', 'lang', 'browser_family', 'os_family', 'search_engine', 'num_referrals', 'day']
 output_header = ['time', 'country', 'lang', 'browser_family', 'os_family', 'search_engine', 'num_referrals']
 mode = 'w'
 if os.path.exists(args.output_tsv) and args.append:
 mode = 'a'
 with open(args.output_tsv, mode) as fout:
 tsvwriter = csv.writer(fout, delimiter='\t')
 if mode == 'w':
 tsvwriter.writerow(output_header)
 for tsv in tsvs:
 print("Processing:", tsv)
 with open(os.path.join(args.input_tsv_dir, tsv), 'r') as fin:
 tsvreader = csv.reader(fin, delimiter='\t')
 if args.header:
 input_header = [c.strip() for c in next(tsvreader)]
 for line in tsvreader:
 line_json = {c:v for c,v in zip(input_header, line)}
 line_json['num_referrals'] = int(line_json['num_referrals'])
 line_json['time'] = datetime.strptime(line_json['day'], '%Y%m%d').strftime('%Y-%m-%d')
 tsvwriter.writerow([line_json[c] for c in output_header])
if __name__ == "__main__":
 main()
== Read More ==
* https://techblog.wikimedia.org/2021/06/07/searching-for-wikipedia/
== See also ==
* Research:Wikipedia_clickstream
Category:Traffic data
Category:Data platform
Data Platform/Data lifecycle management

The data life cycle refers to the sequence of stages that data goes through from its initial creation or capture to its eventual archiving and possible deletion.These stages include:
* Data Creation. Data is generated or collected from various sources.
* Data Processing: Data is cleaned, sanitized, and organized for use
* Data Maintenance: Data Maintenance includes
** Data Versioning due to schema, semantic or calculation changes
** Data Quality Monitoring
** Addressing Data Quality Incidents
** Storage: Data/metadata is stored in Hive, or other datastores & documented in the data glossary(i.e. datahub)
* Data Usage. Data is used for analysis, decision-making, reporting, research, etc.
* Data Sharing and Distribution. Data is shared with or distributed to data consumers outside of WMF.
* Data Deprecation. Data is marked for archiving, and should no longer be actively used.
* Data Archiving. Where applicable, data that is no longer actively used is moved to a storage where it can be accessed if needed in future.
* Data Deletion. Data that is no longer needed or has become obsolete is deleted. Data can also partially be deleted in compliance with the data retention policy, which includes deletion to redact and suppress sensitive data. 
Understanding the data lifecycle, and having a well-defined management process bring benefits in terms of:
* Ensuring that data is created and managed according to FAIR principles
* Quality control
* Compliance, security and privacy
* Efficiency In standardizing and streamlining data management activities
* Ensuring that the business and technical data stewards are engaged to facilitate the right decisions, and data lifecycle events communicated to all affected parties to prevent any adverse downstream effects
The following pages define the dataset lifecycle processes:
* Data Incident management
* Dataset creation
* Dataset archiving and deletion
* Dataset versioning
Category:Data platform
Data Platform/Dataset archiving and deletion

== Motivation ==
Deprecating, archiving or deleting datasets helps maintain system efficiency, security and compliance as well as improves data relevance and accuracy.
== Process ==
=== Identification and Review ===
* Candidates for depreciation can be nominated any time. Whenever a respective team works in an area where a legacy dataset qualifies for deprecation it should be flagged.
* Integrated into the quarterly planning the Data Engineering (DE) team, with input from data stewards and others, identifies candidates for dataset deprecation. 
** Criteria for candidates
*** Expired TTL
*** Stale data, no updates in last x months
*** No downstream dependencies
*** Expired ownership
* The review may also be triggered by deprecation of a legacy system, data migration needs, disk space constraints, or other factors such as usage or dependency metrics.
=== Pre-deprecation Preparations ===
* In case a data steward exists for the candidate data set, the steward confirms that the dataset can be deprecated without adverse effects. Otherwise the data engineering team performs the mandatory investigations.
* This involves:
** Impact Analysis
*** Identifying and verifying downstream datasets, data pipelines, and dashboards derived from the dataset. Checks against dependencies are primarily made through Datahub, Airflow, GitHub and Notebook repositories.
** Phabricator ticket
*** Creating a Phabricator ticket to document the reasons for deprecation, affected downstream assets, and tagging relevant stakeholders.
** Communication (2x)
*** Posting the intent to deprecate on relevant Slack channels (#data-engineering-collab, #working-with-data) and data-related email lists to provide a grace period for feedback. This should be repeated at least once before the deprecation deadline.
=== Feedback Grace Period ===
* The grace period for feedback should be 30 days. Multiple reminders should be sent in advance.
=== Deprecation Execution ===
* Establish child tasks for deprecating associated dashboards, pipelines, and downstream datasets, and sets a timeline for completion.
* After confirming all affected data usages have migrated or been deprecated, the data steward officially deprecates or archives the dataset.
=== Archiving vs. Deletion Decision ===
* Post-deprecation, the decision to archive or delete the dataset is made based on:
*# The impact of keeping the dataset around (e.g., risks, disk space).
*# The presence of subscribers or dependents.
*# Core vs. non-core dataset considerations.
* Official channels (#data-engineering-collab and #working-with-data) are used for communications.
=== Archiving ===
* If archiving is chosen, the dataset is renamed by adding an "_archived" suffix, indicating it is no longer active but still accessible for historical reference or legal compliance.
* If technically possible the dataset should be marked as read-only.
* The archived status is documented in DataHub.
=== Deletion ===
*All data, data definitions, and associated code are removed.
* The deletion is documented in Phabricator and communicated through established channels.
Category:Data platform
Data Platform/Discover data

This page provides links to data documentation for private and public Wikimedia data sources. Its primary audience is WMF data analysts, product teams, and researchers who have an official non-disclosure agreement with the Wikimedia Foundation. 
* Private data requires production data access. It includes datasets in WMF's Data Lake: a large, analytics-oriented repository of data about Wikimedia projects.
* A selection of public data sources are linked here, but public Wikimedia data is described more fully at meta:Research:Data.
== Traffic data ==
Analytics data about wiki pageviews and site usage. 
== Content data ==
Datasets that contain full content of revisions for Wikimedia wikis.
== Contributing and edits data ==
Data about wiki revisions, pages, and users. Includes data about editors and their characteristics. 
== Instrumentation and events data ==
== How to query private data ==
Visit Analyze data to learn how to run queries and generate visualizations using WMF private datasets and analysis tools.
== Report data issues ==
Data Issue reports
Category:Landing page
Category:Data platform