nyuuzyou/wikis huggingface dataset:
Analytics/Archive/EventLogging/NotErrorLogging

== Eventlogging is not well suited to do error logging ==
There had been many talks in the past (and present, see: ) about using eventlogging to do client side errorl ogging. While eventlogging is well suited to handle events, and ingesting client side traces shares similarities with ingesting events, Eventlogging is really not well suited to be a client side error logging library for several reasons:
1) Eventlogging is designed around handling data that validates to a schema, while error messages might be json there is really no value on validating them against a schema. The error happened and it should be ingested by backend regardless of whether it validates, it is not "curated data". 
2) Any system we use to handle error logging should be tier-1, EventLogging is tier-2.
3) Any system to handle errors needs to be able to group by stacktraces and be good at handling free text, EventLogging is not for reasons in 1). It is made to deal with data that abides to a schema. It does not group events by free text (like stack traces) and an error that appears for a million pageviews in 1 hour will appear in the database a million times rather than appearing once with a count of 1 million. Because EventLogging is made for distinct events, and errors are not distinct events. All users of Chrome 68 might be running on the same error for the same reasons. This is a big deal and why a solution customized to the error space is needed. See 4).
4) There is no need to reinvent the wheel, [Sentry https://sentry.io/welcome/] is a well-stablished software to do this very thing: client side error logging. See attempts to install Sentry at the foundation: https://phabricator.wikimedia.org/tag/sentry/
While grouping server side stack traces is a core usage of sentry in order to deal with bursty traffic an error logging solution probably needs to do some client side normalization of stack traces and deduplication so errors are somewhat processed by the time they get to the server side. Sentry comes with a client side library that does part of this pre-processing.
5) Privacy concerns. This is a smaller concern that the ones listed prior but since it has come up listing it for completion: A log system is normally short retention. EL data is retained for 90 days and the whitelisted data is retained for a longer term. Erroneously whitelisting error messages for longer retention can lead to privacy concerns such as these: https://phabricator.wikimedia.org/T136851
Category:Data platform
Category:Data platform systems
Analytics/Archive/EventLogging/Publishing

WMF's EventLogging database is private, because it may hold sensitive information during a certain time window. To access it, one must be an employee of the Wikimedia Foundation or have signed an NDA. Hence, any reports or data sets based on EventLogging data are potentially harmful and need to be subject of review before they can be published.
== Publishing reports ==
We consider a report: any collection of prose, graphs and statistics that has been drafted by a human for the purpose of communicating some learning. It typically does NOT contain actual records of the database, nor parts of them.
Before publishing any report, you should ensure that it does not contain any potentially sensitive information, as defined bellow. If you are unsure if your report contains private data, please consult with the Research or Analytics teams.
== Publishing data sets ==
We consider a data set: a collection of (whole or partial) records extracted from the database for the purpose of enabling future analyses.
The preferred option is NOT to release any such data sets publicly. If you'd like to open an exception, please contact the Legal team AND also the Community Advocacy team to review your data set, and ensure that it contains no sensitive data. If you have other questions, please ask the Analytics team or the Research team.
=== Potentially sensitive data ===
* PII (Personally identifiable information), like clientIp, userAgent, userName, userId, editCount, and in general, any piece of information that can uniquely identify a physical or virtual person.
* User-inputed textual fields, like pageTitle, imageTitle, summary, userName, userText, etc. Schemas containing this kind of data are marked as such in the schema talk page.
Category:Data platform
Category:Data platform systems
Analytics/Archive/EventLogging/Sensitive Fields

=== Scenario 1: Entering personal information by mistake ===
A user enters sensitive personal information (credit card number, or others) into a text field like username, pageTitle, imageTitle, summary, etc. by mistake. After a while they realize that and immediately ask WMF to delete this information. A risk exists, that in the meantime, after the sensitive data was submitted and before its records were deleted, an EventLogging event was stored featuring this data.
* Thus, all fields that store textual information potentially inputed by the user (username, pageTitle, etc.) should be auto-purged in events older than 90 days.
=== Scenario 2: Editing as anonymous by mistake ===
An editor usually edits logged in, because they don't like their IP being stored in the revision table. However, this day they edit as anonymous by mistake. After a while they realize that and immediately ask WMF to delete this information. A risk exists, that in the meantime, after the sensitive data was submitted and before its records were deleted, an EventLogging event was stored featuring this data.
* Thus, all fields that store the IP address of an anonymous editor should be auto-purged in events older than 90 days.
Category:Data platform
Category:Data platform systems
Analytics/Archive/EventLogging pipeline

=== Value Proposition ===
For the Analytics team who maintain the EventLogging platform, the new EventLogging Pipeline is an architecture and implementation that leverages our big data infrastructure yielding capacity and reliability. Unlike the current implementation, the new architecture will make it easier to backfill data and handle 10x more data.
For product managers and analysts, the new EventLogging Pipeline will appear more reliable (because of capacity and ease of backfilling) unlike the current implementation where events are dropped or backfilling is slow.
=== Objective ===
Leverage existing infrastructure (Kafka & Hadoop) to scale the throughput capacity of EventLogging.
=== Key Result ===
* Current maximum throughput is roughly 500 events / second
* Handle at least 5000 events / second
=== Requirements ===
* Preserve teams ability to query events via SQL over a small amount of data (depending on throughput)
* Preserve all existing dashboards
* Teams can test their instrumentation in Beta Cluster
* Teams can join their data with mediawiki databases
=== Design ===
Given the complexity of the project, having multiple phases seems reasonable:
==== Phase 1 ====
After that phase, EventLogging still works full scale on MySQL, but also provides data to Hadoop for us to check everything works fine.
* Use Kafka as a new transportation layer for event messages between forwarders - processors - consumers processes (no data loss when processes crash).
* Have processor publish into schema_version Kafka topics to ease consuming coherent batches
* Rework MySQL consumers to take advantage of schema_version batches (optional values with default still to be fixed).
* Create a new Camus job to import Kafka schema_version topics to HDFS (one shot job to initialize HDFS with existing EventLogging and mediawiki MySQL data using Sqoop)
* Use graphite to chart per topic throughput and validation/error.
==== Phase 2 ====
After that phase, EventLogging becomes available for customers on Hadoop, and MySQL gets gently relieved of it services.
* Automate hive table creation based on schema_version definition.
* Automate data import from mediawiki into a single HDFS store (easier to query, using Sqoop and oozie?)
* Ask customers to move to Hive (Hue + JDBC? file copy? RESTBase instance?)
* Truncate old MySQL data.
==== Phase 3 ====
At the end of that phase, we have an easily scalable, fault-tolerant system.
* Rework stream processing (python forwarder, processor, consumer) with Spark Streaming.
* Anything else ?
=== Roadmap & Timeline ===
Work is being done as part of our ongoing priority for operational excellence. Resources dedicated to this priority will take on the work, not extra resources or projects are needed.
Analytics/Archive/FAQ

== Data Requests ==
The job of the Wikimedia Foundation's Data Engineering team is to provide infrastructure and services so you can put/get your statistics into a system via a fairly simple code snippet. We are an infrastructure team that provides analytics infrastructure. Please have in mind that the data engineering team does not provide teams with numbers to measure their KPIs against. We will support your data access request, grant access to our systems if it pertains and help you as needed be. You can find us on IRC on #wikimedia-analytics.
Please see the Research FAQ on Meta to understand who at WMF owns research-related processes and resources, and on where to find data or statistics about a specific Product Audience (such as editors or readers).
== What is the Analytics Cluster? ==
The Analytics Cluster is a catch-all term for compute resources and services running inside of the Analytics VLAN, which itself is inside of WMF production networks. Individual systems in the Analytics Cluster can be referred to more specifically, e.g. Analytics Hadoop Cluster, Druid Analytics, Hive, etc.
== How do I transfer files between stat boxes? ==
See https://wikitech.wikimedia.org/wiki/Analytics/Systems/Clients#Rsync_between_clients
== What do I do if I don't know what to do ==
Wait 5 minutes and try again. It might work, and even if it doesn't you'll be generally happier for having taken a short break.
Analytics/Archive/Hue

Hue is a web interface for the Hadoop cluster, most notably for running Hive queries. Our Hue instance is at https://hue.wikimedia.org.
General documentation for Hue can be found on our instance at https://hue.wikimedia.org/help/.
== Access ==
To access Hue, you will just need wmf or nda LDAP access. For more details, see Analytics/Data access#LDAP access.In the past, you also needed someone from Analytics Engineering to manually sync your LDAP account to Hue. This is no longer necessary.
If you want to use Hue to query private data, you will also need analytics-privatedata-users shell access. 
== Hive query errors with Kerberos ==
In T242306 it was reported the following error while using the Hive query editor:
Error while compiling statement: FAILED: SemanticException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
The error is related to Kerberos authentication, and can be fixed forcing the renewal of the Hive session via the following steps:
* In the Hue Hive editor, locate the 3-dots button at the top-right corner of the screen and hit it.
* Then hit "Session" and the "Recreate" button.
==Package and release a new version of Hue==
We package Hue from the cloudera upstream repository, https://github.com/cloudera/hue/
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/hue/+/refs/heads/master/debian/README.debian
== Notes ==
Analytics/Archive/Hue/Administration

Information about Hue can be found in the parent wiki page Analytics/Cluster/Hue. The Analytics team uses Hue as handy UI to check the status of the Hadoop jobs, since it offers a nice integration with Oozie and Hive. If this UI is down other tools can be used to gather the same information, so it is not critical for the Analytics team.
== Syncing new Hue accounts ==
When a new Hadoop user is added, an admin needs to give them a Hue account. Once T127850 is resolved, this process should be automatic.
#Log into http://hue.wikimedia.org
#In the upper right, click on your username, and select Manage Users (you will only be able to do this if you are Hue admin. Another admin can make you one.)
#Click 'Add/Sync LDAP User'
#Fill in the form with their UNIX shell username (not their Wikimedia developer account username), deselect both 'Distinguished name' and 'Create home directory', and click 'Add/Sync user'
== Restarting Hue ==
There is an Apache httpd proxy server running in front of Hue, so in case of HTTP failures there are two places to check:
* /var/log/apache2/etc. for the proxy part (even if it rarely gives problems)
* /var/log/hue/etc.. in which the CherryPy server logs the Hue python application errors
Usually a simple systemctl restart hue fixes the problem.
== Python processes ==
Hue usually creates only a process for the CherryPy Python server, but if kerberos is enabled then there will also be another process called kt-renewer responsible to periodically refresh its kerberos credentials via kinit. Logs of both can be found in /var/log/hue.
Analytics/Archive/Limn

Limn can do stuff
== Graph types ==
=== Time series ===
=== Maps ===
;Projections
:Accepted values are: 'albers', 'albersUsa', 'azimuthal', 'azimuthalEqualArea', 'azimuthalEquidistant', 'equirectangular', 'gnomonic', 'mercator', 'orthographic', 'stereographic'
== Examples ==
*reportcard
*ee-dashboard
*multimedia-metrics
Analytics/Archive/Pageviews

=== Summary ===
In 2014/15, the Analytics and Research Teams at Wikimedia developed a new and more comprehensive definition and algorithm to count pageviews. "Pageviews" or "Current Pageviews" refers to a tally using the new algorithm. As of 2015, many dashboards and reports continue to use the legacy definition of pageviews and those counts should be referred to as "Legacy Pageviews". 
[Note: In the comparison below, "legacy pageviews" refers to pageview numbers aggregated at the project level. Page-level pageview numbers were available based on unsampled webrequest logs even in the pre-2015 version, see e.g. pagecounts-raw.]
 	Legacy Pageviews 	Pageviews
(Current Pageviews)
Data Source 	sampled web-request logs	un-sampled web-request logs
Cons 	
* the data source is sampled
* the definition is several years old
* excludes the apps	
Pros 	 	
* the data source is un-sampled
* better detection and exclusion of automated traffic (spiders, web crawlers, bots, ....)
Examples 	WMF Quarterly Report 	
Specification	Here 	Here
Uses 	
* Wikimedia Report Card
Eventually some dashboards will be deprecated 
or migrated to use the current pageview definition.	
* Pageviews for Wikipedia on stats.wikipedia.org
* WMF Quarterly Report
=== Details ===
* The Research team has developed a new definition of Pageviews: https://meta.wikimedia.org/wiki/Research:Page_view
* The new definition is better because:
** it is based on all the web-request logs (not a 1:1000 sampling of them);
** it can detect and flag more automated traffic (which can be a significant fraction of total traffic on some wikis);
* The new definition will evolve over time as new ways of viewing content are developed (e.g. a mobile app using and APIs)
* The new definition is generated on the analytics cluster using Hadoop and web request logs
* Current pageview counts are being tallied starting May 1 2015
* Where appropriate, we will slowly transition uses of legacy pageviews to use current pageviews or we will point out where legacy pageviews are used.
* The previous definition is documented here: https://phabricator.wikimedia.org/diffusion/ANME/browse/master/pageviews/webstatscollector/pageview_definition.png
* The previous definition relied on a tool named webstatscollector: Analytics/Webstatscollector
=== Comparing current and legacy pageviews ===
Legacy pageview counts can/are larger than current pageview counts because of automated traffic. The current definition makes a better effort at counting traffic from real persons and excluding automata. Please take this into account when trying to plot year over year changes in traffic. For example, when looking at http://stats.wikimedia.org/EN/TablesPageViewsMonthlyCombined.htm there is a discontinuity in traffic on May 2015 because that is when the current pageview definition is used to report traffic.
Category:Pageviews
Analytics/Archive/Pageviews/Aggregation

Project Animal 	{wren}
Phabricator Ticket	https://phabricator.wikimedia.org/T100027
ETA 	Q1 FY2016
== Value Proposition ==
For Analysts, Product Managers, Researchers and the Wikimedia Community interested in analytics 
Who are dissatisfied with a lack of easy to get & granular pageview data 
Our Product is a table you can query via SQL 
That provides useful breakdowns (dimensions) while maintaining privacy 
Unlike our current pageview data dumps. 
== Deliverable ==
A table in a database that is query-able using SQL.
Dimension Name	No. of Possible Values	Examples
project 	1000 	frwiki
country 	300 	France
access_method 	3 	mobile_web | mobile_app
agent_type 	2 	spider
referer_class 	3 	
day 	? 	
Dimension Name			
project 			
country 			
access_method 			
agent_type 			
referer_class 			
article 			
day 			
== Privacy ==
It is possible the pageview count for a particular bucket will be so small it could be used to identify someone because perhaps only one person could have produced a particular pageview. Therefore when pageview counts are lower than X, the database will return 0.
== Implementation ==
Category:Pageviews
Analytics/Archive/Pentaho

== Pentaho instance on labs ==
Disclaimers: This setup is on labs and very much alpha, there is no guarantees of support.
=== Start/Stop Pentaho ===
 ssh <user>@dan-pentaho.eqiad.wmflabs
 nuria@dan-pentaho:/home/milimetric/biserver-ce$ ./start-pentaho.sh
=== Import Cube ===
To import a cube into Pentaho you need to:
==== Dump the data you want to import into a TSV file ====
This example is specific for the analytics research team
Within stat1003 type:
 mysql --defaults-extra-file='/etc/mysql/conf.d/research-client.cnf' -h analytics-store.eqiad.wmnet \
 -Dstaging -e "select * from my_table;" | cat > my_table.tsv
Note that | cat > my_table.tsv is necessary, because it transforms the mysql common tabular output into TSV format.
Also, Be sure your user belongs to research group or you'll get an error similar to:
 unknown variable 'defaults-extra-file'
Now you can compress it and scp it to Pentaho's instance. From your local machine:
 scp stat1003.wikimedia.org:my_table.tgz .
 scp my_table.tgz dan-pentaho.eqiad.wmflabs:
And you're done.
==== Create a table in the database Pentaho is looking at ====
From you local machine, enter Pentaho's instance:
 ssh dan-pentaho.eqiad.wmflabs
Open MySQL client:
 mysql -uroot -p -Dwarehouse
And create the table normally. Note that the columns must correspond exactly to those in the TSV file.
==== Import the data into the newly created table ====
Within Pentaho's instance, decompress the data and import it:
 mysqlimport -uroot -p --local --ignore-lines=1 warehouse my_table.tsv
Note that the TSV file must have the same name as the database table.
==== Create an XML file with the cube description ====
 <Schema name="My Schema">
 <Cube name="My Cube">
 <Table name="my_table"/>
 <Dimension name="Dimension1">
 <Hierarchy hasAll="true">
 <Level name="Dimension1" column="dimension_1" type="String"/>
 </Hierarchy>
 </Dimension>
 <Dimension name="Dimension2">
 <Hierarchy hasAll="true">
 <Level name="Dimension2" column="dimension_2" type="Boolean"/>
 </Hierarchy>
 </Dimension>
 ...
 <Measure name="Measure 1" column="measure_1" aggregator="sum"/>
 <Measure name="Measure 2" column="measure_2" aggregator="sum"/>
 ...
 </Cube>
 </Schema>
For more information, visit Mondrian documentation on cube file format:
http://mondrian.pentaho.com/documentation/schema.php
==== Upload the cube file to Pentaho ====
Go to http://pentaho.wmflabs.org/ and login:
 Login as an Evaluator -> Administrator
Then open the Data Sources Manager:
 File -> Manage Data Sources
Add a new analysis data source:
 +(Add) -> Analysis
When prompt, browse ... for the cube file and click Import.
Close the Data Sources Manager and create a new Saiku view:
 Create New -> New Saiku Analytics
Select the cube from the drop-down in the left. And that's it.
Analytics/Archive/Team/MailingList

The public Analytics mailing list is the main place where researchers, WMF analytics team members, data enthusiasts, other WMF staff, etc., talk about a variety of Analytics topics. We want to bring more discussions public but we want to establish some subject-line tags, to help us filter out the conversations we're not interested in. We can deliberate over these tags, the following is just a draft, please be bold. But when the list is stable, I propose we keep it stable so we can have dependable filters based on it.
* [Outage] - this thread includes details about a service or data product not being available for a while
* [Technical] - this thread contains technical implementation details or infrastructure details
* [Release] - new product, service, or data is being announced
* [Announcement] - general public announcement
Analytics/Archive/Team/Office Hours

The Analytics Engineering Team will have office hours on the second Monday of every month. Join us in . 
Some topics we're happy to discuss, but we're open to suggestions and general questions:
* Infrastructure: the storage and compute resources we have, and the technologies we support
* Datasets: what's available now and what our plans and timelines are
* Work: what our priorities are and how we see our work in context with the larger movement
We're going to be available starting at 15:00 UTC (16:00 CET, 10:00 EST, 07:00 PST), This is fairly early if you are west of the Rockies, so if you need a later meeting just let us know and we'll schedule an additional hour later in the day. We'll talk on IRC but are happy to use video/audio chat to go more in depth.
Process note: we should probably publish this on the engineering calendar or some other public calendar so we can get a publicly shareable link. Google Calendar doesn't seem to provide any other mechanism for sharing a single event.
Analytics/Archive/Team/Quarterly Reviews

This page lists links to Analytics Quarterly Reviews notes
* Apr–Jun 2012
* Jan–Mar 2014
* Apr–Jun 2014 
* Jul–Sep 2014
* Oct–Dec 2014
* Jan–Mar 2015
* April–June 2015
* July–Sept 2015
* Oct–Dec 2015
* Jan–Mar 2016
* Apr–Jun 2016 
* Jul–Sep 2016
* Oct–Dec 2016
* Jan–Mar 2017