nyuuzyou/wikis huggingface dataset:
Data Platform/Evaluations/2021 data catalog selection/Rubric/Amundsen

=== Core Service and Dependency Setup ===
=== Ingestion Configuration ===
=== Progress Status ===
=== Perceptions ===
=== Outcome ===
== Razzi's take on Amundsen ==
Pros:
- simple architecture of 3 flask services all in python (as opposed to Datahub using java and python)
- ingestion architecture is simple: python scripts or airflow dags that make http api requests
- "social" ui features, like frequent users and owners
- loose coupling means you can use a relational database as the data store rather than neo4j (https://github.com/amundsen-io/amundsenrds)
Cons:
- seems like the community is losing steam: https://github.com/amundsen-io/amundsen#blog-posts-and-interviews has a flurry of events in 2019/2020 but nothing in 2021
- only supports polling for data updates, unless we also deploy atlas. Push ingest api is on their roadmap
- documentation is somewhat lacking; few ingestion examples, and broken links in docs
- some dependencies are getting out of date: elasticsearch version 6 (v7 was released 2019), nodejs version 12 (v13 was released 2019)
Amundsen was created by Lyft and is now hosted by the Linux Foundation.
Category:Data platform
Data Platform/Evaluations/2021 data catalog selection/Rubric/Atlas

=== Core Service and Dependency Setup ===
Atlas 2.2.0 was downloaded as a tarball and compiled on an-test-coord1001 without using any root privileges.
At a later date the HEAD from the GitHub repository was also tried as version 3.0.0-SNAPSHOT.
We used Maven to build the project and selected the BerkeleyDB & Apache Solr profile, which automatically built and started the Solr and Zookeeper dependencies on the same host.The daemons were executed with bin/atlas_start.sh and the web service was available on port 21000.
=== Ingestion Configuration ===
The key ingestion elements that we wanted to get working were the Hive Hook and Bridge components, along with the utility script import_hive.sh
The hook and bridge provide for real-time metadata synchronization in Atlas, whenever data is changed in Hive.
The import_hive.sh script provides for a one-off import of existing Hive data.
The bulk of the time spent on the evalution was in trying to get the script working.
=== Progress Status ===
Getting the import script to talk to the Hive Metastore using an individual's existing Kerberos session took some time to get working. Unfortunately, once this had been achieved we discovered that the Hive integration with Atlas depends on having Hive version 3.1.0 or later. We currently have Hive version 2.3.6
Therefore, the only ways in which we could proceed with Atlas and its Hive integration were either:
* Upgrade our existing Hive services, along with the underlying Hadoop services
* Downgrade Atlas to version 1.2.0
The upgrade option was the more appealing of the two, but it would require a great deal of work before we could continue with the evaluation.
=== Perceptions ===
In some ways, Atlas really seems like it might have hit the spot for this requirement. The real-time integration with Hive would have been particularly useful and other projects, such as Amundsen, can build upon this further.
However, there are also a number of ways in which the project did not impress.
* The community did not respond to our requests for assistance on the mailing list.
* There were errors in the pristine 2.2.0 tarball that prevented building, suggesting a lack of quality.
* The monolithic nature of the project makes it difficult to address a single component, such as the Hive connector.
=== Outcome ===
We ceased working on the prototype once the requirement for Hive 3.1 became clear.
Category:Data platform
Data Platform/Evaluations/Event Platform/EventStreams

The Main EventStreams documentation page is at Event_Platform/EventStreams
== RCStream Deprecation ==
RCStream has needed a revamp for a while now. EventStreams is an effort to build a service like RCStream that fits better into our HTTP based architecture, and that can expose more than just the recentchanges stream.
Analytics plans on working with the RCStream community over Q3 to get them to switch to EventStreams, and hope to turn off RCStream by the end of Q4, if not sooner.
== Meetings ==
===2016-11-30. RCStream deprecation and Ownership===
Attendance: Brandon, Faidon, Mark, Nuria, Otto
Action Items:
- Analytics to own communication on RCStream deprecation
- Brandon to merge otto's patches regarding service hosting 
Rough notes:
There are two pieces to this conversation: generic stream service and RCStream https://wikitech.wikimedia.org/wiki/RCStream. RCStream is understood to be low volume, not really user facing (rather tool-facing). We haver it set up through 
cache misc . What about the future? Open pipes is an architecture challenge, not cacheable
RCstream data is not cacheable by its very nature. On near term we want to make sure we are supporting just low volume use cases. We see it as a way for people to build tools. 
Need to document the fact that we only support a low volume use case. 
Analytics would own RCStream deprecation and transition to new EventStreams architecture, we will talk to product (who? Roan's team in editing maybe?) to keep them in the loop.
We plan on deprecating RCStream in a 6 months timeframe. 
Future plans would require an owner from product, let's try to give editing an FYI. 
There are issues with RCStream not being https -able (issues with the currently supported socket io libraries), ops was going to do chnages to this extent but we will not be doing those as the migration woudl take care of them. 
We also need to clean up the mediawiki end as events are published to the RC streams in different ways. We can probably make IRC recent changes https://www.mediawiki.org/wiki/Manual:RCFeed read from kafka and deprecate that piece of publishing code entirely.
Category:Data platform
Data Platform/Mysql/Utility Datasets

The analytics MySQL server on analytics-store.eqiad.wmnet contains a datasets database. This is intended to serve as a repository of useful, fixed datasets of general information - examples would be conversions between ISO country codes and the actual country names, or
=== country_info ===
A dataset of information about various countries. Useful for geolocation, particularly around grouping countries together or rendering the output of our geolocation software human-readable. Contains:
mysql:research@analytics-store.eqiad.wmnet [datasets]> DESCRIBE country_info;
+----------------+------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------+------+------+-----+---------+-------+
| country_code | blob | YES | | NULL | |
| continent_code | blob | YES | | NULL | |
| country_name | blob | YES | | NULL | |
+----------------+------+------+-----+---------+-------+
3 rows in set (0.01 sec)
*country_code is the ISO country code of a particular nation;
*continent_code is the corresponding ISO continent code;
*country_name is the corresponding human-readable, English-language name for that country.
Category:Data platform
Data Platform/Systems

These subpages explain in technical detail the systems that process data for analytics at Wikimedia Foundation. 
They include information about setup, maintenance, architecture, and more.
== Search within the Data Platform/Systems docs ==
type=fulltext
prefix=Data_Platform/Systems
width=40
arialabel=Search the Data Platform systems docs
searchbuttonlabel=Search the Data Platform systems docs
== Child Pages of ==
<splist
 listStyle=bar
 kidsonly=yes
/>
----
== All Subpages of ==
Category:Data platform
Category:Data platform systems
Data Platform/Systems/AQS/Scaling/2017/Cluster Expansion

== Goals ==
Expand the current AQS Restbase cluster with more hosts to improve fault tolerance and available storage space. aqs100[789] nodes are added to the current cluster composed by aqs100[456].
== Current cluster ==
* aqs100[123].eqiad.wmnet
* Two Cassandra instances per node.
* Cassandra 2.2.6
=== Problems ===
* More Cassandra dedicated storage space is needed to avoid deleting old data (setting a retention policy).
* More Cassandra instances in the cluster improve fault tolerance to hardware failures.
== Related tasks ==
https://phabricator.wikimedia.org/T155654
== Remarks ==
The cluster is expanded with three hosts, running two Cassandra instances each (to a total of 6 more instances added to the cluster). We decided to use the following configuration for rack_awareness:
* 1007,1004 -> rack01
* 1008,1005 -> rack02
* 1009,1006 -> rack03
This keeps our replication factor of 3 equal to the number of Cassandra logical racks. The major benefit is more predictability when reasoning about the cluster.
Example: bootstrapping all the new 6 instances on the new nodes starting from the existing ones on aqs100[456]
* 1007-a would stream data from 1004-a and 1004-b
* 1007-b would stream from 1004-a, 1004-b, and 1007-a (and afterward we would run cleanups on 1004-a, 1004-b, and 1007-a)
* 1008-a would stream from 1005-a and 1005-b
* 1008-b would stream from 1005-a, 1005-b, and 1008-a (and afterward we would run cleanups on 1004-a, 1004-b, and 1007-a)
* 1009-a... 
The thing to keep in mind to understand the above example is that Cassandra always tries to stream data from the smallest subset of instances to avoid complex consistency validation issues. Since the replication factor is 3 and the number of logical racks is 3, all the data is available on each rack (and split among multiple instances).
Category:Data platform
Category:Data platform systems
Data Platform/Systems/Airflow/Developer guide/Normalize a DAG

This will serve as a guide to take an existing DAG and update it with the latest Best Practices of our Airflow Platform.
===== Setting up Developer Environment =====
* Install Conda and other dependencies 
*Create a new env to run tests, see how to here
** pytest runs unit tests, if it runs cleanly you have a working environment
===== Create your branch =====
* Make the required changes
** (Copy result of Scope discussion here)
===== Testing the DAG =====
* Push changes to a new branch
* Login into your preferred stat box
* Clone the airflow-dags repository and checkout your branch
* Confirm you are working on your branch and you can see your changes
* Configure the DAG for testing
** Set the DagProperties or VarProperties to read from production data but write to test outputs
*** in some cases you may need to create test tables that mirror the production ones
*** if the job you are testing goes over a lot of data, like a month, it may be appropriate to limit the input to speed up testing
** use /user/<<your-user>> as the output folder base path in HDFS
* from the airflow-dags folder run 
 kinit
 ./run_dev_instance.sh -p 8080 analytics
* This takes a couple of minutes the first time, follow the instructions
** Set up the SSH Tunnel
** Access the Airflow UI
* Turn on the DAG in the UI, it should automatically run with your settings from above. You can check the "Rendered Template" of each task to see what property values it is using.
===== PYTEST =====
* Run pytest again
*Go through fixtures and verify they make sense
*Then rebuild fixtures by adding the keyword
 find tests -name \*.expected -exec rm '{}' \;
 REBUILD_FIXTURES=yes pytest
===== Check Data =====
* compare your version of data with the official production version
Data Platform/Systems/Bigtop Packages

=== Overview ===
We use Apache Bigtop (https://bigtop.apache.org/) as our Hadoop distribution and use their toolchain to build Debian packages for all of the components. The upstream source for Bigtop is here: https://github.com/apache/bigtop/
They maintain a branch called branch-1.5 which was the version that we used until recently. Unfortunately for us, they have decided not to support Debian 11 Bullseye or later as an installation target, whereas we have had to do so. Therefore, we have had to create our own fork of their repository, which is https://gitlab.wikimedia.org/repos/data-engineering/bigtop
The gradle build system is used throughout bigtop, along with puppetized build slaves running under docker.
An example command to build a single component, such as hadoop, under bullseye is as follows:
docker run --rm -v `pwd`:/ws --workdir /ws bigtop/slaves:1.5.0-debian-11 bash -c '. /etc/profile.d/bigtop.sh; ./gradlew allclean hadoop-pkg'
The list of components that we currently build with bigtop is currently:
* bigtop-groovy
* bigtop-jsvc
* bigtop-tomcat
* bigtop-utils
* hadoop
* hbase
* hive
* mahout
* oozie
* solr
* spark
* sqoop
* sqoop2
Unlike most other software that we build, bigtop packages are currently built on an engineer's workstation and then uploaded to the APT repository for serving
=== WMF Build Script ===
We have added a couple of scripts to our branch-1.5 since:
# We have a specific set of components that we need to build
# We use an operating system that is unsupported
The top level script to run is: build_all_bigtop_distros_wmf.sh
This script does the following:
# Creates build slaves for debian-10 and debian-11
# Uses each build slave to run build_bigtop_wmf.sh
=== Package Amendment ===
The default build mechanism in bigtop does not include any information about which distribution the packages were built for.
This causes an issue for us, since we have to ensure that we have packages available for both buster and bullseye on apt.wikimedia.org for distribution.
In order to get around this, the build_bigtop_wmf.sh script does the following for each package file generated.
# Unpack the deb file including the metadata files with dpkg-deb -R
# Modify the DEBIAN/control file and append either -deb10 or -deb11 to the Version: field
# Re-pack the deb file with dpkg-deb -b including either -deb10 or -deb11 to the file name
This will allow us to host all of these packages with reprepro concurrently.
Category:Data platform
Category:Data platform systems
Data Platform/Systems/CloudnativePG

==== What it is ====
cloudnative-pg is an open-source Kubernetes operator used to provision PostgreSQL clusters in Kubernetes. 
It supports the following features:
* read-write/read-only service discovery
* primary -> standby replication
* automatic standby -> primary failover
* connection pooling (via pgBouncer)
* backups / PIT recovery (via pgBarman)
* WAL archiving (via pgBarman)
* placement of cluster pods accross network zones 
* and many more
The operator relies on Kubernetes Custom Resource Definitions , such as Cluster, ImageCatalog, Pooler, etc, each defining how the cluster should be deployed and operated. These CRDs are defined in the cloudnative-pg-crds chart, and the operator chart itself is defined in cloudnative-pg.
The upstream chart integration was discussed and approved in Helm/Upstream Charts/cloudnative-pg.
==== How we build it ====
Both the operator and PostgreSQL images (used to deploy clusters themselves) are defined and built in https://gitlab.wikimedia.org/repos/data-engineering/postgresql-kubernetes/. The pgbouncer image, used when deploying Poolers, is defined and built in https://gitlab.wikimedia.org/repos/data-engineering/pgbouncer. 
==== Where we run it ====
At the time of writing, we only plan to deploy and use it in the dse-k8s-eqiad Kubernetes cluster, to support on-demand provisioning PG clusters for Airflow, with the actual data written to Ceph, via PersistentVolumes, as described in our Airflow - High Availability Strategy document.
We run it in the cloudnative-pg-operator namespace:
root@deploy1003:~# kubectl get pod -n cloudnative-pg-operator
NAME READY STATUS RESTARTS AGE
cloudnative-pg-756fdbb74c-7txv9 1/1 Running 0 117m
==== How we deploy it ====
root@deploy1003:~# kube_env admin dse-k8s-eqiad
root@deploy1003:~# cd /srv/deployment-charts/helmfile.d/admin_ng
root@deploy1003:~# helmfile -e dse-k8s-eqiad -i diff/apply
==== Clusters ====
Clusters details are kept up to date in Data Platform/Systems/CloudnativePG/Clusters
Data Platform/Systems/Cluster/Geotagging

Geotagging functions in Hadoop are provided by jars available at hdfs:///wmf/refinery/current/artifacts
== Libraries ==
==== refinery-core.jar ====
org.wikimedia.analytics.refinery.core.Geocode exposes two functions
Function Name 	Data Returned
getCountryCode(String ip) 	country code
getGeocodedData(String IP)	<map> containing geocoding information:
* continent
* country_code
* country
* subdivision
* city
* postal_code
* latitude
* longitude
* timezone
==== refinery-hive.jar ====
This library provides wraper functions usable as a hive UDF
Hive UDF 	Wrapped Function
org.wikimedia.analytics.refinery.hive.GetCountryISOCodeUDF	org.wikimedia.analytics.refinery.core.Geocode.getCountryCode
org.wikimedia.analytics.refinery.hive.GetGeoDataUDF 	org.wikimedia.analytics.refinery.core.Geocode.getGeocodedData
== Updates ==
These functions use a regularly updated version of the MaxMind database that is downloaded on every node of the cluster in the folder /usr/share/GeoIP.
Currently we download MaxMind daily at 0300UTC on two servers: puppetmaster1001 (legacy puppet 5) and puppetserver1001 (new puppet 7). All the rest of the servers pull the updates from their respective puppet server (depending on their puppet version), every 30 minutes.
The pipeline is orchestrated with systemd. We should be alerted on failures. If you have access to puppet servers, you can check systemd logs with:
journalctl -u geoip_update_main.service
Category:Data platform
Category:Data platform systems
Data Platform/Systems/Coordinator

== The coordinator hosts ==
The Analytics team relies on two multi-purpose nodes called "an-coord100[1-2]" to run critical services like Hive and Presto.
In T257412 we started a major effort to reduce SPOFs in the Analytics infrastructure, since there was only one coordinator host (an-coord1001) and its hostname was hardcoded almost everywhere (puppet, refinery, etc..). This meant that the failover procedure was really complex and long to execute, so the Analytics team tried to reduce the gap. The final goal would be to have a very quick and easy (order of minutes) failover procedure for all the services, but it will take time. This document tracks the status of the failover procedures to give a good overview to how things run on the coordinator nodes.
== Services ==
This is the list of services broken down by node (As of January 2024):
=== an-coord1003 ===
Hive Metastore (port 9083)
Hive Server 2 (port 10000)
Presto coordinator
=== an-coord1004 ===
Hive Metastore (port 9083)
Hive Server 2 (port 10000)
== Hive ==
The Hive servers (port 10000) on every coordinators use their local metastore (port 9083) when needed (this is a standard for Hive). The metastores all fetch and store Hive sessions/tokens to the same mariadb database, in this case an-mariadb1001. See Data Engineering/Systems/Analytics Meta for more information on that database and how we are working to achieve high-availbility for it.
The analytics-hive.eqiad.wmnet is a DNS CNAME that points only to one of the coordinators, and it is set for all the Hive clients. This means that only one coordinator is active at any given time, the other one is in "standby" state.
How does it work from the Kerberos point of view? The keytab and principal that Hive uses is hive/analytics-hive.eqiad.wmnet@WIKIMEDIA on both coordinator nodes, so they share the same credentials and can answer to the same queries.
With a simple change of the analytics-hive.eqiad.wmnet CNAME in the DNS it is possible to change the coordinator that serves Hive traffic transparently (already tested, it works) without changing any job config!
== Failover ==
See Analytics/Systems/Cluster/Mysql_Meta#Failover
Category:Analytics cluster
Category:Data platform
Category:Data platform systems
Data Platform/Systems/Dashiki/Configuration

The Dashiki dashboarding system depends on pages on meta.wikimedia.org to configure dashboards and related metadata, such as annotations. Here we describe these configuration files and the schemas they use. Currently, these schemas are not enforced when you save an article, so users are responsible for making sure the configuration works.
== Annotations ==
Articles that specify annotations to use on dashboards need to have a path such as
Config:Dashiki:Annotations/<<project name>>/<<metric name>>
Where <<project name>> is something like Wikistats and <<metric name>> is something like editors. So, a concrete example is :m:Config:Dashiki:Annotations/Wikistats/totalPageViews.
Annotations can be defined here in the following format:
description: -- text describing this set of annotations
documentation: -- link to the metric's research page or other context
annotations: -- a list of annotations using one of these patterns:
 - date:
 title: 
 note: 
 - date:
 title:
 note: 
 relevantBreakdowns: -- list of splits to limit annotations to
 - dimension: -- the split (eg. editor_type) or "total" for no breakdown
 values: -- list of breakdown values to limit to, eg.:
 - group-bot
 - anonymous
Take a look at the examples or ask questions on the talk page here. The slightly complicated thing is that if you choose to use relevantBreakdowns, you need to include the "total" dimension in the list, with values: [ "total" ] if you want to also show the breakdown on the main, un-split graph.
== Dashboards ==
Articles that specify dashboard configurations need to have a path such as
Config:Dashiki:<<dashboard name>>
The format for the configuration depends on what type of layout your dashboard is using. These are detailed below:
=== Tabs Layout ===
=== Compare Layout ===
=== Metric by Project Layout ===
Category:Dashiki
Category:Data platform
Category:Data platform systems
Data Platform/Systems/DataHub/Upgrading

The upstream DataHub repository is: https://github.com/linkedin/datahub/
At the moment we maintain a fork of DataHub here: https://gerrit.wikimedia.org/r/admin/repos/analytics/datahub
Update - We are about to unfork from the upstream project and we will shortly migrate our build pipeline to GitLab.
== Deploy datahub CLI tool ==
The version of the CLI tool has to match the server version, so we have to:
* Update the datahub-cli version on the packaged virtual environment
* Build and publish to Archiva: Analytics/Systems/Archiva#Uploading_Dependency_Artifacts
* Update the artifact version and metadata ingestion jobs in the airflow jobs repository
Category:Data platform
Category:Data platform systems