Data Platform/Data access

In addition to a variety of publicly-available data sources, Wikimedia has the internal Data Platform, which allows a restricted, carefully-vetted set of users to perform research and analysis on confidential data (such as the IP addresses of readers and editors). This private data is stored according to our privacy policy and data retention guidelines.
The Data Platform also provides duplicate copies of publicly-available data for ease of use.
== Do you need it? ==
Private data lives in same server cluster that runs Wikimedia's production websites. Often, this means you need production access to access it.
However, since this access gets you closer to both those production websites and this confidential data, it is not freely given out. First, you have to demonstrate a need for these resources. Second, you need to have a non-disclosure agreement with the Wikimedia Foundation. If you're a Foundation employee, this was included as part of your employment agreement. If you're a researcher, it's possible to be sponsored through a formal collaboration with the Wikimedia Foundation's Research team.
=== User responsibilities ===
You must remember this access is extremely sensitive. You have a duty to protect the privacy of our users. As Uncle Ben says, "with great power comes great responsibility". Always follow the rules outlined in the Acknowledgement of Server Access Responsibilities, even if you don't have requested ssh access to stat100x clients, since it contains good guidelines about how to handle sensitive data.
In addition, keep in mind the following important principles:
* Read data access guidelines, 
*Be paranoid about personally identifiable information (PII). Familiarize yourself with the data you are working on, and determine if it contains any PII. It's better to double and triple check than to assume anything. If you have any doubt ask the Data Engineering team. Please see the data retention guidelines.
*Don't copy sensitive data (for example, data accessible only by users in the analytics-privatedata-users group) from its origin location to elsewhere (in HDFS or on any other host/support) unless strictly necessary. And most importantly, do it only if you know what you are doing. If you are in doubt, please reach out to the Data Engineering team first.
*Restrict access. If you do need to copy sensitive data somewhere, please make sure that you are the only one able to access the data. For example, if you copy Webrequest data from its location on HDFS to your /user/$your-username directory, make sure that the permissions are set to avoid everybody with access to HDFS to read the data. This is essential to avoid accidental leaks of PII/sensitive data or retention over our guidelines.
*Clean up copies of data. Please make sure that any data that you copied is deleted as soon as your work has been done.
If you ever have any questions or doubts, err on the side of caution and contact the Data Engineering team. We are very friendly and happy to help!
== Requesting access ==
If after reading the above you do need access to WMF analytics data and/or tools, you'll need to submit a request on Phabricator and add the project tag SRE-Access-Requests: Follow the steps at SRE/Production access#Access Request Process.
If you already have access and you only need to get kerberos credentials, it is sufficient to create a task with the project tag Data-Engineering: Create a ticket requesting kerberos credentials.
Read the following sections to figure out what you'll access levels you should request in your ticket.
Please follow the instructions Production access request instructions for any of the access types. We need a paper trail and a standard form in order to keep track of requests and understand why they are happening. When submitting the Phabricator ticket, you may edit the description accordingly to match the request you are asking for. E.g. if you don't need SSH access, you don't need to provide an SSH key.
== Access Levels ==
There are a few varying levels and combinations of access that we support. 
Levels of access
 	Requesting 	Gives you
Required for everything below	Developer account added to
wmf/nda LDAP group 	Basic access to Superset & Turnilo, but not private data-based dashboards†
Required for everything below	Shell account added to analytics-privatedata-users group	Everything above, plus:
Access to private data-based dashboards†
Required for everything below	SSH access 	Everything above, plus:
* access to analytics client servers ("stat boxes")
* access to internal Jupyter (like PAWS but more)
* querying the MediaWiki replicas
* using a GPU (on some stat boxes)
Optional 	Kerberos principal 	Everything above, plus:
* access to data in Hadoop (HDFS)
* querying with Hive/Spark/Presto/wmfdata
* loading data into the Data Lake
* running/scheduling jobs
Optional 	Shell account with access to a team group 	Everything above, plus:
Managing shared jobs, files in HDFS, and data in Hive
† If a dashboard/chart in Superset uses a dataset that is accessible in Turnilo, it is not private data-based. There are certain extra steps that need to be taken to make datasets available to users with just basic Superset/Turnilo access, so early-in-development and experimental dashboards are often built using datasets that require private data access.
This might all be confusing if you are just trying to figure out what to put in your Phabricator SRE-Access-Requests ticket. Below are a some common use cases of what you might be trying to request.
== What access should I request? ==
If you need access to...
=== Dashboards in web tools like Turnilo and/or Superset that do not access private data ===
* LDAP membership in the wmf or </tt>nda</tt> LDAP group. 
=== Dashboards in Superset / Hive interfaces (like Hue) that do access private data ===
* LDAP membership in the wmf or </tt>nda</tt> LDAP group.
* Shell membership in the `analytics-privatedata-users` group
Note to SREs granting this access: This can be done by declaring the user in Puppet as usual, but with an empty array of ssh_keys.
''
=== SSH login to analytics clients without Hadoop access ===
This is a rare need, but you might want it if you just want to use a GPU on a stat box, or access to MediaWiki analytics MariaDB instances.
* LDAP membership in the wmf or </tt>nda</tt> LDAP group. 
* Shell membership in the `analytics-privatedata-users` group
* An ssh key for your shell user
=== SSH login to analytics clients with Hadoop access ===
* LDAP membership in the wmf or </tt>nda</tt> LDAP group. 
* Shell membership in the `analytics-privatedata-users` group
* An ssh key for your shell user
* A Kerberos principal
=== All of the above ===
If you are a WMF engineer wanting to work with analytics data, most likely you'll want all of these access levels together:
* LDAP membership in the wmf or </tt>nda</tt> LDAP group.
* Shell membership in the `analytics-privatedata-users` group
* An ssh key for your shell user
* A Kerberos principal
If needed for work on your team, you may also want Team specific shell (posix) group membership (see below).
== Analytics shell groups explained ==
=== Generic data access ===
;analytics-privatedata-users (no kerberos, no ssh)
:The Analytics team offers various UIs to fetch data from Hadoop, like Turnilo and Superset. They are both guarded by CAS authentication (requiring the user to be in either the wmf or the nda LDAP groups), fetching data from Druid (currently not authenticated). Superset is also able to fetch data from Hadoop/Hive on behalf of the logged in user via a (read-only) tool called Presto. There are two use cases:
:* Sql-lab panel: the user is able to make sql-like queries on Hadoop datasets (pageviews/event/etc..) without the need to log in on a stat100x host.
:* Dashboards: data visualized in dashboards fetched from Hadoop.
:In both cases, Superset works on behalf of the user, so eventually the username will need to hold read permissions for Hadoop data to correctly visualize what requested. This is guaranteed by being into analytics-privatedata-users, that gets deployed on the Hadoop master nodes (without ssh access) to outline user permissions on HDFS. This is why some users might want to be in the group without either kerberos or ssh.
:Access of this kind, a shell group without the actual shell access, is managed by SRE. To request it, use the Request shell access template in Phabricator and clarify in the title "no server access" and leave the "SSH public key" point blank. Additionally, if you are not yet in either of the "wmf" or "nda" LDAP groups, make sure to ask for this at the same time in the task. (Check https://ldap.toolforge.org/group/wmf or https://ldap.toolforge.org/group/nda to know if you're already in the LDAP group.). For example request, see T305634.
;analytics-privatedata-users (no kerberos)
:Grants access to the analytics clients, GPUs and to MariaDB replicas (using the credentials at /etc/mysql/conf.d/analytics-research-client.cnf).
;analytics-privatedata-users (with kerberos)
:Grants access to all the analytics clients, the analytics cluster (Hadoop/Hive) and the private data hosted there, and to MariaDB replicas, using the credentials at /etc/mysql/conf.d/analytics-research-client.cnf.
:Users in this group also need a Kerberos authentication principal. If you're already a group member and don't have one, follow the instructions in the Kerberos user guide. If you're requesting membership in this group, the SRE team will create this for you when they add you to the group.
;analytics-admins
:This and similar groups (like statistics-admins, eventlogging-admins, and statistics-web-users) are for people doing system maintenance and administration, generally as part of a WMF engineering team. The analytics-admins group, for example, is for people working on the Data Engineering team or collaborating with Data Engineering through a value stream. The list of users currently in each group is available in this configuration file. Note that access to analytics-admins is not required if the user is an SRE, since they will be added to the ops group which includes all permissions applied to analytics-admins.
=== Team-specific ===
;analytics-wmde-users
:For Wikimedia Deutschland employees, mostly used for crons running automation jobs as the analytics-wmde system user. Grants access to all stat100x hosts, to the MariaDB replicas via /etc/mysql/conf.d/research-wmde-client.cnf and to the analytics-wmde system user. It is not required that every WMDE user is placed into this group, only those who needs to take care of the aforementioned automation will require access (so they'll ask it explicitly).
;analytics-search-users
: For members of the Wikimedia Foundation Search Platform team , used for various Analytics-Search jobs). Grants access to all stat100x hosts, an-airflow1001 and to the analytics-search system user.
;analytics-product-users
:For members of the Product Analytics & Movement Insights teams, used for various analytics jobs. Grants access to all stat100x hosts, and to the analytics-product system user.
;analytics-research-users
:For members of the Research team, used for various jobs. Grants access to all stat100x hosts, an Airflow instance, and to the analytics-research system user.
;analytics-platform-eng-users
:For members of the Research team, used for various jobs. Grants access to all stat100x hosts, an Airflow instance, and to the analytics-platform-eng system user.
=== Groups to avoid (deprecated) ===
;researchers
;analytics-users
===Host access granted===
There used to be a lot of differences in what hosts an Analytics POSIX group could have had access to, but now there is none anymore.
===Data access granted===
Access Groups 	Hadoop access
(No private data)	Hadoop access
(Private data)	Mariadb credentials 	System user 	Other
analytics-privatedata-users	yes	yes	analytics-research-client.cnf 	analytics-privatedata	
analytics-wmde-users 	 	 	research-wmde-client.cnf (only on stat1007)	analytics-wmde 	
analytics-search-users 	 	 	 	 	Airflow admin
analytics-product-users 	 	 	 	analytics-product 	
=== Shell access expiration ===
Data access is given to collaborators and contractors with a time limit. Normally the end date is set to be the contract or collaboration end date. For staff data access terminates upon employment termination unless there is a collaboration in place.
Once a user is terminated their home directory is deleted, if the team wishes to preserve some of the user work (work, not data as data as strict guidelines for deletion) it can be done via archiving that work to hadoop. Please file a phab ticket to have this done. Archival to hadoop would happen in the following directory:
 
/wmf/data/archive/user/<username>
== LDAP access ==
Some Analytics systems, including Superset, Turnilo, and Jupyter, require a developer account in the wmf or nda LDAP groups for access. 
If you need this access, first make sure you have a working developer account (if you can log into this wiki, you have one). If you need one, you can create one at mw:Developer_account.
Note that a developer account comes with two different usernames; some services need one and some services need the other. You can find both by logging into this wiki and visiting the "user profile" section of Special:Preferences. Your Wikitech username is listed under "Username", while your developer shell username is listed under "Instance shell account name". Thankfully, there's only one password!
Then, create a Phabricator task: Read and follow the instructions for LDAP-access-requests to request getting added to the appropriate group. Make sure you include both your usernames.
Note that this access has similar requirements to shell access: you will need to either be a Wikimedia Foundation employee or have a signed volunteer NDA.
== Accounts and passwords explained ==
There are too many different accounts and passwords one has to deal with in order to access analytics systems. For now it's what we've got. Let's try to explain them all explicitly.
* LDAP AKA Wikitech AKA Developer accounts are the same. There are 2 usernames for this account, but only one password.
* POSIX AKA shell AKA ssh accounts are the same. The username is the same as your 'shell username' for your LDAP account. There is no password, only an ssh key pair.
* Kerberos uses your shell username and a separate Kerberos account password, and grants you access to distributed systems like Hadoop.
=== Developer account ===
LDAP is used mostly for web logins. An LDAP account has 2 usernames, the 'Wikitech' username and the shell username, as described above. The password for these is the same. 
Since LDAP account creation is handled by Mediawiki and also allows you to log into Wikitech (this wiki), LDAP accounts are sometimes referred to as your 'Wikitech' account or your 'Mediawiki developer account'. These terms all mean the same thing. 
Analytics web UIs (like Jupyter, Turnilo, Superset, etc.) require that you have an LDAP account in specific groups. Membership in these groups authorize access.
=== Shell ===
To log into a production server, you need an explicit POSIX shell account created for you. This is handled by SRE. POSIX user accounts are often also referred to as your shell or ssh account, as ssh allows you to remote login and get a shell (terminal) on a production server. At WMF, POSIX user accounts do not use passwords. Instead, you login via ssh using an ssh key pair.
Access to specific production servers is managed by membership of your POSIX account in specific groups, e.g. analytics-privatedata-users.
=== Kerberos ===
Kerberos is only needed when using a distributed system like Hadoop. You can ssh into a single production server with your POSIX account, but other production servers that you are not directly logged into have no way of knowing you are authorized to access them. Kerberos solves this problem. After logging into a server with ssh, you authenticate to Kerberos with kinit and your Kerberos password (this is a totally different password than your LDAP one). Then, when using a distributed system, other servers can interact with Kerberos to determine if your access should be authorized.
Category:Data platform