nyuuzyou/wikis huggingface dataset:
Maps

This page describes the technical aspects of deploying Maps service on Wikimedia Foundation infrastructure. 
The service is being actively redesigned, documentation can be found under Maps/v2
== Intro ==
The maps service consists of Kartotherian - a nodejs service to serve map tiles, Tilerator - a non-public service to prepare vector tiles (data blobs) from OSM database into Cassandra storage, and TileratorUI - an interface to manage Tilerator jobs. There are 20 servers in the maps group: maps20[01-10].codfw.wmnet and maps10[01-10].eqiad.wmnet that run Kartotherian (port 6533, NCPU instances), Tilerator (port 6534, half of NCPU instance), TileratorUI (port 6535, 1 instance). Also, there are four Varnish servers per datacenter in the cache_maps group.
== The infrastructure ==
* OSM Database
* Tile storage
* Kartotherian
* Tilerator
* Maintenance
== Miscellaneous ==
* Dynamic tile sources
* Maps/Debugging
== Development processes ==
* Kartotherian/Tilerator deployment
* Kartotherian/Packages
== Puppetization and Automation ==
=== Prerequisites ===
* passwords and postgres replication configuration is set in Ops private repo (root@puppetmaster1001:/srv/private/hieradata/role/(codfw|eqiad)/maps/server.yaml)
* other configuration in puppet/hieradata/role/(codfw|common|eqiad)/maps/*.yaml
* cassandra::rack is defined in puppet/hieradata/hosts/maps*.yaml
* the role::maps::master / role::maps::slave roles are associated to the maps nodes (site.pp)
== Monitoring ==
* KPI dashboard
* Usage dashboard
* Usage - varnish
* Kartotherian - Grafana
* Kartotherian - Logstash
* Maps Cassandra - Logstash
* Tilerator - Grafana
* Tilerator - Logstash
== Subpages ==
Maps-migration

This page describes lesson learn and steps for Migrating Maps servers.
== Intro ==
In early 2019 we migrated maps from Debian Jessie to Debian Stretch (phab:T198622). The following lessons were learnt to help aid other migrations.
== Steps taken for stretch migration ==
 
* downtime the maps server
* disable puppet
* depool
* merge patch e.g: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/486062/
* run puppet on the other maps node e.g if maps1001 is being migrated, run puppet on maps100[234] to ensure that iptables rules are updated and maps1001 is excluded from the old cluster and can join the new one.
* run puppet on install server (for the new stretch DHCP configuration)
* reimage (make sure it isn't pooled after reimage)
 -- wmf-auto-reimage --phab-task-id T198622 --mask tilerator --conftool maps1001.eqiad.wmnet
* mask tilerator until we can check that cassandra / postgresql are up to date
* check that maps1001 postgres slave is initialiized, otherwise run:
 sudo systemctl stop postgresql@9.6-main.service
 rm /srv/postgresql/9.6/main/*
 sudo -u postgres /usr/bin/pg_basebackup -X stream -D /srv/postgresql/9.6/main -h <new_maps_master fqdn> -U replication -w && sudo systemctl restart postgresql@9.6-main.service 
* While the above command is running in the background, make sure recovery.conf is created with the necessary parameters in /srv/postgresql/9.6/main
* run puppet to create recovery.conf file
* restart postgresql@9.6-main
* check that cassandra joined the cluster
* unmask tilerator
* check that tilerator / kartotherian are working correctly
* increase cassandra replication factor
** SELECT * FROM system.schema_keyspaces; - To check replication factor
** ALTER KEYSPACE "v4" WITH replication = {'class':'SimpleStrategy', 'replication_factor':3}; # for v4 keyspace, make sure replication_factor is the equal to number nodes
** ALTER KEYSPACE "system_auth" WITH replication = {'class':'SimpleStrategy', 'replication_factor':2}; #for system_auth keyspace, replication factor should be total_nodes - 1
* nodetool repair on all maps nodes, sequentially
* (nodetool repair can continue in background while we continue with the next steps)
* pool
== Things to note for cassandra ==
Increasing replication_factor for system_auth keyspace causes the downtime because privileges are cleared and clients accessing cassandra cannot read or right. See https://wikitech.wikimedia.org/wiki/Incident_documentation/20190122-maps. So when doing this make sure servers in the cluster is depooled and downtimed and Immediately run the following commands: to restore privileges back.
GRANT SELECT ON ALL KEYSPACES to kartotherian;
GRANT SELECT ON ALL KEYSPACES to tilerator;
GRANT MODIFY ON ALL KEYSPACES to tilerator;
GRANT CREATE ON ALL KEYSPACES to tilerator;
GRANT MODIFY ON ALL KEYSPACES to tileratorui;
GRANT SELECT ON ALL KEYSPACES to tileratorui;
GRANT CREATE ON ALL KEYSPACES to tileratorui;
Maps/ClearTables

== Setup ==
A new machine will need imaging by ops. Once it is imaged, stop kartotherian, tilerator, and tileratorui services. With services stopped, load the map data. After map data is loaded, create a new Cassandra keyspace.
Once the setup is complete, launch tilerator, tileratorui, and kartotherian. Check for new errors with tail-tilerator, tail-tileratorui, and tail-kartotherian.
Maps/Dynamic tile sources

=== Dynamic tile sources ===
==== Cassandra ====
To create a new Cassandra data source, post something like this to the /sources as a text body. Default table name is tiles. If table or keyspace is not there, you have to use createIfMissing parameter.
v2a:
 uri: cassandra://
 params:
 keyspace: v2
 table: tiles2a
 cp: [maps-test2001.codfw.wmnet, maps-test2002.codfw.wmnet, maps-test2003.codfw.wmnet, maps-test2004.codfw.wmnet]
 username: {var: cassandra-user}
 password: {var: cassandra-pswd}
# repfactor: 4
# durablewrite: 0
# createIfMissing: true
# copyInfoFrom: {ref: gen}
==== Dynamic Layer Generator ====
To generate just a few layers from database, create a layer filter and a layer mixer:
gentmp:
 uri: bridge://
 xml:
 npm: ["osm-bright-source", "data.xml"]
 xmlSetDataSource:
 if:
 dbname: gis
 host: ""
 type: postgis
 set:
 host: localhost
 user: {var: osmdb-user}
 password: {var: osmdb-pswd}
 xmlLayers: [admin, road]
mixtmp:
 uri: layermixer://
 params:
 sources: [{ref: v2}, {ref: gentmp}]
Once set, POST a job to copy mixtmp into the storage v2 e.g.
src=mixtmp dst=v2 baseZoom=0 fromZoom=5 beforeZoom=6 parts=10
Maps/External usage

maps.wikimedia.org tiles may only be used by Wikimedia wikis, and sites hosted by Wikimedia Affiliates. Please see this maps-l posting and the Maps terms of use for more details.
Currently external usage is limited by a regex in the Maps varnish configuration. To have your site added to that list, please file a Phabricator task in the SRE project with the following information:
* Link to site which you want to use Wikimedia Maps on
* Purpose of using maps / details about your project
* Which Wikimedia Affiliate is backing/supporting this project
Using this link will prefill the form for you.
Prior to August 2021, requests were made on T261694, that task is no longer used.
Maps/Infrastructure plans

This page describes plans and not the current state of maps servers
== Constraints ==
When updating vector tile styles you may run into a scenario where there is a schema change, and the new style and old style vector and raster parts are not forward or backwards compatible with each other. These should be avoided, but are sometimes necessary.
Additionally, we need to withstand the following while handling a schema change:
- Two specific servers failing 
- One specific server failing and a datacenter failing
Desirable properties are the abilities to
- take a database server offline to run maintenance (e.g. reclustering),
- reimport with updated software
- upgrade other software
== Architecture ==
The maps cluster will consist of four machines in each datacenter. Under normal operation, requests can go to any of the four machines. When a schema upgrade is necessary, the following needs to be done
== Recommendations 2022-12 ==
These are the shared recommendations discussed in a meeting between WMF Product Infrastructure developers and WMDE Technical Wishes developers:
* : Service containerization
** Moving to k8s will make it easier to upgrade Node (now and in the future).
** Split geoshapes into a separate service.
*** Containerization made it much easier to refactor the service. There's a proof-of-concept port into TypeScript.
** Will lower the maintenance barrier.
* : Node 14+
** Any earlier Node is already end-of-life.
** This upgrade will be a strategic requirement for moving to mapslibre or generally, upgrading upstream packages.
* : Mapnik 3.1 which is the final "LTS" for the old stack.
* : Plan what a mapslibre.org migration would look like
** Will be a lot of work if we keep our current schema and styles.
** Have pitched https://openmaptiles.org/ in the past. Has a very similar style available: https://openmaptiles.org/styles/#osm-bright
** Feasible. The API does allow us to port what has already been implemented.
** Need to keep language support in mind.
*** Mini-investigation into benchmarking language support with the WebGL renderer which is part of mapslibre: T281230
* : Client-side rendering of vector tiles
** Looked into client-side rendering in the past, the obstacle was a product concern of "anything that will change the experience".
** The performance tradeoff depends on loading a client library for maps rendering, maybe.
** Now that we have vector tiles available, it's a short leap to making a client-side rendering proof-of-concept.
* : Separate geoshapes DB from map rendering DB using imposm capabilities
** Depends on the geoshapes split above.
** Will make it much easier to maintain the full-planet OSM sync database for rendering.
** Geoshape database will be smaller.
* Could look at enabling "nearby" on Android again.
* : Current work to expand geoshapes at page parse time
** Is expected to take some load off of the servers.
** Could result in eliminating components eg. the mapdata library.
Devs agree that the most important next step is containerization. WMDE will have to drop out but we hope this work is continued.
Maps/Kartotherian packages

=== Kartotherian service ===
Forked npm dependencies required by kartotherian.
Name 	Forked repo? 	Upstream status 	Wikimedia plans
abaculus 	@kartotherian/abaculus 	 	Unfork. May include a bugfix that can be upstreamed.
geojson-mapnikify 	@kartotherian/geojson-mapnikify 	 	Unfork
kartotherian-tilelive-switch	@wikimedia/kartotherian-tilelive-switch	 	Unfork
libmapnik 	 	 	Unfork, update major revision.
makizushi 	@kartotherian/makizushi 	Abandoned in 2018	Fork harder, support new versions of Maki using an improved backend strategy.
mapnik 	@kartotherian/mapnik 	New major version	We want to upgrade to the latest version and get rid of our fork.
osm-bright-source 	@kartotherian/osm-bright-source 	 	Can probably be decommissioned, this was specific to tilerator.
osm-bright-style 	@kartotherian/osm-bright-style 	 	These are our style customizations, keep forked.
tilelive-bridge 	@kartotherian/tilelive-bridge 	 	Unfork
tilelive-overlay 	@kartotherian/tilelive-overlay 	 	Unfork
tilelive-tmsource 	@kartotherian/tilelive-tmsource 	 	Unfork
tilelive-vector 	@kartotherian/tilelive-vector 	 	Unfork
Maps/Keyspace Setup

This page contains information on how to do a bunch of tasks. It leaves to other pages information on what tasks to do. Frequently the keyspace name will be referenced. These can be found in /etc/tilerator/config.yaml and come from the deploy process. In cases involving multiple keyspaces, the names will depend on what's being done.
Access to the tilerator account will be required for most commands.
==Prerequisites==
For most steps described here, the map data loading must be finished.
Several steps call for commands to be run in Cassandra. This is done with the cqlsh command, which is called like CQLSH_NO_BUNDLED=TRUE cqlsh -u tilerator -p <password> <hostname>, where <hostname> is one of the Cassandra IPs in the cassandra_servers array of /etc/tilerator/config-vars.yaml, and <password> is the cassandra_password variable from the same file. It's essential to use the tilerator user consistently.
==Removing data==
Connect to Cassandra and run DROP KEYSPACE <keyspace_name>;
==Creating a keyspace==
This section differs significantly from the old documentation, which called for a create-sources.yaml file which contained only the Cassandra source. That approach seems to run into errors where tileshell can't get the metadata for the source layer information
Tilerator can create a keyspace if createIfMissing: true is part of the params for the layer. By default this is false, so there is a special process for creation with tileshell.js. Tileshell needs a configuration file containing just the layers, but telling it the Cassandra repfactor and durablewrite settings, and where to get the metadata for the source like layer names, fields, etc.
1. Copy /etc/tilerator/config.yaml to your home directory with cp /etc/tilerator/config.yaml $HOME/create-sources.yaml
2. Remove everything from create-sources.yaml except the sources list, which is the content of the sources key of the conf list item of the services key. This requires removing text from the beginning and end, but does not require adjusting indentation. Don't forget to remove the line with sources: on it.
Typically the file will then start with 
 # Generates vtiles from database
 genallraw:
 public: true
 formats: [pbf]
 uri: tmsource://
 yaml:
 npm: ['@kartotherian/meddo', 'data.yml']
3. Set repfactor, durablewrite, and createIfMissing: true as keys under params for the source which needs a keyspace created. Normally this can be done by uncommenting some lines and changing a false to true.
4. Set which source is used for the tileset metadata of the keyspace. This is done by adding copyInfoFrom: {ref: gen} to params. Normally this can be done by uncommenting a line.
5. As the tilerator user, call tileshell.js with sudo -u tilerator node /srv/deployment/tilerator/deploy/node_modules/tilerator/scripts/tileshell.js --config /etc/tilerator/config.yaml --source $HOME/create-sources.yaml.
Tileshell should run and exit, with the information
info Loading variables from the config file
info Loading sources from /home/pnorman/create-sources.yaml
done
==Copying tiles from an existing keyspace==
Maps/Maintenance

= Maps maintenance =
This document outlines the maintenance activities and points to further documentation explaining each process. The idea is to make the maps infrastructure:
Understood
Teams responsible for aspects of the service understand where their responsibilities begin and end, and have the information required to fulfill those responsibilities (alerting, documentation, SLO, etc)
Supported
Modern components, up to date (where possible and realistic) versions and if possible internally standardized (I’m thinking Prometheus here, but also possibly the discussion around Cassandra that emerged in our meeting, running maps services in Buster where metal is needed, nodejs updates, etc)
Automated
Wherever possible, manual intervention isn’t required for updates and self-healing. This isn’t a problem as such at the moment but if we could avoid things like resyncing the databases in the way we do now it would be excellent.
Distributed/fault tolerant
Currently, if we lose an individual maps node, we lose four services at once. The plan to move components to k8s where possible greatly improves this situation. Avoiding tightly coupling application components and state.
== Known issues ==
* Services are not in k8s
* Currently if we lose an individual maps node, we lose four services at once
* Components tightly coupled
* Resyncing the DB has a high cost
* Metrics needs to move from Graphite to Prometheus
* Service is not paging
* Kartotherian is not publicly available for 3rd parties
== Maintenance activities responsibility and support ==
What's needed for the maintenance work?
R = Responsible, S = Support
=== Application layer ===
Kartotherian
Activity 	Responsible	Consulted	Informed	Automated?
Investigate and fix application production errors	 	 	 	
Package kartotherian code for deployment 	 	 	 	
Deploy code and configuration into maps clusters 	 	 	 	
Configure load balance between maps sources 	 	 	 	
Tegola
Activity 	Responsible	Consulted	Informed	Automated?
Investigate and fix application production errors and submit code to upstream	 	 	 	
=== Infrastructure ===
Beta Cluster (*.maps-experiments.eqiad1.wikimedia.cloud)
Activity	Responsible	Consulted	Informed	Automated?
 	 	 	 	
Varnish
Activity 	Responsible	Consulted	Informed	Automated?
Purge tile cache when vandalism occurs	 	 	 	
PostgreSQL/OSM
Activity 	Responsible	Consulted	Informed	Automated?
Restore replica out of sync with main DB 	 	 	 	
Initial OSM import 	 	 	 	
Restore DB causing disk space issue 	 	 	 	
Restore OSM replication because OSM lag is falling behind 	 	 	 	
Make sure that the proper binaries are successfully installed in the infrastructure	 	 	 	
Swift
Activity	Responsible	Consulted	Informed	Automated?
 	 	 	 	
Kafka
Activity 	Responsible	Consulted	Informed	Automated?
Wipe Kafka topic (empty queue) by moving skipping all events in the stream	 	 	 	
Tegola
Activity 	Responsible	Consulted	Informed	Automated?
Enable pre-generation on eqiad/codfw tegola
Maps/Maps experiments

Because maps-experiments is not configured in deployment-prep (they are different projects in Cloud VPS) a hack is needed to deploy kartotherian code to maps-master01. The problem is that the deploy instance in deployment-prep doesn't recognize connections to maps-experiments project, but we can fetch the needed code the other way around.
First, proceed with deployment as normal. This is important to create the deployment rev that we will fetch from maps-master01:
ssh deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud
cd /srv/deployment/kartotherian/deploy
# make sure you are in the imposm branch
git checkout imposm
git pull && git submodule update
git log -n 1
# Create the deployment at deployment-prep
scap deploy --environment beta "Deploy kartotherian imposm at maps-master01.maps-experiments"
# It will fail 
# Don't execute rollback. Press "n"
Then log into maps-master01.maps-experiments and execute the following commands manually as root:
ssh maps-master01.maps-experiments.eqiad1.wikimedia.cloud
sudo -u deploy-service /usr/bin/scap deploy-local -v --environment beta --repo kartotherian/deploy -g default fetch --refresh-config
sudo -u deploy-service /usr/bin/scap deploy-local -v --environment beta --repo kartotherian/deploy -g default config_deploy --refresh-config
sudo -u deploy-service /usr/bin/scap deploy-local -v --environment beta --repo kartotherian/deploy -g default promote --refresh-config
This should replicate the same process as deploying code to instances inside the deployment-prep project.
Maps/OSM

= OpenStreetMaps Extract Transform Load (ETL) procedure =
The WMF maps stack use OSM to render static and dynamic maps. Imposm3 is the tool behind the ETL procedure that loads data into a PostgreSQL DB schema that conforms with the maps applications' needs.
== Database requirements ==
* Postgres 11
* PostGIS 3.1+ (this version allows the use of the function ST_GeomAsMVT to generate vector tiles with Tegola)
== Imposm3 mapping and DB schema ==
The database schema is defined in a configuration file that is available in the operations/puppet repository.
== Import OSM data ==
In order to import OSM data, we have a production-ready script that needs to be executed with root access and the latest planet dump from OSM, with a caveat, always choose a specific revision to execute the script instead of the one tagged with latest, here is an example of the command to be executed:
 sudo -s
 screen -DR # always execute with screen because the script takes several hours to finish
 imposm-initial-import \
 -d 210906 \
 -x webproxy.eqiad.wmnet:8080
The script will take care of the whole procedure, but if you want to know how the process works and execute it locally, please refer to the guide present in maps-config
= Rendering Maps =
== Layers Functions ==
As part of the Maps 2.0 work, it was introduced the concept of SQL functions to query the layers of the map. See the SQL functions that can be accessed.
=== Updating Layers Functions ===
When changing a SQL function behavior that introduces significant changes or needs to be easily rolled-back, we decided to add version suffix in the function name and updating the applications usage of the functions accordingly. This means that, the next changes in public.layer_admin will be applied in public.layer_admin_v2.
Maps/Services deployment

== Deploying maps services ==
Before you actually start deploying check out some basic tips for deployment.
=== Developer setup for updating deployment repo ===
The deployment repo is used to compose the kartotherian and tilerator code and it's dependencies. Before a new versions can be deployed there must be an update to the deployment repo.
We build the deploy repo in Docker in order to ensure that the Node modules have the correct binaries. We build the deploy repo using Docker for Mac or a Linux machine. Before you start the first build of the deploy repo run through the setup instructions.
Kartotherian is developed on Gerrit, and the old Github repository is abandoned and is no longer used by WMF infrastructure.
=== Publish library dependencies to NPM before building for deployment ===
In order to be able to package kartotherian and tilerator, the process to bump the version and publish to npm is crucial, therefore you should have permission to publish packages in the @wikimedia npm organization. The version bump occurs using manually package by package or using the lerna dependency manager.
=== Building for deployment ===
Kartotherian and Tilerator are deployed according to the standard process for deploying Wikimedia Node.js services.
=== Update and build the deployment repo ===
Sync the code and deploy repos with current master.
cd ~/path/to/kartotherian/deploy
git status
git reset --hard origin/master
git checkout .
git clean -fd
git checkout master
git pull
git --no-pager log --decorate -n 1
git submodule update --init
cd ./src
git checkout master
git pull
cd ..
git add src
git branch
git status
At this point, commit any updates to the src submodule.
If using Docker for Mac start the Docker daemon by clicking on the Whale icon in the menu bar. (Should work automatically on Linux.)
Run the tests in Docker and build the new commit for the deploy repo:
./server.js build --deploy-repo --force
And push to Gerrit:
git review
=== Deploy to Beta Cluster ===
The steps are similar to #Deploy to Production but using different machines, of course.
To deploy:
ssh deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud
scap deploy --environment beta "`git log --pretty=format:'%s' -n 1`"
To verify service deployment you can ssh into deployment-maps-master01.deployment-prep.eqiad1.wikimedia.cloud.
Use #wikimedia-releng (instead of #wikimedia-operations) to see if there are issues. You may want to log manually in this channel using !log until phab:T156079 is resolved. This logs to the Releng team's Server admin log
=== Deploy to Production ===
Maps/v2/Common tasks#Deploy new code
=== Deploying services ===
Refer to Services/Deployment for general instructions.
In most cases, Kartotherian and Tilerator should be deployed together, to ensure that all dependencies (in particular, styles) are in sync.
=== Monitoring ===
To watch logs on the maps server, a custom script has been included:
tail-kartotherian -f