Maps/v2/Common tasks

== Common maps tasks ==
=== Access map nodes ===
ssh maps1009.eqiad.wmnet
=== Deploy new code ===
==== Kartotherian ====
===== Package the code =====
Currently, kartotherian main code is available in the master branch.
In order to deploy kartotherian you need to package the deploy repo locally and submit to gerrit, kartotherian uses service-runner to run a Docker container for packaging, the following script assumes you have both the kartotherian repo and the deploy repo at the same directory level:
cd ../kartotherian
git status
git reset --hard origin/master
git checkout .
git clean -fd
git checkout master
git pull origin master
git status
git --no-pager log --decorate -n 1
rm -rf node_modules/
npm i service-runner --ignore-scripts
cp config.test.yaml config.yaml # Needed just to infer some packaging attributes
cd ../deploy
git status
git reset --hard origin/master
git checkout .
git clean -fd
git checkout master
git pull
git --no-pager log --decorate -n 1
git submodule update --init
git status
# Change to the kartotherian source directory
cd ../kartotherian
./server.js build --deploy-repo --force
NOTE: In order for the script to work you need to first merge the changes to kartotherian/package repo and then build the deploy package.
===== Submit changes to production with scap =====
After the code is reviewed and merged, it should be available at deployment.eqiad.wmnet
ssh deployment.eqiad.wmnet
cd /srv/deployment/kartotherian/deploy
git pull && git submodule update --init
git log -n 1
# deploy to eqiad
scap deploy --environment eqiad -l 'maps1*' `git log --pretty=format:'%s' -n 1`
# deploy to codfw
scap deploy --environment codfw -l 'maps2*' `git log --pretty=format:'%s' -n 1`
==== Tegola ====
*Build a new image using CI
** Each merge builds an image:
*** https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/tegola/+/refs/heads/wmf/v0.14.x
*** https://docker-registry.wikimedia.org/wikimedia/operations-software-tegola/tags/
* Bump version on deployment-charts
* Deploy new version
=== Configuration changes ===
==== Tegola ====
The tegola config is maintained using helm in the deployment charts repo and populated by the helm values per env:
* Config
** https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/charts/tegola-vector-tiles/templates/_config.tpl
* Values
** Base values
*** https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/charts/tegola-vector-tiles/values.yaml
** Per env overrides
*** https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/helmfile.d/services/tegola-vector-tiles/
==== Kartotherian ====
Kartotherian is using scap for deployment. The configuration is maintained here:
* https://gerrit.wikimedia.org/r/plugins/gitiles/maps/kartotherian/deploy/+/refs/heads/master/scap/templates/config.yaml.j2
* https://gerrit.wikimedia.org/r/plugins/gitiles/maps/kartotherian/deploy/+/refs/heads/master/scap/vars.yaml
==== Postgres ====
Postgres config is maintained using puppet. Here are the relevant sections:
* https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/postgresql/
==== Imposm ====
OSM import process is configured using puppet. Here are the relevant sections:
* https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/osm/
==== EventGate ====
* Schema definitions
** https://gerrit.wikimedia.org/r/plugins/gitiles/schemas/event/primary/+/refs/heads/master/jsonschema/maps/tile_change
** https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/wmf-config/InitialiseSettings.php
*** Go to section "Stream config"
=== Enable/disable import process ===
In order to disable the import process you need to switch the boolean value to false and run puppet:
* profile::maps::osm_master::disable_replication_cron: false
=== Run a planet import ===
When running a planet import, make sure that the following checklist is followed:
* disable disable_replication_service and disable_tile_generation_cron in hieradata 
* execute initial-imposm-import script using the following command
# change -d accordingly when performing the import with the most updated values
sudo -s
screen -DR
imposm-initial-import -d 210906 -x webproxy.eqiad.wmnet:8080
* Post a log message in IRC #wikimedia-operations !log maintenance: trigger full planet re-import for maps eqiad
* monitor the full planet import
** Is there any disk space issue?
** Did the script finished properly?
** Are the logs sufficient?
* monitor and check the APIs, in case of rollback, run: sudo -u osmupdater imposm-rollback-import
* REINDEX DATABASE gis;
* Monitor and check the APIs, if needed rollback with the following command
* Deploy data to the public schema by running:
 sudo -u osmupdater imposm-deploy-import
* re-enable the replication service and check for its health
* re-enable the tile-generation cron and check for its health
* remove imposm backup to free disk space
* check if sql functions and indexes are applied (if not apply them manually)
* re-enable the replication service and check for its health
* re-enable the tile-generation cron and check for its health
* clean-up old data in backup
** Post a log message in IRC #wikimedia-operations !log <your_username>@maps1009 imposm-removebackup-import (T305845)
 imposm-removebackup-import
=== Pooling/depooling nodes ===
TODO
=== Get a DB shell ===
> ssh maps1005.eqiad.wmnet
> sudo su 
> su - postgres 
> psql 
postgres=# \c gis
=== Run an example query ===
Select all landuse geometries included in the bounding box:
* SRID 3857 (the one we use in our layers config)
* Box: left=2632692 bottom=4572168 right=2648887 top=4581940
This bounding box is a rectangle that includes the center of Athens, Greece.
gis=# SELECT osm_id, ST_AsMVTGeom(geometry, ST_MakeEnvelope(2632692,4572168,2648887,4581940, 3857)) AS geom FROM layer_landuse(ST_MakeEnvelope(2632692,4572168,2648887,4581940, 3857), 9);
=== Reindex DB ===
WARNING this operation might take even ~10 hours to be completedFrom a pgsql shell, connect to the `gis` DB and run:
gis #= REINDEX DATABASE gis;
=== Example HTTP requests ===
==== Kartotherian ====
Get the raster tile x=0 y=0 z=0 for source osm-intl
curl https://maps.wikimedia.org/osm-intl/0/0/0.png
==== Tegola ====
Tegola currently is only used internally. In order to reach out for the current example we will ssh to the deployment node (eq. in eqiad)
> ssh deployment.eqiad.wmnet 
To get the endpoint information for different maps and layers:
> curl https://tegola-vector-tiles.svc.eqiad.wmnet:4105/capabilities | jq .
To get the vector tile x=0 y=0 z=0 for map osm
> curl https://tegola-vector-tiles.svc.eqiad.wmnet:4105/maps/osm/{z}/{x}/{y}.pbf
==== EventGate ====
Post an event:
curl -X POST -d '
{
 "\$schema": "/maps/tile_change/1.0.0",
 "meta": {
    "dt": "$(date --iso-8601=s)"
   "stream": "maps.tile_expiration"
 },
"tile": "1/0/2",
"state": "expired"
}' https://eventgate-main.svc.eqiad.wmnet:4492/v1/events/
=== Example tegola commands ===
* Run server
** tegola -c </path/to/config> serve 
* Cache a tile
** tegola -c /path/to/config cache seed tile-name 0/0/0
* Purge a tile from cache
** tegola -c /path/to/config cache purge tile-name 0/0/0
=== Clean-up Kafka queue ===
Reset the queue:
kafka-commit-last-message --topic "eqiad.maps.tiles_change" --broker kafka-main1001.eqiad.wmnet:9092 --group-id "poppy-eqiad.maps.tiles_change"
Verify that there are no messages to be consumed from last offset
kafka-consume-messages --topic "eqiad.maps.tiles_change" --broker kafka-main1001.eqiad.wmnet:9092 --offset earliest --group-id "poppy-eqiad.maps.tiles_change"
=== List tiles on Openstack Swift ===
cd /root
source .tegola_credentials
swift -A $ST_AUTH -U $ST_USER -K $ST_KEY list tegola-swift-container
=== Load balance maps requests ===
In order to perform safe switchovers in maps we have developed a simple load balancer for kartotherian.
To configure the load balancer you should edit kartotherian config templates with the following code:
Enable load mirroring
# Switch requests to a secondary source
switch:
 uri: mirror://
 params:
 source: { ref: tm2source }
 secondarySource: { ref: tegola }
 enableMirror: true
 loadBalancer:
 bucketSize: 1000 # milliseconds
 totalBuckets: 10 # number of buckets in the histogram
 threshold: 0.1 # 10% of load to the mirrored source
Enable load switch
# Switch requests to a secondary source
switch:
 uri: switch://
 params:
 source: { ref: tm2source }
 secondarySource: { ref: tegola }
 enableSwitch: true
 loadBalancer:
 bucketSize: 1000 # milliseconds
 totalBuckets: 10 # number of buckets in the histogram
 threshold: 0.1 # 10% of secondary source load will be returned in the responses
Here's a real world example where kartotherian is configured to mirror 100% of the requests to eqiad tegola.
=== Check tile pregeneration status ===
Currently the only way we have to check for pregeneration status is through tile pregeneration logs.
Here are the links to logstash:
* eqiad tile pregeneration
* codfw tile pregeneration
Codfw is running every day at 00:00
Eqiad is running every day at 12:00
To filter for errors we can just query for "errors".
Category:Maps V2