Wikidata Query Service/Manual maintenance

This page describes the maintenance tasks that are currently manual. This page is not meant to be a documentation but rather track the actions done/required by members of the mw:Wikimedia Search Platform team. Ideally this page should not exist.
== Workaround for missing deletes in the current updater ==
* Context: phab:T272120
* Workaround: re-sync deleted entities polling the deletion log using an ad-hoc tool.
* Tool: https://people.wikimedia.org/~dcausse/wdqs_manual_deletes/
* Last run: 2021-09-27 for the period 2021-09-17T00:00:00Z to 2021-09-27T00:00:00Z
== Streaming updater: test with wdqs1009 ==
* Context: phab:T266470
As of 2021-02-19 the streaming updater is running a custom build:
* user: analytics-search
* machine: stat1004
* flink base: /home/dcausse/flink-1.12.0-wdqs
* session cluster started with: sudo -u analytics-search kerberos-run-command analytics-search sh -c HADOOP_CLASSPATH="`hadoop classpath`" ./bin/yarn-session.sh -tm 4g -jm 2600m -s 4 -nm "WDQS Streaming Updater"
* application started with: sudo -u analytics-search kerberos-run-command analytics-search sh -c 'export HADOOP_CLASSPATH=`hadoop classpath`; ./bin/flink run -p 12 -s swift://updater.thanos-swift/wdqs_streaming_updater/bootstrap_savepoint_20210201 -c org.wikidata.query.rdf.updater.UpdaterJob ~/streaming-updater-producer-0.3.64-SNAPSHOT-jar-with-dependencies.jar ../updater-job-partial-reordering.properties'
* custom build: /home/dcausse/streaming-updater-producer-0.3.64-SNAPSHOT-jar-with-dependencies.jar (master minus https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/649715)
Verify that it's running by checking that the rate of offset commits is not zero: https://grafana-rw.wikimedia.org/d/000000484/kafka-consumer-lag?orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-cluster=main-eqiad&var-topic=All&var-consumer_group=wdqs_streaming_updater
It might fail because:
* stat1004 was rebooted
* the jobmanager has its node being killed by yarn/analytics
== How to recover it ==
Find the last valid checkpoint from application log using:
sudo -u analytics-search kerberos-run-command analytics-search yarn logs -applicationId application_1612875249838_7657 | less
Find the proper application id using https://yarn.wikimedia.org/cluster/apps and search for WDQS. If yarn still has the data there should be one line with:
* User: analytics-search
* Name: WDQS Streaming Updater
* Application Type: Apache Flink
In the logs search for the last instance of Completed checkpoint NNNN for job XYZ
2021-02-19 13:42:54,810 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 3087 (type=CHECKPOINT) @ 1613742173917 for job b85e02696673c5d09d41918872c98898.
2021-02-19 13:43:00,980 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed checkpoint 3087 for job b85e02696673c5d09d41918872c98898 (103619620 bytes in 5275 ms).
2021-02-19 13:43:24,763 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 3088 (type=CHECKPOINT) @ 1613742203917 for job b85e02696673c5d09d41918872c98898.
2021-02-19 13:43:30,731 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed checkpoint 3088 for job b85e02696673c5d09d41918872c98898 (103072267 bytes in 5037 ms).
2021-02-19 13:43:54,776 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 3089 (type=CHECKPOINT) @ 1613742233917 for job b85e02696673c5d09d41918872c98898.
End of LogType:jobmanager.log.This log file belongs to a running container (container_1612875249838_46677_01_000001) and so may not be complete.
*******************************************************************************
here 3088 is right one, 3089 is not yet complete.
the checkpoint path should be wdqs_streaming_updater/checkpoints/b85e02696673c5d09d41918872c98898/chk-3088.
If yarn has purged the application logs you can still try to dig into swift to find the checkpoint. The command:
swift -A https://thanos-swift.discovery.wmnet/auth/v1.0 -U wdqs:flink -K CHANGEME list updater -l
will list the file on the store used for checkpointing, you must find the last valid checkpoint created around the date of the failure. It is a folder named:
wdqs_streaming_updater/flink application id / chk-NNNN
It must have a _metadata file in it, example:
 0 2021-02-17 10:11:41 application/octet-stream wdqs_streaming_updater/checkpoints/a2f9c2f3d78e904a87e08bade5a44786
 0 2021-02-17 10:28:09 application/octet-stream wdqs_streaming_updater/checkpoints/a2f9c2f3d78e904a87e08bade5a44786/chk-10349
 0 2021-02-17 10:44:36 application/octet-stream wdqs_streaming_updater/checkpoints/a2f9c2f3d78e904a87e08bade5a44786/chk-10353
 0 2021-02-17 10:45:25 application/octet-stream wdqs_streaming_updater/checkpoints/a2f9c2f3d78e904a87e08bade5a44786/chk-10355
 413964 2021-02-17 10:45:29 application/octet-stream wdqs_streaming_updater/checkpoints/a2f9c2f3d78e904a87e08bade5a44786/chk-10355/_metadata
 0 2021-02-17 10:11:41 application/octet-stream wdqs_streaming_updater/checkpoints/a2f9c2f3d78e904a87e08bade5a44786/shared
 169742 2021-02-17 10:29:11 application/octet-stream wdqs_streaming_updater/checkpoints/a2f9c2f3d78e904a87e08bade5a44786/shared/033fa5a9-4fe8-4cfe-ae8f-367b589b345c
 90547 2021-02-17 10:44:56 application/octet-stream wdqs_streaming_updater/checkpoints/a2f9c2f3d78e904a87e08bade5a44786/shared/1186aa00-32cb-41de-8fe7-c3e961c77100
 1029972 2021-02-17 10:29:10 application/octet-stream wdqs_streaming_updater/checkpoints/a2f9c2f3d78e904a87e08bade5a44786/shared/13ae0214-c902-4c6f-943f-27cc5d1ac5c9
 800483 2021-02-17 10:29:07 application/octet-stream wdqs_streaming_updater/checkpoints/a2f9c2f3d78e904a87e08bade5a44786/shared/162aae61-11fd-4ace-8440-6cf62a9f58f0
 1045523 2021-02-17 10:29:08 application/octet-stream wdqs_streaming_updater/checkpoints/a2f9c2f3d78e904a87e08bade5a44786/shared/1ed541ce-a530-4853-8ad2-b0f6b6799709
 32453 2021-02-17 10:44:57 application/octet-stream wdqs_streaming_updater/checkpoints/a2f9c2f3d78e904a87e08bade5a44786/shared/2743e36e-7d98-43a8-85cc-f75845473148
wdqs_streaming_updater/checkpoints/a2f9c2f3d78e904a87e08bade5a44786/chk-10355 sounds like a good checkpoint to use if the failure happened around 2021-02-17T10:45:29.
== How to stop it ==
List of flink apps:
sudo -u analytics-search kerberos-run-command analytics-search sh -c 'HADOOP_CLASSPATH="`hadoop classpath`" ./bin/flink list'
Stop with a savepoint:
sudo -u analytics-search kerberos-run-command analytics-search sh -c 'HADOOP_CLASSPATH="`hadoop classpath`" ./bin/flink stop c4ebc03ec0c5431fb6b8232f45b54aa2 --drain -p swift://updater.thanos-swift/wdqs_streaming_updater/savepoints'
Note the savepoint path on the last line e.g.:
Savepoint completed. Path: swift://updater.thanos-swift/wdqs_streaming_updater/savepoints/savepoint-c4ebc0-e5907d976b64
The swift path is the path to use as the -s param when starting the application.
== Downstream consumer (wdqs1009) ==
As of writing the downstream consumer is still reloading the data but as soon as the data reload is complete and the flag file /srv/wdqs/data_loaded is created the updater (consumer) will start on this machine.
It will start consuming the topic kafka-main eqiad.rdf-streaming-updater.mutation at whatever offset the consumer group wdqs1009 is currently at. Note that it is currently well positioned according to the last run (2021-02-18 09:33:53).
Even though this should not be needed for this particular reload here is how it works:
After the initial load it must be positioned at the offset corresponding to the first message produced from flink when the app is started using the bootstrap savepoint.
Use stat1004@/home/dcausse/set_offsets.py to manipulate offsets:
The backfill from the bootstrap state started around 2021-02-18T09:33:53 position the offset around this time:
dcausse@stat1004:~/flink-1.12.0-wdqs$ python3.7 ~/set_offsets.py -t eqiad.rdf-streaming-updater.mutation -c wdqs1009 -b kafka-main1002.eqiad.wmnet:9092 -s 2021-02-18T09:33:53
Configuring offsets for wdqs1009@kafka-main1002.eqiad.wmnet:9092 for topics ['eqiad.rdf-streaming-updater.mutation'] at time 2021-02-18 09:33:53
Setting: TopicPartition(topic='eqiad.rdf-streaming-updater.mutation', partition=0) to OffsetAndTimestamp(offset=12080891, timestamp=1613640833827)
Check that the data at this offset correspond to the event time of the bootstrap state (date of the dumps):
dcausse@stat1004:~/flink-1.12.0-wdqs$ kafkacat -b kafka-main1003.eqiad.wmnet:9092 -t eqiad.rdf-streaming-updater.mutation -o 12080891 -c 1 | jq .event_time
% Auto-selecting Consumer mode (use -P or -C to override)
"2021-01-29T23:06:26Z"
Double check that it's the first message by looking at the previous one (offset minus one):
dcausse@stat1004:~/flink-1.12.0-wdqs$ kafkacat -b kafka-main1003.eqiad.wmnet:9092 -t eqiad.rdf-streaming-updater.mutation -o 12080890 -c 1 | jq .event_time
% Auto-selecting Consumer mode (use -P or -C to override)
"2021-01-30T17:22:00Z"
Useful dashboard: https://grafana-rw.wikimedia.org/d/fdU5Zx-Mk/wdqs-streaming-updater?orgId=1