Push notifications

Push Notifications provide the capability to send information based on subscribed content to devices. The push-notifications service accepts notification requests via HTTP for batching and submission to native application push provider APIs. It is deployed in Kubernetes, and its request intake API endpoints are exposed only internally to the cluster. In the future it might be updated to handle web push (see phab:T266138).
== Lifecycle of a push notification request ==
Push notifications build on the longstanding MediaWiki notification system implemented in the Echo extension. It adds a third notification type, currently labeled "Apps," to the preexisting "Web" and "Email" types. It is the responsibility of the app clients to register users' push subscriptions via the echopushsubscriptions API module and update their notification preferences in order to benefit from push notifications.
When an Echo event occurs and a recipient's notification preferences indicate that the user has chosen to be notified of that event type by push, Echo pushes a NotificationRequestJob onto the job queue. Upon execution of the job, the database is queried for all of the user's registered push subscriptions, and requests are made to the push-notifications service (to endpoints /v1/events/fcm and /v1/events/apns, for Firebase Cloud Messaging (Android) and Apple Push Notification Service (iOS), respectively) to request that a message be sent to each subscribed device.
Upon receiving a push notification request, the push-notification service enqueues the request to be batched with other requests for a configured period of time. This period can be either defined explicitly or configured to be randomized between a provided pair of lower and upper bounds. After the configured batching period passes, the request is consolidated with other requests received during the batching period into one or more multicast requests, and submitted to the push provider API. Consolidation with other messages into generic multicast requests is possible because the system currently sends only a generic message prompting the apps to contact the MediaWiki API to receive full message content.
[TO BE IMPLEMENTED: see T260247 and subtasks] When a push provider API responds that a subscriber token is invalid, the push-notifications service submits a request to the MediaWiki API to delete that subscription from the database.
== Dashboards ==
The service is sending metrics to Prometheus via service-runner.
* Grafana
* Logstash: push notifications @ eqiad
* Logstash: push notifications @ codfw
Job queue processing metrics for NotificationRequestJobs can also be seen on the JobQueue EventBus dashboard:
* https://grafana.wikimedia.org/d/000000400/jobqueue-eventbus
==Connecting to push-notifications==
===Production===
The service is exposed by service proxy at http://localhost:6012.
To access the service directly, by cluster:
*https://push-notifications.svc.eqiad.wmnet:4104
*https://push-notifications.svc.codfw.wmnet:4104
To quickly verify that the service is up and running, request /_info:
 mholloway-shell@deploy1001:~$ curl -s http://localhost:6012/_info | jq .
 {
   "name": "push-notifications",
   "version": "0.0.1",
   "description": "A service for sending push notification messages",
   "home": "https://www.mediawiki.org/wiki/Wikimedia_Product_Infrastructure_team/Push_Notifications_Infrastructure"
 }
===Beta Cluster===
A role::beta::docker_services instance of the service is running on deployment-push-notifications02 and exposed internally at http://deployment-push-notifications02.deployment-prep.eqiad1.wikimedia.cloud:8900.
== Deployment process ==
=== Production ===
See Deployments_on_kubernetes
=== Beta Cluster ===
In Horizon, navigate to the deployment-prep (Beta Cluster) project and view the deployment-push-notifications tab under "Prefix Puppet". You will see a configuration definition like the following:
profile::docker::engine::declare_service: true
profile::docker::engine::packagename: docker.io
profile::docker::engine::settings: {}
profile::docker::runner::service_defs:
 mediawiki-services-push-notifications:
 config:
 logging:
 level: trace
 metrics:
 port: 9100
 type: prometheus
 num_workers: 0
 services:
[...]
 environment:
 DEBUG: apn
 namespace: wikimedia
 override_cmd: -c /etc/mediawiki-services-push-notifications/config.yaml
 port: 8900
 version: 2021-08-20-154251-publish
Update the version setting with the desired image tag from the pipeline. The update will be picked up and the service will be restarted with the new image on the next Puppet run.
== In case of emergency ==
Push notifications support can be disabled without negative effects on other wiki operations. [TODO: Verify job queue retry behavior and ensure there are no spillover effects]
To disable push notifications support in MediaWiki (Echo), set $wgEchoEnablePush to false. This will result in the push subscription management API module being removed, and push notification request jobs no longer being sent to the job queue for submission to the push-notifications service.
== Service level objectives (SLOs) ==
TODO: T264118
== See also ==
* Project page on mediawiki.org