Service/IPoid

The ipoid service provides an API for retrieving reputation data for individual IP addresses, as well as retrieving lists of IP addresses matching a particular set of labels (e.g. list of IP addresses that are associated with VPNs).
As of October 2023, the primary consumer of the service is the IPInfo extension, which allows a small number of users with privileged access to view IP reputation data.
=== Querying ===
From mwmaint:
To find IPs matching a prefix:
from deploy servers:
=== Data pipeline ===
The extract, transform, load pipeline is described in more detail in IPoid's README. At a high level:
* Once a day, download IP reputation data from Spur.us
* Compare existing data with newly downloaded data and update IPoid's database with the latest information
The import process, as of October 2023, peaks at about 600mb of RAM in local development environment tests.
=== Service Dependencies ===
==== Database ====
MariaDB instance on m5. m5-master.eqiad.wmnet is anchored to eqiad for writes, while it can read from m5-master.codfw.wmnet, that means that, the service will be active/active from the user perspective, but its cronjob, which is the only path where there are writes in the database, will run always from eqiad.
=== Service Monitoring ===
Service Kubernetes dashboard: https://grafana-rw.wikimedia.org/d/6C9Bm6uVz/ipoid?orgId=1
=== Ownership ===
Trust and Safety Product team.
=== Cronjobs ===
There is a cron job for processing daily updates, ipoid-production-daily-updates.
Note that if you update the schedule for the Kubernetes cronJob, helmfile apply will not show any update in its diff about the newly applied schedule.
== Deployment ==
=== Prepare the deployment patch ===
Make a patch in operations/deployment-charts that updates the value of the main_app.version field in helmfile.d/services/ipoid/values.yaml, to the new image tag was built automatically with the merging of a GitLab merge request in the iPoid repo. The image tag can be found at IPoid's Docker registry but may take some time to show up. If the registry hasn't updated yet, the tag can also be found in the logs of the merge's `publish-production-build-and-publish-image` job.
==== Commit message guidelines ====
* Subject line can always be: "ipoid: Bump version"
* Add a bullet point for patch in repos/mediawiki/services/ipoid that is part of this release. The first line should specify what relevant code is affected (api, app, etc) followed by the subject line of the commit. On the second line, include a reference to the task from the patch and a reference to the relevant merge request number(s)
* Finally, the last line should include "Bug: " and reference the relevant phabricator task for this deployment.
All of the above guidelines in the commit message are helpful for paper trail and for documenting what was deployed, and when.
==== See also ====
See Deployments on kubernetes for tips, and note that 1) self merges are OK in this repository, and 2) a cron script on the deployment server will fetch the latest contents of the repository every minute.
=== Additional dependencies ===
If you are deploying a patch with a schema change you'll need to get the ipoid user the necessary database rights before deploying, otherwise the next import using the new schema will fail on a permissions issue. You can do this by creating a Phabricator task and tagging DBA on it. For an example, see this past request.
=== Deploy the patch ===
Now, SSH to a Deployment server. Using tmux is recommended.
==== staging ====
==== eqiad ====
==== codfw ====
==== Checking output from a container ====
=== Monitor progress after deployment ===
To use the kubectl command:
$ export KUBECONFIG=/etc/kubernetes/ipoid-deploy-eqiad.config
* To see the running (and stopped but not deleted) pods: kubectl get pods
* To see logs for a running pod: kubectl logs -f <pod NAME>
* To explore file system for a running pod: kubectl exec --stdin --tty <pod NAME> -- /bin/bash
To explore the database (will prompt for password):
$ mysql -uipoid_ro -p -h m5-master.eqiad.wmnet ipoid
=== Supporting documentation and relevant information ===
* ipoid project plan
* Weekly meeting notes
* Phabricator