Mathoid

Mathoid is a stateless Node.js service hosted in the Kubernetes cluster. It receives POST requests from MediaWiki containing mathematical formulas in LaTeX, and responds with the MathML and SVG renderings of that formula (both encoded in a JSON response).
An early version of Mathoid used PhantomJS to render formulas using client-side JavaScript inside a headless web page. This was replaced with invoking MathJax directly within the Node.js process. As such, this service does not shell out or store anything in the file system.
== Deployment ==
Mathoid is currently built and deployed using the Deployment pipeline. When a change is merged to the master branch, a Docker image is automatically built and pushed to the Docker-registry. A change is automatically generated (similar to this change), which itself is then merged upon receiving a CR+2. This repo is automatically synced to the deploy server and then is deployed to the cluster using the Deployments on kubernetes process. 
== Monitoring ==
* grafana
* Icinga: look at mathoid.svc.{eqiad,codfw}.net for lvs mappings, or kubernetes for issues with the services themselves
== Troubleshooting ==
The service listens on port 4001.
Example request:
 # Staging
 curl -XPOST -d'q=e=mc^2' https://staging.svc.eqiad.wmnet:4001
 service-checker-swagger staging.svc.eqiad.wmnet https://staging.svc.eqiad.wmnet:4001
 
 # Production
 curl -XPOST -d'q=e=mc^2' https://mathoid.discovery.wmnet:4001
This should return JSON with 'mml' and 'svg' members. This query can also be run directly against any kubernetes host, e.g. kubernetes10XX.eqiad.wmnet.
=== Logs ===
Logs go to logstash. But you can get logs as well from every pod. First fetch a pod name
 deploy1001:~$ kube_env mathoid eqiad
 deploy1001:~$ kubectl get pods
 NAME READY STATUS RESTARTS AGE
 mathoid-production-7b9999dd5f-4ml9j 2/2 Running 0 9d
 mathoid-production-7b9999dd5f-5mpmd 2/2 Running 0 9d
 mathoid-production-7b9999dd5f-65phz 2/2 Running 1 9d
 mathoid-production-7b9999dd5f-mvvcm 2/2 Running 1 9d
 ...
Let's choose `mathoid-production-7b9999dd5f-4ml9j`. Note that the names of the pods are random as they change with every new deployment. Then run kubectl logs for the logs
 deploy1001:~$ kube_env mathoid eqiad
 deploy1001:~$ kubectl logs mathoid-production-7b9999dd5f-mvvcm mathoid-production
 {"name":"mathoid","hostname":"mathoid-production-7b9999dd5f-mvvcm","pid":1,"level":40,"levelPath":"warn/service- runner","msg":"Startup finished","time":"2019-02-26T15:48:07.170Z","v":0} 
 TeX parse error: Undefined control sequence \emph
 TeX parse error: Undefined control sequence \emph
 TeX parse error: Double subscripts: use braces to clarify
 TeX parse error: Double subscripts: use braces to clarify
...
Note the mathoid-production part. That's the name of the first of the 2 containers running in this pod and it's the actual application. The second one is the statsd prometheus exporter. Logs for the latter are as easy to obtain. 
 deploy1001:~$ kube_env mathoid eqiad
 deploy1001:~$ kubectl logs mathoid-production-7b9999dd5f-mvvcm production-metrics-exporter
 time="2019-02-26T15:48:03Z" level=info msg="Starting StatsD -> Prometheus Exporter (version=0.8.0+ds1, branch=master, revision=0.8.0+ds1-4)" source="main.go:158" 
Physically, those are logs that are stored /var/log/pods, so you could find your way through that too, but chances are logstash and kubectl logs are going to be more efficient
=== Fail-over ===
The pods are managed by kubernetes and will fail over automatically, within a configurable at the cluster level time frame. The default is 5 mins. The failure of an individual node does not affect the availability of the service.
=== Restarting a Pod ===
If you want to restart a specific pod, just delete it. The infrastructure will restart it automatically. e.g. 
 deploy1001:~$ kube_env mathoid eqiad
 deploy1001:~$ kubectl delete pods mathoid-production-7b9999dd5f-cr2dr
Error from server (Forbidden): pods "mathoid-production-7b9999dd5f-cr2dr" is forbidden: User "mathoid" cannot delete pods in the namespace "mathoid"
Note the forbidden part. That means that the regular deployer does not have the ability to do that (and for good reason). But SREs can do:
 deploy1001:~$ sudo KUBECONFIG=/etc/kubernetes/admin-eqiad.config kubectl -n mathoid delete pods mathoid-production-7b9999dd5f-cr2dr
 pod "mathoid-production-7b9999dd5f-cr2dr" deleted
=== Restarting all pods for the service ===
See: Kubernetes#Recreate pods (of deployments, daemonsets, statefulsets, ...)
== See also ==
* Puppet module (outdated?)
* code
Category:Services