Citoid/Deploying citoid

These are directions for deploying citoid, a nodejs service. More detailed but more general directions for nodejs services are at Migrating_from_scap-helm#Code_deployment/configuration_changes.
===Locate build candidate===
From gerrit, locate the candidate build. PipelineBot will post a message with the build name, i.e.
PipelineBot
Mar 15 9:22 PM
Patch Set 4:
Wikimedia Pipeline
Image Build SUCCESS
IMAGE:
 docker-registry.discovery.wmnet/wikimedia/mediawiki-services-citoid
TAGS:
 2019-03-15-000-production, 950e3b4468f2f84d3bb23c33ba68d8a26725
'2019-03-15-000-production' is the name of the build.
===Add change via gerrit===
====Pipeline bot====
Pipeline bot automatically creates commits to the deployments repo with the subject line "citoid: pipeline bot promote".
# Find the commit by going to https://gerrit.wikimedia.org/r/q/topic:%22pipeline-promote%22+(status:open). If there is none labelled "citoid" in the list, then there are no new versions to merge. 
# +2 the commit to merge it.
====Manually====
# Clone deployment-charts repo.
# cd helmfile.d/services/citoid
=====Update image=====
# vi values.yaml
main_app:
  image: wikimedia/mediawiki-services-citoid
  limits:
    cpu: 10
    memory: 4Gi
  liveness_probe:
    tcpSocket:
      port: 1970
  port: 1970
  requests:
    cpu: 200m
    memory: 200Mi
  version: 2019-01-17-114541-candidate-change-me
# Make a CR to change the version value in values.yaml and after a successful review, merge it.
# After merge, log into a deployment server, there is a cronjob (1 minute) that will update the /srv/deployment-charts directory with the contents from git.
=====Config change=====
Increment chart version in: 
charts/citoid/Chart.yaml 
Changes values in the template: 
charts/citoid/templates/_config.yaml
===Navigating the server===
Ssh into the deploy machine.
 ssh deployment.eqiad.wmnet
citoid runs on two of the available server farms, codfw and eqiad. You can confirm this by doing
Change into the citoid directory.
 cd /srv/deployment-charts/helmfile.d/services/citoid
List the files in the directory. This is what you should see. 
 > ls
helmfile.yaml values-codfw.yaml values-staging.yaml values.yaml
===Staging===
 >myusername@deploy1001:/srv/deployment-charts/helmfile.d/services/citoid$ 
 
 helmfile -e staging -i apply
Apply may take awhile. 
 
 >helmfile -e staging status (to check status again)
To confirm the staging cluster works, try a request:
===Deprecated API===
curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/api?format=mediawiki&search=979%201029801297'
curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/api?format=mediawiki&search=https%3A%2F%2Fjournals.plos.org%2Fploscompbiol%2Farticle%3Fid%3D10.1371%2Fjournal.pcbi.1002947'
curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/api?format=mediawiki&search=https%3A%2F%2Fwww.wsj.com%2Fworld%2Fasia%2Fhistoric-moon-mission-gets-china-one-small-step-ahead-in-space-race-41894d41'
===Restbase shim API===
curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/mediawiki/979%201029801297'
curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/mediawiki/https%3A%2F%2Fjournals.plos.org%2Fploscompbiol%2Farticle%3Fid%3D10.1371%2Fjournal.pcbi.1002947'
curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/mediawiki/https%3A%2F%2Fwww.wsj.com%2Fworld%2Fasia%2Fhistoric-moon-mission-gets-china-one-small-step-ahead-in-space-race-41894d41'
These should successfully retrieve a JSON list of citations. 
The following should return an error: 
curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/mediawiki/https%3A%2F%2Fupload.wikimedia.org%2Fwikipedia%2Fcommons%2F9%2F98%2FColoring_page_for_Wikipedia_Day_2019_in_NYC.pdf'
===Production===
codfw cluster:
 >myusername@deploy1001:/srv/deployment-charts/helmfile.d/services/citoid$ 
 helmfile -e codfw -i apply
 helmfile -e codfw status
Verify citoid is running with a curl request:
 curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.svc.codfw.wmnet:4003/api?format=mediawiki&search=979%201029801297'
 curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.svc.codfw.wmnet:4003/api?format=mediawiki&search=https%3A%2F%2Fjournals.plos.org%2Fploscompbiol%2Farticle%3Fid%3D10.1371%2Fjournal.pcbi.1002947'
Repeat the same process for the eqiad cluster. 
 >myusername@deploy1001:/srv/deployment-charts/helmfile.d/services/citoid$ 
 helmfile -e eqiad -i apply
 helmfile -e eqiad status
Then verify with a curl request: 
 curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.svc.eqiad.wmnet:4003/mediawiki/9791029801297'
 curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.svc.eqiad.wmnet:4003/mediawiki/https%3A%2F%2Fjournals.plos.org%2Fploscompbiol%2Farticle%3Fid%3D10.1371%2Fjournal.pcbi.1002947'
curl for all servers: 
 curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.discovery.wmnet:4003/mediawiki/9791029801297'
===Testing behind restbase===
And finally, you can test the restbase implementation which should give the same results@
https://en.wikipedia.org/api/rest_v1/#!/Citation/getCitation
===Graphing and Logs===
For an overview of how your deploy is doing, you can look at overall stats and logs:
https://grafana.wikimedia.org/d/NJkCVermz/citoid?orgId=1&refresh=5m&from=now-30m&to=now
https://logstash.wikimedia.org/app/dashboards#/view/5eaf4e40-f6b6-11eb-85b7-9d1831ce7631
===Probe===
https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?forceLogin&from=now-30m&orgId=1&to=now&var-site=codfw