Miscweb

miscweb is a new service on kubernetes. 
Since 2022-01-20 it serves production traffic for static-bugzilla. Other miscweb releases in the same namespace contain different services, like annual.wikimedia.org and transparency.wikimedia.org.
It was requested in to replace the legacy service "miscweb" running on Ganeti VMs in production.
Also see: miscweb1002, miscweb2002 for the legacy machines still serving other microsites.
----
== Sites running on miscweb k8s ==
The first of the sites hosted on miscweb-k8s was static-bugzilla.
Since 2022-01-20 static-bugzilla.wikimedia.org is served from k8s. Other microsites are running on miscweb as well. All services in miscweb:
* annual.wikimedia.org and 15.wikipedia.org
* bienvenida.wikimedia.org
* transparency.wikimedia.org
* static-bugzilla.wikimedia.org
* tendril.wikimedia.org and dbtree.wikimedia.org
* research.wikimedia.org
* wikiworkshop.org
* static-codereview.wikimedia.org
* design.wikimedia.org
== Where does the code live? ==
Most code for the html files and blubber container images is hosted under https://gitlab.wikimedia.org/repos/sre/miscweb. Each service has a project there. The projects contain 4 important files and folders:
* .pipeline folder contains blubber configuration for the image build
* html folder contains the static html which is served by the container image. Note: this folder is also called html-compressed for big sites which use lfs and compressed storage of html files.
* production folder contains the apache configuration for the apache2 process in the container
* .gitlab-ci.yml contains the CI configuration for building and publishing the image to our image registry.
The helm charts for kubernetes can be found in operations/deployment-charts.
== How to deploy changes ==
Most service changes contain of updates of the used container image and updates to the Kubernetes helm deployment.
=== Update the container image ===
To update the container image, checkout the project from https://gitlab.wikimedia.org/repos/sre/miscweb/. Create your changes to the container image and get a review in a merge request. Once the merge request is merged to the main branch, a dedicated publish pipeline job is executed. Find the job in GitLab in your project under CI/CD -> Jobs. In the last lines in the job log you can find the new image tag. For example:
#13 pushing layers
#13 pushing layers 4.9s done
13# pushing manifest for docker-registry.discovery.wmnet/repos/sre/miscweb/annualreport:2023-08-14-083827
77bf6661c95d2ee40d007b0e9 0.6s done
#13 DONE 10.2s
Copy the new image tag 2023-08-14-083827 for the next step. Note: if you browse https://docker-registry.wikimedia.org/ it can take some time for your image to show up due to caching.
=== Deploy to Kubernets/wikikube ===
Deployment workflow follows the standard process described in Kubernetes/Deployments#Code deployment/configuration changes.
Checkout /operations/deployment-charts. Update the image tag in the matching miscweb values file with the tag from the previous step (for example in /helmfile.d/services/miscweb/values-annualreport.yaml#3) or do any other change to the Kubernetes service if needed. Get a review and a +2 from SRE team.
Once the change to deployment-charts repo is merged, login to Deployment server and cd to /srv/deployment-charts/helmfile.d/services/miscweb
==== Staging ====
* helmfile -e staging diff --context 5
* helmfile -e staging -i apply --context 5
==== Production ====
* helmfile -e codfw diff --context 5
* helmfile -e codfw -i apply --context 5
* helmfile -e eqiad diff --context 5
* helmfile -e eqiad -i apply --context 5
All helm deployments are atomic. So either they work or will be reverted automatically after 5 minutes.
==== Deploy a single release/service ====
To deploy just one release, you can use --selector flag in helmfile. Replace annualreport with the correct release name:
* helmfile -e eqiad -i --selector name=annualreport apply
==== Toubleshooting ====
For troubleshooting, use kube_env and kubectl:
* kube_env miscweb staging
* kubectl get pods
== Service names ==
 miscweb.svc.eqiad.wmnet has address 10.2.2.58 (eqiad)
 miscweb.svc.codfw.wmnet has address 10.2.1.58 (codfw)
 miscweb.discovery.wmnet has address 10.2.2.58 (DNS/Discovery)
== LVS / discovery ==
https://config-master.wikimedia.org/pybal/eqiad/miscweb
https://config-master.wikimedia.org/pybal/codfw/miscweb
https://config-master.wikimedia.org/discovery/
== Metrics and dashboards ==
https://grafana.wikimedia.org/d/exdTE7kSk/miscweb?orgId=1
== How this service was made ==
Here I am trying to compile a table / list of all the changes made to get this service from scratch into WMF production, in chronological order of how they were merged.
 steps for miscweb
# 	action 	link 	
1 	created a new service request ticket 	https://phabricator.wikimedia.org/project/profile/1305/ 	
2 	read docs 	https://wikitech.wikimedia.org/wiki/Kubernetes#Add_a_new_service 	
3 	reserved a service port 	https://wikitech.wikimedia.org/wiki/Kubernetes/Service_ports 	
4 	added tokens in private repo to CI::master and deployment_server in private repo 	cd /srv/private/.. on the puppetmaster (ask an SRE with root access if needed) 	
5 	added dummy tokens in the labs/private repo 	https://gerrit.wikimedia.org/r/684000 	
6 	created a new namespace in kubernetes, use helmfile apply on deployment servers 	https://gerrit.wikimedia.org/r/683743 	
7 	added new namespace to CI and deployment_server 	https://gerrit.wikimedia.org/r/681500/ , https://gerrit.wikimedia.org/r/685116 	
8 	requested a new Gerrit repo to host your (Blubber) code 	https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests 	
9 	read about deployment pipeline 	https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Migration/Tutorial 	
10	added initial config stub for pipeline lib 	https://gerrit.wikimedia.org/r/690678 	
11	read about Blubber 	https://wikitech.wikimedia.org/wiki/Blubber , https://wikitech.wikimedia.org/wiki/Blubber/Pipeline 	
12	added initial Blubber file 	https://gerrit.wikimedia.org/r/690768 	
13	added pipelines and config in integration/config 	https://gerrit.wikimedia.org/r/690788 (asked releng) 	
14	added bespoke pipeline in integration/config if needed 	https://gerrit.wikimedia.org/r/690794 (asked releng) 	
15	added LVS service IPs 	https://gerrit.wikimedia.org/r/693966 	
16	added entrypoint.sh in Blubber 	https://gerrit.wikimedia.org/r/697140 	
17	tried staging/test variants 	https://gerrit.wikimedia.org/r/697142 	
18	simplified apache config 	https://gerrit.wikimedia.org/r/697654/ , https://gerrit.wikimedia.org/r/697663 , https://gerrit.wikimedia.org/r/697691 	
19	installed vim, curl in container for testing 	https://gerrit.wikimedia.org/r/697655/ , https://gerrit.wikimedia.org/r/697666 	
20	dropped/merged unused pipeline 	https://gerrit.wikimedia.org/r/697657 	
21	switched service to not run 'insecurely' (as a separate user) 	https://gerrit.wikimedia.org/r/697662/ 	
22	added virtual site inside webserver 	https://gerrit.wikimedia.org/r/697695 	
23	tested cloning from repo, letting Blubber generate a Dockerfile and got shell inside container 	https://phabricator.wikimedia.org/T281538#7128132 	
24	stopped loading modules not used 	https://gerrit.wikimedia.org/r/698079 	
25	reserved a public port for LVS 	https://wikitech.wikimedia.org/w/index.php?title=Service_ports&type=revision&diff=1914806&oldid=1913236 	
26	opened firewall on deployment server to dump data from pre-k8s service 	https://gerrit.wikimedia.org/r/699064 	
27	rsynced data over to deployment server 	https://phabricator.wikimedia.org/T281538#7147262 	
28	added config to serve data gzipped to reduce image size, installed browser in container to test 	https://gerrit.wikimedia.org/r/698079 , https://gerrit.wikimedia.org/r/699320 	
29	load mod_rewrite and mod_headers, add headers/encoding settings for gziped content 	https://gerrit.wikimedia.org/r/699319 	
30	read about helm and deployments on kubernetes 	https://wikitech.wikimedia.org/wiki/Helm , https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments 	
30	cloned the repo 'operations/deployment-charts' where the helm files live 	https://gerrit.wikimedia.org/r/admin/repos/operations/deployment-charts 	
31	read README in the repo about how to create charts 	https://gerrit.wikimedia.org/g/operations/deployment-charts 	
32	read and ran 'create_new_service.sh' 	https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/create_new_service.sh https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/Rakefile	
33	adjusted values in new files generated by script and uploaded to the repo 	https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/698895/ 	
34	created a new app type for a httpd without php-fpm, added a prometheus (metrics) exporter 	https://gerrit.wikimedia.org/r/700522 	
35	added helmfile.yaml and values under services.d, copying from another service 	https://gerrit.wikimedia.org/r/713441 	
36	set the docker registry name specifically to use the discovery name 	https://gerrit.wikimedia.org/r/714014/ 	
37	added uncompressed content of the first 1000 Bugzilla bugs 	https://gerrit.wikimedia.org/r/714460 	
38	cleaned up and added comments for others to delete files they don't use 	https://gerrit.wikimedia.org/r/713639 	
39	set a main_app version and added some CPU/RAM limits 	https://gerrit.wikimedia.org/r/714022 	
40	added reserved port as nodePort 	https://gerrit.wikimedia.org/r/714053 	
41	added version tags for staging and production 	https://gerrit.wikimedia.org/r/714368 	
42	linked staging httpd config to prod httpd config 	https://gerrit.wikimedia.org/r/714458 	
43	added httpd rewrite rules from pre-k8s config 	https://gerrit.wikimedia.org/r/714459 	
44	set service deployment to production, not minikube 	https://gerrit.wikimedia.org/r/714034 	
45	bumped staging version to latest build created by CI 	https://gerrit.wikimedia.org/r/714755 , https://gerrit.wikimedia.org/r/715236 etc .. (skipping these in the future, needed after every change) 	
46	loaded missing mod_alias for Redirect directive 	https://gerrit.wikimedia.org/r/715727 	
47	added HTML content for the first 10000 bugs, checked image size 	https://gerrit.wikimedia.org/r/717347 	
48	compressed content with gzip and added more bug HTML 	https://gerrit.wikimedia.org/r/728668 	
49	various changes to add all the content in batches of 10k bugs, then the same for activities HTML files 	https://gerrit.wikimedia.org/r/730275 , https://gerrit.wikimedia.org/r/730281 and various others up to https://gerrit.wikimedia.org/r/730334 	
50	added and gzipped index and "all" pages 	https://gerrit.wikimedia.org/r/730336 	
51	added old Bugzilla Wikimedia skin directory 	https://gerrit.wikimedia.org/r/730339 	
52	read about adding a new service to LVS 	https://wikitech.wikimedia.org/wiki/LVS#Add_a_new_load_balanced_service 	
53	added service IPs in DNS 	https://netbox.wikimedia.org/ (ask infra foundations) 	
53	added LVS config and had it merged 	https://gerrit.wikimedia.org/r/694625 (coordinate with serviceops/traffic for this step) 	
54	switched service_state from service_setup to lvs_setup 	https://gerrit.wikimedia.org/r/694628 	
55	enabled TLS in helm chart 	https://gerrit.wikimedia.org/r/739675 	
56	removed nodePort, added public_port, enabled TLS, multiple attempts to get the order right, then TLS worked 	https://gerrit.wikimedia.org/r/739810 , https://gerrit.wikimedia.org/r/739848 , https://gerrit.wikimedia.org/r/739945 , https://gerrit.wikimedia.org/r/742819 	
57	switched service_state from lvs_setup to monitoring_setup, checked new Icinga monitoring being added, further testing to confirm it at all works 	https://gerrit.wikimedia.org/r/694629 , https://phabricator.wikimedia.org/T281538#7578691 	
58	debugged gzip encoding issue in cloud VPS, confirmed can pull and run directly from prod docker registry 	https://phabricator.wikimedia.org/T281538#7606684 	
59	fixed content type for HTML, which was set to CSS, service now working in cloud 	https://gerrit.wikimedia.org/r/752235 , https://staticbz.wmcloud.org/bug10001.html 	
60	further version bumping / deploying / testing 	https://gerrit.wikimedia.org/r/752750 	
61	confirmed working with curl directly from production service names with right content-type and content-encoding 	https://phabricator.wikimedia.org/T281538#7620703 	
62	switched service_state from monitoring_setup to production (make it page) but only very carefully after checking confd templates on DNS servers, downtiming services in Icinga	https://gerrit.wikimedia.org/r/694630 , https://phabricator.wikimedia.org/T281538#7620961 	
63	read about discovery DNS 	https://wikitech.wikimedia.org/wiki/DNS/Discovery 	
64	added discovery DNS as an active-active service, confirmed could now curl from discovery name 	https://gerrit.wikimedia.org/r/693968 , https://phabricator.wikimedia.org/T281538#7620995 	
65	switched ATS (traffic servers/caching layer) from old backend to new backend, the discovery name on our reserved service port 	https://gerrit.wikimedia.org/r/753813 	
66	added service to disc_desired_state.py 	https://gerrit.wikimedia.org/r/753846 	
67	ATS servers got 502, did not work, reverted, turned out the reason was a missing SAN on the TLS cert 	 	
68	addded SAN to cert, created new cert, checked it 	https://phabricator.wikimedia.org/T281538#7635115 	
== How to switch microsites hosted on legacy VMs from one datacenter to another ==
This section is not about the kubernetes service but about the ganeti VMs hosting microsites that have not moved yet.
This is how to switch those from one DC to another:
# Check the google doc "sre collab service matrix" to see which services are hosted on miscweb* and pick one to switch
# go to the puppet repo and find your service in the file common/profile/trafficserver/backend.yaml
# confirm that it points to the name "https://webserver-misc-sites.discovery.wmnet"
# look in the DNS repository in the file ./templates/wmnet and find this section:
;webserver-misc-sites 300 IN CNAME miscweb1003.eqiad.wmnet.
webserver-misc-sites 300 IN CNAME miscweb2003.codfw.wmnet.
# Now decide to either switch ALL sites at once and switch around which server is commented out and which is not
## or introduce a different name, like webserver-misc-sites but separately, that points to miscweb1003 instead of miscweb2003 (or vice versa)
## if you are doing the second, go back to the ATS config (backend.yaml) mentioned above and switch just one of the sites to your new DNS discovery name
# run puppet on all cp* hosts via cumin, or just on the hosts for your data center (ex. "cp4*" for ulsfo) or wait half an hour
# on old and new miscweb machine do a tail -f /var/log/apache2/*.log and watch the access logs, each site has its own access and error log file
# open the site in your browser and confirm the request hits your new backend and the site looks like before
# run a httpbb test against your new (or old and new) server and confirm there are no errors or differences (from a deployment server, example:)
## [deploy1002:~] $ httpbb --hosts miscweb1003.eqiad.wmnet /srv/deployment/httpbb-tests/miscweb/test_miscweb.yaml
# mark the site as switched in doc/ticket. !log in SAL 
# double check sites that have external deployers (deploy method column in the service matrix doc)
Category:Services
Category:SRE Service Operations