Application servers

The Application servers (or app servers) are the several hundred Apache servers that run MediaWiki (PHP application).
==Service==
Puppet roles:
* mediawiki::canary_appserver (only used by mwdebug)
* mediawiki::maintenance
* mediawiki::jobrunner (only used for videoscalers)
* profile::kubernetes::deployment_server::mediawiki Release configuration for MediaWiki On Kubernetes
MediaWiki On Kubernetes deployments and their purpose
Relevant puppet classes:
* profile::mediawiki::webserver, this provisions Apache, and any other packages or resources needed by MediaWiki on app servers.
** profile::mediawiki::httpd, the Apache service.
** mediawiki::web::prod_sites, the Apache configuration for all production websites (including wikipedia.org).
** Additional Apache configurations are at modules/mediawiki/files/apache/sites/. Prior to 2012, Apache configuration were in a Subversion repository.
==Architecture==
The application servers are load-balanced via LVS. Connections between our CDN (HTTP cache proxies) and app servers are encrypted with TLS, which is terminated locally on the app server using Envoy. Envoy then hands the request off to the local Apache.
Apache is in charge of handling redirects, rewrite rules, and determining the document root. It then uses php-fpm to invoke the MediaWiki software.
The Apache MPM we use is mod_worker</code>, which decides how php-fpm processes are spawned.
==Logging==
Apache errors are logged to /srv/mw-log/apache2.log on mwlog1001.
Apache access logs are mostly disabled. Statistics are drawn from Varnish front ends instead.
==Update our PHP packages==
We use custom PHP packages, which are co-installable and more recent that what is shipped in the underlying Debian releases. If you want to add/update a patch you can do the following:
* On an existing app server obtain the source with "apt-get source php7.4"
* Copy the source files (*debian.tar.xz, *dsc, *orig.tar.xz(.asc)) to build2001.codfw.wmnet
* Unpack the souce with "dpkg-source -x $DSCFILE"
* Make the modification within the source tree (e.g. applying a local patch or a backport from upstream)
* Run "dpkg-source --commit" to record your changes in a debian/patches/foo patch file (debian/patches/series is automatically amended)
* Bump the changelog with "dch -i" (which spawns an editor)
* Finally build the updated package with "PHP74=yes DIST=buster-wikimedia pdebuild"
* Test the resulting build (and if all is fine, import to apt.wikimedia.org)
== Hardware repair ==
When taking down application servers (running mediawiki) for things like disk replacement or other hardware repair, _do not forget to_:
* before: remove from dsh group
These are in puppet, operations/puppet repo, in modules/dsh/files/group. The important one for Mediawiki sync is "mediawiki-installation".
* before: de-pool in pybal
* TODO: Document what to do if it's a scap proxy (see hieradata/common/dsh/config.yaml)
See pybal. You can just grep for the server name and set 'enabled': False and save.
* before: check nobody is scapping right now (best: announce with a !log line in IRC)
This is an IRC thing on libera.chat in //
* during: acknowledge Icinga monitoring checks (best: with related ticket number as comment)
Do this by logging in via browser on icinga.wikimedia.org. search for the hostname, check all services and use the "acknowledge" option. You'll see the IRC bots outputting this as well and they will stop repeating things over and over in the channels.
* after: re-add to dsh groups
Revert the above.
* after: re-pool in pybal
Revert the above.
== See also ==
* Apache log format
* UID
 Apache
 
Category:SRE Service Operations