Incidents/2018-12-12 Train-1.33.0-wmf.8

== Summary ==
There were several problems with the 2018-12-11 1.33.0-wmf.8 train (phab:T206662) resulting in 1 rollback.
=== Stats ===
* changes: 559
* repos: 84
* committers: 94
* finished on-time: yes
* rollbacks: 1
* issues: 6
* :mw:MediaWiki_1.33/wmf.8/Changelog
=== Issues ===
* phab:T211769 Wikimedia\Rdbms\Database::selectSQLText called from ApiBase::filterIDs with incorrect parameters: $conds must be a string or an array
* phab:T211774 Full table scans on oldimage table
* phab:T211805 Call to a member function format() on a non-object (boolean)
* phab:T211806 Passing in the "body" request option as an array to send a POST request has been deprecated
* phab:T211885 ErrorException from line 47 of /srv/mediawiki/php-1.33.0-wmf.8/extensions/Kartographer/includes/ApiQueryMapData.php: PHP Warning: data error
* phab:T211886 Fatal error: request has exceeded memory limit in /srv/mediawiki/php-1.33.0-wmf.8/vendor/guzzlehttp/psr7/src/Stream.php on line 97
== Timeline ==
This is a step by step outline of what happened to cause the incident and how it was remedied.
=== 2018-12-11 ===
* 13:57 zeljkof: starting branch cut for 1.33.0-wmf.8
* 14:43 <zfilipin@deploy1001> Pruned MediaWiki: 1.33.0-wmf.1 (duration: 11m 40s)
* 14:46 <zfilipin@deploy1001> Pruned MediaWiki: 1.33.0-wmf.2 (duration: 03m 12s)
* 14:50 <zfilipin@deploy1001> Pruned MediaWiki: 1.33.0-wmf.3 (duration: 02m 51s)
* 15:31 <zfilipin@deploy1001> Finished scap: testwiki to php-1.33.0-wmf.8 and rebuild l10n cache (duration: 36m 08s)
* 15:49 <zfilipin@deploy1001> rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.8
==== End of Day Summary ====
* Group0: 1.33.0-wmf.8
* Group1: 1.33.0-wmf.6
* Group2: 1.33.0-wmf.6
* Status: 
=== 2018-12-12 ===
* 14:16 <zfilipin@deploy1001> Synchronized php: group1 wikis to 1.33.0-wmf.8 (duration: 00m 51s)
* 14:29 <zfilipin@deploy1001> rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.33.0-wmf.8"
* 14:35 phab:T211769 Wikimedia\Rdbms\Database::selectSQLText called from ApiBase::filterIDs with incorrect parameters: $conds must be a string or an array
* 15:31 phab:T211769 Full table scans on oldimage table
* 15:37 phab:T211769 Wikimedia\Rdbms\Database::selectSQLText called from ApiBase::filterIDs with incorrect parameters: $conds must be a string or an array
* 17:02 <zfilipin@deploy1001> Synchronized php: group1 wikis to 1.33.0-wmf.8 (duration: 00m 50s)
* 17:11 phab:T211769 Full table scans on oldimage table
* 18:35 phab:T211805 Call to a member function format() on a non-object (boolean)
* 18:47 phab:T211806 Passing in the "body" request option as an array to send a POST request has been deprecated
* 21:48 phab:T211805 Call to a member function format() on a non-object (boolean)
==== End of Day Summary ====
* Group0: 1.33.0-wmf.8
* Group1: 1.33.0-wmf.8
* Group2: 1.33.0-wmf.6
* Status: 
=== 2018-12-13 ===
* 01:00 phab:T211806 Passing in the "body" request option as an array to send a POST request has been deprecated
* 14:21 <zfilipin@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.8
* 14:46 phab:T211885 ErrorException from line 47 of /srv/mediawiki/php-1.33.0-wmf.8/extensions/Kartographer/includes/ApiQueryMapData.php: PHP Warning: data error
* 15:03 phab:T211886 Fatal error: request has exceeded memory limit in /srv/mediawiki/php-1.33.0-wmf.8/vendor/guzzlehttp/psr7/src/Stream.php on line 97
* 22:40 phab:T211886 Fatal error: request has exceeded memory limit in /srv/mediawiki/php-1.33.0-wmf.8/vendor/guzzlehttp/psr7/src/Stream.php on line 97
==== End of Day Summary ====
* Group0: 1.33.0-wmf.8
* Group1: 1.33.0-wmf.8
* Group2: 1.33.0-wmf.8
* Status: 
=== 2018-12-18 ===
* 11:36 phab:T211885 ErrorException from line 47 of /srv/mediawiki/php-1.33.0-wmf.8/extensions/Kartographer/includes/ApiQueryMapData.php: PHP Warning: data error
== Conclusions ==
What weakness did we learn about and how can we address them?
== Links to relevant documentation ==
* Heterogeneous deployment/Train deploys
== Actionables ==
Explicit next steps to prevent this from happening again as much as possible, with Phabricator tasks linked for every step.
NOTE: Please add the #wikimedia-incident Phabricator project to these follow-up tasks and move them to the "follow-up/actionable" column.
* To do #1 (TODO: Create task)
* To do #2 (TODO: Create task)
* …