Server Admin Log/Archive 22

== July 31 ==
* 23:40 logmsgbot: rmoen synchronized wmf-config/InitialiseSettings.php 'Remove config variables for VE gender survey'
* 23:39 logmsgbot: rmoen synchronized wmf-config/CommonSettings.php 'Remove config variables for VE gender survey'
* 23:32 logmsgbot: rmoen Finished syncing Wikimedia installation... : update visualeditor to master
* 23:25 logmsgbot: rmoen Started syncing Wikimedia installation... : update visualeditor to master
* 23:18 logmsgbot: bsitu synchronized php-1.22wmf12/extensions/Echo 'Update Echo to master'
* 22:14 logmsgbot: csteipp synchronized php-1.22wmf12/extensions/CentralAuth 'eventlogging patch'
* 18:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:33 cmjohnson1: amaranth currently unreachable according to nosy....going to try rebooting 
* 18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.22wmf12
* 15:42 logmsgbot: hashar synchronized wmf-config ' set some paths to use 'bits' and 'test''
* 02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 31 02:46:48 UTC 
* 02:31 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Wed Jul 31 02:29:48 UTC 
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Wed Jul 31 02:15:56 UTC 
* 01:09 springle: restarting db45 slave threads
* 01:08 springle: restarting db43 slave threads
== July 30 ==
* 23:36 logmsgbot: catrope synchronized php-1.22wmf12/extensions/VisualEditor 'Update VE to master'
* 23:35 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor 'Update VE to master'
* 23:15 logmsgbot: bsitu synchronized php-1.22wmf12/extensions/Echo/modules/alert/ext.echo.alert.js 'Refresh js file'
* 23:14 logmsgbot: bsitu synchronized php-1.22wmf11/extensions/Echo/modules/alert/ext.echo.alert.js 'Refresh js file'
* 23:11 logmsgbot: bsitu synchronized php-1.22wmf12/extensions/Echo/modules/alert/ext.echo.alert.js 'Refresh js file'
* 23:10 logmsgbot: bsitu synchronized php-1.22wmf11/extensions/Echo/modules/alert/ext.echo.alert.js 'Refresh js file'
* 23:07 awight: listeners updates from to 
* 22:20 awight: payments updated from to 
* 21:35 awight_: payments updated from to 
* 21:29 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:28 awight_: listeners updated from to 
* 21:18 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 20:35 mutante: gracefull'ing apaches, RT- & bug 
* 20:19 paravoid: switching wikidata back to squid; reports of "bad token" messages
* 19:24 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo, Thanks, PageTriage to master
* 19:19 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo, Thanks, PageTriage to master
* 19:17 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Revert Enable Secure Login everywhere'
* 19:16 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings-labs.php 'Revert Enable Secure Login everywhere'
* 18:41 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo, Thanks, PageTriage to master
* 18:20 mutante: deploying fix for Bug - Correct the mime type for WOFF to application/font-woff
* 18:05 awight: fr-civicrm updated from to 
* 18:04 awight: fr-civicrm updated from to 
* 17:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Translate on collabwiki'
* 17:37 Reedy: Created Translate tables on collabwiki
* 17:15 logmsgbot: reedy synchronized wmf-config/ 'Disable UploadBlacklist'
* 17:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 17:09 logmsgbot: reedy synchronized wmf-config/
* 15:22 Reedy: Purging old SecurePoll private data from all wikis
* 12:21 mark: Put all non-European wikidata traffic on cp1052 and cp1065
* 11:21 mark: Pooled cp1052 as text varnish box in PyBal group 'text-varnish' with weight 1, to serve a tiny fraction of wikidata traffic
* 08:56 Nikerabbit: Translation memory migrated from vanadium to zinc and repopulated
* 08:12 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'Translation memory to zinc from vanadium'
* 02:58 sbernardin: racked and plugged back pappas 
* 02:54 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 30 02:52:51 UTC 
* 02:34 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Tue Jul 30 02:33:27 UTC 
* 02:19 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Tue Jul 30 02:17:54 UTC 
== July 29 ==
* 23:23 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php 'Enabling wgNoticeUseLanguageConversion for automatic zh conversion in CentralNotice'
* 23:21 logmsgbot: mwalker synchronized php-1.22wmf12/extensions/CentralNotice 'Updating CentralNotice to master on wmf12'
* 23:18 logmsgbot: mwalker synchronized php-1.22wmf11/extensions/CentralNotice 'Updating CentralNotice to master on wmf11'
* 23:03 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'enable captcha on ptwiki'
* 22:35 andrewbogott: moved private puppet manifests into modules
* 22:31 mutante: creating operations/debs/etherpad-lite gerrit project
* 22:21 logmsgbot: maxsem synchronized php-1.22wmf11/extensions/MobileFrontend/javascripts/modules/editor/EditorApi.js
* 22:19 logmsgbot: maxsem synchronized php-1.22wmf12/extensions/MobileFrontend/javascripts/modules/editor/EditorApi.js
* 21:18 mutante: removing unused mysql-server from hooper, the only service uses db9
* 21:16 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Hack for Cirrus in labs'
* 21:13 mutante: upgrading packages on hooper (etherpad/racktables)
* 20:34 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php 'More Cirrus goodies'
* 20:17 logmsgbot: reedy synchronized php-1.22wmf11/extensions/DataTypes
* 20:13 logmsgbot: reedy synchronized wmf-config/
* 20:10 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Syncing Cirrus config'
* 20:09 logmsgbot: demon synchronized wmf-config/CirrusSearch-labs.php 'Syncing Cirrus config'
* 20:08 logmsgbot: demon synchronized wmf-config/CirrusSearch-common.php 'Syncing Cirrus config'
* 20:03 logmsgbot: reedy synchronized php-1.22wmf12/extensions/CirrusSearch
* 19:49 logmsgbot: reedy synchronized php-1.22wmf12/extensions/DataTypes
* 19:41 springle: delaying slave db45 during OSC bug 
* 19:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikibooks to 1.22wmf12
* 19:06 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Actually disable VE for anons on dewiki, hopefully'
* 19:01 Jeff_Green: thulium kernel update, reboot, tweak CPU power settings
* 19:01 springle: delaying slave db43 during OSC bug 
* 19:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikisource to 1.22wmf12
* 19:00 andrewbogott: set root password to ! on all labs instances.
* 18:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews, wikivoyage and wikiversity
* 18:55 ^demon: restarted gitblit service
* 18:54 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, wikimedia and special to 1.22wmf12
* 18:42 sbernardin: server pappas being taken offline to ship to EQIAD
* 18:41 logmsgbot: reedy synchronized php-1.22wmf12/extensions/Wikibase
* 18:18 mutante: DNS update - added voyagewiki.org and .com
* 18:12 springle: restarting db56 slave threads
* 16:44 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable VE for anons on es/fr/he/it/pl/ru/svwiki; set dewiki back to opt-in mode'
* 16:35 logmsgbot: catrope synchronized php-1.22wmf12/extensions/VisualEditor 'Update VE to master'
* 16:34 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor 'Update VE to master'
* 14:50 logmsgbot: maxsem synchronized wmf-config/squid.php 'Amsterdam varnish IPv6'
* 14:25 andrewbogott: restarted apache on virt0, fixing (at least temporarily) bug 
* 13:37 Jeff_Green: removed spammy deprecated cron from gadolinium
* 13:24 ottomata: stopping nginx on maerlant.esams.wikimedia.org (this machine should probably be fully decommissioned)
* 04:31 springle: delaying slave db56 during OSC bug 
* 02:34 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 29 02:33:00 UTC 
* 02:20 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Mon Jul 29 02:19:08 UTC 
* 02:19 springle: enabling slave db45
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Mon Jul 29 02:10:33 UTC 
* 01:23 springle: delaying slave db45
== July 28 ==
* 22:37 MaxSem: mw1157 was running out of /tmp space, nuked all *.v older than 1 day. VIPS bug?
* 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 28 02:07:37 UTC 
* 02:04 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sun Jul 28 02:02:52 UTC 
* 02:03 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Sun Jul 28 02:02:01 UTC 
== July 27 ==
* 23:43 Krinkle: Graceful reload of Zuul (in attempt to purge the ghost entries caused by bug )
* 02:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 27 02:49:20 UTC 
* 02:32 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Sat Jul 27 02:31:17 UTC 
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Sat Jul 27 02:16:34 UTC 
== July 26 ==
* 23:12 mutante: activated softwarewikipedia.com/.org
* 23:08 mutante: activated http://wikimediacommons.pt/
* 23:02 mutante: DNS update - removing zones that link to other links. use wikipedia.org not wikipedia.com
* 20:05 mutante: synced apache, gracefull'ed, activate visualwikipedia.com and .net
* 18:22 mutante: installing package upgrades on iron
* 18:15 binasher: db1042 unresponsive on serial console, power cycling
* 18:13 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1042 from s4'
* 17:07 mark: Rebooting cp3011
* 15:34 logmsgbot: reedy synchronized php-1.22wmf12/extensions/Wikibase
* 15:32 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
* 02:36 ori-l: ran the git-clone update component of l10nupdate (l10nupdate-1, lines 17-45) as 'l10nupdate' user to try and reproduce "git pull of extensions failed" error, but it completed successfully for me.
* 02:04 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
* 01:11 logmsgbot: reedy synchronized php-1.22wmf11/languages/messages/MessagesTl.php
* 00:48 logmsgbot: bsitu synchronized php-1.22wmf11/extensions/Echo/modules/special/ext.echo.special.js 'update cache for resourceloader'
* 00:47 logmsgbot: bsitu synchronized php-1.22wmf11/extensions/Echo/modules/overlay/ext.echo.overlay.js 'update cache for resourceloader'
* 00:17 logmsgbot: maxsem synchronized php-1.22wmf11/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c//'
* 00:16 logmsgbot: maxsem synchronized php-1.22wmf12/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c//'
* 00:11 logmsgbot: bsitu Finished syncing Wikimedia installation... : IE ajax browser cache fix for 1.22wmf11 and scap to update message cache for 1.22wmf12
* 00:00 logmsgbot: bsitu Started syncing Wikimedia installation... : IE ajax browser cache fix for 1.22wmf11 and scap to update message cache for 1.22wmf12
== July 25 ==
* 20:50 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/GettingStarted 'E3 deployment: bugfixes for GuidedTour and GettingStarted (2/2)'
* 20:49 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/GuidedTour 'E3 deployment: bugfixes for GuidedTour and GettingStarted (1/2)'
* 20:47 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/GettingStarted 'E3 deployment: bugfixes for GuidedTour and GettingStarted (2/2)'
* 20:47 logmsgbot: olivneh synchronized php-1.22wmf12/extensions/GuidedTour 'E3 deployment: bugfixes for GuidedTour and GettingStarted (1/2)'
* 20:35 logmsgbot: olivneh synchronized php-1.22wmf11/extensions/GettingStarted 'E3 deployment: bugfixes for GuidedTour and GettingStarted (2/2)'
* 20:35 logmsgbot: olivneh synchronized php-1.22wmf11/extensions/GuidedTour 'E3 deployment: bugfixes for GuidedTour and GettingStarted (1/2)'
* 20:29 MaxSem: Ran sync-common on mw1041 - looked outta sync
* 18:51 paravoid: authdns-update: change ns1's IP to new service IP; glues were updated
* 18:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:40 logmsgbot: reedy synchronized wmf-config/
* 18:32 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.22wmf11
* 18:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, testwikidatawiki, loginwiki and mediawikiwiki to 1.22wmf12
* 18:24 logmsgbot: reedy Finished syncing Wikimedia installation... :
* 18:18 Jeff_Green: authdns-update to change IPs for db1008
* 18:06 logmsgbot: reedy Started syncing Wikimedia installation... :
* 17:54 logmsgbot: reedy synchronized php-1.22wmf12/extensions/Wikibase
* 17:44 logmsgbot: reedy synchronized php-1.22wmf11/cache/l10n/
* 17:30 logmsgbot: reedy Started syncing Wikimedia installation... :
* 17:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki back to wmf11
* 17:12 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
* 17:11 logmsgbot: LocalisationUpdate completed (1.22wmf12) at Thu Jul 25 17:10:36 UTC 
* 16:42 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Thu Jul 25 16:40:47 UTC 
* 16:30 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Thu Jul 25 16:29:28 UTC 
* 16:14 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to php-1.22wmf12 and rebuild l10n cache
* 15:55 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to php-1.22wmf12 and rebuild l10n cache
* 15:30 logmsgbot: reedy synchronized docroot and w
* 15:11 logmsgbot: reedy synchronized php-1.22wmf12
* 12:49 hashar: gallium cleared out /etc/update-motd.d/05* which adds the roles in motd. They got out of sync. Rerunning puppet
* 08:25 paravoid: running swift->ceph thumb sync
* 07:58 paravoid: upgrading ceph to 0.67-rc2
* 02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 25 02:47:27 UTC 
* 02:30 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Thu Jul 25 02:29:13 UTC 
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Thu Jul 25 02:15:32 UTC 
* 00:15 Ryan_Lane: upgrading wikitech-static to 1.22wmf11
== July 24 ==
* 23:55 logmsgbot: krinkle synchronized php-1.22wmf11/extensions/VisualEditor 'Idc7b094f8eb2788c48'
* 23:50 logmsgbot: krinkle synchronized php-1.22wmf10/extensions/VisualEditor 'Idc7b094f8eb2788c48'
* 23:15 Ryan_Lane: upgrading wikitech to 1.22wmf11
* 23:14 logmsgbot: maxsem synchronized php-1.22wmf11/extensions/MobileFrontend/
* 23:12 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/
* 22:46 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/
* 22:45 logmsgbot: maxsem synchronized php-1.22wmf11/extensions/MobileFrontend/
* 21:59 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:44 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 21:30 binasher: dropped all databases on s3 that were migrated to s7 [rt5506]
* 21:29 mwalker: updating fundraising civicrm from to 
* 20:21 paravoid: authdns-update: change ns0's IP to new service IP; glues were updated
* 19:04 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor 'Fix copyright notice bug'
* 19:03 logmsgbot: catrope synchronized php-1.22wmf10/extensions/VisualEditor 'Fix copyright notice bug'
* 18:04 logmsgbot: catrope synchronized php-1.22wmf11/resources/mediawiki/mediawiki.notification.js 'Fix mw.notify positioning bug'
* 18:04 logmsgbot: catrope synchronized php-1.22wmf10/resources/mediawiki/mediawiki.notification.js 'Fix mw.notify positioning bug'
* 17:42 logmsgbot: catrope synchronized php-1.22wmf11/extensions/FlaggedRevs 'Fix FlaggedRevs fatal'
* 17:42 logmsgbot: catrope synchronized php-1.22wmf10/extensions/FlaggedRevs 'Fix FlaggedRevs fatal'
* 17:32 logmsgbot: catrope synchronized php-1.22wmf11/extensions/FlaggedRevs 'Fix excessive FlaggedRevs notices'
* 17:31 logmsgbot: catrope synchronized php-1.22wmf10/extensions/FlaggedRevs 'Fix excessive FlaggedRevs notices'
* 17:21 RobH: torrus deadlocked, fixing
* 17:10 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable VisualEditor on 8 more wikis'
* 17:09 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Show either visualeditor-enable or visualeditor-betatempdisable, not both'
* 16:48 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor/modules/ve-mw/dm/annotations/ve.dm.MWNowikiAnnotation.js 'touch'
* 16:47 logmsgbot: catrope synchronized php-1.22wmf10/extensions/VisualEditor/modules/ve-mw/dm/annotations/ve.dm.MWNowikiAnnotation.js 'touch'
* 16:47 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor/modules/ve-mw/dm/annotations/ve.dm.MWExternalLinkAnnotation.js 'touch'
* 16:47 logmsgbot: catrope synchronized php-1.22wmf10/extensions/VisualEditor/modules/ve-mw/dm/annotations/ve.dm.MWExternalLinkAnnotation.js 'touch'
* 16:42 logmsgbot: catrope synchronized php-1.22wmf11/extensions/VisualEditor 'Fix nowiki bug'
* 16:41 logmsgbot: catrope synchronized php-1.22wmf10/extensions/VisualEditor 'Fix nowiki bug'
* 16:32 RoanKattouw: Fixed /etc/dsh/group/bits ... again :(
* 16:28 logmsgbot: catrope Finished syncing Wikimedia installation... : Deploy code ahead of VE roll-out to another 8 wikis
* 16:25 logmsgbot: catrope synchronized php-1.22wmf10/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.ViewPageTarget.js 'touch'
* 16:22 logmsgbot: catrope synchronized php-1.22wmf10/resources/startup.js 'touch'
* 16:04 logmsgbot: catrope Started syncing Wikimedia installation... : Deploy code ahead of VE roll-out to another 8 wikis
* 15:32 logmsgbot: reedy synchronized wmf-config/
* 12:18 logmsgbot: mark synchronized wmf-config/squid.php
* 11:00 MaxSem: Aborted TTM migration, will file a bug report
* 10:25 MaxSem: Attempting to migrate TTM to zinc
* 02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 24 02:26:44 UTC 
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Wed Jul 24 02:15:04 UTC 
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Wed Jul 24 02:08:46 UTC 
* 00:12 ^demon: rebooting formey
* 00:07 ^demon: dist-upgrading formey lol
== July 23 ==
* 23:41 MaxSem: Rebuilding Solr index to catch up new schema
* 23:35 andrewbogott: done messing with sockpuppet, stafford, puppet-merge, etc for now.
* 21:28 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 20:56 mwalker: updating civicrm on aluminium from to 
* 20:50 mwalker: kludging payments data -- adding 7 hours to contribution timestamps between IDs and 
* 20:36 mwalker: paused all audit and QC jenkins jobs on the payments cluster in preparation for fixing timestamps
* 19:55 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
* 19:47 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
* 19:31 Jeff_Green: dist-upgrade & reboot for payments*
* 18:40 logmsgbot: reedy synchronized php-1.22wmf10/extensions/Wikibase
* 18:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:07 logmsgbot: reedy synchronized wmf-config/
* 18:03 logmsgbot: reedy synchronized database lists files:
* 18:01 logmsgbot: reedy synchronized wmf-config/
* 17:59 Reedy: Populated sites table on all wikivoyage wikis
* 17:41 logmsgbot: reedy synchronized wmf-config/
* 17:32 logmsgbot: reedy synchronized wmf-config/
* 17:28 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
* 17:27 logmsgbot: reedy synchronized php-1.22wmf10/extensions/Wikibase
* 17:14 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php ''
* 16:41 Jeff_Green: authdns-update to propagate fundraisingdb-* cname changes
* 15:59 Jeff_Green: stop icinga notifications for db1025 mysql
* 15:58 Jeff_Green: stop icinga notifications for db1008
* 15:22 Jeff_Green: dist-upgrade &reboot db1025
* 15:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 15:14 logmsgbot: reedy synchronized database lists files:
* 15:14 Jeff_Green: dist-upgrade & reboot payments2-4
* 14:58 Jeff_Green: dist-upgrade & reboot payments2
* 14:48 ottomata: stopping puppet on gadolinium, looking into filters causing socat process packet loss
* 14:23 hashar: Jenkins: lanthanum has been added as a Jenkins slave node though NO job should be running there.e
* 10:48 logmsgbot: mark synchronized wmf-config/squid.php 'Add ssl1005/ssl1006'
* 05:45 paravoid: pooling ssl1005/ for https/ipv6
* 05:35 paravoid: authdns-update for cr1-ulsfo <-> cr2-eqiad, dns-auth-lb -> nsN(-new), ssl100x fixed IPv6
* 02:22 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 23 02:21:20 UTC 
* 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Tue Jul 23 02:12:36 UTC 
* 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Tue Jul 23 02:08:03 UTC 
* 01:02 Ryan_Lane: enabled hyperthreading on ssl1005
* 00:40 Ryan_Lane: depooling ssl1005/6 due to ipv6 issues
* 00:40 Ryan_Lane: pooled new https node ssl1006.wikimedia.org (with hyperthreading enabled)
* 00:40 Ryan_Lane: pooled new https node ssl1005.wikimedia.org
== July 22 ==
* 22:32 logmsgbot: reedy synchronized php-1.22wmf11/extensions/UploadWizard
* 22:05 cmjohnson1: restarting ns2
* 22:01 cmjohnson1: dns update
* 21:48 LeslieCarr: restarting powerdns on ns1
* 21:05 logmsgbot: asher synchronized wmf-config/StartProfiler.php 'only randomly profile web reqs in eqiad'
* 20:40 sbernardin: hot swapping disk 13 on db78 storage array
* 20:21 paravoid: deploying bmc-config fix across all pmtpa/eqiad mgmt
* 19:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everyhting else no wikipedia to 1.22wmf11
* 19:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote, wikinews and wikibooks to 1.22wmf11
* 19:30 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, wikivoyage and wiktionary to 1.22wmf11
* 19:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikimedia, private, fishbowl and closed to 1.22wmf11
* 19:26 paravoid: restarting & repooling cp1047/cp1059
* 19:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Move 3 wikis with UW to 1.22wmf11
* 16:04 hashar: Jenkins back up
* 15:53 hashar: jenkins accidentally killed Jenkins :(
* 15:18 hashar: gallium / jenkins : blacklisting a bunch of user agent 
* 14:42 apergos: (btw docs would be nice, is that really the right way to kick it?)
* 14:42 apergos: shot and restarted gitblit: on antinomy, cd /var/lib/gitblit, java -jar gitblit.jar & (see bug )
* 12:47 paravoid: depooling cp1047/cp1059 for further investigation
* 12:29 paravoid: restarting cp1046/cp1060's varnish-frontend
* 11:22 hashar: Jenkins: deleting -merge jobs ( sequentially to avoid deadlocking jenkins 
* 11:20 hashar: jenkins restarted
* 11:02 hashar: restarting Jenkins, some threads are deadlocked ( see )
* 09:27 paravoid: adding new ns0/ns1 service ip static routes to dobson/linne on cr1-sdtpa/cr2-pmtpa
* 07:19 paravoid: restarting pybal and manually ipvsadm removing dns_auth services from lvs1/lvs5/lvs1002/lvs1005
* 02:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 22 02:17:55 UTC 
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Mon Jul 22 02:10:16 UTC 
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Mon Jul 22 02:06:01 UTC 
== July 21 ==
* 20:20 hashar: crashed Jenkins while debugging it :(
* 19:29 hashar: Jenkins deleting remaining -merge jobs (not triggered anymore since that is handled by Zuul now )
* 19:28 hashar: logging bot is dead, no idea for how long though.
== July 20 ==
* 14:05 Nemo_bis: antimony was overloaded (100 % CPU, around 15 load) again, still serving internal errors
* 08:46 netsplit kills morebots
* 07:01 paravoid: apt: including dclass 2.2.2-1
* 02:31 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 20 02:30:42 UTC 
* 02:20 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Sat Jul 20 02:18:54 UTC 
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Sat Jul 20 02:10:26 UTC 
== July 19 ==
* 20:32 logmsgbot: hashar synchronized wikiversions-labs.dat
* 20:31 logmsgbot: hashar synchronized all-labs.dblist
* 20:21 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php 'Slightly increase Parsoid dequeue rate'
* 19:44 sbernardin: shutting down spence for decommissioning
* 19:38 logmsgbot: gwicke synchronized php-1.22wmf11/extensions/Parsoid/php/ParsoidCacheUpdateJob.php 'restore Parsoid to previously deployed newer code'
* 19:38 logmsgbot: gwicke synchronized php-1.22wmf11/extensions/Parsoid/php/Parsoid.php 'restore Parsoid to previously deployed newer code'
* 19:37 logmsgbot: gwicke synchronized php-1.22wmf10/extensions/Parsoid/php/ParsoidCacheUpdateJob.php 'restore Parsoid to previously deployed newer code'
* 19:36 logmsgbot: gwicke synchronized php-1.22wmf10/extensions/Parsoid/php/Parsoid.php 'restore Parsoid to previously deployed newer code'
* 18:56 gwicke: updated Parsoid to 
* 14:51 mark: Upgraded mobile caches to varnish 3.0.3~rc1-wm14
* 14:40 hashar: continuous integration : slightly adjusted main page https://integration.wikimedia.org/ (see integration/docroot.git )
* 10:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable ShortUrl on testwiki'
* 10:12 logmsgbot: reedy synchronized php-1.22wmf11/includes/Setup.php
* 09:47 logmsgbot: reedy synchronized wmf-config/
* 07:56 paravoid: stopping radosgw across the ceph cluster, post-upgrade issues
* 02:51 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 19 02:49:50 UTC 
* 02:35 logmsgbot: LocalisationUpdate completed (1.22wmf11) at Fri Jul 19 02:33:48 UTC 
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Fri Jul 19 02:15:29 UTC 
== July 18 ==
* 23:55 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor and CentralAuth to master
* 23:53 Tim: deploying new php5-fss, to fix segfault bug
* 23:41 springle: restarting db45 slave threads
* 23:39 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor and CentralAuth to master
* 23:18 springle: completed wikidatawiki wb_terms.term_weight OSC
* 22:48 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Second scap attempt for E3 deployment of GettingStarted and GuidedTour
* 22:31 logmsgbot: mflaschen Started syncing Wikimedia installation... : Second scap attempt for E3 deployment of GettingStarted and GuidedTour
* 22:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove bigdelete from sysyop'
* 22:30 Ryan_Lane: dist-upgrading all virt nodes
* 21:57 superm401: Scap failed due to Wikibase issue
* 21:54 Aaron|home: Finally running "mwscriptwikiset maintenance/purgeDeletedFiles.php all.dblist --starttime "
* 20:27 logmsgbot: reedy synchronized php-1.22wmf11/extensions/Wikibase
* 19:59 springle: starting wikidatawiki wb_terms.term_weight OSC
* 19:53 springle: delaying slave db45 for wikidatawiki wb_terms OSC duration
* 19:15 notpeter: adding elasticsearch v. 0.90.2 package to brewster
* 18:21 LeslieCarr: adding new links into cr1/cr2-eqiad asw-c-eqiad bundle
* 18:14 Reedy: Added term weight to wb_terms on testwikidatawiki
* 18:13 Reedy: Created wb_property_info on testwikidatawiki
* 18:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwikidatawiki to 1.22wmf11
* 18:07 Reedy: Ran extensions/UploadWizard/maintenance/migrateCampaigns.php against testwiki
* 18:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, wikidatawiki and loginwiki to 1.22wmf11
* 18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: All wikipedias to 1.22wmf11
* 17:58 cmjohnson1: authdns update
* 17:58 logmsgbot: reedy synchronized wmf-config/
* 16:49 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf11 and build l10n cache
* 16:35 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf11 and build l10n cache
* 16:28 logmsgbot: reedy synchronized docroot and w
* 16:23 logmsgbot: reedy synchronized php-1.22wmf11/ 'Initial code sync'
* 15:55 logmsgbot: csteipp synchronized php-1.22wmf9/extensions/CentralAuth 'Fix SUL issue - wmf9'
* 15:53 logmsgbot: csteipp synchronized php-1.22wmf10/extensions/CentralAuth 'Fix SUL issue'
* 14:33 ^demon: zuul back up too
* 14:32 ^demon: gerrit service back online
* 14:04 cmjohnson1: shutting down manganese
* 14:04 ^demon: stopped gerrit & zuul services on manganese and gallium respectively
* 14:03 cmjohnson1: removing/swapping sdb2 on manganese
* 12:28 mark: Inserted varnish 3.0.3plus~rc1-wm14 packages into the precise-wikimedia APT repository
* 02:44 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 18 02:43:17 UTC 
* 02:29 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Thu Jul 18 02:28:28 UTC 
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Thu Jul 18 02:15:04 UTC 
== July 17 ==
* 22:17 awight: chongo
* 22:02 mwalker: updating payments-listeners from to 
* 20:43 logmsgbot: csteipp synchronized wmf-config/InitialiseSettings.php
* 20:26 logmsgbot: csteipp synchronized php-1.22wmf9/extensions/CentralAuth 'hooks for SUL2 - wmf9'
* 20:23 logmsgbot: csteipp synchronized php-1.22wmf10/extensions/CentralAuth 'hooks for SUL2'
* 20:19 Ryan_Lane: deploying OpenStackManager commit to virt0
* 19:54 Ryan_Lane: clarification (those two hashes for OpenStackManager)
* 19:54 Ryan_Lane: deploying and to virt0
* 18:54 ^d: killed *another* pywikibot/compat clone. Please stop cloning this right now, you're gerrit cry.
* 18:43 awight: payments cluster updated from from to 
* 18:03 ^d: killed a clone of pywikibot/compat. Until I figure out why it's soooo big, please don't kill gerrit trying to clone it
* 17:55 ^d: force running jgit gc on pywikibot/compat
* 17:41 ^d: restarting gerrit
* 16:58 logmsgbot: reedy synchronized wmf-config/
* 16:52 logmsgbot: reedy synchronized wmf-config/
* 16:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Revert live hack of setting testwiki to zh'
* 16:24 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'debugging over'
* 16:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'testwiki wgLanguageCode to zh'
* 16:10 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'debug'
* 15:52 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'debug'
* 14:32 akosiaris: adding bigtop-tomcat package to apt for analytics
* 14:13 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'debug'
* 11:32 paravoid: updating gdnsd in apt to 1.9.0-1~precise1
* 08:41 akosiaris: added oozie and hue packages to apt.wikimedia.org as per analytics team request
* 02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 17 02:47:27 UTC 
* 02:32 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Wed Jul 17 02:31:39 UTC 
* 02:18 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Wed Jul 17 02:17:00 UTC 
* 00:09 logmsgbot: krinkle synchronized php-1.22wmf9/extensions/VisualEditor 'I12f52719ecafd7488bb00419'
* 00:07 logmsgbot: krinkle synchronized php-1.22wmf10/extensions/VisualEditor 'I12f52719ecafd7488bb00419'
== July 16 ==
* 23:38 logmsgbot: krinkle Finished syncing Wikimedia installation... : 'Deploying Ifeea7885d67af6e53bf5c19ccaa231f86c7ead90'
* 23:31 logmsgbot: krinkle Started syncing Wikimedia installation... : 'Deploying Ifeea7885d67af6e53bf5c19ccaa231f86c7ead90'
* 23:15 logmsgbot: bsitu synchronized php-1.22wmf9/extensions/Echo 'Update Echo to master'
* 23:14 logmsgbot: bsitu synchronized php-1.22wmf10/extensions/Echo 'Update Echo to master'
* 21:00 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 20:58 springle: finished rechecking tag_summary enwiki re bug comment 11
* 20:49 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 18:04 springle: rechecking tag_summary enwiki re bug comment 11
* 17:54 Ryan_Lane: deployed LdapAuthentication to virt0 and purged memcache to remove bad memcache entries for users
* 17:10 ^demon: gitblit restarted to grab a bugfix
* 16:59 mwalker: updated payments-listeners from to 
* 16:33 cmjohnson1: that's sdb2 on manganese
* 16:31 cmjohnson1: swapping sdb1 on manganese (software raid1) rt5442
* 15:34 logmsgbot: reedy synchronized php-1.22wmf10/extensions/Wikibase
* 14:54 hashar: zuul / jenkins -merge jobs are no more triggered by Zuul since it detects whether a patch apply cleanly on the tip of the branch. That saves up a bunch of overhead and 
* 12:58 mark: Sending all European mobile traffic to esams
* 12:21 mark: Sending dutch mobile traffic to esams
* 12:06 mark: Created georecord m.wikimedia.org
* 10:30 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php ' now uses ['bits'] '
* 10:29 logmsgbot: hashar synchronized wmf-config/CommonSettings.php ' now uses ['bits'] '
* 10:22 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '$wgCopyrightIcon now vary using $wmfHostnames[bits]'
* 10:21 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '$wgCopyrightIcon now vary using $wmfHostnames[bits]'
* 09:55 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS: Bug '
* 09:53 logmsgbot: nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/ 'ULS to master'
* 09:49 logmsgbot: nikerabbit synchronized php-1.22wmf10/extensions/UniversalLanguageSelector/ 'ULS to master'
* 09:42 ori-l: Resumed reporting of MediaWiki fatals & exceptions to Ganglia
* 09:28 hashar: upgrading packages on gallium
* 09:13 apergos: restarting gerrit
* 02:45 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 16 02:43:55 UTC 
* 02:29 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Tue Jul 16 02:28:30 UTC 
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Tue Jul 16 02:15:23 UTC 
* 01:01 logmsgbot: catrope Finished syncing Wikimedia installation... : Update VisualEditor to master again
* 00:53 logmsgbot: catrope Started syncing Wikimedia installation... : Update VisualEditor to master again
== July 15 ==
* 23:40 cmjohnson1: reinstalling carbon
* 23:06 logmsgbot: csteipp synchronized php-1.22wmf10/extensions/CentralAuth 'Fix sul2 regression'
* 23:03 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php 'Slightly increase Parsoid dequeue rate'
* 22:31 logmsgbot: kaldari synchronized php-1.22wmf10/extensions/WikiLove/WikiLove.hooks.php 'Fixing WikiLove regression on wmf10'
* 21:51 ori-l: All EventLogging services start/running; data looks good.
* 20:47 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable VisualEditor for all users (anons too) on enwiki'
* 20:45 Krinkle: Graceful reload of Zuul to fast-forward deployment to Ia53d412b029205
* 20:41 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master
* 20:30 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
* 20:26 ori-l: Disabled MediaWiki errors Ganglia module on vanadium
* 20:03 ori-l: Rebooting vanadium to complete kernel upgrade to 3.2.0-49.
* 19:49 ori-l: Shutting down EventLogging services on vanadium ahead of Iba8cc5d7b deployment.
* 18:59 hashar: labstore3 system CPU rocketed from ~10% to ~60%
* 18:43 cmjohnson1: powercycling mw1163
* 18:42 cmjohnson1: depooling mw1163 to troubleshoot DIMM error
* 18:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: everything non wikipedia to 1.22wmf10
* 18:23 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wiktionary to 1.22wmf10
* 18:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks, wikivoyage, wikiversity to 1.22wmf10
* 18:17 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia, closed, special to 1.22wmf10
* 17:55 cmjohnson1: powering down mw1173
* 17:36 cmjohnson1: depooling mw1173 to troubleshoot DIMM Failure
* 17:10 RobH: Fixing blog setup of themes, theme may reset to defaults for next few minutes as i tinker with it.
* 15:28 MaxSem: Recreating Solr index
* 12:41 apergos: rebooting kaulen to pick up some upgrades, per rt 
* 02:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 15 02:42:20 UTC 
* 02:29 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Mon Jul 15 02:27:57 UTC 
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Mon Jul 15 02:14:48 UTC 
== July 14 ==
* 18:44 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php 'Throttle titles per template update job further from 10 to 6'
* 17:50 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php 'Re-enable Parsoid updates after throttling template update rate'
* 16:55 logmsgbot: reedy synchronized php-1.22wmf10/extensions/Gadgets/
* 16:54 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Gadgets/
* 14:05 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Resync for boxes that came back up'
* 14:02 mark: Repooled stolen apaches back in the appserver cluster
* 14:00 logmsgbot: catrope synchronized php-1.22wmf9/resources/startup.js 'touch'
* 13:51 mark: Depooled stolen apaches from the api cluster
* 13:21 mark: Upping PyBal weight to 15 for stolen appservers
* 13:20 RoanKattouw: Hand-syncing CommonSettings.php with dsh to set $wgParsoidSkipRatio = 0
* 13:18 mark: Stealing 10 app servers, for the API pool
* 13:06 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Set $wgParsoidSkipRatio to 1 to let the API cluster breathe'
* 12:55 RoanKattouw: Restarted Apache on mw1118, had to stop by hand using killall -9 (/etc/init.d/apache stop didn't work)
* 12:39 RoanKattouw: Restarted pybal on lvs1003
* 12:38 RoanKattouw: Restarted pybal on lvs1006 (set api depool threshold to .4 from .6)
* 06:39 apergos: powercycled mc1005, unreachable via console, from ganglia guess it was in swap death
* 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 14 02:07:10 UTC 
* 02:03 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Sun Jul 14 02:02:31 UTC 
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Sun Jul 14 02:01:43 UTC 
== July 13 ==
* 02:30 mutante: kernel upgrade, rebooting zirconium
* 02:30 mutante: copied etherpad-lite from lucid to precise
* 02:24 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 13 02:22:51 UTC 
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Sat Jul 13 02:13:55 UTC 
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Sat Jul 13 02:07:36 UTC 
* 00:17 mutante: swift install on iron (swift cleaner & utils only) had a depends issue python-swift (= 1.5.0-3) but 1.7.4-0ubuntu2+wmf1 is installed, upgraded anyway with -f
* 00:07 mutante: installing package upgrades on iron
== July 12 ==
* 23:40 gwicke: updated Parsoid to 
* 23:38 springle: restarted db32 slave
* 23:32 springle: finished tag_summary rebuild
* 22:40 logmsgbot: aaron cleared profiling data
* 22:34 mutante: completely stopped and then started squid front and back on cp1001 (had socket already in use errors)
* 22:26 springle: rebuilding tag_summary on enwiki
* 22:25 mutante: restarting squid on cp1001
* 22:07 mutante: restarting job runners
* 21:07 springle: stopped db32 mysql slave threads
* 19:58 logmsgbot: gwicke synchronized php-1.22wmf9/extensions/Parsoid/php/ParsoidCacheUpdateJob.php 'updating Parsoid'
* 19:57 logmsgbot: gwicke synchronized php-1.22wmf9/extensions/Parsoid/php/Parsoid.php 'updating Parsoid'
* 18:45 RobH: authdns-update all ulsfo now 4XXX
* 18:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove wgOldChangeTagsIndex on all remaining wikis'
* 18:00 logmsgbot: gwicke synchronized php-1.22wmf9/extensions/Parsoid/php/ParsoidCacheUpdateJob.php 'updating Parsoid master'
* 17:59 logmsgbot: gwicke synchronized php-1.22wmf9/extensions/Parsoid/php/Parsoid.php 'updating Parsoid master'
* 16:27 Krinkle: Graceful reload of Zuul to forward deployment to I4de1670e11d58e
* 16:10 logmsgbot: reedy synchronized docroot and w
* 15:51 logmsgbot: reedy synchronized docroot and w
* 15:35 logmsgbot: reedy synchronized docroot and w
* 15:33 logmsgbot: reedy synchronized docroot and w
* 15:31 logmsgbot: reedy synchronized docroot and w
* 15:26 logmsgbot: reedy synchronized docroot and w
* 15:20 Krinkle: Resetting commons:User:CommonsDelinker account e-mailaddress (bug )
* 10:43 mark: Sending European wikidata and wikivoyage traffic to esams
* 05:13 mutante: restarting ircecho on neon
* 03:03 mwalker: and again... payments from to 
* 02:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 12 02:47:23 UTC 
* 02:41 mwalker: attempting again on the payments cluster! from to 
* 02:35 logmsgbot: LocalisationUpdate completed (1.22wmf10) at Fri Jul 12 02:34:20 UTC 
* 02:21 mwalker: aaand nope; didn't work, reverting payments back to 
* 02:18 mwalker: trying again -- updating payments from to 
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Fri Jul 12 02:16:00 UTC 
* 01:40 mwalker: aaaahhhhh reverting payments to because banners are sending the wrong gateway (and I was ignoring it before!)
* 01:33 mwalker: updating payments (including the formchooser) from to 
* 00:49 springle: paused updating change_tags indexes on remaining wikis in bug comment 6 (adds done, drops todo)
* 00:11 logmsgbot: maxsem synchronized php-1.22wmf9/extensions/GeoData/solrupdate.php
* 00:09 logmsgbot: maxsem synchronized php-1.22wmf10/extensions/GeoData/solrupdate.php
== July 11 ==
* 23:46 logmsgbot: catrope Finished syncing Wikimedia installation... : Update VE to master
* 23:38 logmsgbot: catrope Started syncing Wikimedia installation... : Update VE to master
* 22:53 springle: updating change_tags indexes on remaining wikis in bug comment 6
* 22:42 logmsgbot: tstarling synchronized wmf-config
* 22:41 logmsgbot: tstarling restarted twemproxy on all servers
* 22:40 logmsgbot: tstarling synchronized wmf-config/mc.php
* 22:37 logmsgbot: tstarling synchronized wmf-config/twemproxy.yaml
* 22:37 logmsgbot: tstarling synchronized wmf-config/twemproxy-pmtpa.yaml
* 22:33 logmsgbot: springle synchronized wmf-config/InitialiseSettings.php 'simplewiki change_tags indexes updated, bug '
* 21:49 logmsgbot: spage synchronized wmf-config/InitialiseSettings.php 'Config changes for VE survey'
* 21:38 RobH: move internal ip, not wiki. bleh
* 21:38 RobH: authdns update for zinc to move internal wiki
* 21:38 Ryan_Lane: deployed squid change to switch test to mw1017
* 21:30 Ryan_Lane: depooling mw1017 to make it the new test.wm.o
* 21:23 logmsgbot: spage Finished syncing Wikimedia installation... : E3 deploy latest GettingStarted and VE gender survey config
* 21:22 RobH: authdns-update for tmc1/2 and mgmt
* 21:12 logmsgbot: spage Started syncing Wikimedia installation... : E3 deploy latest GettingStarted and VE gender survey config
* 20:21 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Wikibase
* 20:21 logmsgbot: reedy synchronized php-1.22wmf10/extensions/Wikibase
* 19:11 LeslieCarr: clearing out stuck icinga processes , except higher than normal icinga spam and do not worry
* 18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: loginwiki testwiki testwikidatawiki mediawikiwki to 1.22wmf10
* 18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: all 'pedias to 1.22wmf9
* 17:22 cmjohnson1: commenting out mc1-16 in node groups
* 17:17 sbernardin: bringing down ms-be5 for hardware replacement
* 16:56 gwicke: updated Parsoid to 
* 15:13 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf10 and rebuild l10n cache
* 15:00 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf10 and rebuild l10n cache
* 14:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.22wmf10 and rebuild l10n cache
* 14:46 logmsgbot: reedy synchronized docroot and w
* 14:45 logmsgbot: reedy synchronized php-1.22wmf10 'initial sync of 1.22wmf10'
* 10:21 Krinkle: Graceful reload of Zuul (forward to I3fc6baba8c6d65)
* 07:39 hashar: gallium : fix permissions for android-common nightly build (need to be jenkins owned, not jenkins-slave) 
* 02:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 11 02:51:16 UTC 
* 02:38 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Thu Jul 11 02:37:40 UTC 
* 02:19 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Thu Jul 11 02:17:47 UTC 
* 00:26 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor 'Update VE to pick up cssText fix'
* 00:26 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Update VE to pick up cssText fix'
== July 10 ==
* 23:39 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor 'Update VE to master'
* 23:38 RoanKattouw: [00:23]	gwicke	!updated Parsoid to 
* 23:38 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Update VE to master'
* 23:30 logmsgbot: csteipp synchronized php-1.22wmf9/extensions/CentralAuth 'Updating CentralAuth on wmf9 for SUL'
* 22:27 mutante: reprepro copying etherpad from wikimedia-lucid to wikimedia-precise
* 21:36 mutante: removing old racktables Apache site and other remnants from hooper
* 21:03 mutante: package upgrades on hooper (etherpad)
* 18:29 logmsgbot: nikerabbit synchronized php-1.22wmf8/extensions/UniversalLanguageSelector/ 'ULS perf fix'
* 18:25 logmsgbot: nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/ 'ULS perf fix'
* 18:10 gwicke: updated Parsoid to 
* 17:58 mutante: reverting removing spence, NTP server issue
* 17:47 mutante: DNS update - killing spence
* 17:21 Jeff_Green: dist-upgrade and reboot aluminium, db78
* 17:21 Jeff_Green: grosley switched to frack puppetmaster
* 17:21 Jeff_Green: db78 switched to frack puppetmaster
* 16:35 mutante: upgrading ruby and other packages on sockpuppet and stafford
* 14:40 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/'
* 14:36 apergos: restarted gerrit, its queue was backed up
* 14:24 hashar: gerrit being super slow, it has 32 items in its queue ( lot of git-upload-pack p/mediawiki/core.git ). Ganglia for manganese does not show anything obvious.
* 14:22 logmsgbot: maxsem synchronized php-1.22wmf8/extensions/GeoData/solrupdate.php
* 14:20 logmsgbot: maxsem synchronized php-1.22wmf9/extensions/GeoData/solrupdate.php
* 11:34 mark: Upgraded and restarted eqiad mobile caches (front/back)
* 09:48 mark: Upgrading Varnish on bits servers
* 09:17 mark: Inserted varnish 3.0.3plus-rc1-wm12 packages into the precise-wikimedia APT repository
* 03:03 logmsgbot: tstarling Finished syncing Wikimedia installation... : ProofreadPage update for bug 
* 02:59 logmsgbot: tstarling Started syncing Wikimedia installation... : ProofreadPage update for bug 
* 02:53 logmsgbot: tstarling synchronized php-1.22wmf9/extensions/ProofreadPage/ProofreadPage.body.php
* 02:43 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 10 02:42:32 UTC 
* 02:39 logmsgbot: tstarling synchronized php-1.22wmf8/maintenance/showJobs.php
* 02:38 logmsgbot: tstarling synchronized php-1.22wmf8/includes/job/JobQueueRedis.php
* 02:37 logmsgbot: tstarling synchronized php-1.22wmf8/includes/job/JobQueue.php
* 02:37 logmsgbot: tstarling synchronized php-1.22wmf8/includes/MappedIterator.php
* 02:30 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Wed Jul 10 02:29:28 UTC 
* 02:22 Tim: on tin: attempted to do a git pull in wmf8 but it updated a whole lot of extensions for some reason. Checking out the old versions manually.
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Wed Jul 10 02:15:40 UTC 
* 01:16 gwicke: finished Parsoid config update with latest dependencies
* 01:14 gwicke: starting Parsoid config update with latest dependencies
== July 9 ==
* 23:17 logmsgbot: mwalker synchronized php-1.22wmf9/extensions/ContributionReporting 'Updating ContributionReporting for fundraiser'
* 22:16 mwalker: updating payments.wm.o from to 
* 22:04 logmsgbot: maxsem synchronized php-1.22wmf9/extensions/MobileFrontend
* 22:03 logmsgbot: maxsem synchronized php-1.22wmf8/extensions/MobileFrontend
* 21:42 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:27 Jeff_Green: db78 switched to mariadb
* 21:21 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 21:13 Robh: authdns-update for bastion5001.mgmt
* 20:13 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'activating Disambiguator extension on all WMF wikis'
* 18:26 LeslieCarr: authdns-update on ns0
* 18:21 logmsgbot: mlitn synchronized wmf-config/ 'Enable ArticleFeedbackv5 auto-archive'
* 18:02 logmsgbot: mlitn synchronized wmf-config/ 'Update ArticleFeedbackv5 & Echo config'
* 17:58 logmsgbot: mlitn synchronized php-1.22wmf9/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 17:57 logmsgbot: mlitn synchronized php-1.22wmf8/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 16:50 logmsgbot: anomie synchronized php-1.22wmf9/extensions/AbuseFilter 'Update AbuseFilter to master for global filter bug fixes'
* 16:39 cmjohnson1: going to tinkering with dysprosium...will see some icinga alerts
* 15:10 ottomata: re-starting puppet on analytics1006,,
* 15:01 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
* 14:40 ottomata: stopping puppet on analytics1006, and to manually apply udp2log puppetization
* 13:20 paravoid: upgraded ceph to 0.66
* 13:04 logmsgbot: hashar synchronized php-1.22wmf9/extensions/Wikibase/lib/resources/wikibase.ui.SiteLinksEditTool.js
* 13:04 hashar: touched and synced extensions/Wikibase/lib/resources/wikibase.ui.SiteLinksEditTool.js on php-1.22wmf8 and php-1.22wmf9
* 13:04 logmsgbot: hashar synchronized php-1.22wmf8/extensions/Wikibase/lib/resources/wikibase.ui.SiteLinksEditTool.js
* 11:48 paravoid: apt: updating to ceph 0.66
* 09:34 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug '
* 09:33 logmsgbot: nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/ 'ULS to master'
* 09:30 logmsgbot: nikerabbit synchronized php-1.22wmf8/extensions/UniversalLanguageSelector/ 'ULS to master'
* 08:56 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS phase 5'
* 08:53 logmsgbot: nikerabbit synchronized php-1.22wmf8/extensions/UniversalLanguageSelector/ 'ULS to master'
* 08:44 logmsgbot: nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/ 'ULS to master'
* 04:17 logmsgbot: tstarling synchronized php-1.22wmf8/includes/api/ApiPurge.php
* 04:16 logmsgbot: tstarling synchronized php-1.22wmf8/includes/WikiPage.php
* 02:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 9 02:38:31 UTC 
* 02:28 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Tue Jul 9 02:26:54 UTC 
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Tue Jul 9 02:14:12 UTC 
* 01:57 logmsgbot: tstarling synchronized php-1.22wmf9/includes/api/ApiPurge.php
* 01:57 logmsgbot: tstarling synchronized php-1.22wmf9/includes/WikiPage.php
* 00:46 gwicke: updated Parsoid to 
== July 8 ==
* 23:56 mutante: graceful'ing Apaches
* 23:53 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VE and AbuseFilter
* 23:43 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VE and AbuseFilter
* 22:48 mutante: restarting pdns on ns0
* 22:47 mutante: DNS update - point download.mw away from empty dir on kaulen
* 21:36 Robh: updating blog
* 19:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 19:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non 'pedia to 1.22wmf9
* 19:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wiktionary to 1.22wmf9
* 18:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikiquote to 1.22wmf9
* 18:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks, wikimania and wikivoyage to 1.22wmf9
* 18:36 logmsgbot: reedy synchronized php-1.22wmf8/extensions/TimedMediaHandler/
* 18:35 logmsgbot: reedy synchronized php-1.22wmf9/extensions/TimedMediaHandler/
* 17:31 mark: Gave row A and row C upload caches equal share of traffic again
* 17:16 logmsgbot: csteipp synchronized php-1.22wmf9/includes/SkinTemplate.php 'Deploy missing function for SUL2'
* 16:44 mutante: installing package upgrades on kaulen (bugzilla)
* 15:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 14:38 paravoid: apt: installing gdnsd 1.8.3-1~precise1
* 12:44 hashar: jenkins: reduced number of executors on master from 2 to 0.
* 10:28 paravoid: (re)installing colby, rubidium, eeden
* 09:09 hashar: srv281 and srv1173 have ssh timeout when synchronizing configuration.
* 09:09 logmsgbot: hashar synchronized images/wikimedia-button.png 'added to git as of '
* 08:36 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'touch config for $wgLoadScript beta/prod variance '
* 08:35 logmsgbot: hashar synchronized wmf-config/CommonSettings.php '$wgLoadScript beta/prod variance '
* 08:33 hashar: Testing out $wgLoadScript beta/prod variance , pulled change and running sync-common on srv193
* 04:44 Tim: on enwiki: discarded refreshLinks2 entries for templates Yesno, If_pagename, Category_handler/numbered, Category_handler, by setting their root job timestamp to a fake new timestamp. See bug 
* 02:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 8 02:15:21 UTC 
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Mon Jul 8 02:09:13 UTC 
== July 7 ==
* 23:29 Tim: on fenari: removed /a/common directory so that mwscript will work
* 23:28 Tim: restarted morebots
* 02:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 7 02:14:42 UTC 
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Sun Jul 7 02:09:01 UTC 
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Sun Jul 7 02:05:09 UTC 
== July 6 ==
* 02:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 6 02:38:04 UTC 
* 02:28 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Sat Jul 6 02:27:28 UTC 
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Sat Jul 6 02:14:24 UTC 
== July 5 ==
* 21:55 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor/ 'Update VE to master in wmf8 (did wmf9 earlier)'
* 21:25 mutante: graceful Apaches, activate "ontology" alias for wikidata
* 21:10 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor/ 'Update VE to master in wmf9'
* 20:56 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/'
* 18:34 sbernardin: bringing down colby to remove extra drives
* 13:54 hashar: 
* 13:48 hashar: Jenkins: reduced # of executors on [master] from 4 to 2. Will bring it to 0 next monday (see )
* 05:13 logmsgbot: tstarling Finished syncing Wikimedia installation... :
* 05:09 logmsgbot: tstarling Started syncing Wikimedia installation... :
* 03:01 logmsgbot: tstarling Finished syncing Wikimedia installation... :
* 02:58 logmsgbot: tstarling Started syncing Wikimedia installation... :
* 02:52 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 5 02:51:29 UTC 
* 02:45 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Fri Jul 5 02:44:41 UTC 
* 02:40 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Fri Jul 5 02:38:54 UTC 
* 02:34 Tim: testing l10nupdate
* 02:04 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jul 5 02:03:10 UTC 
* 02:04 logmsgbot: LocalisationUpdate failed (1.22wmf9) at Fri Jul 5 02:03:09 UTC 
* 02:04 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Fri Jul 5 02:03:09 UTC 
== July 4 ==
* 22:10 LeslieCarr: icinga restarted, was down due to bad check
* 16:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 13:58 pp-pdf1: restarted all services
* 13:57 pp-pdf1: - update mwlib to 0.15.10
* 12:01 akosiaris: uploaded buck version 0+git20130612-0+wmf1 at apt.wikimedia.org
* 09:37 logmsgbot: tstarling Finished syncing Wikimedia installation... :
* 09:34 logmsgbot: tstarling Started syncing Wikimedia installation... :
* 07:07 logmsgbot: tstarling synchronized README
* 02:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jul 4 02:34:13 UTC 
* 02:28 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Thu Jul 4 02:27:40 UTC 
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Thu Jul 4 02:14:32 UTC 
* 01:47 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor 'Fix <a href bug'
* 01:47 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Fix <a href bug'
* 01:15 logmsgbot: catrope synchronized php-1.22wmf9/skins/vector/collapsibleTabs.js 'touch'
* 01:15 logmsgbot: catrope synchronized php-1.22wmf9/skins/vector/vector.js 'touch'
* 01:15 logmsgbot: catrope synchronized php-1.22wmf9/resources/startup.js 'touch'
* 01:14 logmsgbot: catrope synchronized php-1.22wmf9/resources/Resources.php 'collapsibleTabs fix'
* 01:14 logmsgbot: catrope synchronized php-1.22wmf8/skins/vector/collapsibleTabs.js 'touch'
* 01:14 logmsgbot: catrope synchronized php-1.22wmf8/skins/vector/vector.js 'touch'
* 01:13 logmsgbot: catrope synchronized php-1.22wmf8/resources/startup.js 'touch'
* 01:13 logmsgbot: catrope synchronized php-1.22wmf8/resources/Resources.php 'collapsibleTabs fix'
* 00:52 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor 'Latest fixes'
* 00:51 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Latest fixes'
* 00:46 RoanKattouw: Add cp1056, cp1057, cp1069 and cp1070 to /etc/dsh/group/bits , they were missing and so the purge-varnish script didn't actually work
== July 3 ==
* 23:30 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor 'Update VE again'
* 23:30 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Update VE again'
* 23:25 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master
* 23:18 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
* 22:12 ^d: restarting gerrit for firefox workaround
* 20:42 mark: Trippled PyBal weights of row A upload caches to reduce traffic on row C
* 20:38 ottomata: restarting gmetad on nickel to reload an rrd file
* 20:37 hashar: reloading Zuul to lower gerrit debug messages 
* 19:47 logmsgbot: mwalker synchronized php-1.22wmf9/extensions/CentralNotice 'Updating CentralNotice to master on wmf9 (mobile stability and emergency messaging)'
* 19:46 logmsgbot: mwalker synchronized php-1.22wmf8/extensions/CentralNotice 'Updating CentralNotice to master on wmf8 (mobile stability and emergency messaging)'
* 19:45 binasher: migrated ishmael to neon (eqiad)
* 19:45 binasher: pdns hung on ns2, restarted
* 19:41 gwicke: updated Parsoid to 
* 18:07 logmsgbot: yurik synchronized php-1.22wmf9/extensions/ZeroRatedMobileAccess/
* 17:59 logmsgbot: yurik synchronized php-1.22wmf8/extensions/ZeroRatedMobileAccess/
* 17:32 gwicke: updated Parsoid to 
* 17:28 logmsgbot: yurik synchronized php-1.22wmf8/extensions/ZeroRatedMobileAccess/
* 17:25 logmsgbot: yurik synchronized php-1.22wmf9/extensions/ZeroRatedMobileAccess/
* 17:18 Ryan_Lane: deploying change to virt0
* 17:10 RobH: authdns-update rt 
* 16:56 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Wikibase
* 15:39 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c//'
* 15:30 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c//'
* 15:26 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c//'
* 14:13 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Wikibase changes'
* 14:12 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Wikibase changes'
* 14:06 logmsgbot: demon synchronized php-1.22wmf9/extensions/Wikibase/ 'Updating wikibase to '
* 13:40 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Wikibase: I2e9d03b2'
* 13:37 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c//'
* 02:39 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jul 3 02:38:05 UTC 
* 02:29 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Wed Jul 3 02:28:35 UTC 
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Wed Jul 3 02:15:00 UTC 
* 00:02 ^d: manganese back up
* 00:00 ^d: rebooting manganese (gerrit)
== July 2 ==
* 22:47 logmsgbot: bsitu synchronized php-1.22wmf8/extensions/Echo 'update Echo to master'
* 22:46 logmsgbot: bsitu synchronized php-1.22wmf9/extensions/Echo 'update Echo to master'
* 20:57 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 20:46 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 19:41 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo, GettingStarted, PageTriage, Thanks to master
* 19:32 mwalker: updated production civi from to 
* 19:32 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo, GettingStarted, PageTriage, Thanks to master
* 18:08 bblack: upgraded vhtcpd to 0.0.8 on brewster
* 17:48 sbernardin: replacing disks slot5 & slot6 on db45
* 17:38 sbernardin: replacing disk on db44
* 17:29 logmsgbot: csteipp rebuilt wikiversions.cdb and synchronized wikiversions files: bumping loginwiki to wmf9
* 16:30 cmjohnson1: disabling puppet freshness check
* 16:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 16:16 sbernardin: rebooting mw80 for bios update
* 16:11 cmjohnson1: depooling mw80
* 16:10 bblack: updated vhtcpd package on brewster to 0.0.7-1
* 16:09 Jeff_Green: authdns update, switching mx records for donate.wikimedia.org to mchenry/sodium
* 16:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 15:15 paravoid: apt: include jmxtrans 242-1
* 14:59 pp-pdf1: restarted all services
* 14:59 pp-pdf1: - update mwlib to 0.15.9
* 10:18 logmsgbot: maxsem synchronized wmf-config/mobile.php
* 10:10 logmsgbot: maxsem synchronized wmf-config/mobile.php
* 10:03 logmsgbot: maxsem synchronized wmf-config/mobile.php
* 10:01 logmsgbot: maxsem synchronized wmf-config/mobile.php
* 09:27 hashar: jenkins upgraded.
* 09:15 apergos: reinstalling dataset1001
* 09:01 hashar: gallium : upgraded Jenkins from 1.509.1 to 1.509.2 
* 08:46 hashar: gerrit : added Tntnet user to the 'openzim' group. Grants him push/submit rights on the openzim git repository. Requested by Kelson.
* 08:33 logmsgbot: nikerabbit synchronized php-1.22wmf9/extensions/UniversalLanguageSelector/ 'ULS to master'
* 08:17 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'ULS on enwiki part 2'
* 08:16 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS on enwiki part 1'
* 08:11 logmsgbot: nikerabbit synchronized php-1.22wmf8/extensions/UniversalLanguageSelector/ 'ULS to master'
* 05:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 2 05:25:52 UTC 
* 05:22 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Tue Jul 2 05:21:48 UTC 
* 05:15 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Tue Jul 2 05:14:00 UTC 
* 05:06 Tim: testing LU with a real shell (not /bin/false)
* 02:24 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jul 2 02:23:13 UTC 
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Tue Jul 2 02:15:57 UTC 
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Tue Jul 2 02:09:03 UTC 
* 01:22 gwicke: updated Parsoid to 
* 00:56 logmsgbot: ori testing dologmsg. mutante ftw.
* 00:06 mutante: restarted logmsgbot
== July 1 ==
* 23:51 ori-l: logsmsgbot's cidr-filtering was broken by https://gerrit.wikimedia.org/r/#/c//, manifested now because of neon restart earlier
* 22:49 mutante: restarted snmptt and snmptrapd on neon
* 22:43 mutante: deleting files from snmptt spool dir on neon
* 22:18 LeslieCarr: restarting neon for new kernel
* 21:37 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable VisualEditor for all logged-in users on enwiki'
* 21:36 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Plumbing for VE settings'
* 21:34 gwicke: updated Parsoid to 
* 21:24 logmsgbot: catrope Finished syncing Wikimedia installation... : Scap for i18n changes in VE
* 21:14 logmsgbot: catrope Started syncing Wikimedia installation... : Scap for i18n changes in VE
* 21:04 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor/ 'Update VE to master in wmf8 (did wmf9 earlier)'
* 21:02 logmsgbot: catrope synchronized php-1.22wmf8/extensions/TimedMediaHandler/ 'Deploy TMH API cherry-pick to wmf8 (already in wmf9)'
* 20:55 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor 'Updating VE to master in wmf9; for real this time'
* 20:47 logmsgbot: catrope synchronized php-1.22wmf9/extensions/VisualEditor 'Updating VE to master in wmf9'
* 20:36 Krinkle: Zuul was clogging up with ~ queued up events. Added 1 extra executor on Jenkins master and slave.
* 20:11 gwicke: updated Parsoid to 
* 18:02 logmsgbot: csteipp synchronized wmf-config/CommonSettings.php 'enablign silent redirects for test wikis'
* 18:00 logmsgbot: csteipp synchronized wmf-config/InitialiseSettings.php 'pushing config flag for centralauth silent login'
* 17:22 logmsgbot: csteipp synchronized php-1.22wmf9/extensions/CentralAuth 'Updating wmf9 to latest CentralAuth'
* 16:00 cmjohnson1: reseating disk 3 db44
* 14:58 ^demon: gerrit: brought offline, removed /var/lib/gerrit2/review_site/cache/*, brought back up
* 13:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 12:09 mark: Depooled old eqiad mobile cache servers
* 11:25 mark: Depooled old eqiad upload varnish servers
* 10:54 apergos: rebooting labstore3, /srv reporting errors, couldn't xfs_check it
* 10:08 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/Campaigns 'I58a424c70: Restore extensions/Campaigns submodule; was accidentally removed by change I2cafcb595.'
* 03:53 Tim: restarted morebots
* 03:33 TimStarling: testing new scap
== June 29 ==
* 02:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 29 02:11:56 UTC 
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Sat Jun 29 02:06:08 UTC 
* 02:04 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Sat Jun 29 02:03:41 UTC 
* 00:00 jvandavier: jvandavier is the best
== June 28 ==
* 23:59 K4-713: updated payments cluster to 
* 20:55 gwicke: updated Parsoid to 
* 19:20 mutante: attempting clean stop and restart of snmptt service on neon
* 19:08 mutante: deleting /var/spool/snmptt/ files on neon running out of inodes
* 17:57 ^demon: restarting gerrit, bringing back missing documentation lost during upgrade
* 17:33 mutante: upgrading binutils and libldap on bast1001
* 17:20 mutante: installing some pkg upgrades on fenari
* 16:56 paravoid: cause of brief outage: operator error caused restart of varnish on all upload caches eqiad, cascaded into a swift overload, cascaded into imagescaler outage & page
* 16:39 LeslieCarr: restarting snmptrapd and smptt on neon
* 16:28 hashar: created operations/debs/jenkins-debian-glue and force pushed upstream sources from https://github.com/mika/jenkins-debian-glue
* 15:46 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.22wmf9
* 14:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 28 14:05:42 UTC 
* 13:58 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Fri Jun 28 13:56:55 UTC 
* 13:52 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Fri Jun 28 13:51:49 UTC 
* 13:44 logmsgbot: reedy Finished syncing Wikimedia installation... : Scaptastic
* 13:42 logmsgbot: reedy Started syncing Wikimedia installation... : Scaptastic
* 13:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything back from wmf7 to wmf8
* 13:39 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Wikibase
* 13:28 logmsgbot: reedy Finished syncing Wikimedia installation... : Ensure all l10n caches are built and properly built
* 13:22 logmsgbot: reedy Started syncing Wikimedia installation... : Ensure all l10n caches are built and properly built
* 13:21 logmsgbot: reedy synchronized wmf-config/
* 13:16 hashar: jenkins: reducing # of executor on master from 4 to 2
* 12:24 hashar: jenkins: migrating mwext-Wikibase-client-tests and mwext-Wikibase-repo-tests from master to slaves
* 09:57 hashar: gallium : updating some files under /srv/org to belong to jenkins-slave:jenkins-slave instead of jenkins.
* 09:24 hashar: jenkins: made the .erb linting jobs to be voting 
* 06:28 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki to wmf8
* 05:07 Krinkle: Note, anons were eventually getting HTTP too (even for enwiki Main Page) as action=view threw as well squid cache got purged somehow and apaches unable populate it back. And then LVS depooled stuff for apache serving on action=view for main page.
* 05:07 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 're-enabling GeoData'
* 05:05 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
* 05:00 Tim: reverting to 1.22wmf7
* 04:58 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'disable GeoData'
* 04:55 RoanKattouw: sync-dir php-1.22wmf8/cache/l10n
* 04:55 Krinkle: catrope Removing that extension from extension-list
* 04:54 Krinkle: Cause from GeoData "coordinates" magic word. Deployed through broken i18n stuff from WikibaseDataModel deployment earlier that caused fatal during the scap just now
* 04:54 Krinkle: About 20 minutes ago the site went down for logged in users (and anon users hitting an uncached page) throwing HTTP 
* 04:54 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.22wmf8.php
* 04:53 logmsgbot: tstarling synchronized wmf-config/ExtensionMessages-1.22wmf9.php
* 04:51 logmsgbot: catrope Finished syncing Wikimedia installation... : Trying to fix the site
* 04:47 logmsgbot: catrope Started syncing Wikimedia installation... : Trying to fix the site
* 04:47 RoanKattouw: Running scap in an attempt to fix things
* 04:46 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.22wmf8.php
* 04:31 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master
* 04:22 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
* 02:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 28 02:18:38 UTC 
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Fri Jun 28 02:10:26 UTC 
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Fri Jun 28 02:07:05 UTC 
* 01:53 mwalker: updating payments from to 
* 00:57 gwicke: updated Parsoid to 
* 00:00 mutante: graceful Apaches, activate wiikipedia and wekipedia
== June 27 ==
* 22:31 gwicke: updated Parsoid to 
* 20:52 hashar: jenkins: migrating mediawiki-core-phpunit-parser from master to slaves
* 20:44 logmsgbot: spage synchronized php-1.22wmf8/extensions/Campaigns 'updating Campaigns in 1.22wmf8'
* 20:43 hashar: deployment-prep jenkins: migrating mediawiki-core-phpunit-misc from master to slaves
* 20:42 logmsgbot: spage synchronized php-1.22wmf8/extensions/GuidedTour 'updating GuidedTour in 1.22wmf8'
* 20:36 logmsgbot: spage synchronized php-1.22wmf9/extensions/GuidedTour 'updating GuidedTour in 1.22wmf9'
* 20:33 hashar: jenkins: migrating mediawiki-core-phpunit-databaseless form master to slaves
* 20:21 hashar: jenkins: migrating mediawiki-core-phpunit-api from master to slaves
* 18:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add Extension:NewUserMessage to de.wikiversity'
* 18:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:35 Reedy: Created Echo tables on enwikivoyage
* 18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, mediawikiwiki and testwikidatawiki to 1.22wmf9
* 18:03 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Rest of wikipedias to 1.22wmf8
* 18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.22wmf8
* 17:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki back to 1.22wmf9
* 17:38 logmsgbot: reedy synchronized php-1.22wmf8/extensions/Wikibase 'Updating to master of 1.22wmf6 branch'
* 17:37 logmsgbot: reedy synchronized php-1.22wmf7/extensions/Wikibase 'Updating to master of 1.22wmf6 branch'
* 17:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki back to 1.22wmf8
* 17:21 hashar: zuul: removing --force-message from layout and reloading zuul. Caused 
* 17:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 27 17:15:49 UTC 
* 17:13 hashar: Zuul/Jenkins merging bug is 
* 17:12 logmsgbot: LocalisationUpdate completed (1.22wmf9) at Thu Jun 27 17:11:43 UTC 
* 17:11 hashar: Jenkins/Zuul is no more merging changes due to --force-message options no more being available in Gerrit :/
* 17:02 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 27 17:01:20 UTC 
* 17:01 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Thu Jun 27 17:00:37 UTC 
* 16:45 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf9 rebuild l10n cache
* 16:30 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf9 rebuild l10n cache
* 16:28 mark: Corrected IPv6 problem, repooled the new servers, depooled the old bits servers
* 16:19 logmsgbot: reedy synchronized php-1.22wmf9/extensions/WikibaseDataModel
* 16:15 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Wikibase/
* 16:15 logmsgbot: reedy synchronized php-1.22wmf9/extensions/Diff
* 16:14 logmsgbot: reedy synchronized php-1.22wmf9/extensions/DataValues
* 16:12 logmsgbot: reedy synchronized php-1.22wmf9/
* 16:03 mark: Depooled row C bits caches, repooled old bits caches to investigate IPv6 problem
* 15:56 logmsgbot: reedy synchronized docroot and w
* 15:47 mark: Depooled old eqiad bits caches
* 15:41 mark: Pooled new eqiad bits caches
* 14:50 logmsgbot: mark synchronized wmf-config/squid.php
* 14:50 logmsgbot: mark synchronized wmf-config/squid.php
* 14:11 ^demon: gerrit: running puppet and restarting service
* 09:39 mark: Pooled new eqiad upload caches with 1% load
* 03:09 Tim: on fenari: moved away the outdated source copy that was at /home/wikipedia/common . Will update noc.
* 02:56 Tim: added springle to wmf and ops LDAP groups
* 02:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 27 02:19:05 UTC 
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 27 02:13:40 UTC 
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Thu Jun 27 02:07:37 UTC 
* 01:35 gwicke: Roan cleared Parsoid caches
* 01:33 gwicke: updated Parsoid to 
* 00:24 gwicke: updated Parsoid to 
* 00:17 logmsgbot: catrope synchronized php-1.22wmf8/resources/startup.js 'touch'
* 00:17 logmsgbot: catrope synchronized php-1.22wmf7/resources/startup.js 'touch'
* 00:10 logmsgbot: catrope Finished syncing Wikimedia installation... : Scap for VE update, contained i18n changes
* 00:03 logmsgbot: catrope Started syncing Wikimedia installation... : Scap for VE update, contained i18n changes
== June 26 ==
* 23:48 logmsgbot: yurik synchronized php-1.22wmf7/extensions/ZeroRatedMobileAccess/
* 23:47 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Update VE to master'
* 23:47 logmsgbot: catrope synchronized php-1.22wmf7/extensions/VisualEditor 'Update VE to master'
* 23:46 logmsgbot: yurik synchronized php-1.22wmf8/extensions/ZeroRatedMobileAccess/
* 23:22 gwicke: updated Parsoid to 
* 22:22 LeslieCarr: killed slow query on db1025
* 21:15 Jeff_Green: changing fundraising db config to point to service aliases
* 18:37 Jeff_Green: commented out cron-msp queue flush in stat1:/etc/cron.d/sendmail to stop pointless cronspam, host is running exim
* 18:13 logmsgbot: yurik synchronized php-1.22wmf7/extensions/ZeroRatedMobileAccess
* 18:10 logmsgbot: yurik synchronized php-1.22wmf8/extensions/ZeroRatedMobileAccess
* 16:38 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Update VisualEditor to master'
* 16:38 logmsgbot: catrope synchronized php-1.22wmf7/extensions/VisualEditor 'Update VisualEditor to master'
* 16:08 gwicke: updated Parsoid to 
* 14:26 logmsgbot: reedy synchronized php-1.22wmf8/extensions/VisualEditor/modules/ve/ve.js
* 14:24 logmsgbot: reedy synchronized php-1.22wmf7/extensions/VisualEditor/modules/ve/ve.js
* 14:01 logmsgbot: reedy synchronized php-1.22wmf7/resources
* 14:00 logmsgbot: reedy synchronized php-1.22wmf8/resources
* 13:59 logmsgbot: reedy synchronized wmf-config 'touch 1'
* 12:08 hashar: Jenkins let jobs roam on gallium slave node and keep master only for tied jobs.
* 10:39 hashar: jenkins moving unit test jobs for mw extensions to use the slave tmpfs and let them roam on slaves
* 08:42 MaxSem: recreating Solr index
* 08:40 hashar: jenkins installed Simple Theme Plugin
* 08:35 hashar: jenkins reducing number of executors on (master) from 10 to 4. Most jobs are now running on the 'gallium' slave node.
* 08:35 hashar: jenkins: restarted it twice to finally have it up and running.
* 08:15 hashar: restarting Jenkins to apply plugins updates.
* 07:13 Tim: deploying squid config change to block API DoS attack
* 06:41 paravoid: upgrading ceph/ms-fe.eqiad.wmnet to 0.65, ignore warnings/alerts for the next few hours
* 03:59 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Updating VisualEditor to master'
* 03:58 logmsgbot: catrope synchronized php-1.22wmf7/extensions/VisualEditor 'Updating VisualEditor to master'
* 02:57 RoanKattouw: Updated Parsoid config for bug 
* 02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 26 02:27:09 UTC 
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Wed Jun 26 02:13:32 UTC 
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Wed Jun 26 02:07:35 UTC 
* 01:41 logmsgbot: catrope synchronized php-1.22wmf8/skins/vector/collapsibleTabs.js 'I6e277a9503a1e6003bc7bf9f9468ed7b35552e60'
== June 25 ==
* 23:26 logmsgbot: catrope synchronized php-1.22wmf8/extensions/VisualEditor 'Updating VisualEditor to master'
* 23:25 logmsgbot: catrope synchronized php-1.22wmf7/extensions/VisualEditor 'Updating VisualEditor to master'
* 21:47 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving pool4 and 5 search traffic back to eqiad for realz'
* 21:34 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving pool2 and 3 search traffic back to eqiad for realz'
* 21:28 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:25 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving en and prefix search traffic back to eqiad for realz'
* 21:19 LeslieCarr: restarted pybal on lvs1003
* 21:16 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 19:34 Ryan_Lane: test
* 19:32 Ryan_Lane: test
* 19:26 logmsgbot: reedy synchronized .
* 19:26 Ryan_Lane: upgrading adminbot to 1.7.4 on wikitech-static
* 19:04 logmsgbot: reedy synchronized docroot and w
* 18:34 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving en and prefix search traffic back to pmtpa'
* 18:27 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving en and prefix search traffic back to eqiad'
* 18:14 logmsgbot: reedy synchronized php-1.22wmf8/extensions/UploadWizard
* 17:19 logmsgbot: aaron cleared profiling data
* 16:57 logmsgbot: reedy synchronized wmf-config/
* 15:25 hashar: jenkins: raising # of executors on 'gallium' slave node from 4 to 8
* 15:03 gwicke: updated Parsoid to 5ef05e6c9
* 14:54 paravoid: deluser icinga; rm -rf /home/icinga on all hosts but neon
* 14:45 paravoid: restarting nrpe on all hosts
* 14:25 logmsgbot: reedy synchronized php-1.22wmf8/extensions/ProofreadPage/
* 12:37 mark: Pooled the new eqiad mobile frontend caches with lowest weight in PyBal
* 12:27 hashar: jenkins: all mediawiki extension jobs got updated.
* 12:12 logmsgbot: mark synchronized wmf-config/squid.php
* 11:45 mark: Depooled old parsoid cache servers - parsoidcache migration complete
* 11:17 hashar: jenkins: migrating extensions jobs to use gallium slave ( lint , phpcs-HEAD ) or be bound on master (test extensions)
* 10:56 mark: Pooled cp1045 and cp1058 Varnish frontends in the parsoid cache pool
* 08:13 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS phase3'
* 02:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 25 02:26:25 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Tue Jun 25 02:13:22 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Tue Jun 25 02:07:28 UTC 2013
* 01:51 logmsgbot: krinkle synchronized php-1.22wmf7/extensions/VisualEditor/ ''
* 01:49 logmsgbot: krinkle synchronized php-1.22wmf8/extensions/VisualEditor/ ''
* 01:23 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings-labs.php 'Iac6a3f2f4d: wmgMFLogEvents => true on beta labs'
== June 24 ==
* 23:49 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'I82fe6063a: Disable AFTv4 (Bug 43892)'
* 23:06 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/GettingStarted 'I329f59929: Don't log when schemaAction is falsy. (Bug: 50065)'
* 23:06 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/GettingStarted 'I329f59929: Don't log when schemaAction is falsy. (Bug: 50065)'
* 22:44 Krinkle: Graceful reload of Zuul to deploy 
* 20:54 mutante: DNS update - switching RT over to magnesium and 4.x
* 19:53 logmsgbot: reedy synchronized php-1.22wmf8/extensions/ProofreadPage/
* 19:47 gwicke: updated Parsoid to 6ab78c8dd8a
* 19:46 mutante: moving backups from db9 to tridge
* 19:39 mutante: temp shutting down webserver on streber (RT scheduled downtime)
* 18:22 gwicke: updated Parsoid to b3a872fa7
* 18:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiooks to 1.22wmf8
* 18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote, wikiversity, wikinews, wiktionary to 1.22wmf8
* 18:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage and wiktionary to 1.22wmf8
* 18:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikimedia and closed to 1.22wmf8
* 17:09 apergos: er, "shot" many dead snmptt processes
* 17:09 apergos: while kicking icinga into working again on neon, also cleared out a pile of inodes in /var/spool/snmptt again, show many many dead processes too
* 16:55 logmsgbot: krinkle synchronized wmf-config/CommonSettings.php 'I94d74fd8abeba465f'
* 16:55 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'I94d74fd8abeba465f'
* 16:48 logmsgbot: krinkle synchronized php-1.22wmf7/extensions/VisualEditor/ 'I73522f63d9a49a4b7'
* 16:47 logmsgbot: krinkle synchronized php-1.22wmf8/extensions/VisualEditor/ 'I73522f63d9a49a4b7'
* 15:54 logmsgbot: reedy synchronized wmf-config/
* 15:50 ^demon: restarting gerrit
* 15:49 Reedy: Created WikiLove tables on enwikivoyage
* 14:51 hashar: jenkins: migrating pyflakes and pep8 jobs to slaves having the label 'hasContintPackages'
* 14:09 hashar: jenkins raising 'gallium' slave # of executors from 1 to 4. I have migrated some jobs to run on that slave.
* 02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 24 02:24:40 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Mon Jun 24 02:12:57 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Mon Jun 24 02:07:08 UTC 2013
== June 23 ==
* 07:06 logmsgbot: krinkle Finished syncing Wikimedia installation... : Update VisualEditor I7f875f48ce2d
* 06:39 logmsgbot: krinkle Started syncing Wikimedia installation... : Update VisualEditor I7f875f48ce2d
* 06:32 RoanKattouw: Made ExtensionMessages-1.22wmf8.php group-writeable
* 06:28 Krinkle: Aborted scap due to php fatal errors with 1.22wmf8 (Permission error on ExtensionMessages-1.22wmf8.php; Missing files: Narayam.i18n.php, WebFonts.i18n.php and WebFonts.alias.php) 
* 06:27 logmsgbot: krinkle Started syncing Wikimedia installation... : Update VisualEditor I7f875f48ce2d
* 02:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 23 02:16:43 UTC 2013
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Sun Jun 23 02:09:09 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Sun Jun 23 02:05:29 UTC 2013
== June 22 ==
* 15:46 Reedy: tin is down, dead to SSH
* 02:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 22 02:19:26 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Sat Jun 22 02:13:00 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Sat Jun 22 02:07:29 UTC 2013
== June 21 ==
* 22:47 binasher: created a wb_entity_per_page view in labsdb wikidatawiki_p
* 21:24 mutante: deleting /home/daniel on fenari (confirmed not needed)
* 20:31 mutante: installing upgrades on bast1001
* 20:27 mutante: cleaning up /home/daniel/ (the other Daniel) ssh keys on fenari and letting puppet recreate them
* 19:31 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js 'Gerrit change I3db3cf47b / bug 49967'
* 19:30 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js 'Gerrit change I3db3cf47b / bug 49967'
* 18:53 gwicke: updated Parsoid to 21e1e7d
* 18:09 mutante: installing package upgrades on kaulen (bugzilla)
* 18:03 gwicke: updated Parsoid to 37cd852f
* 14:38 hashar: jenkins: updating all phpcs-HEAD jobs to rely on git-changed-in-head script 
* 10:38 hashar: upgrading packages on gallium
* 06:09 gwicke: updated Parsoid to 90515ab5
* 02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 21 02:24:03 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Fri Jun 21 02:13:50 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Fri Jun 21 02:07:39 UTC 2013
* 00:05 gwicke: updated Parsoid to 393a3263
== June 20 ==
* 23:30 gwicke: updated Parsoid to 6dadde3
* 22:58 gwicke: updated Parsoid to 6240c19
* 22:06 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/GuidedTour 'Updating GuidedTour to 950ee2c70417be517f4cdce3a1b590f6fc28d388'
* 22:02 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/GuidedTour 'Updating GuidedTour to 950ee2c70417be517f4cdce3a1b590f6fc28d388'
* 21:01 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'I24bbe0d4e: Update EventLogging config to utilize API'
* 20:52 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/GettingStarted 'GettingStarted to bf05656766'
* 20:52 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/GuidedTour 'GuidedTour to a6fdf3c910'
* 20:51 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/GettingStarted 'GettingStarted to bf05656766'
* 20:51 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/GuidedTour 'GuidedTour to a6fdf3c910'
* 20:28 Krinkle: Gracefully reloading Zuul to deploy Ie7513f356cd8
* 20:24 logmsgbot: reedy synchronized php-1.22wmf8/extensions/ProofreadPage/
* 20:24 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/EventLogging 'EventLogging to 2351b4ccbb'
* 20:21 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/EventLogging 'EventLogging to 2351b4ccbb'
* 20:20 Krinkle: Gracefully reloading Zuul to deploy I8c0ac58d9498979b
* 20:11 logmsgbot: olivneh synchronized php-1.22wmf8/extensions/CoreEvents 'CoreEvents to d291b64248'
* 20:11 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/CoreEvents 'CoreEvents to d291b64248'
* 20:07 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings-labs.php 'I83fcaa5ec: enable NavigationTiming on beta cluster'
* 19:49 logmsgbot: maxsem synchronized php-1.22wmf7/extensions/MobileFrontend
* 19:47 logmsgbot: maxsem synchronized php-1.22wmf8/extensions/MobileFrontend
* 19:23 logmsgbot: reedy synchronized wmf-config/
* 18:06 gwicke: updated Parsoid to b206b54
* 18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rest of wikipedias to 1.22wmf7
* 18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki, test2wiki and testwikidatawiki to 1.22wmf8
* 17:58 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 20 17:57:01 UTC 2013
* 17:55 logmsgbot: reedy synchronized wmf-config/
* 17:52 logmsgbot: LocalisationUpdate completed (1.22wmf8) at Thu Jun 20 17:51:50 UTC 2013
* 17:41 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 20 17:40:46 UTC 2013
* 17:41 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 20 17:39:59 UTC 2013
* 17:38 logmsgbot: reedy synchronized php-1.22wmf8/extensions/ 'Sync Campaigns and CentralAuth'
* 17:34 logmsgbot: reedy Finished syncing Wikimedia installation... : rebuild localisation cache and testwiki to 1.22wmf8
* 17:10 logmsgbot: reedy Started syncing Wikimedia installation... : rebuild localisation cache and testwiki to 1.22wmf8
* 16:56 logmsgbot: reedy synchronized docroot and w
* 16:54 logmsgbot: reedy synchronized php-1.22wmf8/ 'initial sync'
* 16:09 Jeff_Green: nuked neon puppet.log b/c /var/log was 99% full
* 15:55 gwicke: updated Parsoid to bf8d3df
* 14:17 hashar: jenkins updating all mediawiki extensions unit testing jobs ( mwext-.*-testextensions-master'
* 13:28 cmjohnson1: dns update
* 02:23 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 20 02:22:16 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 20 02:12:59 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 20 02:07:13 UTC 2013
* 00:13 gwicke: updated Parsoid to 45944a0
== June 19 ==
* 21:06 RobH: rebooting tridge due to updates
* 20:57 Ryan_Lane: updating OpenStackManager on virt0 to 3e92211313ed8b73ee44fb395230bafb14dc9bdc
* 20:34 ottomata: analytics1019 and analytics1020 are fully puppetized. rebalancing them back into the hadoop cluster
* 18:42 logmsgbot: nikerabbit synchronized php-1.22wmf6/extensions/UniversalLanguageSelector/ 'ULS to master again'
* 18:35 gwicke: updated Parsoid to ddc47d8
* 18:35 logmsgbot: nikerabbit synchronized php-1.22wmf7/extensions/UniversalLanguageSelector/ 'ULS to master'
* 18:32 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'pull db64'
* 18:29 mutante: sync-apache and graceful for votewiki rewrites (change 67203)
* 18:15 logmsgbot: nikerabbit synchronized php-1.22wmf6/extensions/UniversalLanguageSelector/ 'ULS to master'
* 15:02 cmjohnson1: authdns update
* 14:58 cmjohnson1: powering down and relocating searchidx1001
* 14:55 Jeff_Green: update & reboot boron
* 14:36 cmjohnson1: powering down mw1171 to replace hard drive and reinstalling
* 13:52 cmjohnson1: powering down carbon to replace bad disk
* 10:01 logmsgbot: hashar synchronized wmf-config ' beta: $wmg for AbuseFilterCentralDB'
* 10:00 logmsgbot: hashar synchronized wmf-config/CommonSettings.php ' beta: $wmg for AbuseFilterCentralDB'
* 02:19 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 19 02:18:52 UTC 2013
* 02:16 logmsgbot: catrope synchronized php-1.22wmf7/resources/startup.js 'touch'
* 02:16 logmsgbot: catrope synchronized php-1.22wmf6/resources/startup.js 'touch'
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Wed Jun 19 02:14:07 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Wed Jun 19 02:07:42 UTC 2013
* 02:05 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master
* 01:50 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
* 00:57 gwicke: updated Parsoid to 0445108
== June 18 ==
* 22:52 logmsgbot: olivneh synchronized wmf-config/CommonSettings-labs.php 'Iaade2591b: Configure $wgEventLoggingBaseUri & $wgEventLoggingFile on beta cluster'
* 22:31 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings-labs.php 'Ibeb07a4b9: Re-enable EventLogging on beta cluster'
* 22:30 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'I7fc1a5ad4: Ensure forward-compatibility by setting '
* 22:16 mutante: scheduled 24 hour downtime for eqiad search-pools to stop paging
* 22:04 mutante: rebooting iron to finish upgrade
* 22:04 logmsgbot: maxsem synchronized php-1.22wmf6/extensions/MobileFrontend/
* 21:55 logmsgbot: maxsem synchronized php-1.22wmf6/extensions/MobileFrontend/
* 21:50 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:39 mutante: apache-graceful-all for redirects
* 21:37 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 20:41 LeslieCarr: added temp logging filter to cr2-eqiad analytics4 filter
* 20:34 mutante: upgrading iron to precise
* 19:44 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Turn Disambiguator on on en.wiki'
* 19:17 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 18:58 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 18:50 mutante: rebooting iron
* 18:44 mutante: installing kernel and package upgrades on iron
* 18:33 mwalker: updating payments cluster (listeners) from aa0039104e71417845ec14a5ad666c2de464cfab to 8ca174296d570b6fda365cab37b2222d81c47c3d for adyen USD capture bug
* 18:12 greg-g: it's better than bad, it's good!
* 18:09 RobH: log log log log
* 18:09 mutante: LVS monitoring pages caused by neon recovering
* 18:08 RobH: ignore eqiad based search errors for now, as tampa is hosting all search traffic while hardware is repaired
* 18:07 RobH: search is based in tampa
* 17:40 cmjohnson1: dns update
* 17:28 cmjohnson1: wtp1008 is going down for main board replacement
* 17:09 mutante: reloading squid frontends in squid_esams_text and eqiad.text
* 17:07 mutante: reloading squid frontends in pmtpa.text
* 14:01 cmjohnson1: powering down search1001-1024 to relocate to row c5
* 08:26 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'ULS phase 2'
* 08:16 logmsgbot: nikerabbit synchronized php-1.22wmf6/extensions/UniversalLanguageSelector/ 'ULS to master'
* 08:14 logmsgbot: nikerabbit synchronized php-1.22wmf7/extensions/UniversalLanguageSelector/ 'ULS to master'
* 07:19 ori-l: Force-confirmed e-mail for 160 accounts. full list in fluorine:/home/olivneh/users-forced-confirm-email-18-Jun-2013.list
* 07:12 ori-l: Marking e-mail addresses of accounts created in last 24h with no password as "confirmed" on all wikis on which VE is enabled.
* 04:13 ori-l: Manually marked the e-mail address of 54 enwiki users with e-mail address configured but no password (due to bug 49727) as confirmed.
* 03:41 logmsgbot: mflaschen synchronized php-1.22wmf7/extensions/VisualEditor/VisualEditor.hooks.php 'Hotfix to fix account creation'
* 03:15 logmsgbot: mflaschen synchronized php-1.22wmf6/extensions/VisualEditor/VisualEditor.hooks.php 'Emergency hotfix to fix account creation'
* 03:10 logmsgbot: mflaschen synchronized php-1.22wmf6/extensions/VisualEditor/VisualEditor.php 'Emergency hotfix to fix account creation'
* 03:10 superm401: Started hotfix for account creation on wmf6.
* 02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 18 02:25:42 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Tue Jun 18 02:13:24 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Tue Jun 18 02:07:31 UTC 2013
* 01:04 Krinkle: Gracefully reloading Zuul to deploy 
* 00:51 mutante: reloading squid config on pmtpa.txt
* 00:48 mutante: reloading squid config on esams text squids
* 00:40 mwalker: updating the payments cluster (listeners) from 5f67ceddd9d495cffd22f6b303443aba58041d47 to aa0039104e71417845ec14a5ad666c2de464cfab for additional smashpig content
* 00:11 mutante: touch /etc/exim4/defer_domains on streber
== June 17 ==
* 23:52 mutante: restarting exim on streber
* 23:52 logmsgbot: catrope Finished syncing Wikimedia installation... : Updating VisualEditor to master
* 23:40 logmsgbot: catrope Started syncing Wikimedia installation... : Updating VisualEditor to master
* 23:36 ^demon: restarting gerrit
* 23:31 RoanKattouw: Running scap to update VisualEditor
* 23:11 mutante: install package upgrades on streber
* 23:06 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Change I95a8785c7 / Bug 49312: Add 'Programs' NS to MetaWiki'
* 23:05 Krinkle: Gracefully reloading Zuul to deploy 
* 22:22 gwicke: updated Parsoid to b2a7b101e
* 22:20 mutante: built package and upgrading to php-luasandbox 1.7-1
* 19:38 Jeff_Green: dist-upgrade and reboot indium
* 19:29 gwicke: updated Parsoid to d1ae92720c4e
* 18:50 hashar: upgrading packages on gallium
* 18:41 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'bookkeeping changes for pmtpa dbs'
* 18:16 logmsgbot: aaron synchronized php-1.22wmf7/includes/db/DatabaseMysql.php 'ac676a2d83af283701dee111a28a52ea1730d657'
* 18:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: everything non 'pedia to 1.22wmf7
* 17:15 notpeter: rebooting es7 for lvm snapshot cleanup (again)
* 17:05 apergos: neon should be back in business
* 16:41 apergos: neon out of inodes again, clearing out /var/spool/snmpttp, will return to service shortly
* 13:11 apergos: need to find out why sfo-intranet.corp.wikimedia.org can't be resolved in order to have opendj back and restore mail service
* 12:40 apergos: restarting opendj on sanger
* 10:50 hashar: manually rebased mediawiki-config on fenari 7d10d3f --> 1d2093e
* 02:13 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 17 02:12:33 UTC 2013
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Mon Jun 17 02:06:18 UTC 2013
* 02:04 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Mon Jun 17 02:03:45 UTC 2013
== June 16 ==
* 23:36 paravoid: restarting pybal on lvs1003/lvs1006 for ms-fe4.eqiad.wmnet ip change
* 02:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 16 02:13:03 UTC 2013
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Sun Jun 16 02:06:29 UTC 2013
* 02:05 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Sun Jun 16 02:03:54 UTC 2013
== June 15 ==
* 08:38 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Mobile host for Commons'
* 02:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 15 02:19:50 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Sat Jun 15 02:13:57 UTC 2013
* 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Sat Jun 15 02:08:17 UTC 2013
* 02:06 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
* 00:58 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
* 00:05 mutante: reloading squid config on cp1001-cp1020 (text eqiad) for RT-5296
== June 14 ==
* 23:25 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
* 22:27 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
* 22:26 logmsgbot: reedy synchronized php-1.22wmf7/extensions/SecurePoll/
* 22:06 logmsgbot: reedy synchronized php-1.22wmf7/extensions/SecurePoll/
* 21:57 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
* 21:56 logmsgbot: py synchronized wmf-config/db-eqiad.php 'repooling es1007 and es1010'
* 21:56 logmsgbot: reedy synchronized php-1.22wmf7/extensions/SecurePoll/
* 21:41 notpeter: reboot es1010 to deal with messed up lvm snapshot
* 21:36 logmsgbot: mholmquist Finished syncing Wikimedia installation... : Fixing UploadWizard in Chrome
* 21:33 notpeter: reboot es1007 to deal with messed up lvm snapshot
* 21:31 logmsgbot: py synchronized wmf-config/db-eqiad.php 'temp depooling es1007 and es1010 for reboots'
* 21:28 logmsgbot: mholmquist Started syncing Wikimedia installation... : Fixing UploadWizard in Chrome
* 21:03 notpeter: rebooting es10 to deal with fucked up snapshot volume
* 19:33 mutante: deploying small changes to bugzilla login screen
* 19:30 notpeter: installin updates on es7 while I'm at it
* 19:28 notpeter: rebooting es7 to deal with messed up vlm snapshot
* 18:46 gwicke: updated Parsoid config and code back to latest after finding and fixing Varnish config bug
* 18:31 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Re-enable VisualEditor on the rest of the wikis'
* 18:01 logmsgbot: aaron synchronized php-1.22wmf7/maintenance/purgeDeletedFiles.php '9e2ffededde6e7752a8bd64d2ae791af768213c0'
* 17:32 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Re-enable VE on mw.org'
* 15:44 cmjohnson1: powering down wtp1008 to swap cpu's
* 15:26 cmjohnson1: carbon going offline to run h/w test at Dell support request.
* 15:15 gwicke: rolled back Parsoid dependency update to investigate issues
* 15:14 cmjohnson1: working mw1041 updating bios
* 14:54 logmsgbot: ariel synchronized wmf-config/InitialiseSettings.php 'emergency disabling of Visual Editor, see bug 49577'
* 04:46 gwicke: updated Parsoid to 64921430b1
* 04:33 gwicke: updating Parsoid dependencies from config repository
* 02:20 cmjohnson1: dns update
* 02:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 14 02:17:44 UTC 2013
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Fri Jun 14 02:11:02 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Fri Jun 14 02:05:57 UTC 2013
* 01:25 paravoid: starting swift->ceph scripts on a screen on ms-fe1002
* 00:29 logmsgbot: catrope Finished syncing Wikimedia installation... : Fixup for VE deployment
* 00:20 logmsgbot: catrope Started syncing Wikimedia installation... : Fixup for VE deployment
* 00:10 RoanKattouw: Scapping to fix up VE deployment
== June 13 ==
* 23:59 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'I3728a4cd9'
* 23:42 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Enabling Campaigns & CoreEvents (4/4)'
* 23:41 logmsgbot: olivneh synchronized wmf-config/extension-list 'Enabling Campaigns & CoreEvents (3/4)'
* 23:41 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling Campaigns & CoreEvents (1/4)'
* 23:40 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings-labs.php 'Enabling Campaigns & CoreEvents (1/4)'
* 23:37 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/CoreEvents 'Adding CoreEvents extension to wmf7'
* 23:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 13 23:35:36 UTC 2013
* 23:31 logmsgbot: olivneh synchronized php-1.22wmf6/extensions/Campaigns 'Adding Campaigns extension to wmf6'
* 23:31 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 13 23:30:06 UTC 2013
* 23:30 logmsgbot: olivneh synchronized php-1.22wmf6/extensions/CoreEvents 'Adding CoreEvents extension to wmf6'
* 23:21 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 13 23:20:43 UTC 2013
* 23:15 logmsgbot: mholmquist synchronized php-1.22wmf6/extensions/UploadWizard/resources/mw.LanguageUpWiz.js 'Opera UploadWizard fix'
* 23:14 logmsgbot: mholmquist synchronized php-1.22wmf6/extensions/UploadWizard/includes/specials/SpecialUploadWizard.php 'Opera UploadWizard fix'
* 23:08 logmsgbot: anomie synchronized php-1.22wmf7/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'Update LocalisationUpdate to master in wmf7'
* 23:07 logmsgbot: anomie synchronized php-1.22wmf7/extensions/LocalisationUpdate/LocalisationUpdate.php 'Update LocalisationUpdate to master in wmf7'
* 22:57 logmsgbot: catrope synchronized php-1.22wmf6/resources/startup.js 'touch'
* 22:52 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Enable EventLogging for VE on enwiki'
* 22:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 13 22:49:02 UTC 2013
* 22:48 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable EventLogging for VE on enwiki'
* 22:43 logmsgbot: LocalisationUpdate completed (1.22wmf7) at Thu Jun 13 22:42:42 UTC 2013
* 22:40 logmsgbot: catrope Finished syncing Wikimedia installation... : VisualEditor update to master, and adding Campaigns in wmf7
* 22:39 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 13 22:38:14 UTC 2013
* 22:34 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 13 22:33:09 UTC 2013
* 22:31 logmsgbot: catrope Started syncing Wikimedia installation... : VisualEditor update to master, and adding Campaigns in wmf7
* 22:18 RoanKattouw: Fixing perms on php-1.22wmf7, not group-writable
* 22:10 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
* 22:08 LeslieCarr: restarted squid on cp1004 due to "Jun 13 21:40:34 cp1004 squid[1961]: commBind: Cannot bind socket FD 23 to *:0: (98) Address already in use"
* 22:04 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
* 21:55 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
* 21:15 logmsgbot: spage synchronized php-1.22wmf6/extensions/GuidedTour 'latest GuidedTour'
* 21:13 logmsgbot: spage synchronized php-1.22wmf6/extensions/EventLogging 'latest EventLogging'
* 20:51 LeslieCarr: mw1173 has bad memory, rt 5294
* 20:48 logmsgbot: spage synchronized php-1.22wmf7/extensions/GuidedTour 'latest GuidedTour'
* 20:48 LeslieCarr: repuppetizing mw1020 after a reinstall
* 20:41 Jeff_Green: authdns update
* 20:41 logmsgbot: spage synchronized php-1.22wmf7/extensions/EventLogging 'latest EventLogging'
* 20:19 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/UploadWizard/resources/mw.LanguageUpWiz.js 'Ia09ace5dc: Fix UploadWizard's config variables (2/2)'
* 20:18 logmsgbot: olivneh synchronized php-1.22wmf7/extensions/UploadWizard/includes/specials/SpecialUploadWizard.php 'Ia09ace5dc: Fix UploadWizard's config variables'
* 20:16 logmsgbot: kaldari synchronized php-1.22wmf6/extensions/Thanks/modules/ext.thanks.thank.js 'fix IE Thanks bug'
* 20:16 logmsgbot: reedy synchronized php-1.22wmf5/cache/
* 20:07 gwicke: updated Parsoid to a33980cf
* 19:44 binasher: rebooted mc1009 as a firedrill, chaos monkey ahoy!
* 19:36 cmjohnson1: dns update
* 19:06 logmsgbot: kaldari synchronized php-1.22wmf7/extensions/Thanks/modules/ext.thanks.thank.js 'fix IE Thanks bug'
* 18:59 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving last of search traffic to pmtpa'
* 18:50 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving more search traffic to pmtpa'
* 18:33 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving search traffic to pmtpa for en, de, fr, and ja'
* 18:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki, testwikidatawiki and mediawikiwiki to 1.22wmf7
* 18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikipedia to 1.22wmf6
* 16:29 logmsgbot: reedy Finished syncing Wikimedia installation... : testwiki to 1.22wmf7 and build l10n cache
* 16:12 logmsgbot: reedy Started syncing Wikimedia installation... : testwiki to 1.22wmf7 and build l10n cache
* 16:04 logmsgbot: reedy synchronized w
* 16:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 15:46 logmsgbot: reedy synchronized docroot
* 15:45 logmsgbot: reedy synchronized php-1.22wmf7 'Initial file sync'
* 07:33 apergos: the last of the c2100s in the swift pool has been replaced (and new rings pushed), finally. now we "only" have the H310 controllers to replace...
* 02:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 13 02:28:27 UTC 2013
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Thu Jun 13 02:14:58 UTC 2013
* 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Thu Jun 13 02:08:12 UTC 2013
* 00:29 logmsgbot: asher synchronized wmf-config/mc-eqiad.php 'disable persistent conns to twmprxy'
* 00:24 paravoid: ceph readding osd.81, osd.125, osd.137, osd.141 after disk replacement (#5202, #5228, #5248, #5263)
* 00:22 logmsgbot: asher synchronized wmf-config/mc-eqiad.php 'trying twemproxy on all eqiad hosts'
* 00:14 logmsgbot: asher synchronized wmf-config/mc-eqiad.php 'large server_failure_limit, small retry_timeout for twemproxy test hosts'
== June 12 ==
* 23:57 logmsgbot: asher synchronized wmf-config 'revert twemproxy'
* 23:48 logmsgbot: asher restarted twemproxy on all servers
* 23:46 logmsgbot: asher synchronized wmf-config/twemproxy-pmtpa.yaml 'pull 10.0.12.15'
* 23:42 logmsgbot: aaron cleared profiling data
* 23:35 logmsgbot: asher synchronized wmf-config 'use twemproxy for memcached access from all hosts'
* 23:33 logmsgbot: tstarling cleared profiling data
* 23:10 paravoid: apt: updating libmemcached/php5-memcached (#5268)
* 23:00 logmsgbot: asher synchronized wmf-config/mc-eqiad.php 'set (server_failure_limit, retry_timeout) = 0 for twemproxy test hosts'
* 22:26 logmsgbot: aaron synchronized php-1.22wmf6/includes/objectcache/MemcachedPeclBagOStuff.php '7f091934ceae0280b44292e5ce426b24c5a579a2'
* 22:24 logmsgbot: aaron synchronized php-1.22wmf5/includes/objectcache/MemcachedPeclBagOStuff.php 'f6287b2e2faba20aea7379dae84ee7c5dd84983f'
* 21:35 mutante: deploying new favicon on wikitech
* 21:22 LeslieCarr: restarting caesium
* 20:56 mutante: PXE booting analytics1020
* 18:51 logmsgbot: asher synchronized wmf-config/mc-eqiad.php 'use twemproxy on mw113* (10 api apaches)'
* 17:29 cmjohnson1: powering down wtp1008 for h/w problem/testing
* 17:23 logmsgbot: yurik synchronized php-1.22wmf5/extensions/ZeroRatedMobileAccess/
* 17:21 yurik: mw1020 complained during depl about fingerprint 81:2e:56:16:83:fd:b1:65:08:7b:ad:f8:66:98:19:c9.
* 17:20 ori-l: mw1020: Warning: Remote Host Identification Has Changed
* 17:18 logmsgbot: yurik synchronized php-1.22wmf6/extensions/ZeroRatedMobileAccess/
* 17:06 cmjohnson1: removing/replacing disk slot 9 ms-be1007
* 16:36 cmjohnson1: removing/replacing failed disks in slot5 and slot9 from ms-be1012
* 16:00 cmjohnson1: powering down carbon for h/w testing/reports
* 15:19 cmjohnson1: mw1041 powering down for h/w issue
* 15:10 gwicke: updated Parsoid to 058da0822
* 02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Jun 12 02:27:19 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Wed Jun 12 02:13:49 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Wed Jun 12 02:07:39 UTC 2013
* 00:21 Ryan_Lane: upgraded mediawiki on wikitech-static
* 00:11 Tim: installing the experimental libmemcached packages on srv193
== June 11 ==
* 22:58 cmjohnson1: wtp1008 rebooting to check bios setting
* 21:26 cmjohnson1: mw1171 powering down via racadm
* 21:21 logmsgbot: maxsem synchronized php-1.22wmf5/extensions/MobileFrontend/
* 21:19 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:11 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 20:33 mutante: deploying bugzilla frontpage css/html changes (bug 22170)
* 20:28 cmjohnson1: powering off asw-c8-eqiad and removing from rack
* 20:18 LeslieCarr: halting asw-c8-eqiad
* 20:17 LeslieCarr: disabling asw-c8-eqiad
* 19:46 gwicke: updated Parsoid to ab8d15596f
* 19:35 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 19:23 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 18:56 Aaron|home: Cleared rows for queued transcodes on commonswiki
* 18:41 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Enable Echo interaction schema and cohort study'
* 18:40 cmjohnson1: depooling and powering down wtp1001-1002 and wtp1005-1tp1008
* 18:40 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable Echo interaction schema and cohort study'
* 18:15 Aaron|home: Parsoid jobs pipeline fixed with 866e66104fa6d285911da1888b9f9dcece2f53a4, pipeline should be working now
* 17:21 logmsgbot: asher restarted twemproxy on all servers
* 17:20 logmsgbot: asher synchronized wmf-config
* 16:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 16:03 logmsgbot: reedy synchronized docroot/bits/favicon/testwikidata.ico
* 15:56 logmsgbot: asher restarted twemproxy on all servers
* 15:22 hashar: cleaned up permissions on operations/debs/buck , that is now a regular ops repo (ops are owner + force push )
* 15:13 logmsgbot: reedy synchronized php-1.22wmf6/extensions/DataValues/
* 15:13 logmsgbot: reedy synchronized php-1.22wmf6/extensions/Wikibase/
* 15:12 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Wikibase/
* 15:09 mark: Inserted varnish 3.0.3plus~rc1-wm11 packages into the precise-wikimedia APT repository
* 14:41 hashar: granted myself force push rights on operations/debs/buck to push in git://github.com/facebook/buck.git@master to 'upstream' branch
* 08:16 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Enable ULS'
* 08:14 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'Disable narayam and WebFonts'
* 08:10 Nikerabbit: Starting ULS deployment phase 1
* 07:21 Jamesofur: updating Board Election translations on cluster
* 02:31 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 11 02:30:24 UTC 2013
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Tue Jun 11 02:16:57 UTC 2013
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Tue Jun 11 02:09:34 UTC 2013
* 01:31 paravoid: restarting all jobrunners per Aaron's request
* 01:07 logmsgbot: aaron synchronized php-1.22wmf6/includes/db/LoadMonitor.php '2e62b818a01d714b8464689eec2394a993209a70'
* 00:56 logmsgbot: catrope synchronized php-1.22wmf5/extensions/Wikibase
* 00:56 logmsgbot: catrope synchronized php-1.22wmf6/extensions/Parsoid
* 00:55 logmsgbot: catrope synchronized php-1.22wmf5/extensions/Parsoid
* 00:46 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Added parsoid jobs to $wgJobTypesExcludedFromDefaultQueue'
* 00:39 RoanKattouw: 	mw1020: @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
* 00:35 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Point Parsoid updates to LVS IP'
* 00:20 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Re-enabled TTM server message jobs.'
== June 10 ==
* 23:33 paravoid: disabling asw-c-eqiad ge-4/0/7, enabling ge-4/0/6, moving description ms-fe1004 from one to the other and fixing VLANs
* 22:50 logmsgbot: reedy synchronized php-1.22wmf6/extensions/Wikibase/
* 22:48 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Wikibase/
* 22:10 mutante: restarting lucene on search prefix hosts
* 21:55 gwicke: updated Parsoid to 6e40256821
* 21:08 mutante: dpkg --configure -a on analytics1020 to fix interrrupted dpkg
* 21:02 mutante: puppet re-signing analytics1020
* 20:48 mutante: restarting lsearchd on all pool4 search nodes
* 19:39 cmjohnson1: dns update
* 19:36 RoanKattouw: Restarted Apache on gallium
* 19:34 logmsgbot: olivneh synchronized php-1.22wmf6/includes/content 'Reverting change Ibfb2cbefe49398'
* 19:34 mutante: reinstalling analytics1020
* 19:26 mutante: powercycling analytics1020
* 19:24 mutante: deleting and recreating /var/spool/snmptt on neon
* 19:08 cmjohnson1: powering down ms-fe1004 to relocate to different row
* 18:58 RoanKattouw: Restarted Varnish on cerium and titanium
* 18:47 mutante: deleting files from /var/spool/snmptt to fix neon again
* 18:35 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikibooks to 1.22wmf6
* 18:29 paravoid: depooling mw1171, broken hardware #5231
* 18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikisource to 1.22wmf6
* 18:26 Reedy: mw1171 is running with readonly file system
* 18:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage and wiktionary to 1.22wmf6
* 18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia, special, private and fishbowl to 1.22wmf6
* 17:48 mutante: rebooting analytics1020
* 17:46 mutante: killed and restared lucene on search indexers after imports, starting incremental updater
* 17:38 mutante: restarting search indexers to let them know about new wikis, running import-db scripts for elwikivoyage, vecwiktionary, testwikidatawiki
* 17:22 logmsgbot: catrope synchronized php-1.22wmf5/extensions/VisualEditor/ApiVisualEditor.php 'Undo live hack'
* 17:15 logmsgbot: catrope synchronized php-1.22wmf5/extensions/VisualEditor/ApiVisualEditor.php 'Live hack for debugging'
* 17:14 logmsgbot: catrope synchronized php-1.22wmf5/extensions/VisualEditor/ApiVisualEditor.php 'Live hack for debugging'
* 16:29 ^demon: restarting apache on gerrit box
* 08:49 apergos: restarted gerrit again, poor thing
* 08:26 logmsgbot: nikerabbit synchronized php-1.22wmf5/extensions/UniversalLanguageSelector/ 'ULS to master'
* 08:10 logmsgbot: nikerabbit synchronized php-1.22wmf6/extensions/UniversalLanguageSelector/ 'ULS to master'
* 07:34 Jamesofur: updated Board Election translations on cluster
* 02:36 gwicke: updated Parsoid to 956117df0e
* 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 10 02:06:49 UTC 2013
* 02:03 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Mon Jun 10 02:02:14 UTC 2013
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Mon Jun 10 02:01:29 UTC 2013
* 00:12 Krinkle: Gerrit has once again closed the Zuul socket, ssh event stream down
== June 9 ==
* 22:33 Nemo_bis: gitblit fixed as of 1h ago: ^demon> Yeah, I saw the e-mails.
* 18:03 gwicke: deploy Parsoid 836ae1e, hopefully fixing the hanging worker issue
* 16:11 gwicke: restarting Parsoid
* 05:20 ^demon: restarted gerrit, again
* 03:29 Krinkle: git.wikimedia.org is 503 Service Temporarily Unavailable
* 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 9 02:07:43 UTC 2013
* 02:03 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Sun Jun 9 02:02:15 UTC 2013
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Sun Jun 9 02:01:31 UTC 2013
* 01:59 Krinkle: Zuul is halted on a broken Gerrit event stream again.
== June 8 ==
* 23:22 RoanKattouw: Restarted Parsoid on a few machines where it was stuck
* 23:02 Jamesofur: synced Board Election translations on cluster
* 16:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Actually enable EducationProgram on the swedish wikipedia'
* 10:40 Nemo_bis: git.wikimedia.org 503: Service Temporarily Unavailable
* 07:54 Jamesofur: synced Board Election translations on cluster
* 02:25 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/
* 02:25 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
* 02:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 8 02:19:43 UTC 2013
* 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Sat Jun 8 02:08:16 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Sat Jun 8 02:05:56 UTC 2013
* 01:37 logmsgbot: reedy synchronized php-1.22wmf6/extensions/SecurePoll/
* 01:37 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/
* 01:00 RoanKattouw: Restarting Parsoid on all backends
* 00:54 RoanKattouw: Restarted Apache on gallium AGAIN. Patch for the underlying issue is https://gerrit.wikimedia.org/r/#/c/67551/
== June 7 ==
* 23:09 RoanKattouw: Restarted Apache on gallium agai
* 22:47 gwicke: update Parsoid to 9ccf7a3365
* 22:46 RoanKattouw: and on cerium and titanium too
* 22:46 RoanKattouw: Rerunning puppet on nickel for parsoidcache change
* 22:40 LeslieCarr: restart snmptrapd and smptt on neon
* 22:24 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php
* 22:03 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php
* 21:58 logmsgbot: reedy synchronized php-1.22wmf6/extensions/EducationProgram/
* 21:57 logmsgbot: reedy synchronized php-1.22wmf5/extensions/EducationProgram/
* 21:49 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Enable REL1_21 for extdist'
* 21:45 cmjohnson1: auth-dns update
* 21:21 mutante: installing package upgrades on neon now that it has inodes again
* 21:08 cmjohnson1: auth-dns update removing tampa parsoid dns entries
* 21:06 mutante: neon ran out of inodes, culprit /var/spool/snmptt has >300k files, deleting -mtime +2 in batches
* 20:49 RoanKattouw: Actually restarting pybal on lvs4 now
* 20:48 RoanKattouw: Running puppet on lvs4
* 20:47 RoanKattouw: Restarting pybal on lvs4
* 20:32 RoanKattouw: Restarting pybal on lvs3
* 19:50 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Tweaked redis queue timeout and enabled gzip'
* 19:43 RoanKattouw: Restarting Parsoid manually because git-deploy is not cooperating
* 19:29 mutante: installing package upgrades on neon (Icinga, nagios-plugins)
* 19:28 mutante: installing package upgrades on bast1001
* 19:24 mutante: installing package upgrades on fenari
* 19:20 RoanKattouw: Restarted Apache on gallium (APC cache pollution)
* 19:13 cmjohnson1: shutting down wtp1001 and wtp1002 to relocate to c8-eqiad
* 19:06 RoanKattouw: Depooled wtp1008 for investigation, Chris saw some sort of error on it
* 19:04 RoanKattouw: Depooled wtp1001, wtp1002 so Chris can move them
* 18:52 cmjohnson1: restarting ns1
* 18:47 cmjohnson1: auth-dns update (removing/renaming db1013)
* 18:28 mutante: DNS update - removing singer
* 17:52 cmjohnson1: shutting down db1013 to move to frack and rename
* 17:45 logmsgbot: aaron synchronized php-1.22wmf5/maintenance/eraseArchivedFile.php '2aa62f6afbd48a3362b78788b81aed847e6e805f'
* 16:29 logmsgbot: reedy synchronized wmf-config/
* 16:22 logmsgbot: reedy synchronized wmf-config/
* 16:04 logmsgbot: reedy synchronized wmf-config/
* 15:59 Reedy: Created EducationProgram tables on hewiki
* 15:58 Reedy: Created EducationProgram tables on sewiki
* 15:56 Reedy: Creating wikilove tables on viwiki
* 14:17 hashar: Zuul was no more triggering Jenkins jobs . Fixed by Chad who restarted Gerrit.
* 13:55 ^demon: restarting gerrit, queue is overloaded and it's killing the bridge to jenkins
* 13:44 ^demon: restarting zuul per https://bugzilla.wikimedia.org/show_bug.cgi?id=49294#c4
* 12:46 logmsgbot: reedy synchronized php-1.22wmf6/extensions/Parsoid/
* 12:45 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Parsoid/
* 02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 7 02:27:05 UTC 2013
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf6) at Fri Jun 7 02:16:10 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Fri Jun 7 02:05:54 UTC 2013
* 00:48 logmsgbot: catrope synchronized php-1.22wmf6/extensions/VisualEditor 'Updating VisualEditor to master'
== June 6 ==
* 23:39 logmsgbot: gwicke synchronized wmf-config/CommonSettings.php
* 23:31 gwicke: updating Parsoid to 020c7b2
* 23:24 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Plumbing for $wmgVisualEditorExperimental'
* 23:23 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Set $wmgVisualEditorExperimental = true on mediawiki.org'
* 23:20 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Parsoid/
* 23:17 logmsgbot: reedy synchronized php-1.22wmf6/extensions/Parsoid/
* 23:15 logmsgbot: reedy synchronized php-1.22wmf6/extensions/Parsoid/
* 23:08 logmsgbot: tstarling cleared profiling data
* 23:04 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Parsoid/php/Parsoid.php 'fix hook call'
* 22:56 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'touch'
* 22:47 logmsgbot: catrope synchronized visualeditor.dblist 'Enabling VisualEditor on the remaining 270 Wikipedias'
* 22:44 logmsgbot: gwicke Finished syncing Wikimedia installation... :
* 22:36 logmsgbot: gwicke Started syncing Wikimedia installation... :
* 22:28 gwicke: deploying https://gerrit.wikimedia.org/r/#/c/67338/
* 22:22 logmsgbot: csteipp synchronized wmf-config/InitialiseSettings.php 'enable SUL2 on testwikidatawiki'
* 22:14 RoanKattouw: WTF, puppet ran just fine on srv283
* 22:13 RoanKattouw: Running puppet on srv283 to figure out what broke
* 22:11 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploying PostEdit change to core, GettingStarted and GuidedTour for E3 deployment
* 22:01 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploying PostEdit change to core, GettingStarted and GuidedTour for E3 deployment
* 21:43 Krinkle: Zuul gave up on connecting to Gerrit. Restarting Zuul made it try again. Restarting Gerrit again made it connect and it seems back up.
* 21:40 logmsgbot: catrope synchronized php-1.22wmf5/extensions/Parsoid 'Adding Parsoid to wmf5'
* 21:26 Krinkle: Restarting zuul on gallium
* 21:11 kaldari: completed updateSpecialPages.php on testwiki
* 21:10 kaldari: Running updateSpecialPages.php on testwiki to test new Disambigutor special pages
* 21:00 logmsgbot: yurik synchronized php-1.22wmf6/extensions/ZeroRatedMobileAccess
* 20:58 RoanKattouw: Restarting zuul on gallium
* 20:57 logmsgbot: yurik synchronized php-1.22wmf5/extensions/ZeroRatedMobileAccess
* 20:45 logmsgbot: reedy synchronized database lists files:
* 20:36 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Send all Parsoid requests to eqiad'
* 20:34 logmsgbot: reedy synchronized php-1.22wmf6/extensions/
* 19:42 logmsgbot: reedy synchronized php-1.22wmf6/extensions/WikimediaMaintenance/
* 19:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 19:35 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
* 19:34 logmsgbot: reedy synchronized php-1.22wmf5/extensions/WikimediaMaintenance/
* 19:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 19:04 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Set wgSecurePollScript'
* 19:01 logmsgbot: reedy synchronized w/auth-api.php
* 18:53 logmsgbot: reedy synchronized database lists files:
* 18:53 logmsgbot: reedy synchronized wmf-config/
* 18:44 logmsgbot: reedy synchronized database lists files:
* 18:39 logmsgbot: reedy synchronized wikidata.dblist
* 18:39 LeslieCarr: removed some extraneous packages from neon which had filled its root partition
* 18:29 Reedy: Created translate tables on testwikidatawiki
* 18:28 logmsgbot: reedy synchronized database lists files:
* 18:27 logmsgbot: reedy synchronized database lists files:
* 18:26 logmsgbot: reedy synchronized wmf-config/
* 18:21 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 18:20 logmsgbot: reedy synchronized database lists files:
* 18:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf6
* 17:25 logmsgbot: reedy Finished syncing Wikimedia installation... : Take 2
* 17:19 logmsgbot: reedy Started syncing Wikimedia installation... : Take 2
* 17:09 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf6 and rebuild l10n cache
* 16:58 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf6 and rebuild l10n cache
* 16:55 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.22wmf6 and rebuild l10n cache
* 16:50 logmsgbot: reedy synchronized w
* 16:49 logmsgbot: reedy synchronized docroot
* 16:47 logmsgbot: reedy synchronized w
* 16:46 logmsgbot: reedy synchronized docroot
* 16:45 logmsgbot: reedy synchronized php-1.22wmf6/
* 15:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 13:44 ^demon: zuul seems to not be yelling about invalid json anymore
* 13:41 ^demon: rolling gerrit back to 2.6-rc0-144-gb1dadd2 (version before any updates). Too many rough edges we didn't catch in testing, zuul is not happy
* 12:53 ^demon: gerrit rollback to 2.7-rc1-420-g5d5c5c3, some of the new work wasn't quite ready for prime time
* 12:30 ^demon: gerrit's back, running 2.7-rc1-424-gef469ac
* 12:26 ^demon: bringing gerrit down for a few minutes to pick up some changes
* 07:28 apergos: wtp1008 rebooted itself a few minutes ago (hence the icinga whine), no idea why
* 07:22 apergos: and the rest of the ms-fes
* 07:18 apergos: restarted swift on most backends as well
* 07:15 apergos: swift init restart all on ms-fe1 and 3
* 05:16 Tim: fixed high enwiki master query load due to GuidedTour
* 05:15 logmsgbot: tstarling synchronized php-1.22wmf5/extensions/GuidedTour
* 05:13 logmsgbot: tstarling synchronized php-1.22wmf5/extensions/GuidedTour/GuidedTourHooks.php
* 05:13 logmsgbot: tstarling synchronized php-1.22wmf5/extensions/GuidedTour/modules/tours/test.js
* 04:19 logmsgbot: tstarling synchronized php-1.22wmf5/includes/Revision.php 'remove debug hack'
* 03:55 logmsgbot: tstarling synchronized php-1.22wmf5/includes/Revision.php 'temp debug hack'
* 03:48 logmsgbot: tstarling synchronized php-1.22wmf5/includes/cache/MessageCache.php 'revert temp hack'
* 03:46 logmsgbot: tstarling synchronized php-1.22wmf5/includes/cache/MessageCache.php 'temp debug log for high master query load'
* 02:16 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 6 02:15:33 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Thu Jun 6 02:06:01 UTC 2013
* 02:06 Krinkle: Graceful reload of Zuul to deploy 
* 00:56 mwalker: updating listeners from a9fdfd328a8d06d4efb5f4037c281cf4341e23ba to e5f30211ae89ba52a1234715a6d59a7a2d421169; updating payments from cf2d1639a9a8b2854d5302db2aba86199b4e4101 to bf6cbd63cb3cc6148f8abf35767c7dba4ce92b0e
* 00:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 00:10 Tim: deployed apache configuration for vote.wikimedia.org
* 00:09 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Enabled message cache debug log'
== June 5 ==
* 23:58 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 23:57 logmsgbot: reedy synchronized wmf-config/
* 23:56 logmsgbot: reedy synchronized database lists files:
* 23:56 logmsgbot: reedy synchronized docroot
* 23:44 logmsgbot: spage Finished syncing Wikimedia installation... :
* 23:40 Tim: added vote.wikimedia.org to DNS
* 23:38 logmsgbot: spage Started syncing Wikimedia installation... :
* 23:29 logmsgbot: mlitn synchronized php-1.22wmf5/extensions/ArticleFeedback/ 'Update ArticleFeedbackv5 to master'
* 23:13 mutante: enabling wikidata "entity" URLs like http://www.wikidata.org/entity/Q1
* 22:52 logmsgbot: maxsem synchronized wmf-config
* 22:21 Ryan_Lane: upgrading salt in labs and production
* 22:09 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 22:00 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 21:39 logmsgbot: dzahn synchronized docroot/testwikidata
* 21:23 RoanKattouw: Killed old process on wtp1004 that was hogging the port
* 21:22 mutante: shutting down mw1171
* 21:06 ^demon: killed gitweb, long live gitblit
* 20:27 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/
* 20:20 mutante: attempting parsoid restart on wtp1004
* 20:17 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/resources/SecurePoll.js
* 20:05 mutante: enabling test.wikidata virtual host
* 19:56 logmsgbot: dzahn synchronized docroot/wikidata_test/
* 19:55 mutante: sync_dir docroot/wikidata_test/ on tin
* 18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything over to 1.22wmf5
* 16:50 paravoid: upgrading Ceph to cuttlefish, ignore warnings for the next half/one hour
* 15:32 andrewbogott: upgraded adminbot on wikitech-static to 1.7.3. Should be more likely to survive netsplits.
* 10:24 paravoid: starting icinga
* 03:58 apergos: restarted morebots
== June 4 ==
* 02:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Jun 4 02:36:22 UTC 2013
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Tue Jun 4 02:16:08 UTC 2013
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Tue Jun 4 02:09:55 UTC 2013
== June 3 ==
* 23:08 mutante: DNS update - point contacts away from broken singer
* 22:58 logmsgbot: kaldari synchronized php-1.22wmf5/extensions/TimedMediaHandler/resources/mw.MediaWikiPlayerSupport.js 'syncing mw.MediaWikiPlayerSupport.js for TimedMediaHandler bug'
* 22:26 binasher: replaced redis-server in precise-wikimedia with 2.6.13
* 21:18 binasher: upgrading php5-redis on all appservers to version bult from github master 2d0f29bdaf29b071aea29b8fb9ee4158c2b69d72
* 20:28 LeslieCarr: dist-upgrading vanadium
* 20:28 LeslieCarr: allowing salt out of analytics subnet
* 19:37 logmsgbot: aaron synchronized php-1.22wmf5/includes/diff/DifferenceEngine.php '22a544be01809226c1069cff90cdc174df81b7e5'
* 19:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf5
* 19:16 logmsgbot: aaron synchronized php-1.22wmf5/includes/WebRequest.php '7715ffc9e4216497edc696fa686103dcc9698e9d'
* 19:15 logmsgbot: aaron synchronized php-1.22wmf4/includes/WebRequest.php '7f31f0029bdc4526b41ad09968e17e10838087a6'
* 17:56 binasher: dropped old rowiki db from s3, left over from the migration to s7 4+ years ago
* 17:01 cmjohnson1: ms-be1011 replacing disk at slot5
* 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 3 02:07:36 UTC 2013
* 02:03 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Mon Jun 3 02:02:19 UTC 2013
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Mon Jun 3 02:01:35 UTC 2013
== June 2 ==
* 23:01 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll
* 23:00 logmsgbot: reedy synchronized php-1.22wmf4/extensions/SecurePoll
* 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 2 02:07:28 UTC 2013
* 02:03 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sun Jun 2 02:02:14 UTC 2013
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Sun Jun 2 02:01:30 UTC 2013
== June 1 ==
* 15:56 paravoid: staggered reboot of ms-fe1001-4, standard maintenance
* 02:08 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 1 02:07:32 UTC 2013
* 02:03 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sat Jun 1 02:02:25 UTC 2013
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Sat Jun 1 02:01:38 UTC 2013
== May 31 ==
* 21:34 hashar: Zuul / Jenkins had some delay most probably related to a lot of patchsets being sent at the same time. Zuul catching up right now.
* 21:15 binasher: completed echo event_page_id schema migrations
* 20:50 binasher: restarted gerrit
* 19:53 AaronSchulz: Running a terbium script to copy missing files to swift as well as ceph
* 19:24 logmsgbot: aaron synchronized php-1.22wmf5/maintenance/copyFileBackend.php '82352634594d0a67d3b9f6597c3bb2c0911f8028'
* 19:23 logmsgbot: aaron synchronized php-1.22wmf4/maintenance/copyFileBackend.php '3d0e3f8e4d09d4a2462f583c25dd284e6ee3e466'
* 14:42 logmsgbot: reedy synchronized php-1.22wmf4/extensions/SecurePoll/
* 14:41 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/
* 14:34 Reedy: Created bv2013_edit tables on all wikis
* 14:30 logmsgbot: reedy synchronized php-1.22wmf4/extensions/SecurePoll/
* 14:29 logmsgbot: reedy synchronized php-1.22wmf5/extensions/SecurePoll/
* 12:04 apergos: uh... it would help if I looked on tin, the actual deploy host so nm. anyways texvc now working
* 12:01 apergos: ran scap-recompile to get texvc built for 1.22wmf5 and btw this had to be done from fenari, there's no dsh on terbium
* 06:57 apergos: ceph osds 50 and 132 on ms-fe1002 are logging 'slow requests', don't know how to restart specific osds in bobtail though
* 06:43 Aaron|home: Running a script on terbium to list out files that were updated (overwritten) but are not synced on all wikis (it found none on commons/enwiki)
* 05:26 Tim: on msfe1001-1004: disabled apache_status ganglia module
* 04:18 Tim: on nickel: restarting gmetad to see if that fixes ceph cluster reporting
* 04:02 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Disabled ceph backends and switched to just swift.'
* 02:32 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 31 02:31:40 UTC 2013
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Fri May 31 02:11:03 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Fri May 31 02:06:06 UTC 2013
* 00:11 mwalker: Updating payments deploy from dce20cbeb501a9945d97b2e64c59dbb4dd0a9620 to cf2d1639a9a8b2854d5302db2aba86199b4e4101 for RTBT attempted fix
* 00:07 logmsgbot: kaldari Finished syncing Wikimedia installation... :
== May 30 ==
* 23:58 LeslieCarr: fixing packages/security upgrades on db45
* 23:56 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 23:55 LeslieCarr: mw1171 appears to have a bad hard drive
* 22:08 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deployment for E3 and E2
* 21:55 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deployment for E3 and E2
* 21:23 LeslieCarr: powercycling mw16
* 20:35 cmjohnson1: ms-be106 replacing disk slot 6
* 20:11 cmjohnson1: es1001 replacing bad disk (slot 11)
* 20:06 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 19:55 LeslieCarr: enabling routing-engine on asw-c1-eqiad
* 19:54 LeslieCarr: asw-c1-eqiad successfully replaced
* 19:48 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 19:21 LeslieCarr: powering on new asw-c-eqiad switch -- possible disruption to row c connectivity (low risk)
* 19:20 LeslieCarr: replacing asw-c1-eqiad with new switch
* 18:15 bblack: updated vhtcpd deb to 0.0.6-1 on brewster, puppet will upgrade the varnishes
* 16:19 andrewbogott_afk: updated OpenStackManager on virt0 to git head.
* 14:59 paravoid: restarting ms-fe4 proxy
* 14:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable SecurePoll on testwiki'
* 13:21 RoanKattouw: (it's not broken, never mind, I'm just incapable of reading things)
* 13:21 RoanKattouw: Synced afe912f7a335ae037b8d8b0ba9616dc55848a2f0 for VisualEditor updates (logmsgbot seems to be broken)
* 13:20 hashar: gallium: killed stalled Jenkins threads that were producing high I/O.
* 13:16 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'afe912f7a335ae037b8d8b0ba9616dc55848a2f0'
* 13:15 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'afe912f7a335ae037b8d8b0ba9616dc55848a2f0'
* 13:15 logmsgbot: catrope synchronized visualeditor.dblist 'afe912f7a335ae037b8d8b0ba9616dc55848a2f0'
* 13:11 hashar: gallium: blackholed some web crawler robot.
* 12:01 Krinkle: Gracefully reloading Zuul to deploy 'Ica1137438a1b9137f01276b4dfe49a82227de518'
* 10:19 logmsgbot: mark synchronized wmf-config/squid.php 'Add cp1037-1040 as text Varnish'
* 02:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 30 02:37:00 UTC 2013
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Thu May 30 02:14:49 UTC 2013
* 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Thu May 30 02:08:18 UTC 2013
== May 29 ==
* 23:36 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'enwikispecies -> specieswiki'
* 23:26 AaronSchulz: Tried running some metawiki TTMServerMessageUpdateJob jobs again, still very slow for some jobs (though at least not hours)
* 23:20 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Enabling new 'VForm' layout for account creation & login interfaces on 31 wikis. (2/2)'
* 23:19 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling new 'VForm' layout for account creation & login interfaces on 31 wikis. (1/2)'
* 23:19 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings-labs.php 'Labs: consolidate 'wgUseVFormCreateAccount' & 'wgUseVFormUserLogin' into 'wmgUseVForm''
* 19:53 logmsgbot: yurik synchronized php-1.22wmf5/extensions/ZeroRatedMobileAccess/
* 19:07 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 18:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.22wmf5
* 18:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 18:57 logmsgbot: reedy synchronized php-1.22wmf5/extensions/Wikibase
* 18:56 logmsgbot: reedy synchronized php-1.22wmf4/extensions/Wikibase
* 18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikinews to 1.22wmf5
* 18:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wiktionary to 1.22wmf5
* 18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikiversity to 1.22wmf5
* 18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, special and wikimedia to 1.22wmf5 (not wikidata)
* 18:10 logmsgbot: reedy synchronized database lists files: private and wikivoyage to 1.22wmf5
* 17:47 cmjohnson1: mw1085 powering down to reseat DIMM
* 17:41 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
* 17:18 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
* 16:18 cmjohnson1: replacing failed disk on dataset1001 slot9
* 15:02 bblack: deployed vhtcpd puppet stuff via sockpuppet...
* 14:06 cmjohnson1: ms-be4..fixing raid cfg
* 02:34 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 29 02:33:19 UTC 2013
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Wed May 29 02:10:48 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Wed May 29 02:05:47 UTC 2013
== May 28 ==
* 21:28 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:17 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 18:54 logmsgbot: mlitn synchronized php-1.22wmf4/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 18:53 sbernardin: taking down ms-be1 for server replacement
* 18:23 logmsgbot: mlitn Finished syncing Wikimedia installation... : Update ArticleFeedbackv5 & Echo to master
* 18:14 logmsgbot: mlitn Started syncing Wikimedia installation... : Update ArticleFeedbackv5 & Echo to master
* 17:28 sbernardin: taking down ms-be4 for server replacement
* 16:46 cmjohnson1: ms-be4 going down for h/w replacement
* 14:37 RoanKattouw: Gracefulled Apache on gallium, looks like there might have been APC corruption
* 14:23 notpeter: reimaging db1033 for testing
* 07:02 apergos: search23 apparently rebooted itself, nothing of use in the logs nor in atop
* 02:18 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 28 02:17:19 UTC 2013
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf5) at Tue May 28 02:11:34 UTC 2013
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Tue May 28 02:01:26 UTC 2013
== May 27 ==
* 20:30 logmsgbot: reedy synchronized php-1.22wmf5/extensions/UniversalLanguageSelector 'Revert back to 1.22wmf4 version of ULS'
* 20:19 apergos: restarted morebots
*18:41 reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf5
*18:39 reedy Finished syncing Wikimedia installation... : Take 3
*18:24 reedy Started syncing Wikimedia installation... : Take 3
*18:16 reedy Started syncing Wikimedia installation... : Take 2
*18:15 reedy synchronized php-1.22wmf5/cache/
*17:30 reedy synchronized wmf-config/squid.php 'Repush'
*17:15 reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf5 and build localisation cache
*17:13 reedy synchronized php-1.22wmf5/extensions/Wikibase/
*17:12 reedy synchronized php-1.22wmf5/extensions/DataValues/
*17:11 reedy synchronized php-1.22wmf5/extensions/Diff/
*17:04 reedy synchronized docroot
*17:04 reedy synchronized w
*17:03 reedy synchronized php-1.22wmf5 'Initial file sync out'
*02.15 LocalisationUpdate ResourceLoader cache refresh completed at Mon May 27 02:15:05 UTC 2013
*02.03 LocalisationUpdate completed (1.22wmf4) at Mon May 27 02:03:50 UTC 2013
== May 26 ==
*18.41 reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf5
*18.39 reedy Finished syncing Wikimedia installation... : Take 3
*18.24 reedy Started syncing Wikimedia installation... : Take 3
*18.16 reedy Started syncing Wikimedia installation... : Take 2
*18.15 reedy synchronized php-1.22wmf5/cache/
*17.30 reedy synchronized wmf-config/squid.php 'Repush'
*17.15 reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf5 and build localisation cache
*17.13 reedy synchronized php-1.22wmf5/extensions/Wikibase/
*17.12 reedy synchronized php-1.22wmf5/extensions/DataValues/
*17.11 reedy synchronized php-1.22wmf5/extensions/Diff/
*17.04 reedy synchronized docroot
*17.04 reedy synchronized w
*17.03 reedy synchronized php-1.22wmf5 'Initial file sync out'
* 15:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 15:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 15:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 14:32 RobH: authdns-update for netmon1001
* 13:21 logmsgbot: mark synchronized wmf-config/squid.php 'Update cache list'
* 13:09 logmsgbot: mark synchronized wmf-config/squid.php 'Add amssq47-62'
* 12:44 Jeff_Green: authdns-update
* 12:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 12:31 logmsgbot: reedy synchronized wikidata.dblist
* 12:22 mutante: DNS update - add test.wikidata cname
* 12:17 ottomata: taking oxygen down for precise upgrade
* 09:43 LeslieCarr: upgrading some pmtpa mw* machines - may be a few alerts
* 09:18 logmsgbot: catrope synchronized wmf-config/throttle.php 'Raise account creation throttle for Amsterdam hackathon'
* 08:49 logmsgbot: tstarling cleared profiling data
* 06:11 Aaron|home: De-duplicated base refreshLinks2 jobs on enwiki, didn't do much though
* 02:28 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 26 02:27:21 UTC 2013
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sun May 26 02:06:36 UTC 2013
== May 25 ==
* 14:51 Jeff_Green: authdns-update
* 13:36 mutante: restarting pdns on ns0
* 13:11 LeslieCarr: rm -rf /home/wikipedia/common/php-1.21wmf12 on fenari
* 12:46 logmsgbot: reedy synchronized php-1.22wmf4/extensions/CentralAuth
* 12:43 Jeff_Green: authdns-update
* 12:07 logmsgbot: reedy synchronized php-1.22wmf4/extensions/CentralAuth
* 12:04 bblack: cp1029.eqiad : manually testing s/varnishhtcpd/vhtcpd/
* 10:08 Krinkle: Gracefully reloading Zuul to deploy I8c95150a688973
* 02:17 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 25 02:17:03 UTC 2013
* 02:05 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sat May 25 02:04:49 UTC 2013
== May 24 ==
* 23:38 logmsgbot: aaron synchronized php-1.22wmf4/maintenance '3d47dae94d7c416a1e2d4acc506c5e6a212592af'
* 23:37 logmsgbot: aaron synchronized php-1.22wmf4/includes/profiler/Profiler.php '3d47dae94d7c416a1e2d4acc506c5e6a212592af'
* 23:03 logmsgbot: reedy synchronized wmf-config/trusted-xff.cdb
* 20:39 logmsgbot: aaron synchronized php-1.22wmf4/includes/job/jobs/RefreshLinksJob.php '88ffcabd2b0039d18b805bd1e5d8d350c26588e1'
* 14:05 mutante: dzahn graceful'ed apaches and api servers , add redirect for be-x-old.wikisource
* 13:56 RobH: tmh1 back online, tmh2 rebooting for kernel updates
* 13:43 RobH: tmh1 rebooting for kernel updates
* 13:01 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'testwiki captcha'
* 12:47 logmsgbot: reedy synchronized wmf-config/
* 10:42 Jeff_Green: authdns-update again...
* 10:28 hashar: jenkins: updating all phplint jobs to get rid of the old ant building script
* 10:22 Jeff_Green: authdns-update to add fundraisingdb service aliases
* 10:21 mutante: installing upgrades on fenari
* 10:07 RobH: tridge upgrading, will require reboot
* 08:29 K4-713: updated payments cluster to dce20cbeb501a
* 02:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 24 02:28:20 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Fri May 24 02:07:17 UTC 2013
== May 23 ==
* 21:23 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable echo email bundling on enwiki'
* 20:31 logmsgbot: spage synchronized php-1.22wmf4/extensions/GettingStarted 'E3 latest GettingStarted'
* 20:29 logmsgbot: spage synchronized php-1.22wmf4/extensions/GuidedTour 'E3 latest GuidedTour'
* 19:17 logmsgbot: kaldari synchronized php-1.22wmf4/extensions/Echo/modules 'syncing Echo modules for css updates'
* 18:54 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to Master
* 18:48 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to Master
* 18:31 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Add new eventlogging schema: EchoPrefUpdate'
* 12:19 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
* 02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 23 02:25:13 UTC 2013
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Thu May 23 02:06:32 UTC 2013
== May 22 ==
* 23:40 logmsgbot: aaron synchronized php-1.22wmf4/maintenance/runJobs.php '2188c14239b74d202b2109933612259193f6fa41'
* 22:44 logmsgbot: krinkle synchronized php-1.22wmf4/extensions/VisualEditor/modules/ve/ve.EventEmitter.js 'touch, attempt to fix VisualEditor cache snafu on dewiki'
* 22:33 paravoid: removing users sara & mwang from ldap/ops
* 19:12 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
* 19:05 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
* 18:51 apergos: restarted gerrit
* 18:42 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Swithced remaining wikis to 1.22wmf4
* 18:26 logmsgbot: kaldari synchronized php-1.22wmf4/includes/Preferences.php 'syncing preferences.php for bug 48693'
* 18:02 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
* 18:00 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
* 17:46 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
* 17:41 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
* 07:50 apergos: restarted swift proxy on ms-fe1
* 02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 22 02:35:44 UTC 2013
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 22 02:14:48 UTC 2013
* 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Wed May 22 02:08:27 UTC 2013
* 00:21 Ryan_Lane: deployed change 64875 to virt0
== May 21 ==
* 21:11 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Regular weekly MobileFrontend deployment
* 21:04 logmsgbot: awjrichards Started syncing Wikimedia installation... : Regular weekly MobileFrontend deployment
* 19:27 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 19:22 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 18:31 mwalker: (previous log referring to fundraising jenkins!) now upgrading fundraising jenkins for auth plugins
* 18:31 mwalker: restarted jenkins for rekeying of jobs dir; then updated non auth plugs
* 18:17 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Disable AFTv5 talk page links'
* 18:16 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Disable AFTv5 talk page links'
* 17:49 cmjohnson1: powering down db26 for last time
* 17:45 Jeff_Green: rebooting several fundraising hosts for kernel upgrades
* 17:14 cmjohnson1: replacing disk10 (L11) on ms-be1009
* 14:38 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'Disable $wgCentralAuthLoginWiki for testwiki'
* 13:59 logmsgbot: anomie synchronized wmf-config/CommonSettings.php 'Set $wgCentralAuthLoginWiki for test & test2 (SUL2 testing)'
* 13:59 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'Set $wgCentralAuthLoginWiki for test & test2 (SUL2 testing)'
* 13:14 logmsgbot: anomie synchronized php-1.22wmf4/extensions/CentralAuth 'Update CentralAuth to master, for SUL2 testing on testwiki and test2wiki'
* 13:13 logmsgbot: anomie synchronized php-1.22wmf3/extensions/CentralAuth 'Update CentralAuth to master, for SUL2 testing on testwiki and test2wiki'
* 09:41 hashar: made Jeroen De Dauw a maintainer of https://packagist.org/packages/mediawiki/data-values
* 08:10 hashar: zuul: running git gc on all Zuul git repositories: find /var/lib/zuul/git -maxdepth 4 -type d -name .git -print -exec git --git-dir={} gc \;
* 08:02 hashar: Running git gc on Zuul git repositories (gallium: /var/lib/zuul/git/operations/puppet )
* 07:18 LeslieCarr: allowed labs access to the new labsdb hosts
* 02:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 21 02:26:30 UTC 2013
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue May 21 02:09:52 UTC 2013
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Tue May 21 02:06:41 UTC 2013
* 01:31 cmjohnson1: cycling titanium to check bios settings
== May 20 ==
* 23:59 mutante: dist-upgrading celsus
* 23:36 logmsgbot: mflaschen synchronized php-1.22wmf4/extensions/GettingStarted/ '-m Sync GettingStarted for E3 lightning deploy'
* 22:26 mutante: rebooting Tampa parsoid servers, one by one
* 22:20 mutante: dist-upgrading, rebooting kuo
* 22:11 mutante: dist-upgrading, rebooting arsenic
* 22:04 mutante: dist-upgrading, rebooting strontium
* 21:48 mutante: dist-upgrading, rebooting palladium
* 21:31 mutante: dist-upgrading, rebooting niobium
* 20:58 logmsgbot: krinkle synchronized php-1.22wmf4/extensions/VisualEditor/modules/ve/ve.EventEmitter.js 'touch, fixup cache snafu'
* 20:51 Ryan_Lane: temporarily disabling nova-api ratelimiting
* 20:46 akosiaris: added cloudera debian packages hadoop, zookeeper, bigtop-jsvc, bitop-utils to apt.wikimedia.org
* 20:07 Ryan_Lane: rebooting all pmtpa virt nodes
* 20:07 Ryan_Lane: shutdown all labs instances
* 19:12 mutante: dist-upgrading cp1(sq51)
* 19:09 logmsgbot: csteipp synchronized php-1.22wmf4/includes/ 'Security fix'
* 19:08 logmsgbot: csteipp synchronized php-1.22wmf3/includes 'Security fix'
* 19:04 logmsgbot: reedy synchronized w/
* 19:01 logmsgbot: reedy synchronized wmf-config/wgConf.php
* 19:00 logmsgbot: reedy synchronized wmf-config/wgConfVHosts.php
* 18:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable translate on commonswiki'
* 18:46 Reedy: Created translate tables on commonswiki
* 18:44 logmsgbot: krinkle synchronized php-1.22wmf4/resources/mediawiki/mediawiki.js 'touched'
* 18:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:30 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache
* 18:21 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache
* 18:15 logmsgbot: reedy synchronized php-1.22wmf4/extensions/Wikibase/
* 18:14 logmsgbot: reedy synchronized php-1.22wmf4/extensions/ProofreadPage/
* 18:08 logmsgbot: reedy synchronized wmf-config/ 'Enable TemplateData on all wikis'
* 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf4
* 17:49 MaxSem: Reindexing solr
* 17:48 mutante: upgrading/rebooting mobile varnish cache cp1041-cp1044
* 17:35 mutante: kernel upgrade cp1043
* 17:00 mutante: resurrecting icinga-wm and nsca
* 16:59 mutante: ping logmsgbot
* 16:52 mutante: rebooting neon for upgrade
* 16:51 cmjohnson1: shutting down lanthanum to add ssd rt5074
* 15:49 ottomata: rebooting gadolinium (source of webrequest multicast stream) to upgrade kernel
* 14:55 ottomata: rebooting emery to upgrade to 3.2.0-43-generic kernel
* 14:53 ottomata: upgraded all analytics* machines to linux 3.2.0-43-generic kernel
* 02:30 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 20 02:29:25 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Mon May 20 02:13:42 UTC 2013
* 02:05 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Mon May 20 02:05:04 UTC 2013
== May 19 ==
* 02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 19 02:06:55 UTC 2013
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sun May 19 02:01:54 UTC 2013
* 02:02 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sun May 19 02:01:19 UTC 2013
== May 18 ==
* 02:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 18 02:29:09 UTC 2013
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sat May 18 02:11:28 UTC 2013
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sat May 18 02:06:30 UTC 2013
* 01:23 mutante: singer boot issue - affects service contacts.wm (but no others)
* 00:13 mutante: dist-upgrading yvon
* 00:09 mutante: dist-upgrading nitrogen
== May 17 ==
* 23:48 mutante: dist-upgrading singer
* 23:46 mutante: dist-upgrading nickel
* 23:45 RobH: all tampa based apaches have had kernel upgrades
* 23:40 mutante: dist-upgrading praseodymium
* 23:33 RobH: upgrading all srv*.pmtpa.wmnet via dist-upgrade in salt.
* 23:28 mutante: dist-upgrading gurvin
* 23:13 RobH: rebooting all pmtpa mw servers
* 23:13 mutante: dist-upgrading hydrogen, manutius
* 23:12 RobH: still doing pmtpa mw upgrades, ignore all icinga alarms for now
* 23:05 mutante: dist-upgrading chromium
* 23:02 RobH: using salt to dist-upgrade all the tampa apaches... nothing could go wrong....right?
* 22:58 akosiaris: upgrading and rebooting cp1001-1020 (3 at a time)
* 22:55 mutante: dist-upgrading capella ( IPv6 tunnel relay)
* 22:34 mutante: shutting down zinc
* 22:23 mutante: !!log doesn't log
* 22:21 Reedy: [23:18:29] <binasher> !!log running hotbackup of db71 to pre-labsdb for s1
* 22:14 mutante: rebooting spence for upgrades
* 22:10 logmsgbot: reedy synchronized w
* 22:08 logmsgbot: reedy synchronized docroot
* 22:08 mutante: rm -rf php-1.21wmf12 on tin per reedy
* 22:03 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
* 21:58 apergos: mw80 memtest failure dimm1, all other image scalers in pmtpa updated
* 21:53 mutante: dist-upgrading zhen (mobile vumi)
* 21:49 mutante: DNS update - kill storage1 and 2
* 21:34 akosiaris: about to reboot fenari
* 21:13 hashar: gallium Manually blackholed some web engine crawler (via ip route)
* 19:55 mutante: Wikimedia IRC server working again
* 19:46 Ryan_Lane: rebooted ssl1/2 ssl3001 ssl1001/1002
* 19:42 hashar: Jenkins restarted successfully.
* 19:40 Ryan_Lane: depooling ssl3001 ssl1/2 ssl1001/2
* 19:40 Ryan_Lane: repooling ssl1003/4 ssl3/4
* 19:31 Ryan_Lane: repooling ssl3002/3
* 19:30 hashar: restarted Jenkins
* 19:29 Ryan_Lane: rebooted ssl3/4 ssl3002/3 ssl1003/1004
* 19:29 Ryan_Lane: depooled ssl3/4 ssl3002/3 ssl1003/1004
* 19:19 RobH: blog back up, whew.
* 19:16 RobH: holmium (blog server) rebooting for kernel upgrade
* 19:12 cmjohnson1: shutting titanium down to add disks (take 2)
* 19:04 apergos: eqiad imagescalers done
* 19:04 mutante: rebooting gallium
* 19:03 mutante: wikimedia irc is back
* 19:00 hashar: gallium: stopping jenkins and zuul
* 18:56 RobH: cerium rebooting for kernel upgrade
* 18:56 Ryan_Lane: dist-upgrading all ssl boxes
* 18:55 mutante: dist-upgrading ekrem (IRC)
* 18:55 cmjohnson1: shutting down titanium for add'l disk
* 18:50 RobH: i hate you morebots.
* 18:50 RobH: caesium rebooting
* 18:47 mutante: dist-upgrading calcium
* 18:42 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php 'Removing france as a redirect country for fundraising (7649276ddb3f20678de265c37fe75e1d5e642956)'
* 18:31 hashar: Jenkins: created a Jenkins slave on gallium using jenkins-slave as user and /srv/ssd/jenkins-slave as working directory.
* 18:17 RobH: carbon rebooting for kernel update, tftp in eqiad will be offline for a few
* 17:50 apergos: updating and rebooting image scalers one by one
* 17:46 cmjohnson1: shutting down cerium to add add'l disk
* 17:39 apergos: rebooted snapshots 1001-4 (new kernel)
* 17:35 mutante: bugzilla is back
* 17:31 ^demon: rebooting formey (svn) in 2 minutes...why are you still using SVN?
* 17:30 mutante: rebooting kaulen
* 17:24 mutante: rebooting kaulen (Bugzilla) in 5 minutes, please save your bugs and expect 5 minutes downtime
* 16:49 ^demon: rebooting antimony
* 15:00 paravoid: setting Varnish send_timeout to 600 on upload, mobile, bits (eqiad), upload, bits (esams) for both frontend & backend
* 13:31 hashar: Zuul: applying project templates 
* 12:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 12:29 logmsgbot: reedy synchronized php-1.22wmf4/maintenance/checkUsernames.php
* 05:06 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
* 02:34 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 17 02:33:21 UTC 2013
* 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Fri May 17 02:12:44 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Fri May 17 02:07:14 UTC 2013
* 00:56 K4-713: Woohoo! Logs!
* 00:51 logmsgbot: reedy synchronized php-1.22wmf3/maintenance/checkUsernames.php
== May 16 ==
* 23:48 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/AccountAudit/ 'Updating AccountAudit to master. Adding aa_method'
* 23:45 logmsgbot: pgehres synchronized php-1.22wmf4/extensions/AccountAudit/ 'Updating AccountAudit to master. Adding aa_method'
* 23:43 logmsgbot: reedy synchronized database lists files:
* 23:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 22:39 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
* 22:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 22:29 logmsgbot: mwalker synchronized php-1.22wmf4/extensions/CentralNotice/ 'Updating CentralNotice to master'
* 22:28 logmsgbot: reedy synchronized wmf-config/
* 22:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 22:02 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
* 22:00 logmsgbot: mflaschen Finished syncing Wikimedia installation... : E3 scap for deployment of GettingStarted and GuidedTour
* 21:53 logmsgbot: mflaschen Started syncing Wikimedia installation... : E3 scap for deployment of GettingStarted and GuidedTour
* 21:50 superm401: Scap failed due to wrong ssh configuration
* 21:49 Ryan_Lane: rebooting virt1005 and virt1007
* 21:49 logmsgbot: mflaschen Finished syncing Wikimedia installation... : E3 deployment
* 21:48 logmsgbot: mflaschen Started syncing Wikimedia installation... : E3 deployment
* 21:47 Ryan_Lane: rebooting virt1000
* 21:40 superm401: Started scap for E3 deployment of GettingStarted and GuidedTour
* 21:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 21:29 binasher: running accountaudit_login aa_method schema migrations on all wikis
* 21:20 Ryan_Lane: rebooting virt0
* 20:56 Ryan_Lane: running dist-upgrade on virt cluster
* 20:51 logmsgbot: reedy synchronized wmf-config/
* 20:40 logmsgbot: reedy synchronized php-1.22wmf4/extensions/Wikibase/
* 20:35 paravoid: restarting a ceph-osd osd.3 for testing
* 20:03 logmsgbot: reedy synchronized wmf-config/ 'Updating interwiki cache'
* 19:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 19:58 logmsgbot: reedy synchronized database lists files:
* 19:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 19:51 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 19:49 logmsgbot: reedy synchronized database lists files:
* 19:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 19:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 19:01 logmsgbot: reedy synchronized wmf-config/
* 18:56 logmsgbot: reedy synchronized wmf-config/
* 18:55 logmsgbot: reedy synchronized wmf-config/session.php
* 18:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'EP on dewikiversity'
* 18:41 mwalker|0_0: updating payments from fdf308c8fa92ca75adbb3154b8b31a9f7237e885 to head
* 18:34 gwicke: updated Parsoid to b0229051eb08c8
* 18:29 mutante: magnesium (racktables): upgrading kernel and rebooting
* 18:21 mutante: zirconium (planet): upgrading kernel and rebooting
* 18:17 paravoid: restarting a ceph-osd osd.0 for testing
* 18:02 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuilding l10n cache for wikidata deploy
* 17:51 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuilding l10n cache for wikidata deploy
* 17:27 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
* 17:13 logmsgbot: reedy synchronized php-1.22wmf4/extensions/
* 16:53 ottomata: rebooting stat1 for kernel upgrade
* 16:24 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pooling db1043'
* 16:18 gwicke: Parsoid update to current master: done
* 16:16 gwicke: Parsoid update to current master: start
* 15:57 Krinkle: Graceful reload of Zuul deploying Ic9194c079fa7d46
* 15:28 Krinkle: Graceful reload of Zuul deploying I7565eff83a7a128e
* 13:50 Krinkle: Graceful reload of Zuul deploying I0ea73ca1439c8aa6
* 07:54 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/GeoData/ ''
* 07:47 logmsgbot: maxsem synchronized php-1.22wmf4/extensions/GeoData/ ''
* 07:30 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
* 06:42 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
* 02:31 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 16 02:30:28 UTC 2013
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Thu May 16 02:11:55 UTC 2013
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Thu May 16 02:06:34 UTC 2013
* 00:37 mutante: puppetstoredconfigclean.rb ocg3.pmtpa.wmnet
* 00:14 mutante: puppetstoredconfigclean.rb ms2.pmtpa.wmnet
* 00:06 logmsgbot: reedy synchronized database lists files:
* 00:05 logmsgbot: reedy synchronized database lists files:
== May 15 ==
* 23:44 mutante: pushing wikimedia-task-appserver 2.9-1
* 23:32 logmsgbot: reedy synchronized database lists files:
* 23:30 logmsgbot: reedy synchronized database lists files:
* 22:30 logmsgbot: krinkle synchronized php-1.22wmf4/resources/mediawiki/mediawiki.js 'touched'
* 22:01 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
* 21:54 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
* 20:57 Ryan_Lane: OpenStackManager is reenabled on virt0. login to wikitech is working again
* 20:56 Ryan_Lane: openstack in pmtpa now at the folsom release
* 20:54 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/
* 20:51 logmsgbot: maxsem synchronized php-1.22wmf4/extensions/MobileFrontend/
* 20:15 Ryan_Lane: force running puppet on all virt nodes to bring them back up
* 20:14 Ryan_Lane: upgrading schema for keystone, then glance, then nova
* 20:13 Ryan_Lane: backing up keystone, nova, and glance databases
* 20:12 logmsgbot: aaron cleared profiling data
* 20:12 logmsgbot: aaron cleared profiling data
* 20:10 Ryan_Lane: upgrading glance, keystone and nova-scheduler on virt0
* 20:09 Ryan_Lane: upgrading nova services on compute nodes
* 20:07 logmsgbot: aaron synchronized php-1.22wmf4/includes/specials/SpecialActiveusers.php ''
* 20:05 Ryan_Lane: disabling OpenStackManager
* 20:05 Ryan_Lane: stopping all openstack services. it won't be possible to log into wikitech during this process
* 19:44 paravoid: running refreshFileHeaders.php on terbium
* 19:25 logmsgbot: aaron cleared profiling data
* 19:19 Ryan_Lane: adding temporary ubuntu-cloud archive apt changes in a non-puppetized way on pmtpa virt nodes
* 19:18 Ryan_Lane: disabling puppet on pmtpa virt cluster
* 19:09 logmsgbot: reedy synchronized wmf-config/ 'touch!'
* 19:09 logmsgbot: reedy synchronized php-1.22wmf4/resources/ 'touch!'
* 18:42 Ryan_Lane: cherry-picked change 62100 on virt0 until permanent fix for jquery.chosen minimum width is added
* 18:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commons back to 1.22wmf4
* 18:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commons back to 1.22wmf3 due to broken UploadWizard
* 18:35 ^demon: restarting gerrit
* 18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: aaaand wikisource to 1.22wmf4
* 18:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikiquote to 1.22wmf4
* 18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikibooks to 1.22wmf4
* 18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage and wiktionary to 1.22wmf4
* 18:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: fishbowl, special, wikimedia and private to 1.22wmf4
* 18:01 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/
* 17:58 RobH: authdns update, killin dataset1 entries
* 17:56 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
* 17:50 Ryan_Lane: deploying new project filter on virt0
* 17:44 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
* 16:57 RobH: authdns update, cleaning out old decom entries
* 15:35 Jeff_Green: rebooting boron
* 11:16 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 11:15 logmsgbot: reedy synchronized wmf-config/squid-labs.php
* 11:14 logmsgbot: reedy synchronized wmf-config/squid.php
* 11:13 logmsgbot: reedy synchronized wmf-config/squid-labs.php 'wmf-config/squid.php'
* 11:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 02:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 15 02:34:37 UTC 2013
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Wed May 15 02:14:58 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 15 02:08:15 UTC 2013
* 00:49 mutante: graceful apache on stat1001
* 00:31 logmsgbot: maxsem synchronized php-1.22wmf4/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/63806/'
* 00:29 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/63806/'
== May 14 ==
* 23:57 mwalker: lies -- went from d895730067e3325fd630d09196ee81eeba24b5ee back to head on civi because it was working -- the tester was just broken
* 23:42 mwalker: rolling back civi from deploy head to d895730067e3325fd630d09196ee81eeba24b5ee because thank you isn't working
* 23:38 logmsgbot: aaron synchronized wmf-config/StartProfiler.php
* 23:23 logmsgbot: maxsem synchronized php-1.22wmf3/resources/startup.js
* 22:38 RobH: authdns-update for a bunch of decom updates
* 22:32 mutante: DNS update - removing owa servers
* 22:23 logmsgbot: tstarling synchronized wmf-config/db-pmtpa.php
* 22:17 Ryan_Lane: finished upgrading wikitech
* 22:10 Reedy: Running sync-common on srv193
* 22:04 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 22:03 Ryan_Lane: upgrading mediawiki to 1.22wmf4 on wikitech
* 22:03 mwalker: upgrading fundraising civi (d895730067e3325fd630d09196ee81eeba24b5ee --> head)
* 21:53 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 21:30 RoanKattouw: Repooled wtp1004, added wtp1005-wtp1024 to the pool
* 21:01 sbernardin: changing HD in ms-be5
* 20:52 binasher: s4 now all mariadb, new master info = MASTER_LOG_FILE='db1059-bin.000012', MASTER_LOG_POS=473448023
* 20:45 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/63596/'
* 20:44 sbernardin: changing drives on ms-be10
* 20:39 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/Gadgets/ 'https://gerrit.wikimedia.org/r/#/c/60954/'
* 20:33 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
* 20:32 mutante: apply umask 0002 for wikidev users fix on tin and terbium, class generic::wikidev-umask
* 20:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'setting s4 to writeable on new master'
* 20:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'setting s4 to read-only for a sec'
* 20:22 logmsgbot: maxsem synchronized docroot/bits/mobile/W.png
* 20:20 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
* 19:53 RoanKattouw: Running salt-call state.highstate on wtp1005-24
* 19:51 logmsgbot: aaron synchronized php-1.22wmf3/includes/filerepo/file/LocalFile.php 'd58dc596a25607deaecf0aa1c3be877f834da533 '
* 19:47 RoanKattouw: Accepted salt keys for wtp1005-1024 on sockpuppet
* 19:22 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to master
* 19:12 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to master
* 18:52 mutante: gracefulling apaches with dsh
* 18:46 Ryan_Lane: running an apache-graceful-all
* 18:44 ottomata: deployed puppet change to make nginx send webrequest logs back to main multicast firehose.  NOTE: These logs are not sent to the udp2log instance on emery.
* 18:35 sbernardin: replacing bad drive in ms-be11
* 18:34 logmsgbot: kaldari synchronized php-1.22wmf4/extensions/Echo 'syncing Echo for 1.22wmf4'
* 18:31 logmsgbot: reedy synchronized wmf-config/
* 18:30 logmsgbot: aaron synchronized php-1.22wmf3/includes/filerepo/file/LocalFile.php '0cf6b27aa43ad2b65725081ee4d36ccb0f8a34c6'
* 18:28 logmsgbot: reedy synchronized wmf-config/
* 18:27 mutante: powercycling frozen mw1173
* 18:25 logmsgbot: kaldari synchronized php-1.22wmf4/extensions/Echo 'syncing Echo for 1.22wmf4'
* 18:12 logmsgbot: reedy synchronized wmf-config/
* 17:56 logmsgbot: reedy synchronized wmf-config/
* 17:53 logmsgbot: reedy synchronized wmf-config/
* 17:28 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Enable upload of XML files on Incubator (for MediaWiki dumps)'
* 17:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Namespace configuration for ur.wikipedia'
* 17:15 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 17:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 17:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 17:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 17:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 16:18 logmsgbot: reedy synchronized wmf-config/
* 15:14 Reedy: Created EducationProgram tables on dewikiversity
* 14:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'thwiki collation'
* 13:53 cmjohnson1: running dns update
* 13:53 hashar: Zuul: applying a patch to prevent it from fetching a change multiple time.
* 10:51 apergos: re-enabling puppet on ssl1001 for a bit of real traffic
* 10:42 apergos: repooled ssl1
* 09:58 apergos: puppet temp disabled on all ssl terminators except ssl1 in pmtpa
* 09:37 apergos: depooled ssl1 in pmtpa for testing
* 08:51 logmsgbot: tstarling synchronized php-1.22wmf4/LocalSettings.php 'remove testwiki special case'
* 08:51 logmsgbot: tstarling synchronized php-1.22wmf3/LocalSettings.php 'remove testwiki special case'
* 08:34 logmsgbot: tstarling Finished syncing Wikimedia installation... :
* 08:26 logmsgbot: tstarling Started syncing Wikimedia installation... :
* 08:11 logmsgbot: tstarling Started syncing Wikimedia installation... :
* 07:44 logmsgbot: tstarling Started syncing Wikimedia installation... :
* 07:34 logmsgbot: tstarling Started syncing Wikimedia installation... :
* 07:16 logmsgbot: tstarling synchronized w/MWVersion.php
* 07:15 logmsgbot: tstarling synchronized refresh-dblist
* 07:14 logmsgbot: tstarling synchronized multiversion
* 07:12 logmsgbot: tstarling synchronized docroot/noc/db.php
* 06:40 logmsgbot: tstarling synchronized README
* 04:34 Tim: moved scap to tin. Copied source from NFS to tin. Fixed up broken git submodule config using sed.
* 04:03 logmsgbot: tstarling synchronized cgi-bin
* 03:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 14 03:09:49 UTC 2013
* 02:31 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Tue May 14 02:31:15 UTC 2013
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue May 14 02:12:10 UTC 2013
* 02:11 logmsgbot: LocalisationUpdate failed: mwversionsinuse returned empty list
* 01:13 logmsgbot: tstarling synchronized README
* 01:00 logmsgbot: tstarling synchronized README
* 00:20 akosiaris: added myself to root email alias at mchenry
* 00:17 akosiaris: installing package-upgrades on zirconium
== May 13 ==
* 23:19 logmsgbot: catrope synchronized php-1.22wmf4/extensions/VisualEditor/ 'VE bug fix'
* 22:46 RobH: authdns-update for wikipedia.is
* 21:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 21:11 hashar: jenkins: updating all phpcs-HEAD jobs to make them ignore i18n.php files / 
* 21:02 logmsgbot: reedy synchronized wmf-config/
* 20:34 logmsgbot: aaron cleared profiling data
* 20:33 cmjohnson1: morebots back
* 14:40 gallium: deleted an ip route blacklist
* 02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 13 02:35:49 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Mon May 13 02:06:03 UTC 2013
== May 12 ==
* 14:59 Krinkle: Graceful reload of Zuul to deploy Ic22d7c6cb811fd7e3
* 14:31 jeremyb: imported missing !log entries from channel log
* 14:23 bblack: restarted adminbot on wikitech-static
* 14:22 bblack: fixed typo in rt.wikimedia.org vhost hostname, was breaking RT http stuff
* 14:19 bblack: morebots is broken, someone should fix that when they read this
* 09:32 logmsgbot: olivneh synchronized wmf-config/throttle.php '(Bug 48301) Add throttle exception for Haifa University workshop'
* 03:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 12 03:03:03 UTC 2013
* 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sun May 12 02:13:17 UTC 2013
* 01:28 logmsgbot: aaron synchronized php-1.22wmf3/maintenance/copyFileBackend.php '3e103e93f01c24d7e454aaa5db07418405d273c3'
* 01:22 logmsgbot: aaron synchronized php-1.22wmf3/includes/filebackend/FileBackendMultiWrite.php '308ebddc237e76a9602b20585386fc26876f0bee'
== May 11 ==
* 13:24 hashar: restarted Jenkins. Got hit by a memory bug again.
* 03:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 11 03:48:00 UTC 2013
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sat May 11 02:16:23 UTC 2013
== May 10 ==
* 23:20 mutante: DNS update - remove various old Tampa hosts
* 22:49 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/CentralAuth/ 'Small little update to maintenance script'
* 21:46 mutante: shutdown -h blondel .. bye
* 21:39 RobH: switch port disabled for blondel & bellin
* 21:27 RobH: decom of bellen/blondel : setting network ports to disabled
* 21:27 AaronSchulz: Running listing based copy-sync scripts in terbium for all originals+transcodes on all wikis for swift->ceph to fix any outdated/missing files
* 21:05 mutante: fixing some more varnishncsa monitoring
* 20:58 logmsgbot: aaron synchronized php-1.22wmf3/includes/filebackend/FileBackendStore.php '2717d876ba2507df26ac2c5ab746ad7895af1bbf'
* 20:31 logmsgbot: aaron synchronized php-1.22wmf3/includes/filebackend
* 20:06 logmsgbot: reedy synchronized docroot/bits/WikipediaMobileFirefoxOS
* 19:57 logmsgbot: aaron synchronized php-1.22wmf3/maintenance
* 19:56 logmsgbot: aaron synchronized php-1.22wmf3/includes '6c01414775bb9c5269b391bee7ffd1611f9ff74f & d1940caa1f4a11e7c7d8d2ff5b178da7de8e2a92'
* 18:50 mutante: killing and restarting nagios-nrpe-server on a bunch of cp10xx boxes to fix monitoring of varnishncsa
* 18:23 mutante: powercycling srv291
* 18:17 RobH: authdns-update to remove blog.wikimediafoundation as it wasnt setup and pointed to decom server
* 17:42 RobH: authdnsupdate - removing racktables2 entry
* 17:19 hashar: jenkins: cleared all occurrences of an old plugin in build histories. . Jenkins should be fine now. Will monitor tonight.
* 13:20 mark: Migrated toolserver routing to cr1-esams and cr2-esams via csw2-esams access
* 12:28 mark: Removed knsq23-knsq30 from the text squid config
* 12:25 hashar: jenkins: cleaning up all occurrences of an old plugin in build histories. The script run in a tmux on gallium (see https://bugzilla.wikimedia.org/show_bug.cgi?id=48025#c19 )
* 12:09 mark: Depooled knsq23-knsq30 text squid frontends
* 11:52 mark: Moved amslvs1 and 2 from csw1-esams to csw2-esams
* 11:52 mark: Started PyBal on amslvs1
* 11:51 mark: Stopped PyBal on amslvs1
* 11:49 mark: Reenabled PyBal on amslvs2
* 11:44 hashar: restarting jenkins
* 11:44 mark: Disabled Puppet and stopped PyBal on amslvs2
* 11:27 mark: Moved ssl3001-3004 from csw1-esams to csw2-esams
* 10:09 hashar: jenkins: upgrading plugins and restarting.
* 09:43 hashar: restarted jenkins twice to get it to finally serve something :(
* 09:27 hashar: killed -9 jenkins :(
* 09:25 hashar: restarting jenkins
* 09:18 hashar: gallium: dumping java heap of Jenkins using jmap -dump:format=b,file=/root/jenkins-bug48025.jmap -F 24508
* 08:34 hashar: Jenkins hit by the evil bug again . Taking as much trace as possible before restarting it
* 04:46 paravoid: swift: pushing new rings, set weight 0 for ms-be5 sdb1, ms-be11 sdh1 (broken disks); balance 999.99, needs further rebalance
* 03:38 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 10 03:37:41 UTC 2013
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Fri May 10 02:15:49 UTC 2013
* 00:15 logmsgbot: pgehres synchronized wmf-config 'Actually enabling wgCentralAuthAutoMigrate'
== May 9 ==
* 23:34 logmsgbot: mwalker synchronized wmf-config/InitialiseSettings.php 'Enabling siteNotice div on all alpha and beta mobile sites'
* 23:27 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/CentralAuth/CentralAuthUser.php 'Moving one more CentralAuth log line'
* 23:22 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/ 'Updating CentralNotice to master for some bugfixes'
* 23:07 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/CentralAuth/ 'Updates to CentralAuth'
* 21:55 hashar: Jenkins / Zuul is back up (for now).
* 21:45 hashar: restarted Jenkins to enable access logs in /var/log/jenkins/access.log 
* 21:42 hashar: forced Zuul to stop. All events have been dropped :(
* 21:31 hashar: restarting Zuul to make it query Jenkins directly without SSL + apache proxy .
* 21:29 Jeff_Green: pdns-update to cut back to aluminium, restarted pdns on nescio b/c it failed
* 21:28 ^demon: gerrit repositories will now recursively merge
* 21:17 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/WikimediaMessages 'Updating WikimediaMessages to master'
* 20:48 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/GuidedTour 'Actually updating GuidedTour to master'
* 20:46 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/EventLogging 'Updating GuidedTour to master'
* 20:46 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/EventLogging 'Updating EventLogging to master'
* 20:46 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/ConfirmEdit 'Updating ConfirmEdit to master'
* 20:21 hashar: restarting Jenkins.
* 20:20 hashar: Jenkins died :/
* 19:17 mutante: fixing file permissions in /a/e3/E3Analysis repo on stat1001
* 19:09 mutante: restarted Apache and then puppetmaster on stafford, that revived it
* 18:56 mutante: attempting to restart broken puppetmaster on stafford
* 18:27 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php 'Enabling CentralAuth debug log'
* 18:00 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php 'Enabling wgCentralAuthAutoMigrate on all wikis that use CentralAuth'
* 15:56 cmjohnson1: powering down barium /relocate server to c1
* 15:27 logmsgbot: anomie synchronized wmf-config/CommonSettings.php 'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653'
* 15:27 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653'
* 15:26 logmsgbot: anomie synchronized wmf-config/InitialiseSettings-labs.php 'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653'
* 11:25 hashar: upgrading Zuul to latest master (makes report times friendlier)
* 08:54 hashar: Jenkins apparently working fine. I have upgraded a few plugins, that seems to have fixed the Gerrit jobs.
* 08:22 hashar: restarting Jenkins again
* 08:21 hashar: upgrading Jenkins plugins 'promoted build' and 'M2 release'. That should fix the Gerrit jobs disappearing (ping qchris)
* 07:39 hashar: restarting jenkins. All its http thread are locked :/
* 07:25 hashar: Jenkins HTTP interface died :/
* 03:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 9 03:39:55 UTC 2013
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Thu May 9 02:15:54 UTC 2013
== May 8 ==
* 22:27 RobH: authdns update to migrate racktables from hooper to magnesium. both servers are already using db1001 as master, which is why hooper is slow to load racktables data
* 22:24 logmsgbot: mholmquist synchronized php-1.22wmf3/extensions/UploadWizard/UploadWizardHooks.php 'Fixing Special:Preferences for wikis with UploadWizard'
* 21:45 logmsgbot: mholmquist Finished syncing Wikimedia installation... : Fix UploadWizard concurrency and minor message updates
* 21:37 logmsgbot: mholmquist Started syncing Wikimedia installation... : Fix UploadWizard concurrency and minor message updates
* 21:07 Krinkle: Running small maintenance script foreachwiki to gather data about odd wgRC2UDPPrefix values (bug 28276)
* 20:21 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/ 'CentralNotice security bugfix 48255'
* 20:15 Jeff_Green: authdns-update to swap in grosley for dead aluminium
* 19:41 RobH: magnesium is future racktables host, disregard errors for now
* 19:41 cmjohnson1: accidently removed power from magnesium
* 19:00 Reedy: Ensuring securepoll database tables exist on all wikis
* 18:35 logmsgbot: reedy synchronized php-1.22wmf3/extensions/CentralAuth
* 18:31 Jeff_Green: updating and rebooting grosley
* 18:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:17 logmsgbot: reedy synchronized wmf-config/
* 18:16 logmsgbot: reedy synchronized database lists files:
* 18:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable EducationProgram on mkwiki'
* 18:07 Reedy: Created EducationProgram tables on mkwiki
* 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.22wmf3
* 18:00 RobH: authdnsupdate for racktables2
* 17:46 apergos: )
* 17:46 apergos: about to graceful apaches *and api servers* (which somehow are not in the actual apache graceful all script
* 17:42 Jeff_Green: kernel updates and reboot on pappas
* 17:29 logmsgbot: yurik Finished syncing Wikimedia installation... : Zero extension
* 17:18 RobH: restarting all mw jobrunners
* 17:13 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Disabled TTMServerMessageUpdateJob jobs'
* 17:12 logmsgbot: aaron synchronized php-1.22wmf3/includes/job/JobQueueGroup.php 'revert hack'
* 17:11 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/JobQueueGroup.php 'revert hack'
* 17:08 logmsgbot: yurik Started syncing Wikimedia installation... : Zero extension
* 15:01 logmsgbot: mark synchronized wmf-config/mc-eqiad.php
* 14:22 paravoid: ceph osd out 39, primary for stuck unclean pg 3.5c6
* 14:21 paravoid: ceph osd out 128, primary for stuck unclean pg 3.19e1
* 13:59 logmsgbot: mark synchronized wmf-config/mc-eqiad.php
* 13:59 mark: Removing mc1009 from the eqiad memcached pool
* 06:14 apergos: restarted adminbot (morebot) on wikitech-static
== May 7 ==
* 23:42 logmsgbot: catrope synchronized php-1.22wmf3/resources/startup.js 'touch'
* 23:42 logmsgbot: catrope synchronized php-1.22wmf3/extensions/VisualEditor/modules/ve/ve.js 'touch'
* 23:39 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
* 23:35 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/ 'Appling update to fix CentralNotice message editing bug'
* 23:30 mutante: running addWiki.php for iegcomwiki
* 23:26 logmsgbot: dzahn synchronized wmf-config/InitialiseSettings.php
* 23:23 mutante: depool mw1041 in pybal
* 23:22 mutante: removed mw1041 from dsh groups mediawiki-installation and apaches
* 23:20 mutante: mw1041 - RT-4517, RT-5065, RT-5105 ...and broken again :p
* 23:17 logmsgbot: mflaschen synchronized php-1.22wmf3/extensions/GettingStarted/ 'E3 Lightning deploy for GettingStarted'
* 23:17 LeslieCarr: upgrading srv* machines
* 22:48 Tim: on marmontel: disabled logrotate
* 22:36 logmsgbot: aaron synchronized wmf-config/mc-pmtpa.php 'removing mc15 for hw maint'
* 22:34 mutante: upgrading PHP_CodeSniffer on gallium
* 22:31 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/62744/'
* 22:10 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/62650/'
* 22:04 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 22:04 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/
* 21:47 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 21:09 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/JobQueueGroup.php
* 20:53 logmsgbot: aaron synchronized php-1.22wmf3/includes/job/JobQueueGroup.php 'Live hack to avoid translationNotificationJob jobs.'
* 20:44 logmsgbot: reedy synchronized database lists files:
* 20:30 andrewbogott: copied passwords from stafford to sockpuppet, then reset --hard on stafford to get things back in sync.
* 20:17 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 20:11 logmsgbot: reedy synchronized docroot/login/
* 20:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 20:02 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 19:57 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: loginwiki
* 19:56 logmsgbot: reedy synchronized wmf-config/
* 19:55 logmsgbot: reedy synchronized database lists files: loginwiki
* 19:37 mutante: DNS update - switch old secure.wm
* 19:30 mutante: apache-graceful-all, move old secure.wm redirects
* 18:54 cmjohnson1: authdns update
* 18:42 logmsgbot: dzahn synchronized docroot
* 18:42 mutante: sync-docroot for login.wm
* 18:38 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1042,1059 to s4'
* 18:03 logmsgbot: awjrichards synchronized php-1.22wmf2/extensions/MobileFrontend/stylesheets/common/ui.css 'Cherry-pick e37e9d9ae6b8a581569c39ede38fc21827983128'
* 17:59 logmsgbot: awjrichards synchronized php-1.22wmf3/extensions/MobileFrontend/stylesheets/common/ui.css 'Cherry-pick e37e9d9ae6b8a581569c39ede38fc21827983128'
* 17:43 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/FundraiserLandingPage/ 'Updating FundraiserLandingPage for WMF fundraising in FR'
* 17:38 kaldari: killed copyExistingEmailPreference.php on enwiki
* 17:29 Reedy: Killed php-1.21wmf1 through 1.21wmf5 on fenari and mediawiki-installation hosts
* 17:26 logmsgbot: mwalker Finished syncing Wikimedia installation... : Update CentralNotice and core for HTMLForms, iframe preview, and devices!
* 17:19 mutante: killed hanging pdns related processes on ns1 and restarted pdns
* 17:15 kaldari: begin running extensions/Echo/maintenance/copyExistingEmailPreference.php on enwiki db
* 17:02 mutante: DNS update - add login.wikimedia and login.m.wikimedia
* 16:57 logmsgbot: mwalker Started syncing Wikimedia installation... : Update CentralNotice and core for HTMLForms, iframe preview, and devices!
* 16:37 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
* 16:35 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
* 16:31 RobH: professor being worked on, network is offline =(
* 16:29 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
* 16:19 gwicke: updated Parsoid to current master
* 16:03 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
* 15:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 15:46 logmsgbot: reedy synchronized docroot
* 14:47 cmjohnson1: authdns update
* 14:33 cmjohnson1: authdns update
* 13:09 mark: Power cycled line card 2 in csw1-esams (which has the SSL servers) to fix esams SSL reachability issues
* 09:15 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'New ULS config part 2'
* 09:14 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'New ULS config part 1'
* 09:09 logmsgbot: nikerabbit synchronized php-1.22wmf3/extensions/UniversalLanguageSelector/ 'ULS to master'
* 08:21 paravoid: ceph: all radosgw & mon + osd.24 upgraded to 0.56.6
* 07:59 paravoid: partially upgrading ceph to 0.56.6
* 07:27 Tim: on holmium: reverted firewall, too hard
* 07:22 Tim: on holmium: added some ad hoc iptables rules intended to keep the rest of the site mostly safe from it overnight
* 07:00 Tim: on holmium: chown'd to root all script files writable by www-data and all directories which are configured to allow PHP execution
* 06:28 Tim: on marmontel: disabled apache logrotate, stopped apache, disabled apache init script
* 04:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 7 04:19:39 UTC 2013
* 02:25 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue May 7 02:25:11 UTC 2013
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue May 7 02:16:06 UTC 2013
* 01:29 logmsgbot: bsitu synchronized php-1.22wmf3/includes/User.php 'Two fixes to User::getNewMessageRevisionId() from kaldari'
* 01:29 logmsgbot: reedy synchronized php-1.22wmf3/extensions/WikimediaMaintenance
* 00:38 logmsgbot: reedy synchronized php-1.22wmf3/includes/User.php 'Live hack to stop fatals'
* 00:07 logmsgbot: bsitu synchronized php-1.22wmf3/extensions/Echo 'Update Echo to master'
* 00:07 logmsgbot: bsitu synchronized php-1.22wmf3/docs/hooks.txt 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
* 00:06 logmsgbot: bsitu synchronized php-1.22wmf3/RELEASE-NOTES-1.22 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
* 00:06 logmsgbot: bsitu synchronized php-1.22wmf3/includes/WikiPage.php 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
* 00:03 logmsgbot: bsitu synchronized php-1.22wmf3/includes/User.php 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
* 00:03 logmsgbot: bsitu synchronized php-1.22wmf3/includes/Skin.php 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
* 00:02 logmsgbot: bsitu synchronized php-1.22wmf3/includes/OutputPage.php 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
== May 6 ==
* 23:25 mutante: restarting prelabsdbdb on db1054
* 22:48 mutante: apache-graceful-all, add iegcom virtual host
* 21:00 mutante: blog is back up
* 21:00 mutante: holmium: bring up interface lo, comment "search" line in resolv.conf, restart varnish, ...
* 20:46 paravoid: swift replication scripts running on ms-fe1002 under screen -- kill if they have any effect on production
* 20:15 mutante: DNS update - sync with ns2
* 20:04 mutante: restarting parsoid on wtp1001
* 20:03 mutante: restarting pdns on ns0
* 20:01 mutante: killing parsoid server.js on wtp1001
* 19:59 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'f257ddac39194f8bfcfd145104999fdba83d33f5'
* 19:59 mutante: DNS update - switch all blog cnames over to holmium
* 19:30 logmsgbot: reedy synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
* 19:25 LeslieCarr: moved gateway from csw1-esams to cr1-esams
* 19:18 Krinkle: Reloading Zuul to deploy I259585e78ee
* 19:18 LeslieCarr: rebooting ns2
* 19:16 logmsgbot: reedy synchronized wmf-config/
* 19:13 binasher: rebuilt db1016 from hotbackup of db1001; pmtpa side of m1 needs to be rebuilt as well
* 18:42 RobH: ns2/nescio appears offline, investigating
* 18:37 RobH: ns2 appears offline
* 18:37 RobH: authdns-update crashed on ns2, rerunning
* 18:33 RobH: holmium
* 18:33 RobH: authdns-update for new server homium
* 18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf3
* 17:18 LeslieCarr: marmontel's interface is now in the sandbox vlan
* 17:16 LeslieCarr: turned off marmontel's network port
* 16:12 logmsgbot: reedy synchronized docroot
* 13:55 hashar: gallium: installed manually libcidr0-dev libanon0-dev . Puppet change is 
* 09:51 paravoid: upgrading Ganglia Web to 3.5.7
* 08:25 hashar: Zuul upgrading to upstream version 7191ee8 + 1 wmf hack.
* 06:45 Nemo_bis: Translate search on vanadium keeps being broken https://toolserver.org/~nemobis/tmp/SearchTranslations.log (now RT-5085)
* 06:15 paravoid: pushing new swift rings: ms-be1 weight 50->0, ms-be2 33->75
* 02:19 paravoid: restarting gerrit
* 02:18 logmsgbot: LocalisationUpdate failed: git pull of extensions failed
== May 5 ==
* 10:56 Nemo_bis: "Search service is temporarily unavailable" since unknown time https://meta.wikimedia.org/wiki/Special:SearchTranslations?query=amen&uselang=en
* 03:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 5 03:36:17 UTC 2013
* 02:26 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sun May 5 02:25:44 UTC 2013
* 02:18 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sun May 5 02:17:22 UTC 2013
== May 4 ==
* 22:42 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
* 21:44 Reedy: Manually force ran TorBlock/loadExitNodes.php
* 21:43 logmsgbot: reedy synchronized php-1.22wmf3/extensions/TorBlock/
* 16:03 logmsgbot: reedy synchronized wmf-config/
* 16:02 logmsgbot: reedy synchronized docroot/bits/WikipediaMobileFirefoxOS/
* 04:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 4 04:36:09 UTC 2013
* 02:21 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sat May 4 02:20:30 UTC 2013
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sat May 4 02:12:01 UTC 2013
* 01:17 Krinkle: binasher restarted gerrit
== May 3 ==
* 23:49 Krinkle: Gerrit is completely unresponsive (all http and ssh time out, serving HTTP 500). Filed details at https://bugzilla.wikimedia.org/show_bug.cgi?id=48061
* 22:19 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'adding new pmtpa slave'
* 21:10 cmjohnson1: db10 is decommissioned...powering down for the final time
* 20:04 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/MobileFrontend.hooks.php
* 19:54 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/includes/MobileFrontend.hooks.php
* 07:52 MaxSem: OMG logmsgbot is a terminator!
* 06:16 logmsgbot: tstarling synchronized wmf-config/missing.php
* 04:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 3 04:03:12 UTC 2013
* 02:26 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Fri May 3 02:26:02 UTC 2013
* 02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Fri May 3 02:15:45 UTC 2013
* 01:25 hashar: Zuul processed all its events, seems to be fine now. *sleep*
* 00:33 logmsgbot: dzahn synchronized docroot
* 00:32 mutante: sync-docroot for iegcom wiki
* 00:25 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php '6512451451ac6617bedbfce7be17045bed794d70'
* 00:16 mutante: DNS update - add iegcom
== May 2 ==
* 23:20 mutante: restarting lucene on search1021, search1022, search1017, search1018 (with some waiting in between)
* 23:18 hashar: yeah after hours and hours of fighting, Jenkins is finally working again.
* 23:11 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php
* 22:59 hashar: restarting jenkins… it got a few threads blocked and the main process is at 100% usage for now reason
* 22:59 mutante: repooling search1016
* 22:40 mutante: gallium, run puppet, graceful Apache to deploy split log files
* 22:34 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1020 for upgrade'
* 22:32 mutante: depooling search1016, restarting lucene, etc.. (Search#Adding_new_wikis)
* 22:29 mutante: repooling search1015
* 22:23 hashar: Zuul restarted. The bug about slowness is 
* 22:15 hashar: restarting Zuul
* 22:13 hashar: Deploying a workaround on Zuul to make it stop querying the Jenkins API when it just want to check whether a job exist. 
* 22:04 mutante: depooling search1015 in pybal
* 21:43 logmsgbot: mflaschen synchronized php-1.22wmf3/extensions/GuidedTour/ 'Sync GuidedTour to 1.22wmf3 for E3 deployment'
* 21:42 logmsgbot: mflaschen synchronized php-1.22wmf2/extensions/GuidedTour/ 'Sync GuidedTour to 1.22wmf2 for E3 deployment'
* 21:38 mutante: importing wikimania2014wiki into search indexers
* 21:35 mutante: restarting search indexers on searchidx2, searchidx1001 to make sure the indexer knows about new wiki
* 21:28 logmsgbot: mflaschen synchronized php-1.22wmf3/skins/common/shared.css 'Sync font-size change for edit section links'
* 21:15 logmsgbot: spage synchronized php-1.22wmf2/extensions/ConfirmEdit 'update 1.22wmf2 to wmf3 version of ConfirmEdit'
* 21:14 LeslieCarr: adding GEANT via fiberring to avoid-paths
* 20:21 logmsgbot: reedy synchronized php-1.22wmf3/includes/
* 20:15 logmsgbot: reedy synchronized docroot/bits/WikipediaMobileFirefoxOS/
* 19:51 Jeff_Green: authdns-update to move db1025 to frack.eqiad.wmnet
* 19:47 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
* 19:40 ottomata: rebooting oxygen
* 19:33 ottomata: re-enabling puppet on cp1031, it was administratively disabled. running puppet there.
* 19:24 RobH: professor was already powered down (why?) starting it back up now
* 19:23 ottomata: all webrequest udp2log loggers (squid and varnish) now send to gadolinium for socat unicast -> multicast relay
* 19:22 RobH: rebooting professor
* 19:16 Jeff_Green: moving db1025 into frack-fundraising1-c-eqiad
* 19:14 logmsgbot: kaldari synchronized php-1.22wmf2/extensions/Echo 'sync Echo ext for en.wiki'
* 19:14 ottomata: deployed squid frontend.conf.php changes to remove locke and send logs directly to gadolinium for multicast relay
* 19:11 logmsgbot: kaldari synchronized php-1.22wmf3/extensions/Echo 'sync Echo ext'
* 19:09 logmsgbot: kaldari synchronized php-1.22wmf3/extensions/Echo 'sync Echo ext'
* 18:52 ottomata: removed varnishcsa-locke instance from varnish hosts: (dsh -c -g varnishncsa-all 'test -f /etc/init.d/varnishncsa-locke && service varnishncsa-locke stop && update-rc.d -f varnishncsa-locke remove && rm -v /etc/init.d/varnishncsa-locke')
* 18:47 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
* 18:41 ottomata: varnishncsa now sends traffic to gadolinium instead of oxygen for multicast relay
* 18:30 Krinkle: Installed Monitoring plugin from Jenkins control panel <https://wiki.jenkins-ci.org/display/JENKINS/Monitoring>
* 17:59 Krinkle: Jenkins restart complete. No visible improvement. Jenkins is still idling most of the time while Zuul is still halted by an unknown factor on spawning jobs.
* 17:56 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 17:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 17:48 logmsgbot: reedy synchronized wmf-config/
* 17:37 Krinkle: Jenkins keeps clogging up. Starting an emergency restart.
* 17:00 Krinkle: Jenkins is nearing 100% CPU on gallium, what is Jenkins doing?
* 16:59 Krinkle: Zuul is somehow having trouble kicking off Jenkins jobs (less than 1 event processed per minute). Jenkins shows that 10/10 executors are idle. Investigating...
* 16:52 logmsgbot: nikerabbit synchronized php-1.22wmf3/extensions/UniversalLanguageSelector/ 'ULS to master'
* 15:13 hashar: Jenkins made Jenkins to instantly restart (was ) ) by deleting the downstream-buildview plugin.
* 15:11 hashar: restarted Zuul
* 14:12 hashar: Upgraded Jenkins from 1.480.3 to LTS 1.509.1. Restarted it.
* 14:10 hashar: shutting down Zuul
* 14:09 hashar: upgrading Jenkins (unscheduled maintenance).
* 13:01 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/61941/'
* 12:59 mark: Migrated amslvs3 and amslvs4 PyBal BGP peerings from csw2-esams to cr2-knams
* 09:55 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/'
* 09:53 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/'
* 09:50 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/'
* 09:40 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/GeoData/GeoData.body.php 'https://gerrit.wikimedia.org/r/#/c/61962/'
* 06:04 apergos: powercycle snapshot1001, upgrade to precise
* 05:46 apergos: powercycle snapshot1004, upgrade to precise
* 05:28 apergos: snapshot1003 powercucle, upgrading to precise
* 05:28 Tim: on professor: manually started carbon-cache.py
* 05:15 Tim: professor is down, no response on serial console, rebooting
* 05:11 logmsgbot: tstarling synchronized wmf-config/jobqueue-pmtpa.php
* 05:08 logmsgbot: tstarling synchronized wmf-config/jobqueue-eqiad.php
* 04:47 Tim: on rdb1001-1002, set requirepass and masterauth in soft state
* 04:44 Tim: on mc1-16 and mc1001-1016, setting requirepass and masterauth to the new password in soft state
* 04:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 2 04:39:48 UTC 2013
* 04:04 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
* 03:59 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
* 03:57 logmsgbot: tstarling synchronized wmf-config/jobqueue-eqiad.php
* 03:47 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/GettingStarted
* 03:40 logmsgbot: tstarling synchronized php-1.22wmf3/extensions/GettingStarted
* 02:46 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 02:32 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Thu May 2 02:32:10 UTC 2013
* 02:22 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Thu May 2 02:21:30 UTC 2013
* 02:21 Tim: on wikibugs-l: disabled bounce processing and re-enabled mail delivery to wikibugs-irc (was disabled due to excessive bounces)
* 02:21 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 01:20 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js 'Poking bits to try and get the new banner controller deployed for CentralNotice'
* 01:17 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js 'Poking bits to try and get the new banner controller deployed for CentralNotice'
* 00:16 mutante: @stafford:~# puppetstoredconfigclean.rb db10.pmtpa.wmnet
== May 1 ==
* 23:28 logmsgbot: tstarling synchronized php-1.22wmf3/extensions/GettingStarted
* 23:23 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/GettingStarted
* 23:18 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/GettingStarted
* 22:42 mutante: powercycling frozen mw1041
* 22:14 logmsgbot: mwalker Finished syncing Wikimedia installation... : Updating CentralNotice from a8797938bc for historical allocations and some refactoring
* 22:06 logmsgbot: dzahn synchronized ./multiversion/MWMultiVersion.php
* 22:02 logmsgbot: dzahn synchronized docroot
* 21:44 logmsgbot: mwalker Started syncing Wikimedia installation... : Updating CentralNotice from a8797938bc for historical allocations and some refactoring
* 21:32 logmsgbot: dzahn gracefulled all apaches
* 21:25 ottomata: syncing new redirects.conf to apachces to redirect wikipediazero.org
* 21:14 Jeff_Green: stopping mysql on db78 for cloning
* 20:50 ottomata: adding new wikipediazero.org zone alias, running authdns-update on dobson
* 19:45 logmsgbot: reedy synchronized php-1.22wmf3/cache/l10n/
* 19:15 mutante: DNS update - remove project2, anthony and yongle :p
* 19:05 logmsgbot: reedy Started syncing Wikimedia installation... : rebuild l10n cache for wikidata
* 18:36 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 1 18:36:02 UTC 2013
* 18:32 mutante: DNS update - add wikimania2014 entries
* 18:31 RobH: disabled owa1-3 switchports, starting remote wipe
* 18:21 RobH: owa1/2/3 offlinging and wiping
* 18:20 logmsgbot: reedy synchronized php-1.22wmf3/extensions/ 'Update extensions for wikibase deploy'
* 18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wiktionary to 1.22wmf3
* 18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource, wikivoyage and wikiversity to 1.22wmf3
* 18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikiquote to 1.22wmf3
* 18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special and wikimedia to 1.22wmf3
* 18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, private and fishbowl to 1.22wmf3
* 17:58 logmsgbot: kaldari synchronized php-1.22wmf3/extensions/Echo/Hooks.php 'syncing Hooks in Echo'
* 17:33 logmsgbot: kaldari synchronized php-1.22wmf2/extensions/Echo/Hooks.php 'syncing Hooks in Echo'
* 17:20 LeslieCarr: puppet working on bits caches again after gerrit change 61793
* 17:01 RobH: torrus compiling, manutilus apache offline
* 16:53 RobH: torrus deadlocked, fixin
* 15:54 mark: Converted link between cr2-knams:xe-1/0/0 and csw2-esams:xe-0/1/1 to LACP aggregated link cr2-knams:ae1 / csw2-esams:ae2
* 14:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 14:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 10:30 logmsgbot: reedy synchronized php-1.22wmf2/extensions/ParserFunctions/ 'Update to master to fix common fatals'
* 10:08 mark: Removed old Toolserver IP range 91.198.174.192/27 on primary and backup router interfaces
* 03:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 1 03:49:55 UTC 2013
* 02:28 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 1 02:27:30 UTC 2013
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed May 1 02:16:40 UTC 2013
* 02:09 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Turned autoResync back on again for multiwrite backend'
* 00:46 logmsgbot: tstarling synchronized wmf-config/PrivateSettings.php
== April 30 ==
* 23:52 Krinkle: Gracefully restarted Zuul on gallium
* 23:52 Krinkle: Ryan_Lane restarted the gerrit service
* 23:44 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Syncing CommonSettings for Echo'
* 23:34 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Syncing InitialiseSettings for Echo'
* 23:28 logmsgbot: kaldari synchronized /php-1.22wmf3/extensions/Echo 'Updating Echo for some minor bugfixes'
* 23:18 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/ 'Revert MobileFrontend'
* 23:15 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/ 'Revert MobileFrontend'
* 23:12 logmsgbot: tstarling synchronized wmf-config/missing.php 'rewrite (I17989fc4)'
* 23:01 Krinkle: Graceful restart of Zuul on gallium
* 22:56 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php 'Enabling Extension:AccountAudit on all wikis and wmgEchoHelpPage'
* 22:09 Jeff_Green: rolling back db1025 change b/c "mysqld: unknown option '--federated'"
* 22:06 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/ 'grrr'
* 22:00 logmsgbot: maxsem synchronized wmf-config
* 21:58 Jeff_Green: restarting mysqld on db1025 to enable federated engine for testing
* 21:56 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/ 'Weekly mobile deployment'
* 21:40 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/ 'Weekly mobile deployment'
* 20:04 logmsgbot: kaldari synchronized echowikis.dblist 'syncing echowikis.dblist for cron jobs'
* 19:59 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning on Echo for English Wikipedia'
* 19:49 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 19:19 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 19:03 cmjohnson1: removing db10 certs from puppet
* 17:48 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Syncing CommonSettings for Echo deployment'
* 17:45 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Syncing InitialiseSettings for Echo deployment'
* 13:57 cmjohnson1: jenkins seems to not be working -- restarting
* 11:27 Nikerabbit: refresh-translatable-pages.php finished for mediawikiwiki and metawiki
* 04:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 30 04:06:11 UTC 2013
* 02:34 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue Apr 30 02:33:50 UTC 2013
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue Apr 30 02:15:03 UTC 2013
* 02:04 paravoid: cleaning up neon's /var/log (100% full), restarting icinga...
* 00:59 logmsgbot: catrope synchronized php-1.22wmf3/extensions/VisualEditor/ 'Update VE'
* 00:44 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php 'Touching InitialiseSettings'
* 00:32 logmsgbot: pgehres Finished syncing Wikimedia installation... : Deploying Extension:AccountAudit and an Echo thing
* 00:00 logmsgbot: pgehres Started syncing Wikimedia installation... : Deploying Extension:AccountAudit and an Echo thing
== April 29 ==
* 23:20 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'updated s5 pmtpa topology'
* 23:09 binasher: s5 pmtpa master changed to db73 - MASTER_LOG_FILE='db73-bin.000090', MASTER_LOG_POS=978211000
* 23:08 AaronSchulz: finished creating AccountAudit tables
* 23:06 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'Re-enable WikiLove, bug fix is deployed'
* 23:04 logmsgbot: catrope Finished syncing Wikimedia installation... :
* 22:40 logmsgbot: catrope Started syncing Wikimedia installation... :
* 22:28 anomie: (Including backport of gerrit change 61494 in the scap)
* 22:27 RoanKattouw: Running scap
* 22:23 RoanKattouw: Running scap again
* 22:19 logmsgbot: catrope Started syncing Wikimedia installation... :
* 22:13 hashar: gallium: restarted apache2 to kill off http connections established by mod_proxy to the jenkins backend.
* 22:03 hashar: jenkins web server is no more responding on port 8080. Might have to restart it.
* 22:00 hashar: gallium / jenkins : blackholed a web drawer using ip route add blackhole <IP>/32
* 21:54 hashar: Jenkins is in trouble
* 21:52 logmsgbot: reedy synchronized php-1.22wmf3/includes/DefaultSettings.php 'Fix wgVersion'
* 21:31 notpeter: upgrading db45 to precise
* 21:24 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enabling VisualEditor on viwiki and test2wiki'
* 20:31 AaronSchulz: Doing another original/math/timeline sync/copy run
* 20:28 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Enabled multiwrites for ceph/swift'
* 19:19 logmsgbot: kaldari synchronized php-1.22wmf2/extensions/Echo/maintenance/setEmailOptionTemp.php
* 19:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf3
* 19:06 logmsgbot: reedy Finished syncing Wikimedia installation... : Scap again to see if that fixes ROOTPAGENAME and ROOTPAGENAMEE. Otherwise I'm reverting it
* 18:46 logmsgbot: reedy Started syncing Wikimedia installation... : Scap again to see if that fixes ROOTPAGENAME and ROOTPAGENAMEE. Otherwise I'm reverting it
* 18:37 logmsgbot: reedy synchronized php-1.22wmf2/.git/modules/extensions/WikiLove
* 18:21 logmsgbot: reedy synchronized php-1.22wmf3/extensions/
* 18:06 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf3 and rebuild l10n cache
* 18:03 notpeter: deploying new frontend squid conf
* 17:51 LeslieCarr: restarting icinga-wm
* 17:35 cmjohnson1: removing asw-c1 for replacement
* 17:30 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf3 and rebuild l10n cache
* 17:09 mutante: fix NRPE checks on ms-be2, running as "weird user id" bug
* 17:05 mutante: removing reinstalled ms-be2 from ssh_known_hosts with ssh-keygen and then making it world-readable again manually so deployers don't get errors :p
* 17:05 logmsgbot: csteipp synchronized php-1.22wmf2/includes/
* 17:05 logmsgbot: reedy synchronized live-1.5
* 17:02 mutante: attempt to fix NRPE checks on ms-be9, restart as correct user
* 16:49 logmsgbot: reedy synchronized docroot
* 16:44 logmsgbot: reedy synchronized php-1.22wmf3/ 'initial sync of 1.22wmf3 files'
* 03:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 29 03:25:35 UTC 2013
* 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Mon Apr 29 02:12:33 UTC 2013
== April 28 ==
* 13:14 hashar: restarting Zuul, it has a finished build stuck in the `postmerge` pipeline.
* 02:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 28 02:48:51 UTC 2013
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sun Apr 28 02:12:10 UTC 2013
== April 27 ==
* 03:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 27 03:06:47 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sat Apr 27 02:13:53 UTC 2013
* 00:08 mutante: make SSL on Bugzilla also use SSLCACertificatePath and c_rehash certs on kaulen
== April 26 ==
* 23:58 mutante: upgrading mysql packages on marmontel (blog)
* 23:53 mutante: make SSL on the Blog use SSLCACertificatePath and c_rehash certs on marmontel
* 20:11 hashar: killed a node process on gallium. Probably a daemon that has gone standalone when jenkins died
* 19:19 hashar: jenkins died again sorry :( But this time I most probably found out the reason why it is slowed down: a web spider attempt to crawl all our builds!
* 18:04 mwalker: Upgraded fundraising civicrm install from a3db312836b6ac004694f831071b1c7d00560fbf to a65fbcec0dbb5bb79a97aca220a998e56d08b067
* 15:56 hashar: jenkins: enabled ExtraSettings.php loading . That enables a bunch of MediaWiki settings for all mediawiki related jobs, namely displaying errors, moving the temp dir to tmpfs, showing exceptions details and enabling development warnings.
* 13:49 mark: Rebooting hooft
* 13:48 mark: dist upgrade of hooft to precise
* 13:48 mark: Moved udpmcast from hooft to nescio
* 13:16 ^demon: gerrit replication freaked out, restarting
* 06:38 hashar: Restarted Jenkins. It was stuck in some while loop doing tons of stat() on unexisting file :( Would be up in roughly 40 minutes :(
* 03:46 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 26 03:45:44 UTC 2013
* 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Fri Apr 26 02:12:38 UTC 2013
* 00:57 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'temporarily disabling WikiLove per bug 47457'
* 00:53 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor/ 'Update VisualEditor to master'
* 00:33 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Clean up VisualEditor config'
* 00:33 Tim: on fenari: fixed permissions on /etc/ssh/ssh_known_hosts
* 00:33 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Remove wmgVisualEditorHide'
== April 25 ==
* 23:09 logmsgbot: mwalker synchronized docroot/bits/favicon/ 'Pushing new favicons for incubator, wiktionary, wiktionary.en, wikibooks, wikinews, and wikispecies'
* 22:32 logmsgbot: mflaschen Finished syncing Wikimedia installation... :
* 22:22 ^demon: new ssl cert (gerrit.wikimedia.org) live on gerrit.
* 20:55 notpeter: reimaging db55
* 20:45 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 20:17 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 20:07 awight: payments cluster updated from 7f70ccd580d to fdf308c8f
* 19:12 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'updating for new schema number'
* 18:52 logmsgbot: reedy synchronized docroot
* 18:19 LeslieCarr: cleaning up old home directories on fenari
* 17:00 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Removing oldskin notice'
* 16:50 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable VisualEditor on 14 more Wikipedias'
* 16:38 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor/ 'Update VisualEditor to master'
* 16:20 hashar: Jenkins is busy reparsing all the build history for some reason. That slow down the continuous integration box by an order of magnitude :/ Should take roughly 40 minutes (no idea when it started)
* 16:09 hashar: Zuul git operations are slow again :/
* 14:57 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php 'rm debug hack'
* 14:56 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Turn mwsearch debug log back off, its not phps fault'
* 14:53 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Turn mwsearch debug log on'
* 14:52 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php 'Debugging'
* 11:56 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'I2f0990fa087a'
* 11:56 logmsgbot: krinkle synchronized docroot
* 03:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 25 03:39:44 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Thu Apr 25 02:14:20 UTC 2013
* 00:45 mutante: upgrading packages on manutius
* 00:30 mutante: DNS update - removing internproxy, removing tesla (minus mgmt)
== April 24 ==
* 23:04 logmsgbot: dzahn gracefulled all apaches
* 22:39 logmsgbot: dzahn gracefulled all apaches
* 22:39 LeslieCarr: upgrading mw's
* 22:38 LeslieCarr: replaced smokeping targets for sdtpa c3 and sdtpa d3
* 22:26 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'New location for icons - Ib5554c54729'
* 22:26 mutante: sync-apache, add coffee
* 22:24 logmsgbot: krinkle synchronized docroot/bits/apple-touch/ 'New location for icons - Ib5554c54729'
* 21:47 mutante: DNS update - remove survey, remove wikitech-old, remove argon
* 21:47 logmsgbot: demon synchronized php-1.22wmf2/extensions/Wikibase 'b4b5fb4046175ca29834a4669c253c97ac5ccd07'
* 21:38 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
* 21:34 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Turn mwsearch log back on'
* 21:33 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php
* 21:26 mutante: DNS update - readding voip
* 21:17 cmjohnson1: removing sq33 from dsh node groups and pybal cfg
* 20:32 RobH: racktables on hooper upgraded to recent stable
* 20:26 RobH: racktables flap again, working on it
* 20:16 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch
* 20:04 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch 'Deploying I48480cb6'
* 20:03 logmsgbot: demon synchronized php-1.22wmf2/includes/api/ApiQuerySearch.php 'Deploying Ib7abe8d5'
* 19:57 logmsgbot: aaron synchronized php-1.22wmf2/includes/filerepo/file/LocalFile.php 'deployed 0e42bdadd9349fe5b6119ff65322e6c52d813a6e'
* 19:17 mutante: deploying Bugzilla wikimedia extension changes for improved wikilinking
* 19:16 RobH: racktables going offline a moment for upgrade
* 18:45 logmsgbot: reedy synchronized wmf-config/ 'wikidata stuffs'
* 18:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non enwiki to 1.22wmf2
* 18:09 mutante: DNS update - removing unused remnants
* 18:09 sbernardin: replacing bad disk on db59
* 17:31 sbernardin: taking down ms-be2 for hw replacement
* 15:09 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/skins/MinervaTemplate.php 'bug 47597'
* 15:05 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/skins/MinervaTemplate.php 'bug 47597'
* 09:05 hashar: restarted Zuul. Upgraded to e403579 (latest master)
* 08:58 hashar: upgrading Zuul
* 06:04 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
* 03:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 24 03:42:12 UTC 2013
* 02:32 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 24 02:31:43 UTC 2013
* 02:19 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed Apr 24 02:18:59 UTC 2013
* 00:03 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Disable mobile sitenotice everywhere but testwiki'
* 00:03 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disable mobile sitenotice everywhere but testwiki'
== April 23 ==
* 23:38 mutante: cleaning up all unused Apache sites on singer, gzip, move to /root for backup
* 23:33 mutante: disabling old techblog apache site config on singer
* 23:32 logmsgbot: maxsem synchronized wmf-config/mobile.php
* 23:15 logmsgbot: mwalker synchronized docroot/bits/favicon/ 'Pushing wikidata, wikimania, wikiquote, and wikiversity favicons from Isarra'
* 23:12 logmsgbot: dzahn gracefulled all apaches
* 23:07 logmsgbot: maxsem synchronized wmf-config 'Mobile banners again, hopefully fixed'
* 23:05 mutante: sync-apache, add wikepdia.org and .com typo domains
* 22:28 logmsgbot: reedy synchronized php-1.22wmf1/extensions/Score
* 22:21 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score
* 22:12 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score/Score.body.php 'debugging'
* 22:05 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score/Score.body.php 'debugging'
* 22:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'rv debugging'
* 22:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Score'
* 21:59 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score/Score.body.php 'debugging'
* 21:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable score debug log'
* 21:40 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/60512/'
* 21:23 logmsgbot: maxsem synchronized wmf-config
* 21:17 logmsgbot: reedy synchronized php-1.22wmf1/extensions/Score
* 21:15 logmsgbot: maxsem synchronized wmf-config 'Mobile banners everywhere - https://gerrit.wikimedia.org/r/#/c/60362/'
* 21:14 logmsgbot: aaron synchronized php-1.22wmf2/includes/db/Database.php 'temp logging'
* 21:13 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/ 'Weekly mobile deployment'
* 21:12 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/ZeroRatedMobileAccess/ 'Weekly mobile deployment'
* 21:06 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/ZeroRatedMobileAccess/ 'Weekly mobile deployment'
* 21:05 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/ 'Weekly mobile deployment'
* 20:56 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Wikibase
* 19:36 logmsgbot: anomie synchronized php-1.22wmf2/extensions/Cite 'Update to master for bug 47291'
* 18:44 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Re-enable AFTv5 on enwiki'
* 17:49 logmsgbot: mlitn synchronized php-1.22wmf2/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 17:49 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 16:34 logmsgbot: reedy synchronized php-1.22wmf2/extensions/TorBlock
* 16:34 logmsgbot: reedy synchronized php-1.22wmf1/extensions/TorBlock
* 12:34 paravoid: reprepro python-statsd 2.0.1-1~wmf1 & voluptuous 0.6.1-1~wmf1 for precise-wikimedia
* 09:57 pp-pdf1: restarted all services
* 09:56 pp-pdf1: - alow numprocs to be configured via initenv.sh
* 09:16 pp-pdf1: restarted all services
* 09:12 pp-pdf1: restarted all services
* 09:10 pp-pdf1: restarted all services
* 08:46 logmsgbot: maxsem synchronized docroot/ 'https://gerrit.wikimedia.org/r/#/c/60375/'
* 08:41 logmsgbot: maxsem synchronized wmf-config 'Beta changes, no-op for production'
* 02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 23 02:25:25 UTC 2013
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 23 02:10:32 UTC 2013
* 02:07 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue Apr 23 02:06:54 UTC 2013
* 00:02 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'bookkeeping for pmtpa slaves (this time for real)'
* 00:01 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'bookkeeping for pmtpa slaves'
== April 22 ==
* 23:51 notpeter: reimaging db44
* 22:57 binasher: mysql query logs / ishmael migrated from db9 to db1001
* 21:41 LeslieCarr: upgrading even more mw servers in tampa
* 20:12 Reedy: Running namespaceDupes on kowikiversity
* 19:57 Reedy: Running namespaceDupes on zhwikibooks
* 19:55 Reedy: Running namespaceDupes on uzwiki
* 19:55 logmsgbot: reedy synchronized wmf-config/
* 19:50 logmsgbot: reedy synchronized images/
* 19:50 logmsgbot: reedy synchronized docroot
* 19:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 19:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Score extension on the cluster'
* 18:49 logmsgbot: catrope synchronized echowikis.dblist
* 18:44 LeslieCarr: sq33 is going to be decomissioned… long live…. nothing else in tampa
* 18:39 mutante: changing #mediawiki to #wikimedia-dev in wikibugs startup script (just in case, should not even be needed by puppetized ircecho)
* 18:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:26 LeslieCarr: still doing package upgrades on mw* machines
* 18:20 LeslieCarr: rebooting nonresponsive sq33
* 18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf2
* 17:37 LeslieCarr: dist-upgrading some mw machines, should be low/no impact
* 16:51 mutante: restarting wikibugs on mchenry
* 16:43 logmsgbot: mark synchronized wmf-config/InitialiseSettings.php
* 16:37 mutante: added abaso to wmf-deployment group on gerrit (new mediawiki deployer)
* 16:11 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'New caching everywhere'
* 16:02 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Wikibase
* 15:57 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php '$wgMFVaryResources on enwiki, https://gerrit.wikimedia.org/r/#/c/60286/'
* 14:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 02:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 22 02:14:58 UTC 2013
* 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 22 02:08:56 UTC 2013
* 02:06 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Mon Apr 22 02:05:30 UTC 2013
* 00:13 Reedy: Running namespaceDupes.php --fix via foreachwiki in screen session on terbium
== April 21 ==
* 02:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 21 02:37:21 UTC 2013
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sun Apr 21 02:16:56 UTC 2013
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sun Apr 21 02:10:46 UTC 2013
== April 20 ==
* 16:54 ottomata: resetarting apache2 on stafford in attempt to fix puppetmaster
* 02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 20 02:35:55 UTC 2013
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sat Apr 20 02:14:31 UTC 2013
* 02:09 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sat Apr 20 02:09:07 UTC 2013
* 00:09 logmsgbot: aaron synchronized php-1.22wmf1/includes/job/JobQueueRedis.php 'deployed c9e702e7f73f41f7bcd2deb045085689d5cb4229'
* 00:03 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/JobQueueRedis.php 'deployed d2e4fdfed03148fdd1a564925ce351bec6f9b935'
== April 19 ==
* 23:07 Coren: brief reboot of labstore3 -> test to see that everything starts up as expected.
* 20:39 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Setup redis job queue debug log'
* 18:36 Jeff_Green: increased php memory limit for fundraising civicrm instance
* 17:55 LeslieCarr: restarting etherpad on hooper
* 17:15 binasher: running enwiki.abuse_filter_log migration - drop afl_user index, added user_timestamp
* 16:10 LeslieCarr: flushed celsus varnish cache
* 06:31 Tim: deploying apache conf change for www.wikidata.org redirect (I7bb872fd)
* 03:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 19 03:32:40 UTC 2013
* 02:27 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Fri Apr 19 02:27:17 UTC 2013
* 02:26 logmsgbot: aaron cleared profiling data
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Fri Apr 19 02:16:42 UTC 2013
* 01:54 logmsgbot: aaron cleared profiling data
* 01:54 logmsgbot: aaron synchronized wmf-config/StartProfiler.php 'Removed cli profiling for now.'
* 00:29 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor 'Update VisualEditor to master'
* 00:29 logmsgbot: catrope synchronized php-1.22wmf1/extensions/VisualEditor 'Update VisualEditor to master'
== April 18 ==
* 23:40 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor 'Update VisualEditor to master'
* 23:40 logmsgbot: catrope synchronized php-1.22wmf1/extensions/VisualEditor 'Update VisualEditor to master'
* 23:38 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Deploy 5c5455c81bc8dee0ae2a0410655da639e2f84e87 ($wgVisualEditorParsoidProblemReportURL)'
* 23:25 logmsgbot: bsitu synchronized php-1.22wmf2/extensions/Echo/processEchoEmailBatch.php 'More echo db fix'
* 23:25 logmsgbot: bsitu synchronized php-1.22wmf2/extensions/Echo/includes/DbEchoBackend.php 'More echo db fix'
* 22:51 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Enable mobile sitenotice on testwiki'
* 22:51 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enable mobile sitenotice on testwiki'
* 22:48 notpeter: reimaging db35
* 22:46 logmsgbot: mwalker synchronized docroot/bits/favicon/wikisource.ico 'New and shiney ws.o favicon by Isarra'
* 22:45 logmsgbot: mwalker synchronized docroot/bits/favicon/mediawiki.ico 'New and shiney mw.o favicon by Isarra'
* 22:35 logmsgbot: reedy synchronized php-1.22wmf2/extensions/EducationProgram
* 22:35 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'adjusting weights on pmtpa slaves'
* 22:26 logmsgbot: reedy synchronized php-1.22wmf1/extensions/EducationProgram
* 22:15 logmsgbot: reedy synchronized php-1.22wmf1/extensions/EducationProgram
* 22:09 notpeter: correction: that was db33 out, not db31
* 22:03 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'db31 and db35 out, db72 and db73 in'
* 21:56 logmsgbot: bsitu synchronized php-1.22wmf2/extensions/Echo/includes/EchoDbFactory.php 'Db cache fix'
* 21:41 logmsgbot: reedy synchronized php-1.22wmf2/extensions/EducationProgram
* 21:39 logmsgbot: reedy synchronized php-1.22wmf1/extensions/EducationProgram
* 21:38 mutante: merging changes on bugzilla to now support RT in "See also" field
* 20:45 logmsgbot: mflaschen synchronized php-1.22wmf2/extensions/GettingStarted/ 'Sync GettingStarted 1.22wmf2 for E3 deploy'
* 20:43 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/ 'Sync GettingStarted for E3 deploy'
* 20:36 logmsgbot: reedy synchronized wmf-config/
* 20:15 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 19:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgMaxImageArea and wgMaxAnimatedGifArea to 50MP'
* 19:52 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 19:14 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Echo email config + db cluster update to testwiki'
* 19:13 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Echo email config + db cluster update to testwiki'
* 18:57 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/jobs 'deployed 1e3eaed6fbe7fd91c15399cc17eac8602b34c9cd'
* 18:28 logmsgbot: reedy Finished syncing Wikimedia installation... : rebuild l10n cache for wikidata deploy
* 17:57 logmsgbot: reedy Started syncing Wikimedia installation... : rebuild l10n cache for wikidata deploy
* 17:50 RobH: authdns0update for git.w.o
* 17:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.22wmf2
* 17:06 logmsgbot: reedy synchronized php-1.22wmf2/extensions/
* 16:58 logmsgbot: reedy synchronized php-1.22wmf1/extensions/Wikibase '5d496b4ea2c8a68b9e1280a282530f7ce5232b68'
* 16:47 logmsgbot: ariel synchronized wmf-config/throttle.php
* 16:10 logmsgbot: mlitn synchronized php-1.22wmf2/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 16:10 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 15:39 cmjohnson1: authdns update
* 15:10 mark: Reversed replication of volume fr_archive, now nas1001-a:fr_archive -> nas1-a:fr_archive
* 14:26 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/59818/'
* 14:24 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/59818/'
* 14:14 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
* 14:06 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/59818/'
* 14:04 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/59818/'
* 13:24 logmsgbot: maxsem synchronized wmf-config/mobile.php
* 13:03 mark: Rebooting cp1041
* 08:48 logmsgbot: maxsem synchronized wmf-config/mobile.php
* 07:57 logmsgbot: olivneh synchronized php-1.22wmf2/extensions/ConfirmEdit 'Updating extension/ConfirmEdit (Bug 46132 / Change I8ee3fd136)'
* 07:57 logmsgbot: olivneh synchronized php-1.22wmf1/extensions/ConfirmEdit 'Updating extension/ConfirmEdit (Bug 46132 / Change I8ee3fd136)'
* 07:43 hashar: jenkins raised max # of executors from 6 to 8
* 07:07 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/59767/'
* 05:59 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
* 05:21 logmsgbot: tstarling synchronized php-1.22wmf1/includes/specials/SpecialSearch.php
* 05:15 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php
* 05:12 logmsgbot: tstarling synchronized php-1.22wmf1/extensions/MWSearch/MWSearch_body.php
* 05:09 logmsgbot: tstarling synchronized php-1.22wmf1/includes/specials/SpecialSearch.php
* 03:57 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/MWSearch
* 03:55 logmsgbot: tstarling synchronized php-1.22wmf2/includes/HttpFunctions.php
* 03:48 logmsgbot: tstarling synchronized php-1.22wmf1/extensions/MWSearch
* 03:46 logmsgbot: tstarling synchronized php-1.22wmf1/includes/HttpFunctions.php
* 03:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 18 03:26:48 UTC 2013
* 02:24 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 18 02:23:54 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Thu Apr 18 02:14:26 UTC 2013
* 01:03 mutante: re-enabling Tomasz' RT account
* 00:42 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'setting db1004 to full weight'
* 00:37 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1004 at a warmup weight'
* 00:09 logmsgbot: demon synchronized docroot
== April 17 ==
* 23:39 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1004'
* 23:21 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/ContributionReporting 'Finished update of CN and CR to latest'
* 23:20 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/CentralNotice/
* 23:14 hashar: If jenkins/zuul goes wild overnight: connect to gallium, killoff puppet in crontab then /etc/init.d/zuul stop and drop me an email. Will take of it whenever I wake up.
* 23:12 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/ContributionReporting/
* 23:12 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralNotice 'Beginning update of CentralNotice and ContributionTracking to latest'
* 22:59 hashar: restarting Zuul
* 22:20 hashar: Jenkins repointing all jobs to use the new Zuul repo ( /srv/ssd/zuul/git ) 
* 22:06 hashar: jenkins restarted. Restarting Zuul.
* 21:20 hashar: stopped Zuul. Will wait a bit till jenkins start up
* 21:19 hashar: restarted Jenkins
* 21:19 hashar: restarted Zuul
* 21:16 hashar: finished migrating zuul git repositories ( rsync -av /var/lib/zuul/git /srv/ssd/zuul )
* 21:09 hashar: Zuul ETA: it is copying MobileFrontend extension right now so half way done
* 21:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionaries to 1.22wmf2
* 21:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiquote to 1.22wmf2
* 21:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikiversity to 1.22wmf2
* 21:00 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki back to 1.22wmf1
* 20:59 hashar: stopping Jenkins that somehow restarted
* 20:59 logmsgbot: reedy synchronized php-1.22wmf2/includes/db/DatabaseMysql.php 'moar'
* 20:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 20:55 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews.dblist and special wikis to 1.22wmf2
* 20:51 logmsgbot: reedy synchronized php-1.22wmf2/includes/db/DatabaseMysql.php 'debugging'
* 20:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage, wikimania and wikimedia to 1.22wmf2
* 20:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Some wikis to create some noise
* 20:42 logmsgbot: reedy synchronized php-1.22wmf2/includes/db/DatabaseMysql.php 'debugging'
* 20:30 hashar: Switching zuul dir to /srv/ssd/zuul ( and )
* 20:18 hashar: stopping Zuul gracefully
* 20:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Revert that due to large number of warnings...
* 20:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: fishbowl, private and special to 1.22wmf2
* 19:57 hashar: gallium slowness is partly due to the ssd install. The git repositories are on the slow raid where as the workspaces are on the ssd. That means `git clone` can't do hardlink anymore :(
* 19:43 binasher: enwiki, dewiki, wikidata are fully on mariadb 5.5.30
* 19:33 hashar: jenkins: reducing number of executors from 10 to 6.
* 19:32 hashar: jenkins gallium has a huge queue of jobs. investigating.
* 19:31 hashar: jenkins / gallium has a huge queue of jobs. Investigating.
* 19:19 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 's1/s5 writeable from pmtpa'
* 19:17 logmsgbot: asher synchronized wmf-config/db-eqiad.php 's1/5 writeable'
* 19:13 logmsgbot: asher synchronized wmf-config/db-eqiad.php 's1/5 read-only'
* 19:12 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'preparing for master swaps'
* 19:03 logmsgbot: aaron synchronized php-1.22wmf2/maintenance/doMaintenance.php 'deployed 1437b57bd475e5ecb06800f9f64b14ccffd0dedf'
* 19:00 logmsgbot: aaron synchronized php-1.22wmf2/maintenance/doMaintenance.php 'deployed 1306792ad24be5b8d17355c58da655eb88c2fcc6'
* 18:27 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1050'
* 18:18 LeslieCarr: installing new linecard in cr2-eqiad , .1% chance of issues
* 18:16 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1050'
* 17:50 logmsgbot: aaron synchronized php-1.22wmf2/includes/User.php 'deployed 6a66776fd7afdf883ad6f82270453fc76833508b'
* 16:20 ^demon: running puppet on manganese (will restart gerrit)
* 13:05 apergos: restarted icinga on neon (the usual: Could not create external command file '/var/lib/nagios/rw/nagios.cmd' as named pipe)
* 13:03 hashar: jenkins: refreshing job mediawiki-core-phpcs-HEAD (points to a wrong git repo path)
* 09:50 mark: Inserted libvpx 1.1.0-1+wmf1 packages into the precise-wikimedia APT repository
* 06:57 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 17 06:57:02 UTC 2013
* 06:51 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed Apr 17 06:51:28 UTC 2013
* 06:48 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 17 06:47:37 UTC 2013
* 03:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 17 03:13:56 UTC 2013
* 02:20 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed Apr 17 02:19:57 UTC 2013
* 02:10 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 17 02:10:22 UTC 2013
* 00:38 mutante: blog: enabled new eventlogging by ori, removed old logging class, converted plugins/WMBlog from svn to git and sync with gerrit
* 00:31 Tim: on hume: changed UID for l10nupdate to be 10002, for consistency with puppet admins.pp
== April 16 ==
* 23:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'raising db1058 weight'
* 23:57 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1005, adding db1058 to s5'
* 23:55 logmsgbot: aaron cleared profiling data
* 23:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 23:48 logmsgbot: reedy synchronized wmf-config/abusefilter.php
* 23:46 logmsgbot: aaron synchronized wmf-config/StartProfiler.php 'Enabled 1:1 profiling for cli scripts and put "cli" in the profile ID'
* 23:22 logmsgbot: awjrichards synchronized php-1.22wmf2/extensions/MobileFrontend/
* 23:22 logmsgbot: awjrichards synchronized php-1.22wmf1/extensions/MobileFrontend/
* 23:00 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/ContributionReporting/ 'Contribution reporting now pulls 2013 data'
* 22:59 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/ContributionReporting/
* 22:51 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 16 22:51:08 UTC 2013
* 22:37 Tim: running l10nupdate manually on fenari to test change 58911
* 22:03 logmsgbot: anomie synchronized php-1.22wmf2/extensions/WikimediaMaintenance 'Update WikimediaMaintenance to master for l10nupdate fix'
* 22:02 logmsgbot: anomie synchronized php-1.22wmf1/extensions/WikimediaMaintenance 'Update WikimediaMaintenance to master for l10nupdate fix'
* 21:49 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/59449'
* 21:46 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/ZeroRatedMobileAccess/ 'https://gerrit.wikimedia.org/r/#/c/59525/'
* 21:43 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:13 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 19:54 logmsgbot: pgehres synchronized php-1.22wmf2/extensions/CentralAuth/maintenance/migrateAccount.php 'Updating CentralAuth/migrateAccount.php'
* 19:53 logmsgbot: pgehres synchronized php-1.22wmf1/extensions/CentralAuth/maintenance/migrateAccount.php 'Updating CentralAuth/migrateAccount.php'
* 19:38 logmsgbot: pgehres synchronized php-1.22wmf2/extensions/CentralAuth/maintenance/migrateAccount.php 'Updating CentralAuth/migrateAccount.php'
* 19:33 logmsgbot: pgehres synchronized php-1.22wmf1/extensions/CentralAuth/maintenance/migrateAccount.php 'Updating CentralAuth/migrateAccount.php'
* 19:18 binasher: upgrading db1005 to precise + mariadb
* 19:07 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1005 from s5 for upgrade'
* 19:02 binasher: streaming s5 hotbackup from db45 to db1058
* 18:49 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1058 from s1'
* 18:34 logmsgbot: aaron synchronized php-1.22wmf2/includes/cache/HTMLCacheUpdate.php 'deployed 344ddf85b6d7b79c40a1b6b9ec7e0a1e05c0705b'
* 18:26 logmsgbot: mlitn synchronized php-1.22wmf2/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 18:26 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 18:17 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Turning on footer contact link for en.wiki'
* 18:13 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable email bundling for test2wiki and mediawikiwiki'
* 18:13 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Enable email bundling for test2wiki and mediawikiwiki'
* 16:42 logmsgbot: aaron synchronized php-1.22wmf2/includes/WikiPage.php 'deployed 38d60cdafe036625a6ec8fa3a9e8a94ef450c85c'
* 16:41 logmsgbot: aaron synchronized php-1.22wmf2/includes/db/Database.php 'deployed fa819f8e3579855f6d95552655de98f92ec1789c'
* 16:40 logmsgbot: aaron synchronized php-1.22wmf2/includes/cache/BacklinkCache.php 'deployed 1a94124385df4c452c68fadaa74bc2714a204761'
* 16:37 logmsgbot: aaron synchronized php-1.22wmf2/includes/OutputPage.php 'deployed 997ab92458946d10fed96ff4ba41c74eeaee0d20'
* 05:04 Tim: moving most cron jobs from hume to terbium
* 03:23 legoktm: test
* 02:35 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue Apr 16 02:34:59 UTC 2013
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 16 02:16:58 UTC 2013
== April 15 ==
* 23:57 logmsgbot: csteipp synchronized php-1.22wmf2/includes/ 'User gerrit versions of security fixes'
* 23:18 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/CentralNotice
* 23:13 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralNotice/ 'CentralNotice banner= fix'
* 22:31 logmsgbot: dzahn gracefulled all apaches
* 22:16 andrewbogott: Moved labs-morebots to tools project.
* 21:43 LeslieCarr: doing a few more random host upgrades of misc hosts
* 21:32 LeslieCarr: upgrading mw1070-mw1079
* 21:24 LeslieCarr: upgrading mw1060-mw1069
* 21:22 RobH: linode wikitech is now shut down \o/
* 21:13 logmsgbot: dzahn gracefulled all apaches
* 20:32 mutante: restarting pdns on ns1
* 20:30 mutante: DNS update - add wikpedia.org
* 19:37 logmsgbot: ariel gracefulled all apaches
* 18:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf2
* 15:54 logmsgbot: reedy Finished syncing Wikimedia installation... : Build localisation cache for 1.22wmf2
* 15:19 logmsgbot: reedy Started syncing Wikimedia installation... : Build localisation cache for 1.22wmf2
* 14:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.22wmf2, scap to come
* 14:56 logmsgbot: reedy synchronized docroot
* 14:55 logmsgbot: reedy synchronized w
* 14:54 logmsgbot: reedy synchronized wmf-config/
* 14:49 logmsgbot: reedy synchronized php-1.22wmf2/ 'Initial sync of 1.22wmf2'
* 12:24 mark: Depooled amssq47-amssq62 frontend squids
* 12:04 mark: Depooled knsq16-knsq22 frontend squids
* 09:20 hashar: jenkins died :(
* 08:22 hashar: updating Jenkins jobs to fetch ZUUL_COMMIT instead of ZUUL_BRANCH. That would solve a race condition between jobs ( / ).
* 05:50 logmsgbot: olivneh synchronized wmf-config/throttle.php 'Adding throttling exception for itwiki GLAM event which starts in four hours (Bug 47221, change Id9ae600ac)'
* 02:04 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 15 02:03:51 UTC 2013
== April 14 ==
* 02:04 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sun Apr 14 02:04:17 UTC 2013
== April 13 ==
* 23:54 Coren: rebooting labstore3. autofs is being silly
* 15:11 Coren: Fixed boot problem on labstore3; back to operating.
* 04:54 Coren: rebooted labstore3 again; kernel and lvm2 upgrade
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sat Apr 13 02:12:27 UTC 2013
* 01:10 Coren: rebooted labstore3 again -> we hates lockd. We hates it!
* 00:41 Coren: rebooting labstore3 -> silly lockd module
== April 12 ==
* 23:31 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Add 'Contact Wikipedia' footer link on test/test2 (Iab1f5f527)'
* 23:31 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Add 'Contact Wikipedia' footer link on test/test2 (Iab1f5f527)'
* 23:14 Coren: rebooted labstore3 -> hung MegaCLI processes
* 22:41 notpeter: temp depooling search1015
* 20:53 mutante: install package upgrades on sanger
* 19:57 Ryan_Lane: finished opendj upgrade on virt0
* 19:23 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Disable mwsearch log'
* 18:54 mutante: restarting parsoid on wtp1004
* 18:25 logmsgbot: aaron synchronized wmf-config/jobqueue-pmtpa.php 'Set pmtpa queue config to match eqiad'
* 18:17 logmsgbot: demon synchronized php-1.22wmf1/extensions/MWSearch/MWSearch_body.php 'Moar logging'
* 18:16 logmsgbot: demon synchronized php-1.21wmf12/extensions/MWSearch/MWSearch_body.php 'Moar logging'
* 18:00 Ryan_Lane: upgrade opendj on virt0
* 16:00 LeslieCarr: adding another link to asw-c-eqiad's uplink
* 15:05 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Debug group for mwsearch'
* 15:02 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Debug group for mwsearch'
* 15:01 hashar: Changing Jenkins workspace directories from ${ITEM_ROOTDIR}/workspace to /srv/ssd/jenkins/workspace/${ITEM_FULLNAME} . My change has not been taken in account early on :(
* 14:54 hashar: jenkins is back up. Starting Zuul
* 14:44 cmjohnson1: analytics1007 has never worked..but working on it
* 14:21 hashar: restarting jenkins
* 14:03 hashar: gallium: created primary partition on /dev/sdb . formatting using: mkfs.xfs /dev/sdb1
* 13:36 hashar: gallium can't find the SSD, shut downing, will attempt another slot
* 12:59 hashar: gracefully shut-downing Jenkins and Zuul for scheduled maintenance. Will shutdown server gallium just after.
* 09:39 hashar: jenkins: updated mediawiki-core-whitespaces job to use ZUUL_COMMIT as a refspec specifier (for )
* 02:15 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Fri Apr 12 02:15:30 UTC 2013
* 00:08 mutante: DNS update - point old bugzilla3 entry over to actual bugzilla server
== April 11 ==
* 23:35 LeslieCarr: upgrading mw1052 to mw1060
* 23:30 logmsgbot: csteipp synchronized php-1.22wmf1/extensions/RSS/
* 23:30 LeslieCarr: upgrading mw1051
* 23:24 logmsgbot: csteipp synchronized php-1.22wmf1/includes/
* 23:22 mutante: shutting down xenon
* 23:20 LeslieCarr: upgrading mw1050
* 23:11 LeslieCarr: upgrading 1040 to 1049
* 23:04 mutante: install package upgrades on neon
* 23:00 LeslieCarr: upgrading mw1030 - 1039
* 22:51 K4-713: synchronized payments to 7f70ccd580d64
* 22:47 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralAuth 'Peter's log line change to CentralAuth'
* 22:41 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php 'Enable translate on CentralNotice'
* 22:39 logmsgbot: mwalker synchronized docroot/bits/favicon/wikipedia.ico 'Bi resolution wiki favicon'
* 22:38 LeslieCarr: upgrading mw1120 to mw1129
* 22:37 mutante: removing spence from dsh group mediawiki-installation
* 22:33 logmsgbot: mwalker synchronized php-1.21wmf12/extensions/CentralNotice
* 22:26 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralNotice 'Updating CentralNotice to the latest and greatest (from 68f0262be838320ecc9c87eb4485c28b0699bc1f)'
* 22:22 LeslieCarr: upgrading mw1110 to mw1119
* 21:41 logmsgbot: kaldari synchronized php-1.22wmf1/extensions/Echo 'syncing Echo for bugfix'
* 21:35 ottomata: emery back up and upgraded to precise (thanks mutante!)
* 21:32 ^demon: fenari: php-1.21wmf12 and php-1.22wmf1 both rebased to remove silly merge commits.
* 21:14 mutante: emery - revoking old puppet cert, resiging, initial puppet run
* 21:04 logmsgbot: olivneh synchronized php-1.22wmf1/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'Remove jquery.clickTracking from AFT dependencies'
* 21:01 logmsgbot: olivneh synchronized php-1.22wmf1/extensions/GettingStarted/SpecialGettingStarted.php
* 20:53 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/ 'Update GettingStarted extension for E3 deployment'
* 20:34 logmsgbot: mflaschen synchronized wmf-config 'Config change for GettingStarted'
* 20:32 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 20:11 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 19:58 mutante: emery - upgrading from lucid to precise
* 19:46 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'syncing InitialiseSettings to turn off VectorSectionEditLinks on testwiki'
* 19:41 Jeff_Green: running authdns-update to propagate frdata cname
* 19:31 AaronSchulz: Refreshed the job queue aggregator hash table
* 19:31 mutante: running puppet on brewster, testing lucid image on emery, rebooting again
* 19:17 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Turn Echo back on for testwiki'
* 19:16 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Add checkDelay to Echo email bundle jobs'
* 19:07 AaronSchulz: All jobs finished copying into redis
* 19:07 sbernardin: taking down ms-be9 for hw replacement
* 19:02 LeslieCarr: uninstalling nagios from spence
* 18:51 mutante: PXE booting emery into installer
* 18:48 ottomata: emery going down for upgrade to precise
* 18:21 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Turn Echo off on testwiki'
* 17:35 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Moved remaining jobs to redis'
* 17:24 sbernardin: taking down db11 for decommissioning
* 17:19 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 11 17:19:14 UTC 2013
* 17:15 cmjohnson1: europium power disconnected during routine data center work
* 13:14 logmsgbot: reedy synchronized wmf-config/ 'Disable ClickTracking :D'
* 13:08 mark: Inserted python-extras packages into the precise-wikimedia APT repository (backport by hashar)
* 10:10 hashar: Jenkins is back up
* 09:23 hashar: restarting Jenkins to get the git verbose mode :(
* 08:44 hashar: jenkins: stop git cloning mw/core over and over by reusing existing clone . That should help with I/O starvation on gallium.
* 08:40 Nikerabbit: translation memory database rebuild finished
* 08:17 Nikerabbit: emptying Translate solr instance and starting rebuilds
* 08:15 logmsgbot: nikerabbit synchronized php-1.22wmf1/extensions/Translate/ 'to master'
* 07:11 logmsgbot: hashar synchronized docroot/noc/conf/highlight.php 'Revert highlight.php recent modifications '
* 07:02 hashar: Put a live hack on fenari for 
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 11 02:16:35 UTC 2013
* 01:10 Ryan_Lane: disabling puppet on labstore3/4 - making sure gluster won't die without them and we're going to rebuild them
== April 10 ==
* 23:31 LeslieCarr: deactivated XO ipv6 transit
* 22:25 Ryan_Lane: added live hack to OpenStackManager on wikitech, ensuring that DynamicSidebar support is only enabled for logged-in users
* 22:24 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'moved refreshLinks to redis.'
* 22:20 ^demon|sick: restarting gerrit, was hung
* 22:15 Ryan_Lane: enable DynamicSidebar support in OpenStackManager
* 22:14 Ryan_Lane: temporarily disabling DynamicSidebar support in OpenStackManager
* 21:59 notpeter: gracefulling apache on stafford
* 21:36 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Moved htmlCacheUpdate jobs to redis'
* 21:19 ottomata: added python-statsd to apt repo for hashar
* 21:18 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php
* 21:07 ottomata: added python-voluptuous to apt repo (for hashar)
* 20:46 ottomata: added python-jsonschema to wikimedia apt repo: ori-l :)
* 20:00 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Moved async upload jobs to redis'
* 19:13 notpeter: running puppetd --enable on all nodes via salt
* 19:09 mwalker: eqiad frack is now using clustered memcache and has site local configuration for DB, ActiveMQ, MemCacheD shtuff
* 18:24 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Enabled use of redis for null test jobs'
* 18:06 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Switched all remaning wikis to 1.22wmf1
* 17:45 RobH: authdns-update to rename db1012 to stat1002
* 17:29 logmsgbot: aaron synchronized php-1.22wmf1/maintenance/copyJobQueue.php 'deployed 54f74111901ddebdc69f26847e3904140b3723d5'
* 17:24 LeslieCarr: restart unresponsive cp1037
* 17:14 RobH: authdns-update
* 15:48 mark: Ran dist-upgrade (for varnish upgrade) on cp1021-1036
* 15:40 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Nostalgia ext'
* 15:40 logmsgbot: demon synchronized wmf-config/extension-list 'Nostalgia ext'
* 15:38 logmsgbot: demon synchronized php-1.22wmf1/extensions/Nostalgia
* 14:33 logmsgbot: maxsem synchronized wmf-config/mobile.php
* 14:19 logmsgbot: maxsem synchronized wmf-config 'Enable $wgMFVaryResources on test'
* 14:12 mark: Inserted varnish 3.0.3plus-rc1-wm10 packages into the APT repository
* 12:23 apergos: removed outright puppet.log on neon, was > 7gb and no room left on device
* 07:53 logmsgbot: krinkle synchronized php-1.22wmf1/resources/startup.js 'Ia54dd738b3ce0995fa'
* 05:57 Tim: on stafford: swapdeath repeat narrowly averted via killall ruby && /etc/init.d/apache2 stop
* 05:34 Tim: on brewster: proxy down due to full root partition, again. Will fix it properly this time.
* 05:28 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Add 'Zero' & 'Zero talk' namespaces to metawiki (If6b3ce5e4)'
* 04:53 Tim: stafford went into swapdeath. I killed all ruby processes a couple of times, and then eventually stopped apache2 for a while, while it recovered
* 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Apr 10 02:28:06 UTC 2013
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 10 02:17:19 UTC 2013
* 00:21 mutante: removing xenon and db1012 from icinga configs, running puppetstoredconfigclean.rb on them, restarting icinga
* 00:07 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'https://gerrit.wikimedia.org/r/#/c/58436/'
* 00:06 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'https://gerrit.wikimedia.org/r/#/c/58436/'
== April 9 ==
* 23:29 logmsgbot: tstarling synchronized php-1.22wmf1/extensions/Nostalgia
* 23:21 notpeter: running upgrades on db56 and rebooting
* 23:19 notpeter: running upgrades on db64 and rebooting
* 22:55 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend/
* 22:53 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/
* 22:36 logmsgbot: reedy synchronized wmf-config/
* 22:34 LeslieCarr: clearing a few varnish caches - fyi if db load is seen to increase
* 22:17 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:45 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 21:45 hashar: Zuul is slowly dequeuing a large number of jobs. There is currently rougly 30 minutes lag between change submission and actually processing of the job. No idea why but investigating.
* 19:56 Ryan_Lane: installed Nuke on wikitech
* 18:11 K4-713: updated payments anti-fraud rules
* 13:07 hashar: jenkins plugin upgrade is tracked in . Luckily git plugin has been fixed just in time ;-D Just need to migrate the config history.
* 12:37 hashar: jenkins plugins seems to have been upgraded overnight :(
* 10:26 mark: Deployed experimental varnish build (3.0.3plus~rc1-wm10) on dysprosium and cp1022
* 08:54 hashar: Jenkins tests for mediawiki/core are broken () due to a faulty change that has been merged in master branch. Root cause is the gating job not testing the proper change :/
* 08:43 Nikerabbit: Got rid of local changes and undeployed commit related to noc
* 08:41 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Narayam and WebFonts'
* 05:42 Tim: on mw1149: enabled apache access log
* 02:31 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr 9 02:30:59 UTC 2013
* 02:19 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 9 02:18:53 UTC 2013
* 00:26 ^demon: jenkins is back. java gc took awhile.
* 00:19 ^demon: jenkins is hung, looking into it.
* 00:16 awight: payments cluster updated from 22943dc28 to d3954768e4dd
== April 8 ==
* 23:16 logmsgbot: catrope synchronized php-1.22wmf1/extensions/WikiEditor 'Deploying 17a90bc74e87341199a50ebe5348576e05be028a'
* 23:15 logmsgbot: catrope synchronized php-1.21wmf12/extensions/WikiEditor 'Deploying 17a90bc74e87341199a50ebe5348576e05be028a'
* 22:55 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Changing from emery to vanadium'
* 22:13 logmsgbot: demon synchronized php-1.21wmf12/includes/AutoLoader.php 'Backported Poolcounter convenience class'
* 22:13 logmsgbot: demon synchronized php-1.21wmf12/includes/PoolCounter.php 'Backported Poolcounter convenience class'
* 21:59 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-pmtpa.php 'Enable poolcounter for search'
* 21:59 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-eqiad.php 'Enable poolcounter for search'
* 21:42 logmsgbot: demon synchronized php-1.21wmf12/extensions/MWSearch/MWSearch_body.php 'Deploying Poolcounter updates'
* 21:17 notpeter: temp stopping slave on db55
* 20:53 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Bumped file journal ttl to a year'
* 20:18 AaronSchulz: Moved ceph sync scripts to terbium and started a second pass
* 20:15 logmsgbot: aaron synchronized php-1.22wmf1/extensions/FlaggedRevs 'deployed c61688baef682b7d97068fefdcc399256786a387'
* 19:08 ^demon: gerrit back
* 19:07 ^demon: gerrit restarting, config change to fix idle timeouts.
* 19:00 ottomata: removing some unused filters from emery's udp2log instance
* 18:34 Jeff_Green: deployed 1.22 on payments
* 18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf1
* 17:09 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1058 to s1 at full weight'
* 17:05 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1058 to s1 at a low warmup rate'
* 11:57 ^demon: gerrit back, deployed 2.6-rc0-154-gfcdb34b which contains a temporary fix for the stream-events timeout issues. See bug 46917 for info.
* 11:53 ^demon: bringing gerrit down for urgent update
* 10:31 hashar: gerrit: created operations/debs/libvpx for j^ . Initialized pristine-tar and upstream branches using empty commits and force push.
* 07:48 hashar: restarting Zuul for demo purposes :-)
* 04:00 Tim: on brewster: root partition was full, removed squid access.log and store.log and started squid
* 03:55 Tim: on all apaches: upgrading libpoppler
* 02:25 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 8 02:25:07 UTC 2013
* 02:18 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 8 02:18:18 UTC 2013
== April 7 ==
* 22:46 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 22:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 22:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 22:31 logmsgbot: reedy synchronized wmf-config/ 'Kill readerfeedback config, it aint coming back'
* 22:26 logmsgbot: reedy synchronized wmf-config/
* 22:20 logmsgbot: reedy synchronized php-1.22wmf1/extensions/CentralAuth
* 11:35 ^demon: restarting gerrit, stream-events seized up again.
* 07:26 hashar: gallium: restarted Zuul just to be sure. Gerrit on manganese is stuck again and need to be restarted 
* 02:17 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sun Apr 7 02:17:09 UTC 2013
* 02:11 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sun Apr 7 02:11:21 UTC 2013
== April 6 ==
* 12:27 ^demon: also restarted zuul
* 12:19 ^demon: restarting gerrit again, stream-events halted like friday.
* 02:23 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sat Apr 6 02:22:48 UTC 2013
* 02:12 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sat Apr 6 02:11:56 UTC 2013
* 01:55 binasher: iwlinks iwl_prefix_from_title migration completed for all projects
* 00:52 K4-713: adjusted anti-fraud rules on payments cluster
== April 5 ==
* 23:11 binasher: running iwlinks index migration on s4 (commons)
* 21:27 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1021 to full weight, pulling db1023'
* 21:13 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1021 at a low weight'
* 20:05 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1021 for upgrade'
* 19:53 binasher: streaming backup of db1056 to db1050 (converting s1 snapshot slave to file_per_table)
* 18:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php 's1: moving watchlist to db1049, pulling db1043, adding db1056'
* 18:50 Jeff_Green: increased jvm heap size on silicon (payments activemq)
* 18:42 notpeter: restarting gerrit
* 18:22 logmsgbot: reedy synchronized php-1.21wmf12 'Killing active users in 1.21wmf12'
* 09:39 hashar: Ariel kindly restarted gerrit on manganese which unbroken the stream-events process in Zuul. It again receives events.
* 09:33 hashar: restarted Zuul
* 09:23 Nemo_bis: varnish esams purging still broken, e.g. https://upload.wikimedia.org/wikipedia/commons/f/fe/Wikipedia-logo-v2-km.png
* 08:58 hashar: for Jenkins / Gerrit issue see 
* 08:48 hashar: Jenkins no more receiving ANY events from Gerrit. Effect: no jobs are triggered.
* 02:23 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Fri Apr 5 02:23:11 UTC 2013
* 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Fri Apr 5 02:12:34 UTC 2013
* 00:49 logmsgbot: csteipp synchronized php-1.21wmf12/includes/
* 00:46 logmsgbot: csteipp synchronized php-1.22wmf1/includes/
* 00:30 Tim: on fenari: testing a new version of mw-update-l10n which doesn't use stdout of mergeMessagesFileList.php
* 00:20 logmsgbot: tstarling synchronized wmf-config/ExtensionMessages-1.22wmf1.php
* 00:19 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to master
* 00:07 Tim: all 1.22wmf1 wikis briefly showed error messages due to display_errors being enabled when mergeMessageFileList.php generated ExtensionMessages-1.22wmf1.php. Fixed temporarily.
* 00:04 logmsgbot: tstarling synchronized wmf-config/ExtensionMessages-1.22wmf1.php
* 00:00 logmsgbot: mflaschen synchronized wmf-config/extension-list 'Remove E3Experiments and LastModified'
== April 4 ==
* 23:51 logmsgbot: demon synchronized wmf-config/extension-list 'Remove e3 since it was deleted'
* 23:49 logmsgbot: mflaschen synchronized wmf-config/CommonSettings.php
* 23:46 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043'
* 23:46 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to master
* 23:45 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043'
* 23:44 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043'
* 23:25 binasher: shutting down mysql on db1043, preparing to reboot
* 23:24 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043'
* 23:17 mutante: wikinews.com now redirects to wikinews.org
* 23:11 mutante: DNS update - adding wikinews.com
* 23:08 logmsgbot: dzahn gracefulled all apaches
* 22:52 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Re-enable Echo on testwiki'
* 22:49 mutante: running updateSpecialPages on cswikinews per request
* 22:14 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Disable Echo on testwiki'
* 22:10 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Bump Echo EventLogging schema for new notification types.'
* 22:02 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GuidedTour/ 'GuidedTour 1.22wmf1 deployment for E3'
* 22:02 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/ 'GettingStarted 1.22wmf1 deployment for E3'
* 22:01 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GuidedTour/ 'GuidedTour 1.21wmf12 deployment for E3'
* 22:00 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GettingStarted/ 'GettingStarted deployment for E3'
* 20:21 LeslieCarr: stopped cp3001 varnish
* 20:18 LeslieCarr: restarting varnish on cp3001
* 20:15 LeslieCarr: restarted udpmcast on hooft
* 20:13 mutante: puppetstoredconfigclean.rb xenon.eqiad.wmnet - Killing xenon ...
* 19:34 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/57525/'
* 19:29 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/57525/'
* 19:20 binasher: enabled 1:100 full query logging on db1042/1043, disabled puppet [planning to revert in 4-8 hours]
* 19:09 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/57528'
* 19:01 binasher: resuming iwlinks migrations on s3
* 17:31 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 17:31 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 17:27 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Updating ArticleFeedbackv5 frwiki config'
* 16:16 paravoid: upgrading image & video scalers for libav
* 15:55 ottomata: now generating webstatscollector dumps on gadolinium. snapshot1 syncs dumps from there over to dataset2 for dumps.wikimedia.org
* 15:38 K4-713: adjusted anti-fraud rules on the payments cluster
* 13:26 ^demon: running jgit gc on all repos
* 05:43 Ryan_Lane: clearing squid logs and restarting squid on brewster
* 04:16 Ryan_Lane: opendj upgrade on virt1000 complete
* 03:56 Ryan_Lane: upgrading opendj on virt1000
* 02:21 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Thu Apr 4 02:21:01 UTC 2013
* 02:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 4 02:13:31 UTC 2013
== April 3 ==
* 23:19 mwalker: Updating CentralNotice schema on testwiki & metawiki with patch-centralnotice-2_3.sql
* 22:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 21:45 binasher: started iwlinks index migrations on all wikis [gerrit 43389]
* 21:32 cmjohnson1: Running DNS update
* 20:34 ottomata: udpated webstatscollector package in apt repo to 0.1-3
* 18:54 mutante: DNS update - also adding softwarewikipedia.org
* 18:49 RoanKattouw: Removed caesium and xenon from /home/wikipedia/common/docroot/noc/pybal/eqiad/parsoid
* 18:43 mutante: restarting pdns on ns1
* 18:42 mutante: DNS update - adding softwarewikipedia.com for RT-4355
* 18:41 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikibooks to 1.22wmf1
* 18:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary to 1.22wmf1
* 18:35 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikisource to 1.22wmf1
* 18:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikivoyage to 1.22wmf1
* 18:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikimania and wikimedia wikis to 1.22wmf1
* 18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: private, closed and fishbowl to 1.22wmf1
* 17:14 logmsgbot: aaron synchronized php-1.22wmf1/includes/objectcache/SqlBagOStuff.php 'deployed b61053ca4b554f6bd18fb6408967839cdb5ccde2'
* 17:12 logmsgbot: aaron synchronized php-1.21wmf12/includes/objectcache/SqlBagOStuff.php 'deployed 61587acc64cb62400ff7978271c54e8bd8b1f02d'
* 16:54 logmsgbot: reedy synchronized php-1.21wmf12/includes/Collation.php
* 16:34 cmjohnson1: authdns update
* 14:22 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php 'setting weight on db1028 to 400'
* 13:15 mark: Stopped and started MySQL on pc1003 after finding mysql deadlocked
* 13:10 logmsgbot: reedy synchronized wmf-config/
* 12:20 mark: Rebooting cp3003 for RAID reconfiguration
* 10:43 mark: Deactivated AS6908 peering on cr2-knams
* 06:09 ^demon: gerrit: finished doing jgit gc on mediawiki/core. Repo size 3G -> 323M. Fresh clone time to <5m to localhost, <1.5m to other hosts inside wmf. I rock. Backup's in /home/demon/core.git in case something goes wrong. Bed time.
* 03:05 LeslieCarr: the final upload varnish is on 2gig instead of 1gig ! win!
* 02:32 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 3 02:31:44 UTC 2013
* 02:19 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Apr 3 02:19:30 UTC 2013
* 01:05 LeslieCarr: applying a big change to varnish interface groups - risk higher than normal
* 00:05 binasher: removed sda3/varnish.persist on cp1021, restarted varnish
* 00:00 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/PostEdit
== April 2 ==
* 23:46 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php '(Ia2665c4fe) Enable PostEdit on bn, br, ca, cs, et, ka and zh wikis'
* 23:46 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php '(Ibc2633f1c) : 10000 => 5000.'
* 23:25 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad'
* 23:11 logmsgbot: anomie synchronized php-1.22wmf1/includes/api/ApiQueryRecentChanges.php 'Fix for API list=recentchanges rccontinue'
* 23:10 logmsgbot: anomie synchronized php-1.21wmf12/includes/api/ApiQueryRecentChanges.php 'Fix for API list=recentchanges rccontinue'
* 23:01 logmsgbot: maxsem synchronized php-1.21wmf12/resources/Resources.php 'https://gerrit.wikimedia.org/r/#/c/57212/'
* 22:49 logmsgbot: maxsem synchronized php-1.21wmf12/resources/Resources.php 'https://gerrit.wikimedia.org/r/57208'
* 22:39 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/56696/'
* 22:34 awight: payments cluster updated from 12bc36c238 to 0ab980b8b
* 22:24 logmsgbot: maxsem synchronized php-1.21wmf12/includes/resourceloader/ResourceLoaderStartUpModule.php
* 22:20 LeslieCarr: authdns-update for cp1039 and cp1040 move
* 22:14 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend 'touch'
* 22:11 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend
* 22:08 logmsgbot: maxsem Finished syncing Wikimedia installation... :
* 21:42 logmsgbot: maxsem Started syncing Wikimedia installation... :
* 21:38 logmsgbot: reedy synchronized wmf-config/
* 21:25 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 20:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1028'
* 20:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 20:41 logmsgbot: maxsem synchronized php-1.21wmf12/includes/OutputPage.php 'https://gerrit.wikimedia.org/r/#/c/49071/'
* 20:34 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/57078/'
* 20:11 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'moving s1 watchlist to db1052, db1043 to full weight'
* 20:09 Ryan_Lane: added marc to the ops ldap group
* 20:01 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1001'
* 19:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 19:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:56 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php 'adding db1028 back'
* 18:54 notpeter: actually starting innopack from db65 to db1054 for pre-labsdb db (sanitarium)
* 18:48 notpeter: starting innopack from db31 to db1054 for pre-labsdb db (sanitarium)
* 18:39 binasher: completed image img_media_mime migration on all projects
* 18:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 17:42 LeslieCarr: restarted varnishncsa-locke on cp1023
* 17:41 LeslieCarr: restarted varnishncsa-multicast_relay on cp1028
* 17:34 cmjohnson1: correction db1028
* 17:34 cmjohnson1: removing disk 0 from db1048 to replace
* 17:28 Jeff_Green: disable fundraising banner log rotation on locke
* 17:21 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php 'adding db1001 back to production removing db1028'
* 17:04 cmjohnson1: setting weight to 100 on db1001
* 15:42 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Message for ancient skin users'
* 15:41 logmsgbot: anomie synchronized php-1.21wmf12/cache/l10n/l10n_cache-en.cdb
* 15:34 logmsgbot: anomie synchronized php-1.21wmf12/extensions/WikimediaMessages/WikimediaTemporaryMessages.i18n.php
* 15:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 2 15:12:45 UTC 2013
* 15:09 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr 2 15:08:37 UTC 2013
* 14:54 logmsgbot: demon synchronized php-1.21wmf12/extensions/WikimediaMessages/WikimediaTemporaryMessages.i18n.php '8th time is the charm'
* 14:51 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'I hate l10nupdate'
* 14:50 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Notice for users of disabled skins'
* 14:24 paravoid: deploying squid config, diverting all of upload to swift
* 04:09 binasher: increased 'wikiadmin'@'10.64.%' max connections to 200 on s1 
* 02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 2 02:17:33 UTC 2013
* 02:10 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr 2 02:10:24 UTC 2013
* 01:15 Ryan_Lane: removing labstore1 and labstore2 entries from projectstorage.wmnet rr dns entry in preparation for shrinking volumes
* 00:44 logmsgbot: reedy synchronized wmf-config/
* 00:43 binasher: now running the image img_media_mime migration on commons (the big one)
* 00:41 mutante: restarting lucene on all pool4 servers (one by one)
* 00:36 mutante: creating search index for transitionteamwiki
* 00:30 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
* 00:14 logmsgbot: reedy synchronized php-1.22wmf1/cache/interwiki.cdb 'Updating 1.22wmf1 interwiki cache'
* 00:10 logmsgbot: reedy synchronized php-1.21wmf12/cache/interwiki.cdb 'Updating 1.21wmf12 interwiki cache'
* 00:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
== April 1 ==
* 23:55 LeslieCarr: search1024 is our guinea pig of using sfp-t's and is connected to asw2-a5-eqiad ge 0/0/18
* 23:48 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
* 23:40 mutante: creating swift containers for private wiki
* 23:39 logmsgbot: dzahn synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
* 23:37 mutante: dumpInterwiki.php to update interwiki cache
* 23:36 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic to pmtpa temporarily'
* 23:33 mutante: adding transitionteamwiki to private.dblist
* 23:25 mutante: running addWiki.php for transitionteamwiki - Access denied for user ...
* 23:21 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
* 23:14 cmjohnson1: swapping disk1 (slot1) db1001
* 23:06 mutante: creating upload dir on upload7 for transitionteamwiki
* 22:57 binasher: beginning image.img_media_mime scehma migrations, starting with s1
* 22:52 logmsgbot: aaron synchronized php-1.22wmf1/includes/job 'deployed ec8f42daf19f386d9af2eece6dfc5f8fa6fc42f3'
* 22:26 logmsgbot: aaron synchronized php-1.22wmf1/maintenance 'deployed c7832c8956d79892be93452e5f4d6b52df1d3bf0'
* 22:26 logmsgbot: aaron synchronized php-1.22wmf1/includes/job 'deployed c7832c8956d79892be93452e5f4d6b52df1d3bf0'
* 22:26 mutante: recreating transitionteam wiki docroot from skel-1.5 and syncing docroot
* 22:26 logmsgbot: root synchronized docroot
* 21:09 LeslieCarr: rebooting lvs1001
* 21:04 logmsgbot: demon synchronized php-1.22wmf1/cache/l10n 'Manually syncing 1.22wmf1 l10ncache'
* 20:56 LeslieCarr: dist-upgrading lvs1001
* 20:48 LeslieCarr: failing over lvs from lvs1001 to lvs1004
* 20:44 LeslieCarr: CORRECTION restarting sshd on lvs1001
* 20:44 LeslieCarr: restarting lvs1001
* 20:21 logmsgbot: demon synchronized php-1.21wmf12/cache/l10n 'Manually syncing 1.21wmf12 l10ncache'
* 20:20 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 1 20:20:29 UTC 2013
* 20:00 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 1 19:59:44 UTC 2013
* 19:54 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 1 19:54:21 UTC 2013
* 19:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki
* 19:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki and testwiki to 1.22wmf1
* 19:11 logmsgbot: reedy synchronized php-1.22wmf1/extensions/ 'Wikidata extensions to same as 1.21wmf12'
* 19:04 logmsgbot: reedy synchronized php-1.22wmf1/includes/
* 18:56 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf1 and rebuild l10n cache
* 18:49 LeslieCarr: testing logbot
* 18:20 Reedy: Ran scap-recompile to build texvc
* 18:20 jeremyb_: re!logged from earlier today while morebots was gone (all times UTC)
* 18:20 jeremyb_: 17:56:29 <logmsgbot> !log reedy synchronized w
* 18:20 jeremyb_: 17:55:45 <logmsgbot> !log reedy synchronized docroot
* 18:20 jeremyb_: 17:55:04 <logmsgbot> !log reedy synchronized php-1.22wmf1 'Initial sync of php-1.22wmf1'
* 18:19 jeremyb_: 17:27:13 <logmsgbot> !log reedy synchronized wmf-config/ExtensionMessages-1.22wmf1.php
* 18:19 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf1 and rebuild l10n cache
* 18:19 jeremyb_: 17:14:31 <Jeff_Green> !log moved aside williams:/opt/otrs-home/.spamassassin/auto-whitelist and restarted spamd to test whether the bloated AWL is resulting in poor filtering
* 18:19 jeremyb_: 14:39:19 <andrewbogott> !log restarted glusterfs-server on labstore1 and labstore2, because of read errors in public/datasets/public in bots project
* 18:19 jeremyb_: 13:55:28 <ottomata> !log added webstatscollector-0.1-2 to apt repo
* 18:18 jeremyb_: 11:54:15 <paravoid> !log repooling ms-fe1; staggered depool/restart/pool of ms-fe2-4
* 18:17 Ryan_Lane: restarted morebots. wikitech-static had 100% / partition. Have also fixed that and its cause.
* 18:16 jeremyb_: 07:45:24 <paravoid> !log depooling ms-fe1 for staging
* 18:16 jeremyb_: 03:58:05 <logmsgbot> !log asher synchronized wmf-config/db-eqiad.php 'returning db1051'
* 18:16 jeremyb_: 03:44:43 <logmsgbot> !log asher synchronized wmf-config/db-eqiad.php 'pulling db1051'
* 18:16 jeremyb_: 03:22:54 <binasher> !log reinstated max_user_connections = 80 for wikiadmin on s1
* 18:15 cmjohnson1: restarted morebots
* 03:23 binasher: reinstated max_user_connections = 80 for wikiadmin on s1
* 02:12 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 1 02:12:24 UTC 2013
== March 31 ==
* 02:10 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sun Mar 31 02:10:18 UTC 2013
== March 30 ==
* 02:14 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sat Mar 30 02:13:49 UTC 2013
* 00:01 LeslieCarr: rebuilding mailman archives …. very slowly
== March 29 ==
* 23:51 LeslieCarr: stopping mailman to remove sensitive message
* 22:38 LeslieCarr: upgrading packages on aluminium
* 22:25 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
* 21:02 hashar: jenkins: raised the number of execution for deployment-bastion from 2 to 4. Will let us execute more jobs in parallel
* 20:14 mutante: scheduling a 2-year downtime for db11 :p
* 19:33 logmsgbot: dzahn gracefulled all apaches
* 19:30 logmsgbot: dzahn synchronized ./docroot/transitionteam
* 19:29 mutante: creating transitionteam docroot
* 19:25 logmsgbot: aaron synchronized php-1.21wmf12/includes 'deployed fa5c0a6a82c9a1cb63f3cedd87e6c4aba59c994d'
* 18:49 sbernardin: hot-swapping disk on db29
* 18:04 ^demon: gerrit back up, now running 2.6-rc0-76-g52fb5ae
* 18:00 mutante: DNS update - adding transitionteam wiki
* 17:47 sbernardin: taking down db29 to replace bad hw
* 17:45 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Re-enabled async upload for commons'
* 17:23 ^demon: gerrit offline for an update, brb
* 09:25 logmsgbot: olivneh synchronized wmf-config/throttle.php 'Updating throttle rules for guwiki workshop (Bug 46686)'
* 04:16 logmsgbot: aaron synchronized php-1.21wmf12/maintenance/runJobs.php 'deployed 651884d4bdc76c29dd717dfdbbd698632223f3b5'
* 04:15 logmsgbot: aaron synchronized php-1.21wmf12/includes/job/JobQueueGroup.php 'deployed 651884d4bdc76c29dd717dfdbbd698632223f3b5 '
* 02:17 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Fri Mar 29 02:17:06 UTC 2013
* 01:24 ori-l: Stopping EventLogging daemons to allow Puppet to change 'eventlogging' user's home dir
* 00:11 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/MoodBar 'Updating MoodBar to remove ClickTracking integration'
* 00:03 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GuidedTour/GuidedTour.php 'Small bug fix to GuidedTour; removing unneeded dependency. https://gerrit.wikimedia.org/r/#/c/56546/1/GuidedTour.php'
* 00:00 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GuidedTour/GuidedTour.php 'Small bug fix to GuidedTour; removing unneeded dependency. https://gerrit.wikimedia.org/r/#/c/56546/1/GuidedTour.php'
== March 28 ==
* 23:24 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Remove E3Experiments and LastModified (2/2)'
* 23:24 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Remove E3Experiments and LastModified (1/2)'
* 22:35 logmsgbot: khorn synchronized php-1.21wmf12/includes/cache/MessageCache.php 'Reverting changes in language fallback behavior'
* 22:34 logmsgbot: khorn synchronized php-1.21wmf12/languages/Language.php 'Reverting changes in language fallback behavior'
* 22:20 logmsgbot: spage Finished syncing Wikimedia installation... : E3: Updated GettingStarted and GuidedTour extensions
* 22:12 LeslieCarr: removed some log files from ms1004 - it was full
* 22:10 RobH: bugzilla admin stuff: noc@w.o is now a disabled account cuz im tired of folks who arent on the alias attempting to sign up with it
* 22:02 logmsgbot: spage Started syncing Wikimedia installation... : E3: Updated GettingStarted and GuidedTour extensions
* 21:46 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Fix mobileMode property name (was: 'mobile')'
* 21:36 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming
* 20:11 Jeff_Green: add hostname samarium.wikimedia.org and pdns-update
* 19:51 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1051'
* 19:44 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1051'
* 19:44 LeslieCarr: killed gmetad on neon
* 19:36 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/ArticleFeedback/populateAFRevisions.php 'Fix fatal error triggered by populateAFRevisions.php cronjob when AFT is not loaded.'
* 19:33 mutante: starting gmetad on neon again
* 19:24 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo 'syncing Echo on wmf12'
* 19:17 mutante: manually deleting db11 references from icinga configs and restarting it
* 19:12 mutante: pkill -u snmptt on neon, restarting snmptt deamon
* 19:10 binasher: replaced collector on professor with locally built version from I2bc117d5620b9e545a8e5a6cf3b654ee835b75d7 (testing fix for issues only under very high load)
* 19:10 mutante: stopping gmetad and ganglia-parser on neon
* 18:53 mutante: adding gjg (greg-g) to wmf ldap group
* 18:23 mutante: restarting lucene on pool4 search nodes to pickup new wikivoyage indices
* 18:12 ottomata: adding reportcard and analytics .wikimedia.org CNAMEs to stat1001
* 18:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 17:56 mutante: creating search indices for new wikivoyages
* 17:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 11:49 hashar: search19 and search20 issue with enwiki.prefix is in 
* 11:42 hashar: search19 and search20 have enwiki.prefix marked as FAILED. (see: curl --silent http://search19.pmtpa.wmnet:8123/status |grep FAILED and curl --silent http://search20.pmtpa.wmnet:8123/status |grep FAILED).
* 10:39 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/Translate/ 'Translate to master'
* 04:19 paravoid: updating jsduck in apt and upgrading it on gallium
* 03:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 02:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 02:19 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Thu Mar 28 02:19:24 UTC 2013
* 00:47 K4-713: synchronized payments cluster to 12bc36c2385d8e56792c
* 00:27 mutante: once again disabling notifications for db11
* 00:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
== March 27 ==
* 23:59 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
* 23:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 23:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 23:40 Ryan_Lane: finished updating wikitech-static
* 23:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 23:38 Ryan_Lane: updating wikitech-static to wmf/1.21wmf12
* 23:24 Ryan_Lane: finished upgrading wikitech
* 23:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 23:23 Ryan_Lane: upgrading wikitech to wmf/1.21wmf12
* 23:17 logmsgbot: reedy synchronized php-1.21wmf12/cache/interwiki.cdb 'Updating 1.21wmf12 interwiki cache'
* 23:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 23:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 23:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 22:50 logmsgbot: maxsem synchronized php-1.21wmf11/extensions/MobileFrontend 'touch'
* 22:47 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend 'touch'
* 22:24 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
* 22:22 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
* 22:05 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment, 2nd attempt
* 21:48 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment, 2nd attempt
* 21:12 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'changing s1 slave weights for testing'
* 20:10 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild localisation cache to fix he related ParserFunctions problems
* 19:56 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild localisation cache to fix he related ParserFunctions problems
* 19:43 logmsgbot: reedy synchronized php-1.21wmf12/extensions/ParserFunctions/
* 19:38 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for WikiData deployment
* 19:05 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for WikiData deployment
* 18:51 cmjohnson1: pushing mw1209-1220 into service
* 18:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:39 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
* 18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf12
* 18:23 mutante: installing package upgrades on neon (Icinga), incl. ganglia-monitor,gmeta
* 18:09 mutante: bunch of mw* hosts: fix puppet runs
* 18:08 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo 'syncing Echo for wmf12'
* 18:05 mutante: mw1130 - kill and fix puppet runs
* 17:41 Jeff_Green: authdns-update, pdns on ns1 croaked, restarted
* 15:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: be_x_oldwiki to 1.21wmf12
* 14:09 hashar: jenkins has been restarted by mistake :-( Job building will be unavailable for up to half an hour.
* 13:58 hashar: Jenkins: disabled the old Gerrit Trigger Plugin 
* 09:56 Nikerabbit: Testing Translate bug fixes on test.wikipedia.org
* 05:20 Tim: on fluorine: running mw-log-cleanup once to test Id2196e7b
* 05:02 Ryan_Lane: disabling Bayes-collector cron on sanger
* 04:55 Ryan_Lane: purging BayesLearning files older than 70 days on mchenry
* 02:45 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Wed Mar 27 02:44:56 UTC 2013
* 02:19 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Mar 27 02:19:11 UTC 2013
* 00:55 LeslieCarr: ms-be3 was crashed with stack traces for swift-container
* 00:54 LeslieCarr: rebooting ms-be3
* 00:38 awight: payments cluster updated from 4862fb8eef to f295d851c
* 00:34 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'db1052 to full weight'
* 00:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1001, adding db1052 at a warmup weight'
* 00:11 awight: payments cluster updated from fa2e2 to a06a03f05 wheee!
== March 26 ==
* 23:54 awight: payments cluster rolled back from e758102 to fa2e240b6e9
* 23:53 awight: payments cluster updated from fa2e240b6e9 to e758102
* 23:53 Tim: on all apaches: replacing /usr/local/apache/common-local/live-1.5 with a symlink to w
* 23:44 logmsgbot: tstarling synchronized w
* 23:35 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/56019/'
* 23:33 awight: payment cluster rollback, from 4862fb8 to fa2e240b
* 23:31 mutante: restarting nrpe on professor
* 23:29 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Workaround for buggy NavTiming implementation in IE9 (bug 46474)'
* 23:20 awight: payments cluster updated from fa2e240b to 4862fb8
* 23:10 mutante: db11 - disabling all notifications
* 23:07 mutante: mw13 - killing puppet and re-running
* 23:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 23:00 mutante: removing mysql packages from db1036 (broken dpkg, status "pi")
* 22:55 mutante: re-installing ganglia-monitor on mw1135, fix dpkg
* 22:54 logmsgbot: reedy synchronized docroot
* 22:52 mutante: starting udpprofile (carbon-cache.py) on professor
* 22:51 logmsgbot: reedy synchronized wmf-config/
* 22:49 mutante: search21 - remove sun-j2sdk , replaced by oracle-jdsdk
* 22:47 mutante: search21 - broken DPKG: iF sun-j2sdk1.6
* 22:45 mutante: mw2 - broken DPKG: pi librsvg2-2 , pi librsvg2-bin
* 22:39 LeslieCarr: did a service apache2 start on mw108*
* 22:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 22:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 22:10 notpeter: icinga mostly trustworthy again :)
* 22:08 andrewbogott: restarted nova-api on virt0 because it appeared to have crashed.
* 21:31 andrewbogott: restarted memcached on virt0 in a futile attempt to address https://bugzilla.wikimedia.org/show_bug.cgi?id=46583
* 21:30 mutante: killed stuck snmptt procs on neon to fix the others.. apparently :p
* 21:28 mutante: stopping gmetad on neon
* 20:52 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'syncing CommonSettings.php for Echo event tracking'
* 20:41 mutante: more degraded RAIDs: db1001->RT-4832, db1028->4834
* 20:35 mutante: broken disks/degraded RAID on db servers: db11->RT-4828, db29->RT-4829, db45->RT-4831
* 20:34 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/ContributionReporting 'syncing ContributionReporting extension'
* 20:07 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'raising db1051 to full weight'
* 19:49 logmsgbot: khorn synchronized php-1.21wmf12/extensions/CentralNotice 'Fixing CentralNotice bug in which it was possible to initialize multiple times'
* 19:46 logmsgbot: khorn synchronized php-1.21wmf11/extensions/CentralNotice 'Fixing CentralNotice bug in which it was possible to initialize multiple times'
* 19:40 logmsgbot: kaldari synchronized php-1.21wmf12/resources/startup.js 'syncing php-1.21wmf12/resources/startup.js'
* 19:33 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase/client
* 19:27 cmjohnson1: mw1209 re-imaging
* 19:20 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo/modules/icons 're-syncing Echo icons dir'
* 19:19 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Include AFTv5 on frwiki'
* 19:12 logmsgbot: mlitn synchronized wmf-config 'Enable AFTv5 on frwiki'
* 19:05 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Collection/
* 19:04 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 19:04 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 19:02 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo/modules/icons 'syncing Echo icons dir'
* 18:58 mutante: starting gmetad on neon - we failed to resolve data source name ms-fe1001.eqiad.wmnet
* 18:56 mutante: restarting ircecho on fenari
* 18:48 mutante: stopping gmetad on neon
* 18:44 mutante: killing all snmp related processes on neon and restarting
* 18:43 logmsgbot: reedy synchronized php-1.21wmf12/extensions/MobileFrontend/includes/specials/SpecialMobileWatchlist.php
* 18:41 logmsgbot: kaldari synchronized php-1.21wmf11/extensions/Echo/modules/icons 'syncing Echo icons dir'
* 18:39 mutante: restarting snmtrapd on neon
* 18:25 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Turning Thanks extension on on MediaWiki wiki'
* 18:13 logmsgbot: krinkle synchronized php-1.21wmf12/resources/mediawiki/mediawiki.js 'I80af730daa815 fixing bug 46575'
* 18:12 Krinkle: Change I34a488247 (mw.loader debug) has been reverted in wmf/1.21wmf12 and pulled on fenari
* 17:51 cmjohnson1: removing disk from slot 0 ms-be1012
* 17:14 Krinkle: Change I34a488247 is now merged in wmf/1.21wmf12 and pulled on fenari for test.wikipedia.org, do not sync.
* 17:04 ottomata: added (crappy, but better than nothing) .deb for webstatscollector to apt.wikimedia.org
* 15:24 cmjohnson1: swapping disk 6 ms-be1012
* 14:34 logmsgbot: hashar synchronized multiversion ' wikiversions.cdb now vary by realm'
* 14:10 cmjohnson1: running sync-common on mw1209
* 10:23 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/UniversalLanguageSelector/UniversalLanguageSelector.php 'Testing fix for wikidata'
* 09:28 pp-pdf1: updated mwlib to 0.15.4
* 08:47 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/Translate
* 08:46 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/UniversalLanguageSelector
* 07:27 mark: Restarted swift-proxy on ms-fe*
* 04:16 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
* 02:46 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Tue Mar 26 02:45:52 UTC 2013
* 02:21 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Mar 26 02:21:08 UTC 2013
* 01:41 mutante: analytics1026 - delete 16G udp2log.log to free disk
* 01:38 mutante: analytics1003 - gzipping jmxtrans logs to free some disk
* 01:36 mutante: analytics1003 - gzipping kafka.log to free some disk
* 00:39 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1051 at a low warmup weight'
== March 25 ==
* 23:36 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'done'
* 23:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'changing s1 weights, returning db1050'
* 23:27 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Quick profiling'
* 22:10 hashar: reloading Zuul
* 21:44 logmsgbot: khorn synchronized wmf-config/InitialiseSettings.php 'Re-enabling CentralNotice'
* 21:39 mutante: killing and restarting nagios-nrpe-server on '*'
* 21:32 logmsgbot: khorn synchronized php-1.21wmf12/extensions/CentralNotice/ 'CentralNotice jQuery fix'
* 21:28 logmsgbot: khorn synchronized php-1.21wmf11/extensions/CentralNotice/ 'CentralNotice jQuery fix'
* 21:12 mutante: killing and restarting nagios-nrpe-server on gadolinium
* 21:07 mutante: --last-lines 20 | RT ..would be nice sometimes:)
* 20:13 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
* 20:07 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Use ffmpeg for ogg thumbs as is done for webm'
* 19:55 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild l10n cache for wikibase
* 19:33 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild l10n cache for wikibase
* 18:57 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 18:46 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
* 18:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf12
* 18:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable CN everywhere'
* 18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rb
* 18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf12
* 17:25 logmsgbot: reedy synchronized wmf-config/
== March 24 ==
* 18:13 Ryan_Lane: restarting glusterd processes on labstore*
* 02:31 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sun Mar 24 02:31:22 UTC 2013
* 02:08 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sun Mar 24 02:08:00 UTC 2013
== March 23 ==
* 02:37 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sat Mar 23 02:36:53 UTC 2013
* 02:14 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sat Mar 23 02:13:44 UTC 2013
* 00:17 RoanKattouw: Deployed master of Parsoid onto wtp1004; the code is broken but the box is depooled for Gabriel's benchmarking work
* 00:15 RoanKattouw: That's me messing with wtp1004
== March 22 ==
* 23:33 ottomata: restarting hadoop setting yarn.nodemanager.resource.memory-mb to 16G
* 23:06 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
* 23:04 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
* 23:04 mutante: sync-wikiversions
* 22:29 logmsgbot: aaron synchronized php-1.21wmf12/extensions/Translate 'deployed 0aab5984415a94bd214088767ff1de33ab57c4e0'
* 22:12 mutante: killall nrpe via salt, then restart nagios-nrpe-server , runs as wrong user
* 22:09 mutante: kill and restart nrpe on oxygen
* 22:07 mutante: installing package upgrades on oxygen
* 21:40 mutante: killing nagios-nrpe on gallium and testing if puppet restores it properly
* 21:03 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Set all non-Wikipedias back to 1.21wmf12 again.
* 20:51 logmsgbot: aaron synchronized php-1.21wmf12/includes/Title.php 'deployed 0f4914bb4ff2d13cb124cad94dc5ded663123a18'
* 20:46 logmsgbot: aaron synchronized php-1.21wmf12/includes/WikiPage.php 'deployed e372b635ae6d6a589f6d87f14fe4c9452d7a1b4d'
* 20:38 logmsgbot: aaron synchronized php-1.21wmf12/includes/Title.php 'Reverted follow-up fixes too'
* 19:32 ottomata: removed unicast udp2logging to gadolinium; gadolinium will consume from multicast stream
* 18:44 K4-713: synchronized payments cluster to fa2e240b6e92440b
* 18:28 K4-713: synchronized payments cluster to df3c8dbc937c971
* 17:33 logmsgbot: aaron synchronized php-1.21wmf12/includes/WikiPage.php 'deployed 1862a57aef08432e8b1194a54dc179649f5cae63'
* 09:08 MaxSem: Recreating Solr index
* 02:52 Tim: on searchidx1001: started incremental indexer, apparently it died on March 21 at 02:08 when it ran at the same time as a cron job import
* 02:50 binasher: power cycling db1050, unresponsive on serial console
* 02:49 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1050, crashed'
* 02:47 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043'
* 02:42 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Fri Mar 22 02:42:32 UTC 2013
* 02:37 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043, oops'
* 02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Fri Mar 22 02:30:31 UTC 2013
* 01:28 ori-l: Finished correcting E3Experiments extension's entry point from 'Experiments.php' (which does not match the submodule name) to 'E3Experiments.php'. Updated extension-list, CommonSettings.php, and the extension itself in both wmf11 and 12.
* 01:24 logmsgbot: olivneh synchronized wmf-config/extension-list 'Remove temporary workaround for E3 entry-point migration (2/2)'
* 01:23 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Remove temporary workaround for E3 entry-point migration (1/2)'
* 01:18 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/E3Experiments 'Updating E3Experiment extensions's entry point'
* 01:18 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/E3Experiments 'Updating E3Experiment extensions's entry point'
* 01:11 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Temporarily accepting either E3Experiments.php or Experiments.php as extension entry point to facilitate clean-up'
* 00:55 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'NavigationTiming: log mobile mode'
* 00:16 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.21wmf12.php 'Remove PHP errors'
* 00:15 logmsgbot: aaron synchronized php-1.21wmf12/includes/Title.php 'deployed cef327e945e2593fe0291880a7d1976cc5a2f248'
* 00:09 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Thanks 'syncing Thanks ext'
== March 21 ==
* 23:24 logmsgbot: robh Finished syncing Wikimedia installation... :
* 23:17 logmsgbot: robh Started syncing Wikimedia installation... :
* 23:04 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo, PageTriage to master, Add Thanks
* 22:44 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo, PageTriage to master, Add Thanks
* 21:50 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Syncing change to CommonSettings for Thanks ext'
* 21:49 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Syncing change to InitialiseSettings for Thanks ext'
* 21:46 mutante: running puppet and reloading zuul on gallium. enables hashar's new shiny https://integration.wikimedia.org/zuul/
* 21:38 RobH: authdns-update for terbium
* 21:05 RoanKattouw: Added fenari back to /etc/dsh/groups/mediawiki-installation per https://rt.wikimedia.org/Ticket/Display.html?id=4794
* 21:02 logmsgbot: cmjohnson Finished syncing Wikimedia installation... :
* 20:58 binasher: pulling mw1070, may have network issues
* 20:57 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/PageTriage/modules/ext.pageTriage.views.list 'syncing PageTriage js on wmf12'
* 20:57 hashar: fenari disappeared from dsh group mediawiki-installation :-(
* 20:52 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 20:51 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 20:48 awight: payments cluster updated from 18fd11a4 to df3c8dbc9
* 20:43 logmsgbot: cmjohnson Started syncing Wikimedia installation... :
* 20:37 LeslieCarr: restarting gmond on mw1153
* 20:19 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 20:19 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming 'Enable NavigationTiming on mobile alpha and beta'
* 20:19 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 20:19 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'Enable NavigationTiming on mobile alpha and beta'
* 19:53 logmsgbot: mflaschen synchronized php-1.21wmf11/extensions/GuidedTour/ 'E3 deploy'
* 19:52 logmsgbot: mflaschen synchronized php-1.21wmf11/extensions/GettingStarted/ 'E3 deploy'
* 19:52 logmsgbot: mflaschen synchronized php-1.21wmf11/extensions/E3Experiments/ 'E3 deploy'
* 19:49 logmsgbot: mflaschen synchronized wmf-config/CommonSettings.php 'E3 deployment'
* 17:39 logmsgbot: csteipp rebuilt wikiversions.cdb and synchronized wikiversions files: Update test2wiki to 1.21wmf12
* 17:32 logmsgbot: csteipp synchronized php-1.21wmf12/includes/Title.php 'update for loadPageData()'
* 17:24 notpeter: running puppet on all lucid nodes via salt
* 17:23 mutante-away: purging wikispecies.net from squid
* 17:08 notpeter: install timeout package on all lucid nodes to get puppet back on track
* 12:49 mark: Rebooting palladium with hyperthreading disabled
* 11:27 mark: Rebooting niobium with hyperthreading disabled
* 09:29 hashar: created git repo operations/debs/python-statsd
* 09:15 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Add 'wikidata.org' to $wgCrossSiteAJAXdomains to allow CORS'
* 09:14 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Add 'wikidata.org' to $wgCrossSiteAJAXdomains to allow CORS'
* 06:34 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: 
* 06:34 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
* 06:32 Tim: reverting all wikis to 1.21wmf11 due to move page being totally broken (bug 46397)
* 06:32 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: 
* 06:32 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
* 01:10 logmsgbot: catrope synchronized php-1.21wmf12/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
* 01:10 logmsgbot: catrope synchronized php-1.21wmf12/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
* 01:10 logmsgbot: catrope synchronized php-1.21wmf11/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
* 01:10 logmsgbot: catrope synchronized php-1.21wmf11/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
== March 20 ==
* 23:57 RobH: yay peter!
* 23:56 notpeter: freeing up roughly 500G of ram across our cluster by calling puppet via cron :)
* 23:52 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning es100[69]'
* 23:52 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning es100[69]'
* 23:51 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s2 servers'
* 23:51 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s2 servers'
* 23:36 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling es100[69]'
* 23:36 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling es100[69]'
* 23:36 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s1 servers'
* 23:35 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s1 servers'
* 23:31 RobH: authdns-update for lanthanum
* 22:45 RobH: ignore mw1209+ they arent deployed
* 22:04 LeslieCarr: restarting ircecho on fenari to ensure logmsgbot is on operations and tech irc channels
* 22:03 ottomata: stopping puppet on gadolinium to test for packet loss weirdness
* 22:01 pgehres: pgehres synchronized php-1.21wmf12/includes/GlobalFunctions.php 'Fix wfWaitForSlaves() so the parameter actually works correctly'
* 22:01 logmsgbot: pgehres synchronized php-1.21wmf11/includes/GlobalFunctions.php 'Fix wfWaitForSlaves() so the parameter actually works correctly'
* 21:35 LeslieCarr: restarted ircecho on fenari
* 21:01 Krinkle: asher synchronized wmf-config/db-eqiad.php 'pulling db1049 from s1'
* 20:53 pgehres: pgehres synchronized php-1.21wmf11/extensions/CentralAuth 'Updating CentralAuth to master'
* 20:53 pgehres: pgehres synchronized php-1.21wmf12/extensions/CentralAuth 'Updating CentralAuth to master'
* 20:42 Reedy: reedy synchronized wmf-config/InitialiseSettings.php
* 19:59 mutante: DNS update - remove zuul CNAME
* 19:53 Reedy: wikibooks to 1.21wmf12
* 19:51 Reedy: sync'd wmf-config/InitialiseSettings.php
* 19:50 Reedy: wikiversity and wikiquote to 1.21wmf12
* 19:49 Reedy: wikivoyage and wiktionary to 1.21wmf12
* 19:47 Reedy: wikinews and wikisource to 1.21wmf12
* 19:47 Reedy: Synchroni[sz]ed php-1.21wmf12/extensions for Wikidata deploy
* 19:46 Reedy: special and wikimedia dblists to 1.21wmf12
* 19:46 Reedy: wikimania, private and fishbowl dblists to 1.21wmf12
* 19:45 Reedy: closed.dblist to 1.21wmf12
* 19:38 RobH: sync-common on mw23
* 19:36 mutante: restarting hung apache on mw1202
* 19:28 LeslieCarr: sync-common on mw1085
* 19:24 mutante: .tar.gz'ing /var/mwdocs on formey (SVN) and then manually delete it
* 19:18 LeslieCarr: upgrading mw118* and then restarting apache2 on these boxes
* 19:11 LeslieCarr: installing package upgrades on mw1170-1179
* 19:10 mutante: installing package upgrades on formey
* 18:50 mutante: apache-graceful-all, remove boards.wm redirect
* 16:03 ottomata: added python-voluptuous package to apt repo
* 15:12 mark: Fixed purging for all sites but eqiad (which wasn't broken)
* 07:56 hashar: gallium: reloaded apache2 to make sure recent changes are applied.
* 02:44 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Mar 20 02:44:03 UTC 2013
* 02:32 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Wed Mar 20 02:32:16 UTC 2013
== March 19 ==
* 23:52 K4-713: synchronized payments cluster to fe4fb96bea1378e
* 23:33 mutante: DNS update - add wikiartpedia.org
* 23:30 logmsgbot: dzahn gracefulled all apaches
* 23:08 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Re-enable AFTv5 on dewiki/testwiki'
* 22:38 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
* 22:36 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
* 22:33 ottomata: adding udp-filter_0.3.22 for precise to apt repo (lucid version already was there)
* 22:17 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
* 22:15 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
* 21:52 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 21:52 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 21:37 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 21:25 mutante: installing package upgrades on kaulen
* 21:24 mutante: installing package upgrades on sockpuppet
* 21:13 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 20:36 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5/SpecialArticleFeedbackv5.php 'Touched AFT Special page file'
* 20:35 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/SpecialArticleFeedbackv5.php 'Touched AFT Special page file'
* 19:42 ottomata1: added gadolinium as a udp2log host on frontend caches
* 19:14 logmsgbot: mlitn synchronized wmf-config/CommonSettings.php 're-include AFTv5'
* 19:08 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 19:07 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
* 19:03 paravoid: adding ruby-multi-json (universe), ruby-parallel, ruby-dimensions, ruby-execjs, ruby-jsduck (main) to apt
* 18:46 mutante: dist-upgrade on gallium
* 18:03 logmsgbot: robh synchronized docroot
* 18:01 logmsgbot: robh synchronized docroot
* 17:21 mark: Ended bits.esams performance testing
* 16:29 mark: Starting load testing with esams bits servers
* 16:20 mark: Rebooted cp3019 with hyperthreading and DAPC disabled
* 16:16 logmsgbot: krinkle synchronized extract2.php
* 14:29 mark: Rebooting arsenic with hyperthreading disabled
* 12:54 Tim: on professor: restarted collector 
* 08:46 hashar: gallium: killed stalled puppet agents. Locked on apt-get and update-java-alt
* 08:41 hashar: gallium: restarted puppet and zuul
* 08:38 hashar: gallium: stopped zuul and upgrading
* 08:37 hashar: gallium: stopped puppet to deploy Zuul whenever the queue is empty
* 08:36 hashar: zuul : giving results processing priority over enqueued jobs. Cherry picked 263fba9 from upstream. That should resolve Zuul being slow to report back to Gerrit 
* 03:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 03:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 03:12 Tim: started incremental updater on searchidx1001 and searchidx2, apparently has not been running since March 13
* 02:50 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Mar 19 02:50:19 UTC 2013
* 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Tue Mar 19 02:29:02 UTC 2013
* 01:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 00:22 Ryan_Lane: adding dns entries for labstore systems in eqiad
* 00:08 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning servers'
== March 18 ==
* 23:34 binasher: i <3 running "dpkg -P mysql-server-5.1 mysqlfb-client-5.1 mysqlfb-common mysqlfb-server-5.1 mysqlfb-server-core-5.1 libmysqlfbclient16 libmysqlclient16"
* 23:11 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db10[10-11],[26-28] at low weights'
* 22:45 binasher: converted rogue s3 tables (moodbar_feedback, trackbacks, transcache) to innodb
* 22:28 ottomata: added python-flask-login-0.1.2-1 to apt
* 22:23 logmsgbot: robh gracefulled all apaches
* 22:20 LeslieCarr: intradatacenter link is flapping , switching links, this may cause some higher latency
* 22:14 logmsgbot: robh gracefulled all apaches
* 22:11 logmsgbot: robh gracefulled all apaches
* 21:58 mutante: purging wikimediafoundation.org from squid
* 21:57 logmsgbot: dzahn gracefulled all apaches
* 21:56 logmsgbot: robh gracefulled all apaches
* 21:36 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling a db from s3-7 for upgrade'
* 21:30 paravoid: authdns-update: removing *.ts.wikimedia.org records
* 21:29 logmsgbot: dzahn gracefulled all apaches
* 21:25 binasher: ran "ddsh -g apaches -cM '/etc/init.d/apache2 start'"
* 21:24 RobH: authdns-update rt4683
* 21:22 RobH: gracefuling all apaches resulted in rendering cluster overload, they didnt restart apache, odd
* 21:21 logmsgbot: robh synchronized docroot
* 21:18 LeslieCarr: service apache2 start on the mw-eqiad group via dsh
* 21:14 binasher: started apache on all imagescalers
* 21:00 logmsgbot: robh gracefulled all apaches
* 21:00 RobH: troubleshooting erros in apache restart script
* 20:59 logmsgbot: robh gracefulled all apaches
* 20:36 RobH: authdns-update rt4694
* 20:20 RobH: authdns-update
* 20:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 20:00 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for 1.21wmf12
* 19:44 ^demon: jetty freaked out again, forcing a gerrit restart
* 19:32 binasher: upgraded all coredb mariadb replicas to 5.5.30
* 19:25 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for 1.21wmf12
* 19:08 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043 db1009'
* 19:04 logmsgbot: reedy synchronized docroot
* 19:03 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'db1050 for special s1'
* 18:56 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043, db1009 for upgrade'
* 18:40 logmsgbot: reedy synchronized php-1.21wmf12 'Initial sync of php-1.21wmf12'
* 18:16 awight: payments cluster updated from c19cc66 to fe4fb96b
* 18:06 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 18:03 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 'Removed old testing cruft'
* 16:19 cmjohnson1: removing/replacing disk 4 ms-be1004
* 16:01 hashar: Hard restarted Zuul which was dead locked again :-(
* 13:23 paravoid: depooled, upgraded, restarted and pooled again all ms-fe[1-4]
* 13:02 paravoid: depooling ms-fe1 for new swift version (patched 1.7.4)
* 11:27 apergos: restarted both the varnishncsas on niobium, they were giant again
* 02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Mon Mar 18 02:30:03 UTC 2013
== March 17 ==
* 02:31 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sun Mar 17 02:31:16 UTC 2013
== March 16 ==
* 14:58 logmsgbot: reedy synchronized php-1.21wmf11/extensions/EducationProgram
* 14:48 logmsgbot: reedy synchronized php-1.21wmf11/extensions/EducationProgram
* 14:40 logmsgbot: reedy synchronized php-1.21wmf11/extensions/EducationProgram
* 13:07 hashar: Mar 16, 2013 1:06:47 PM Jenkins is fully up and running
* 12:55 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/EducationProgram/includes/pages/EducationPage.php 'Fix fatal error in EducationProgram (Bug 46188)'
* 12:33 hashar: restarting Jenkins on a quiet day to have it using umask 0002 (aka g+w). That needs a full restart since the umask is set by /usr/bin/daemon See .
* 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sat Mar 16 02:29:19 UTC 2013
== March 15 ==
* 23:24 logmsgbot: laner synchronized wmf-config/CommonSettings.php 'Disabling aft5'
* 22:45 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning es1007,10 at lower weights'
* 22:35 binasher: rebooted es1007,1010 for kernel upgrades
* 22:33 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'temporarily pulling es1007, es1010'
* 22:23 Ryan_Lane: hid domain dropdown for account creation and login on wikitech
* 22:23 Ryan_Lane: changed wikitech configuration to require email address on account creation
* 22:23 Ryan_Lane: deployed change 54093 to wikitech
* 22:03 hashar: Zuul : opened a bug about this week slowness, breakage: https://bugzilla.wikimedia.org/show_bug.cgi?id=46176
* 21:51 logmsgbot: aaron synchronized php-1.21wmf11/includes 'deployed 08295adcdc4a4130954f52fb1eb5377638cd97bc'
* 21:36 binasher: db59 -> mariadb 5.5.30
* 21:30 hashar: restarted Zuul with the safe version ff79197
* 21:28 hashar: downgrading Zuul to safe version ff79197 (aka stop doing `git remote update` on ref-update events)
* 21:25 binasher: added new build of mariadb 5.5.30 to precise-wikimedia on brewster
* 21:20 hashar: hard restarted Zuul. It was locked down again :/
* 21:18 mutante: installing package upgrades on gallium
* 21:14 hashar: restarting Zuul, it is locked somehow. Definitely need to update it :(
* 21:09 RobH: professor services seem back up
* 20:51 RobH: professor shutting down for fan swap, short downtime (hopefully)
* 20:47 hashar: integration.mediawiki.org is not redirecting to the new integration.wikimedia.org (apache conf is disabled). Patch in 
* 20:33 mutante: puppet runs on neon broken due to mysql puppet issue - mysql-client-5.5 can not create alias mysql-client: object already exists at /var/lib/git/operations/puppet/manifests/mysql.pp:513
* 20:31 mutante: restarting pdns on ns0
* 20:30 mutante: DNS update - removing ipv4/ipv6/ipv6and4/results.labs.wm
* 19:37 mutante: maerlant - re-adding 2620:0:862:1::80:2 to eth0, starting nginx
* 19:22 sbernardin: taking down raskin for decommissioning
* 19:13 mutante: maerlant - dpkg: error processing ganglia-monitor | backing up /var/www/ to tridge /data/other then deleting contents | removing sites-enabled/ipv6and4 | nginx can't bind to [2620:0:862:1::80:2]:443
* 18:50 mutante: installing more package upgrades on maerlant (wikimedia-lvs-realserver, libc6, nginx...)
* 18:46 mutante: installing package upgrades on formey
* 18:44 mutante: fixing broken dpkg/APT on maerlant - dpkg --configure -a , re-run puppet and let it install package upgrades
* 18:34 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/
* 18:19 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/
* 18:19 notpeter: ran puppetd --enable on all mw hosts
* 18:15 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/
* 18:07 logmsgbot: mlitn Finished syncing Wikimedia installation... : Update ArticleFeedbackv5 to master
* 17:52 logmsgbot: mlitn Started syncing Wikimedia installation... : Update ArticleFeedbackv5 to master
* 17:36 hashar: jenkins: updated jenkins configuration to have its URL default to wikimedia.org domain instead of the legacy mediawiki.org
* 17:22 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/ 'Update ArticleFeedbackv5 to master'
* 17:21 hashar: gallium : reloading Zuul so it reports the status url with the wikimedia.org domain
* 17:19 hashar: integration website going down while we deploy a new layout.
* 17:03 hashar: gallium : restarted puppet
* 17:03 hashar: gallium : update /srv/ to latest version of integration/docroot.git ( aaa4b0c ) and moved all content from /srv/org/mediawiki/integration to the new /srv/org/wikimedia/integration
* 17:00 hashar: stopped puppet on gallium to fix some permissions before puppet changes are merged in and applied.
* 05:09 Tim: restarting squid on cp1010 for squid redirector test
* 05:06 Tim: testing new squid redirector
* 02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Fri Mar 15 02:30:02 UTC 2013
* 02:16 logmsgbot: reedy synchronized docroot
* 02:13 logmsgbot: dzahn gracefulled all apaches
* 01:53 logmsgbot: reedy synchronized wmf-config/
* 01:52 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 01:52 logmsgbot: reedy gracefulled all apaches
* 01:44 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/GettingStarted/SpecialGettingStarted.php 'Fix deployed bug in check for editability'
* 01:44 logmsgbot: reedy synchronized wmf-config/
* 01:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 01:35 logmsgbot: reedy synchronized wmf-config/
* 01:35 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 01:34 logmsgbot: reedy synchronized docroot
* 01:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 01:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 01:21 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/GettingStarted/CategoryRoulette.php 'Fix to use single-parameter version of SRANDMEMBER'
* 01:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 01:13 logmsgbot: dzahn gracefulled all apaches
* 01:12 logmsgbot: reedy synchronized docroot
* 01:10 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'Missed a file in earlier sync'
* 00:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'chapcomwiki readonly'
* 00:47 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Additional fields for NavTiming'
* 00:43 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deployment for Mobile and E3
* 00:17 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deployment for Mobile and E3
== March 14 ==
* 23:48 mutante: DNS update - add affcom in prep. to replace chapcom (bug 39482)
* 23:47 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Updating MF copyright logo path for enwiki'
* 23:47 logmsgbot: awjrichards synchronized php-1.21wmf11/extensions/MobileFrontend/ 'Updating MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-03-14'
* 23:44 mutante: installing package upgrades on sockpuppet
* 23:30 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Updating MF copyright logo for testwiki'
* 22:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 22:43 mutante: pa.us.wikimedia.org now redirects to pa-us.wikimedia.org (Wikimedia Pennsylvania) (no sub.sub domains)
* 22:41 logmsgbot: dzahn gracefulled all apaches
* 22:34 mutante: disabling notifications for puppet freshness on celsus
* 22:14 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling NavigationTiming at 1:10,000 sampling factor across cluster.'
* 22:02 logmsgbot: olivneh synchronized wmf-config 'Enable NavigationTiming on test2'
* 21:57 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'Update of extension to use EventLogging'
* 21:51 hashar: Gerrit took roughly 5 minutes to merge the change https://gerrit.wikimedia.org/r/#/c/53412/ ( mediawiki/core ) which means Zuul does nothing during that time waiting for the merge to happen.
* 21:48 hashar: apparently Gerrit takes ages to submit a change that in turn cause a hard lock on Zuul which is waiting for the merge to happen.
* 21:39 hashar: Zuul is hanging out not reporting changes :/
* 21:01 mutante: DNS update - add pa-us wiki to replace pa.us
* 20:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 20:50 cmjohnson1: authdns update (labsdb1002-3 vlan)
* 20:45 Ryan_Lane: deleting the jenkins user from ldap
* 20:36 mutante: DNS update - add zuul as CNAME for gallium
* 20:34 Ryan_Lane: installed WikiLove on wikitech
* 20:30 ori-l: Added article edit event to EventLogging, which upped events/sec from ~3.5 to ~30. Expect increased load on vanadium and db1047; bits caches unaffected.
* 20:20 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/EventLogging 'Log ['HTTP_HOST'] as webHost'
* 20:05 logmsgbot: olivneh synchronized php-1.21wmf11/includes/WikiPage.php 'Adds CategoryAfterPageAdded / CategoryAfterPageRemoved hooks'
* 19:32 logmsgbot: aaron synchronized php-1.21wmf11/includes/job/jobs 'deployed 6f76ede163cb114724bce7a1c1b8938e1e30606f '
* 18:49 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Enable wgEnableAsyncUploads only on testwikis for now'
* 18:44 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/ 'Update ArticleFeedbackv5 to master'
* 18:41 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1009 to full weight'
* 18:32 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1009 at a low weight for warmup'
* 18:12 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Enabled $wgEnableAsyncUploads on all wikis'
* 18:08 binasher: cleaned up myisam cruft (unused t���ables) on s2
* 17:49 logmsgbot: aaron synchronized wmf-config 'Enabled redis job aggregator fully'
* 17:48 cmjohnson1: labsdb1001 powering down
* 17:42 logmsgbot: aaron synchronized wmf-config 'Set redis job aggregator on for testwiki.'
* 17:27 logmsgbot: reedy synchronized php-1.21wmf11/extensions/Math/
* 17:24 binasher: shutting down mysql on db1009, upgrading to mariadb
* 16:59 cmjohnson1: powering down labsdb1002 and labsdb1003
* 16:59 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1009 for mariadb migration'
* 16:57 cmjohnson1: addin mw1041 back to dsh groups
* 16:26 logmsgbot: mlitn synchronized wmf-config 'Prepare AFTv5 config for deployment new features'
* 13:15 mark: Set OSPF cost to 10 on csw1-esams:ve7 to facilitate ip multipath
* 13:15 mark: Fixed OSPF on cr2-knams:xe-0/0/0 - csw1-esams:e8/2 (earlier)
* 12:17 hashar: gallium setting up /srv/ as a copy of integration/docroot.git
* 09:03 apergos: restarted varnishncsa.vanadium on niobium, it was using 9gb memory or so
* 02:25 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Thu Mar 14 02:25:19 UTC 2013
* 02:21 hashar: jenkins: made Zuul to block changes on operations/debs/ircecho whenever pep8/pyflakes fails.
* 01:35 mutante: running puppet on celsus to investigate puppet freshness reports from icinga - it started parsoid, but also celsus is a Wikimedia DECOMMISSIONED server (base::decommissioned).
* 01:10 logmsgbot: reedy synchronized php-1.21wmf11/cache/l10n/
* 01:00 logmsgbot: reedy synchronized php-1.21wmf11/extensions/Scribunto
* 00:53 logmsgbot: reedy synchronized wmf-config/
== March 13 ==
* 23:44 mutante: installing package upgrades on zirconium
* 23:41 logmsgbot: demon Finished syncing Wikimedia installation... :
* 23:38 Ryan_Lane: deploying change 53464 to OpenStackManager on wikitech
* 23:23 RobH: no one sync apaches, we wre doing crazy magic on them
* 23:17 logmsgbot: demon Started syncing Wikimedia installation... :
* 23:05 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad. upgrades over.'
* 23:00 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad. upgrades over.'
* 22:53 logmsgbot: reedy synchronized wmf-config/
* 22:44 logmsgbot: demon synchronized php-1.21wmf11/extensions/Scribunto/ 'Updating Scribunto to master'
* 22:30 logmsgbot: robh gracefulled all apaches
* 21:46 awight: payments cluster updated to 61d3f1f
* 21:43 pp-pdf1: upgraded mwlib to 0.15.3
* 21:38 Tim: on professor: restarted collector
* 21:22 RobH: repooling all ssl servers (except broken ssl3004) in all sites. new unified certificate is now in use.
* 21:15 RobH: depooling ssl servers 3 and 4 in each dc site
* 21:14 RobH: repooling the first two ssl servers per site
* 21:08 logmsgbot: py synchronized wmf-config/lucene-production.php 'temp moving all earch traffic to pmtpa for upgrades in eqiad'
* 21:05 RobH: setting first 2 ssl servers per dc site to false in pybal
* 20:21 Ryan_Lane: repooling ssl1001
* 20:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable collection'
* 20:16 Ryan_Lane: depooling ssl1001
* 20:08 LeslieCarr: cp1022 unresponsive, power cycling
* 19:44 LeslieCarr: switched preferred path between esams and eqiad
* 19:41 LeslieCarr: deploying a possibly mobile front end traffic effecting ipv6 apache request firewall filter
* 19:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable Collection extension'
* 19:08 LeslieCarr: site was popedotted !
* 18:47 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqaid'
* 18:31 logmsgbot: py synchronized wmf-config/lucene-production.php 'temp moving all search traffic to pmtpa for upgrades in eqiad'
* 18:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable scribunto EVERYWHERE'
* 18:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf11
* 17:23 pp-pdf1: downgrade to 0.14.3
* 16:39 pp-pdf1: upgraded mwlib to 0.15.1.
* 16:32 apergos: cleaned up borked puppet agent on ms-be12 and re-enabled/restarted it
* 12:30 logmsgbot: anomie synchronized php-1.21wmf11/extensions/Scribunto/common/Hooks.php 'Fix for bug 46031'
* 12:30 logmsgbot: anomie synchronized php-1.21wmf11/extensions/Scribunto/common/Common.php 'Fix for bug 46031'
* 12:07 Danny_B: reenabled mail delivery to wikibugs 30 mins ago (disabled automatically for excessive bounces)
* 11:42 apergos: restarted morebots
* 01:10 Tim: running updateCollation.php on all uca-* wikis in 6 screens on hume
* 00:45 logmsgbot: dzahn gracefulled all apaches
* 00:39 Ryan_Lane: depooling ssl1 ssl2 ssl1001 ssl1002 ssl3001 ssl3002
* 00:35 Ryan_Lane: pooling ssl3 ssl4 ssl1003 ssl1004 ssl3003
* 00:31 Ryan_Lane: depooling ssl3 ssl4 ssl1003 ssl1004 ssl3003
* 00:30 Ryan_Lane: pooling ssl1 ssl2 ssl1001 ssl1002 ssl3001 ssl3002
* 00:16 Ryan_Lane: depooling ssl1 ssl2 ssl1001 ssl1002 ssl3001 ssl3002
* 00:12 Tim: manually creating missing /etc/apache2/wmf symlinks on servers that lack them
* 00:07 logmsgbot: dzahn gracefulled all apaches
== March 12 ==
* 23:39 Tim: updating PHP to php5-3.5+wmf1 for new ICU
* 23:39 notpeter: upgrading pmtpa search nodes to lucene-search-2 2.1.7wm1
* 23:34 logmsgbot: tstarling synchronized php-1.21wmf11/maintenance/updateCollation.php
* 23:34 logmsgbot: tstarling synchronized php-1.21wmf10/maintenance/updateCollation.php
* 22:33 Ryan_Lane: pooling ssl3003 ssl3 ssl4 ssl1003 ssl1004
* 22:29 Ryan_Lane: depooling ssl3003 ssl3 ssl4 ssl1003 ssl1004
* 22:28 Ryan_Lane: pooling ssl3001 ssl3002 ssl1 ssl2 ssl1001 and ssl1002
* 22:23 mutante: pushing new version 2.7 of wikimedia-task-appserver to wikipedia-precise repo (to fix apache-sanity-check)
* 22:21 Ryan_Lane: depooling ssl3001 ssl3002 ssl1 ssl2 ssl1001 and ssl1002
* 21:23 Ryan_Lane: seems that the unified certificate is missing mediawiki.org, depooling the ssl servers that are serving it
* 21:22 Ryan_Lane: installing the new unified ssl certificate on all ssl servers
* 19:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:54 RobH: authdns-update for europium to gadolinium ip
* 18:47 RobH: europium shuttin down
* 18:43 mark: Imported ganglia 3.5.0 packages into the precise-wikimedia APT repository
* 18:19 RobH: europium reclaimed per rt4689
* 18:02 RobH: en-wp.com/org now redirect to en.wikipedia per rt 4669
* 18:00 RobH: ran apache restart as myself, not root, had a bunch of fails. restarted them as root, hopefully nothing noticable to users
* 17:58 logmsgbot: robh gracefulled all apaches
* 17:53 Jeff_Green: live-hack trivial OTRS patch to support filtering on X-Spam-Score for RT #4713, will check in once project is copied to git
* 17:42 logmsgbot: anomie synchronized php-1.21wmf11/extensions/Scribunto 'Update Scribunto to master'
* 17:29 logmsgbot: reedy synchronized php-1.21wmf10/includes/Collation.php
* 17:28 logmsgbot: reedy synchronized php-1.21wmf11/includes/Collation.php
* 15:42 paravoid: apache graceful to fix bz 46018 / RT 4695
* 15:39 logmsgbot: root gracefulled all apaches
* 15:32 hashar: Restarted Zuul process. Was stuck somehow waiting for a job that has been canceled.
* 13:30 hashar: Jenkins restarted!
* 12:55 hashar: gallium : upgrading Jenkins (unscheduled)
* 08:32 hashar: jenkins: live hacked the ant build script for MediaWiki jobs. Now using the local replicated git directory instead of a handmade replication.
* 05:03 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Math
* 05:02 logmsgbot: tstarling synchronized php-1.21wmf11/extensions/Math
* 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Mar 12 02:52:43 UTC 2013
* 02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Tue Mar 12 02:29:54 UTC 2013
* 02:16 Ryan_Lane: deploying change 53248 and 52951 to OpenStackManager on wikitech
* 02:10 Ryan_Lane: deployed change 53304 to OATHAuth on wikitech
* 00:38 logmsgbot: kaldari synchronized php-1.21wmf11/extensions/UploadWizard/ 'deploying some bugfixes for UploadWizard'
* 00:15 awight: payments cluster updated to 207a112b5b36a9e125999acc6f4098d9058ef594
== March 11 ==
* 23:48 AaronSchulz: syncFileBackend.php Running for ceph for all wikis now
* 23:48 AaronSchulz: copyFileBackend.php finished for ceph for all originals and transcodes (plus math/captcha)
* 23:38 logmsgbot: aaron synchronized php-1.21wmf11/includes/clientpool/RedisConnectionPool.php 'deployed 5af8702a4b4208ac42fab2fb36da69d13f3e3b92'
* 23:37 logmsgbot: aaron synchronized php-1.21wmf10/includes/clientpool/RedisConnectionPool.php 'deployed b9fdedc57295a4dd1da684859871aeb734bf70fe'
* 23:10 Ryan_Lane: adminbot upgraded to 1.6.3
* 23:09 Ryan_Lane: upgrading adminbot on wikitech-static
* 22:31 LeslieCarr: upgrading ircecho on neon
* 22:30 Ryan_Lane: upgrading ircecho on manganese
* 21:20 mutante: gracefulling eqiad Apaches - wikiquote.net now redirects to wikiquote.org
* 21:19 logmsgbot: dzahn gracefulled all apaches
* 21:07 Ryan_Lane: deployed change 53267 to OpenStackManager on wikitech
* 20:59 logmsgbot: reedy synchronized php-1.21wmf10/cache/interwiki.cdb 'Updating 1.21wmf10 interwiki cache'
* 20:59 logmsgbot: reedy synchronized php-1.21wmf11/cache/interwiki.cdb 'Updating 1.21wmf11 interwiki cache'
* 20:30 ^demon: Upgraded gerrit to 2.5.2-1636-g353e384, all's well
* 20:18 Ryan_Lane: deploying change 52950 to OpenStackManager on wikitech
* 19:22 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf11
* 19:05 mutante: patching Bugzilla for security - RT-4553
* 17:45 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
* 17:34 mwalker: Shields up! Payments deploy from a6fe052a73f1421028b44305908ec6984dbc50fa to head for Japanese localization updates.
* 17:29 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
* 17:19 RobH: search23 shutting down for mb sap per rt4323, has been disabled in pybal
* 17:18 RobH: search23 out of search pool1, other servers in pool online
* 17:17 RobH: setting search23 to false for mb swap
* 17:10 sbernardin: taking down search23 for MB replacement: https://rt.wikimedia.org/Ticket/Display.html?id=3423
* 16:48 logmsgbot_: aaron synchronized php-1.21wmf11/includes/limit.sh 'deployed e77397fc1de324dfc6d7b515e9440bd564311dd9'
* 15:57 ottomata: deployed new webstatscollector filter on locke. This replaces spaces in urls.
* 11:39 apergos: ook that was wrong started apache2 on stafford (puppet uses passenger), now things look ok
* 10:51 apergos: that would be 10.5. can't add.
* 10:50 apergos: shot old puppet agents hanging around on stafford, also old apaches from dead puppet master, restarted puppet master, it died around 6.5 hours ago
* 02:52 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Mon Mar 11 02:52:06 UTC 2013
* 02:43 logmsgbot_: krinkle synchronized wmf-config/CommonSettings.php 'I7c8174bcf6ca411014a712a6416d3a74e9fecb3d'
* 02:41 logmsgbot_: krinkle synchronized wmf-config/InitialiseSettings.php 'I7c8174bcf6ca411014a712a6416d3a74e9fecb3d'
* 02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Mon Mar 11 02:29:27 UTC 2013
== March 10 ==
* 22:19 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild message cache
* 21:59 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild message cache
* 21:10 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/CentralAuth/
* 21:00 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
== March 9 ==
* 02:54 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Sat Mar 9 02:54:23 UTC 2013
* 02:37 mutante: deploying blog WMBlog.php changes (gerrit 28835, gerrit 29823)
* 02:33 Reedy: Created cn_notice_log on enwiki. Not sure it's actually needed, but there's dberrors appearing here and htere
* 02:31 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Sat Mar 9 02:31:06 UTC 2013
* 02:01 mutante: deploying WMBlog.php changes for EventLogging (gerrit 52945)
* 01:26 logmsgbot_: olivneh synchronized wmf-config/InitialiseSettings.php '(Change If7ab3d016 / bug 45830) Separate slow parse log destination for private wikis'
* 01:13 Ryan_Lane: cherry-picking in change 52950 on wikitech
* 01:12 Ryan_Lane: scratch that
* 01:11 Ryan_Lane: deploying OATHAuth master to wikitech
* 00:32 mutante: deleting squid logs on brewster to prevent out-of-disk, and setting logrotate to compress
* 00:25 Ryan_Lane: upping OpenStackManager to master on wikitech
* 00:22 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'removing a second node per shard for streaming dumps. stopping slave'
* 00:00 Ryan_Lane: upping OpenStackManager to master on wikitech
== March 8 ==
* 22:28 logmsgbot_: reedy synchronized wmf-config/
* 22:28 logmsgbot_: py synchronized wmf-config/lucene-production.php 'moving pool4 and pool5 search traffic back to eqiad'
* 22:21 logmsgbot_: reedy synchronized php-1.21wmf10/includes/Collation.php
* 21:47 awight: payments cluster update to a6fe052a73
* 20:29 Ryan_Lane: enabling quorum feature for all gluster volumes
* 19:55 RobH: authdns-update, why? because i can.
* 19:34 Ryan_Lane: added Babel extension to wikitech
* 19:20 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'removing a second node per shard for streaming dumps. stopping slave'
* 15:35 ottomata: adding metrics.wikimedia.org domain, running authdns-update on dobson
* 06:21 Tim: updated luasandbox to version 1.4 (with pauseUsageTimer)
* 04:01 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/Translate
* 02:53 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Fri Mar 8 02:52:54 UTC 2013
* 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Fri Mar 8 02:28:34 UTC 2013
* 01:27 mutante: installing package upgrades on ekrem
* 01:12 logmsgbot_: catrope synchronized wmf-config/InitialiseSettings.php '20388a06 - Normalize VE config on testwiki'
* 00:57 mutante: added Set(@ReferrerWhitelist, qw(rt.wikimedia.org:443 rt.wikimedia.org:80));
* 00:55 Ryan_Lane: deploying change 52592 to wikitech
* 00:55 mutante: killall lighttpd and restart lighttpd and apache on streber
* 00:19 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
* 00:06 mutante: installing openjdk/icedtea jre and other package upgrades on gallium (jenkins)
* 00:05 LeslieCarr: upgrading cp1001
== March 7 ==
* 23:54 logmsgbot_: kaldari Started syncing Wikimedia installation... :
* 23:47 LeslieCarr: doing tmh upgrades
* 23:47 mutante: bravely rebooting streber - expect short RT and Observium outage
* 23:25 LeslieCarr: upgrading solr boxes
* 23:14 mutante: installing kernel on streber
* 23:11 ^demon: gerrit: disabling hooks-its plugin for the time being. It's working (yay) but it's overly spammy until we fix it next week.
* 23:04 LeslieCarr: dist-upgrading pmtpa memcache
* 23:04 mutante: dist-upgrading zhen
* 22:38 mutante: rebooting yvon
* 22:34 mutante: installing package upgrades on yvon despite (defunct apt-get procs)
* 22:31 mutante: salt pkg.upgrade on yvon
* 22:28 Ryan_Lane: rebooting marmontel
* 22:25 mutante: salt pkg.upgrade on spence
* 22:21 mutante: salt pkg.upgrade on zirconium
* 22:18 LeslieCarr: upgrading all analytics machiens
* 22:06 mutante: reboot formey after dist-upgrade via salt
* 20:05 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'fa0a56a89 - parsoidcache VIP in pmtpa'
* 19:37 logmsgbot_: bsitu synchronized wmf-config/CommonSettings.php 'Add Eventlogging configuration to Echo'
* 19:17 logmsgbot_: pgehres synchronized php-1.21wmf11/extensions/CentralNotice/ 'Updating CEntralNotice to master, resolving bug 45846'
* 18:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
* 18:39 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
* 17:36 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
* 16:07 cmjohnson1: reseating disk4 on ms-be1004
* 15:28 mark: Reinstalled cp3003
* 15:00 mark: Restarted gmetad on nickel
* 02:53 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Thu Mar 7 02:53:50 UTC 2013
* 02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Thu Mar 7 02:29:38 UTC 2013
* 01:53 mutante: installing package upgrades on sodium
* 00:54 RobH: kicked icinga
* 00:49 LeslieCarr: restarted pybal on lvs3
* 00:40 RobH: puppet locked on ms1004, killall puppet, refired
* 00:39 RobH: puppet halted run on cp1003, left in locked state, killed all puppet instances and refired, cleared.
* 00:33 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/RSS
* 00:15 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
== March 6 ==
* 23:19 LeslieCarr: reinstalling celsus as an internal host
* 23:07 logmsgbot_: reedy synchronized php-1.21wmf10/resources/ 'touch'
* 23:06 logmsgbot_: reedy synchronized php-1.21wmf11/resources/ 'touch'
* 23:05 logmsgbot_: reedy synchronized wmf-config 'touch'
* 22:54 logmsgbot_: reedy synchronized wmf-config/
* 22:53 logmsgbot_: reedy synchronized wmf-config/
* 22:40 mutante: starting haproxy on brewster
* 22:39 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild message cache for Wikibase deploy
* 22:39 notpeter: reslaving db36 and db38
* 22:29 RobH: authdns-update
* 22:20 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/Wikibase
* 22:20 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild message cache for Wikibase deploy
* 22:11 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
* 22:10 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
* 22:09 logmsgbot_: reedy synchronized wikidataclient.dblist
* 22:07 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
* 22:02 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
* 21:55 logmsgbot_: reedy cleared profiling data
* 21:55 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Wikibase Client on all wikipedias'
* 21:41 awight: updated payments cluster to 39a265489
* 21:16 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/
* 20:54 logmsgbot_: reedy synchronized wmf-config/
* 20:38 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/DataValues/
* 20:29 Reedy: Creating and populating site_identifiers table on all Wikipedias
* 20:28 Reedy: Creating and populating sites table on all Wikipedias
* 20:27 logmsgbot_: olivneh synchronized wmf-config/CommonSettings.php 'Adding redis host config. var for GettingStarted'
* 20:12 logmsgbot_: reedy synchronized wmf-config/
* 20:07 logmsgbot_: reedy synchronized wmf-config/
* 20:04 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/Wikibase
* 20:04 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/DataValues
* 20:03 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/Diff
* 19:57 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/Wikibase
* 19:56 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/DataValues
* 19:43 notpeter: restarting all jobrunners
* 19:34 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 19:29 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
* 19:28 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource to 1.21wmf11
* 19:25 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: aawiki back to 1.21wmf10 for Aaron
* 19:21 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 19:20 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia and wikiquote to 1.21wmf11
* 19:18 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikinews to 1.21wmf11
* 19:15 Reedy: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage, wikiversity and wiktionary to 1.21wmf11
* 19:10 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Special wikis to 1.21wmf11
* 19:07 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Closed, private and fishbowl to 1.21wmf11
* 19:03 mutante: gracefulling apaches to remove 1.17 aliases (gerrit 50609)
* 19:03 logmsgbot_: dzahn gracefulled all apaches
* 18:58 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
* 18:47 mutante: regenerating mailman list archives for wmfreqs
* 18:44 mutante: renaming mailman list 'allhands' to 'wmfreqs' (RT-4640)
* 18:36 logmsgbot_: aaron synchronized php-1.21wmf11/maintenance/nextJobDB.php 'deployed cbc98b09fb0a13b387d82bafab825608e2bb0b3b'
* 18:35 logmsgbot_: aaron synchronized php-1.21wmf11/includes/job/JobQueueGroup.php 'deployed cbc98b09fb0a13b387d82bafab825608e2bb0b3b'
* 18:35 logmsgbot_: aaron synchronized php-1.21wmf11/includes/DefaultSettings.php 'deployed cbc98b09fb0a13b387d82bafab825608e2bb0b3b'
* 18:10 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'clear config cache'
* 18:09 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'clear config cache'
* 18:01 logmsgbot_: hashar synchronized wmf-config 'deploying adapt Lucene configuration file to support beta'
* 17:59 hashar: deploying adapt Lucene configuration file to support beta
* 07:08 jeremyb_: 02:52:46 <+logmsgbot> !log LocalisationUpdate completed (1.21wmf11) at Wed Mar 6 02:52:46 UTC 2013
* 07:08 jeremyb_: 02:36:35 <+logmsgbot> !log reedy synchronized wmf-config/InitialiseSettings.php
* 07:08 jeremyb_: 02:28:21 <+logmsgbot> !log LocalisationUpdate completed (1.21wmf10) at Wed Mar 6 02:28:21 UTC 2013
* 07:08 jeremyb_: 01:33:50 <+logmsgbot> !log csteipp synchronized wmf-config/CommonSettings.php 'Revert https patch'
* 07:07 jeremyb_: 01:08:27 <+logmsgbot> !log reedy synchronized wmf-config/CommonSettings.php 'Revert wgSecureLoginDefaultHTTPS'
* 07:07 Ryan_Lane: restarted morebots
* 01:10 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Set wgSecureLoginDefaultHTTPS to false for testing'
* 00:37 binasher: added new deb of twemproxy 0.2.3 built with --enable-debug=log to precise-wikimedia
* 00:21 logmsgbot_: csteipp synchronized wmf-config/CommonSettings.php 'Enabling HTTPS login'
* 00:14 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings-labs.php
* 00:09 hashar: snapshot1002: rsync: change_dir#3 "/apache/common-local" failed: No such file or directory (2)
* 00:09 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings-labs.php
== March 5 ==
* 23:57 logmsgbot_: py synchronized wmf-config/db-eqiad.php 'increasing db1009 to full weight'
* 23:54 LeslieCarr: reinstalling constable as an internal host
* 23:49 RoanKattouw: Removed decommissioned srv226-234 from /etc/dsh/group/mediawiki-installation
* 23:47 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php '6fd144c44f26 - Use new service IP for Parsoid in eqiad'
* 23:46 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'repooling previously crashed mariadb pmtpa slaves'
* 23:36 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'ombudsmenwiki logo'
* 23:26 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'commenting out one slave per shard that's mysqldumping'
* 23:18 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
* 23:15 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
* 23:13 notpeter: stopping slave and starting mysql dumps on dbs 71, 57, 66, 65, 55, 50, and 68
* 23:00 logmsgbot_: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 22:40 logmsgbot_: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 22:36 logmsgbot_: py synchronized wmf-config/db-eqiad.php 'repooling db1009 at 25% weight'
* 22:18 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
* 22:17 logmsgbot_: reedy synchronized wmf-config/
* 22:14 LeslieCarr: restarting pybal on lvs3
* 22:14 LeslieCarr: restarting pybal on lvs4
* 22:13 LeslieCarr: restarting pybal
* 22:07 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 22:05 logmsgbot_: reedy synchronized wmf-config/
* 21:57 LeslieCarr: restarting pybal on lvs1003
* 21:55 LeslieCarr: restarting pybal on lvs1006
* 21:44 Coren: tools rejiggered the webproxy config to be smarter about paths not leading to specific tools
* 21:43 LeslieCarr: adding parsoidcache service to lvs1006
* 21:42 mutante: gracefulling eqiad Apaches - add ombudsmen.wm (bug 45744)
* 21:41 logmsgbot_: dzahn gracefulled all apaches
* 21:35 mutante: restarting pdns on ns2
* 21:34 mutante: DNS update - add ombudsmen (bug 45744)
* 20:49 logmsgbot_: reedy synchronized docroot
* 20:35 mutante: installing package upgrades on hooper
* 20:26 logmsgbot_: reedy synchronized wmf-config/
* 20:22 mutante: installing package upgrades on hume
* 20:00 mutante: wikipedia.ee now redirects to et.wikipedia, wikimedia.ee now redirects to et.wikimedia
* 19:56 mutante: gracefulling eqiad Apaches, push wiki(m|p)edia.ee redirects
* 19:56 logmsgbot_: dzahn gracefulled all apaches
* 19:18 K4-713: updated payments cluster to 9b063ee2e399a3b
* 18:42 logmsgbot_: mlitn synchronized wmf-config/InitialiseSettings.php 'Disable AFTv4 (entirely) and AFTv5 (leaving only opt-in) on enwiki'
* 18:04 cmjohnson1: restarting dhcpd3 service brewster
* 16:55 cmjohnson1: mw1085 powering off to troubleshoot DIMM error
* 05:35 ori-l: EventLogging db on db1047: ALTER DATABASE log CHARACTER SET utf8 COLLATE utf8_unicode_ci; (was: latin1)
* 02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Tue Mar 5 02:48:58 UTC 2013
* 02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Tue Mar 5 02:28:16 UTC 2013
* 02:03 K4-713: updated payments cluster to 5361c795f4bf31980f
* 00:46 mutante: fixing exim/mail permissions on neon to make Icinga paging work. missing +x on /var/log/exim4 dir
== March 4 ==
* 23:12 mutante: starting dhcp3 on brewster
* 23:11 LeslieCarr: p.s. bonding on foundry sucks
* 23:10 LeslieCarr: bringing down virt2 network to bond it
* 22:18 mutante: gracefulling eqiad Apaches to push indiawikipedia redirect
* 22:17 logmsgbot_: dzahn gracefulled all apaches
* 22:03 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'shuffling around pmtpa enwiki slaves'
* 21:50 LeslieCarr: messing around with the network of virt2 - expect possible pmtpa labs network issues
* 21:50 ottomata: staring puppet back up on oxygen, packet loss is better after disabling a filter
* 21:30 cmjohnson1: authdns update
* 21:17 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf11
* 21:16 logmsgbot_: reedy synchronized php-1.21wmf11
* 21:15 logmsgbot_: reedy synchronized php-1.21wmf10
* 21:14 RobH: shutting down professor for a fan swap, should be a short window
* 21:11 cmjohnson1: authdns update
* 21:05 ottomata: stopping puppet on oxygen to see if we can disable filters and reduce packet loss
* 20:55 Ryan_Lane: restarted icinga
* 20:53 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild Message Cache for 1.21wmf11
* 20:52 RobH: had to restart morebots, but it wasnt really dead, opps
* 20:51 RobH: had to restart icinga
* 20:51 RobH: Why arent you taking all the !log
* 20:50 RobH: i hate you morebots
* 20:28 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild Message Cache for 1.21wmf11
* 19:55 logmsgbot_: reedy Started syncing Wikimedia installation... : Build 1.21wmf11 message cache
* 19:41 logmsgbot_: reedy synchronized php-1.21wmf11/
* 19:34 logmsgbot_: reedy synchronized wmf-config/
* 19:34 logmsgbot_: reedy synchronized live-1.5/
* 19:33 logmsgbot_: reedy synchronized docroot
* 19:30 hashar: on gallium: stopping puppet, cleaning defunct and restarting puppet
* 19:23 logmsgbot_: reedy synchronized wmf-config/
* 19:20 mutante: installing package upgrades on sockpuppet
* 19:13 sbernardin: taking down amaranth for hw replacement: https://rt.wikimedia.org/Ticket/Display.html?id=4497
* 18:44 cmjohnson1: authdns update
* 18:17 K4-713: updated payments cluster to 059969c3d29f685c
* 17:44 cmjohnson1: authdns update
* 16:56 logmsgbot_: py synchronized wmf-config/db-eqiad.php 'depooling db1009 while investigating error'
* 16:24 sbernardin: rebooting db70 to reset idrac: https://rt.wikimedia.org/Ticket/Display.html?id=4628
* 16:14 sbernardin: db27 being taken down for decom per: https://rt.wikimedia.org/Ticket/Display.html?id=4623
* 10:12 apergos: cleared some space on brewster by rmeoving old squid access/storage logs (2.2gb worth in 3 files!), its root partition was full, restarted squid
* 10:12 apergos: reinstalled wikimedia-app-taskserver on fenari (how did it get removed?), needed for various cron jobs among other things
* 02:04 logmsgbot_: LocalisationUpdate failed (1.21wmf10) at Mon Mar 4 02:04:10 UTC 2013
== March 3 ==
* 02:25 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sun Mar 3 02:25:44 UTC 2013
== March 2 ==
* 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sat Mar 2 02:27:35 UTC 2013
* 01:17 mutante: powercycling emery
* 00:35 mutante: rebooting emery for kernel upgrade
* 00:29 mutante: installing package upgrades on emery
== March 1 ==
* 23:59 cmjohnson1: authdns update
* 23:56 K4-713: updated payments cluster to c8442435a9e2f5c
* 23:29 notpeter: dropping limesurvey from db9, backup in db9:/a/backup_dumps
* 22:53 RoanKattouw: Make that apache2-utils, that's the actual name of the package
* 22:51 RoanKattouw: Temporarily installing apachebench on wtp1004 per Gabriel's request
* 22:49 RoanKattouw: Depooled wtp1004 so Gabriel can use it for benchmarking
* 22:46 mwalker: reverting payments to dced5328fd6ae4b395622abc41d2afa0ccad21c8 from 4a3ca6cc36b9d78dbb015c1bf93bbafb14eb0799 because of bug report that some people are not able to make donations due to 'required field city' being missing and then being unable to provide it
* 22:34 cmjohnson1: authdns update
* 22:15 cmjohnson1: auth-dns update
* 22:12 mutante: DNS update - remove status cname
* 21:56 cmjohnson1: taking wtp1001 down to replace w/new box
* 21:32 mwalker: pushing 4a3ca6cc36b9d78dbb015c1bf93bbafb14eb0799 to the payments cluster (again, because the instant reviewer, me, was broken)
* 21:23 mwalker: attempted deploy of payments cluster to new version, but then had to roll back to dced5328fd6ae4b395622abc41d2afa0ccad21c8
* 20:02 mutante: blocked en.wikivoyage from bugmenot
* 20:01 logmsgbot: dzahn synchronized ./docroot/wikivoyage.org/block_8708f.html
* 19:55 logmsgbot: dzahn synchronized ./docroot/wikivoyage.org/block_ab2c3.html
* 19:55 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'parser cache max age 30d'
* 19:37 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043 to s1 watchlist service'
* 19:31 mutante: DNS update - add CNAME for status.wp/status.wm
* 19:28 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/51291/'
* 19:13 mutante: install package upgrades on marmontel
* 19:06 mutante: install package upgrades on streber
* 19:04 mutante: create missing backup dir on streber to enable RT_Shredder plugin
* 19:04 Ryan_Lane: maxsem synchronized wmf-config/InitialiseSettings.php 'make upload endpoint for Commons local https://gerrit.wikimedia.org/r/#/c/51612/'
* 19:03 Ryan_Lane: LocalisationUpdate completed (1.21wmf10) at Fri Mar 1 02:31:13 UTC 2013
* 19:03 Ryan_Lane: csteipp synchronized php-1.21wmf10/includes
* 19:03 mutante: db9 - zipping old sql dumps of drupal and civirm to prevent full disk
* 19:03 Ryan_Lane: dzahn synchronized ./docroot/wikivoyage.org/mywotf83102384bfcd1e52152.html
* 19:03 Ryan_Lane: restarted glusterd service on labstore1-4
* 19:03 Ryan_Lane: pgehres Finished syncing Wikimedia installation... : Updating i18n for DonationInterface-langonly
* 19:02 Ryan_Lane: pgehres Started syncing Wikimedia installation... : Updating i18n for DonationInterface-langonly
* 19:02 Ryan_Lane: binasher: shutting down mysql on db1043, converting to mariadb
* 19:02 LeslieCarr: powering back on spence
* 19:02 Ryan_Lane: asher synchronized wmf-config/db-eqiad.php 'pulling db1043 and shifting watchlist q's to db1050'
* 19:02 Ryan_Lane: pgehres synchronized php-1.21wmf10/extensions/DonationInterface/ 'Updating DonatonInterface-langonly'
* 19:02 Ryan_Lane: LeslieCarr: powering off spence
* 19:01 Ryan_Lane: asher synchronized wmf-config/lucene.php 'sending search-pool4 traffic to pmtpa, where the index/host distribution has been rebalanced'
* 19:01 Ryan_Lane: notpeter: upgrading mariadb boxes to 5.5.29
* 19:01 Ryan_Lane: tstarling synchronized wmf-config/InitialiseSettings.php
* 19:01 Ryan_Lane: Jeff_Green: stopped gmetad on neon for testing
* 19:01 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/GettingStarted 'Update to split test (3/3)'
* 19:00 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/E3Experiments 'Update to split test (2/3)'
* 19:00 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/GuidedTour 'Update to split test (1/3)'
* 19:00 Ryan_Lane: mutante: create missing dump directory on streber, enables "RT-shredder" plugin which comes with RT since 3.8
* 19:00 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/EventLogging 'Updates to JavaScript API'
* 19:00 Ryan_Lane: changing A record for wikitech.wm.o and changing labsconsole.wm.o into a cname to wikitech
* 18:59 Ryan_Lane: disabled editing on wikitech.wikimedia.org
* 18:59 Ryan_Lane: binasher: resharding search-pool4 in pmtpa - restarted lsearchd on all local nodes and indexers on searchidx2
* 18:59 mwalker: ... therefore the payments cluster is now running dced5328fd6ae4b395622abc41d2afa0ccad21c8
* 18:59 mutante: restart nagios-nrpe,gmond,apache,salt-minion on bastion hosts
* 18:59 Ryan_Lane: binasher: rebuilding db35 from a hotbackup of db55 (s5); testing a new build of mariadb 5.5.29
* 18:59 Ryan_Lane: reedy synchronized php-1.21wmf10/cache/interwiki.cdb 'Updating 1.21wmf10 interwiki cache'
* 18:59 Ryan_Lane: reedy synchronized php-1.21wmf10/cache/interwiki.cdb 'Updating 1.21wmf10 interwiki cache'
* 18:58 Ryan_Lane: reedy synchronized php-1.21wmf10/extensions/WikimediaMaintenance
* 18:58 Ryan_Lane: apergos: restarted icinga (not sure why it stopped, log file simply said 'caught TERM signal', in any case it complained that Error: Could not create external command file '/var/lib/nagios/rw/nagios.cmd' as named pipe )
* 18:58 mwalker: enabled sofort forms for CH and DE on the payments cluster
* 18:39 mutante: installing package upgrades on bast1001
* 18:37 mutante: installing package upgrades on fenari
* 17:40 ^demon: rescanned all trackingids for gerrit on manganese
* 17:31 ^demon: upgrading gerrit to 2.5.2-1506-g278aa9a on manganese/formey
* 17:07 Ryan_Lane: test
== February 28 ==
* 12:58 apergos: restarted icinga (not sure why it stopped, log file simply said 'caught TERM signal', in any case it complained that Error: Could not create external command file '/var/lib/nagios/rw/nagios.cmd' as named pipe )
* 06:28 apergos: stabbed searchpool4 and restarted lucene search etc gah hate hate hate
* 03:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 03:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 03:12 logmsgbot: reedy synchronized docroot
* 03:10 logmsgbot: reedy synchronized docroot
* 03:09 logmsgbot: reedy synchronized docroot
* 02:30 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Thu Feb 28 02:30:14 UTC 2013
* 01:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 00:58 RobH: dsh restarted eqiad apaches in progress
* 00:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 00:57 logmsgbot: robh gracefulled all apaches
* 00:54 logmsgbot: reedy synchronized wmf-config/
* 00:32 RobH: authdns-update to support wikinews.de
* 00:27 logmsgbot: reedy synchronized wmf-config/
* 00:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Lowering throttle to 5/day for new users, and'
== February 27 ==
* 23:26 Reedy: Running sync-common on mw1111. Reporting failed opening wmf-config/mc.php
* 22:57 mutante: rebooting zirconium (planet)
* 22:54 mutante: installing package upgrades on zirconium
* 22:52 logmsgbot: reedy synchronized wmf-config/
* 22:46 mutante: rebooting bast1001
* 22:36 mutante: installing package upgrades on bast1001
* 22:16 logmsgbot: reedy synchronized wmf-config/
* 21:44 RobH: rdb1001 & rdb1002 allocated, network setup, and ready for handoff, per rt4608
* 21:36 RobH: authdns-update for rdb1001-1002
* 21:36 RobH: authdns-update for sdb1001-1002
* 21:31 Tim: running resetUserTokens.php on all wikis, in screen on hume, to fix user_token field corruption (bug 41586)
* 21:26 logmsgbot: tstarling synchronized php-1.21wmf10/includes/User.php 'user_token fixes'
* 20:05 K4-713: updated payments cluster to fcbf9211c9836b
* 19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf10
* 18:55 LeslieCarr: rebooting arsenic
* 18:42 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for EducationProgram
* 18:25 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for EducationProgram
* 17:57 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/
* 17:51 K4-713: updated payments cluster to eab489d0889a
* 17:13 mutante: rebooting fenari for kernel upgrade
* 08:33 apergos: restarted lucene search on search1015
* 08:04 logmsgbot: reedy synchronized live-1.5/static-1.21wmf4/
* 08:04 logmsgbot: reedy synchronized docroot/bits/static-1.21wmf4/
* 07:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 07:05 Krinkle: done adding 1.21wmf3, 1.21wmf4, 1.21wmf5 (bug 44570)
* 07:04 logmsgbot: krinkle Finished syncing Wikimedia installation... :
* 06:29 apergos: snapshot1002 down for reinstallation
* 06:27 logmsgbot: krinkle Started syncing Wikimedia installation... :
* 06:15 Krinkle: adding 1.21wmf3, 1.21wmf4, 1.21wmf5 (bug 44570)
* 06:15 Krinkle: added 1.21wmf1, 1.21wmf2 (bug 44570)
* 05:35 apergos: restarted lucene search on search1016
* 04:57 logmsgbot: krinkle Finished syncing Wikimedia installation... :
* 04:37 logmsgbot: krinkle Started syncing Wikimedia installation... :
* 02:57 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 27 02:57:22 UTC 2013
* 02:31 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Wed Feb 27 02:31:06 UTC 2013
* 01:35 Krinkle: Bringing back checkouts of MediaWiki 1.21wmf5 - 1.21wmf1 on fenari for bug 44570
* 01:35 Jeff_Green: dist-upgrade and reboot grosley
* 00:25 logmsgbot: tstarling cleared profiling data
== February 26 ==
* 23:29 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/51053 https://gerrit.wikimedia.org/r/51057'
* 23:27 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/51053 https://gerrit.wikimedia.org/r/51057'
* 23:04 K4-713: synchronized payments cluster to 596b854787132c730
* 22:35 mutante: installing package upgrades on fenari
* 22:31 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 22:31 LeslieCarr: nagios is dead! spence.wikimedia.org if you want to access it directly
* 22:16 mutante: gracefulling eqiad Apaches to push redirect for wikivoyage.eu
* 22:15 logmsgbot: dzahn gracefulled all apaches
* 22:15 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 20:58 RobH: stat1 puppet runs fail due to dan account error, i am working on it, but i also want to run out and get something to eat, so leaving in broken state for 10 minutes until i return to work on it
* 20:21 mutante: DNS update - add wikiovyage.eu
* 19:34 Jeff_Green: package updates & reboots of payments hosts
* 19:13 paravoid: freezing all exim4 bounces to restore normal email delivery
* 18:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 17:54 paravoid: restarting opendj on sanger
* 17:45 AaronSchulz: Copying math files into ceph
* 15:14 mark: suspended i-000001d7 and i-0000039e for sending out malicious emails
* 15:06 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/GeoData/solrupdate.php
* 15:05 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/GeoData/solrupdate.php
* 12:26 apergos: temp blacklist live hack on sodium for exim4 to prevent some spam, see /etc/exim4/blacklist for that
* 11:20 logmsgbot: mlitn synchronized php-1.21wmf10/extensions/ArticleFeedback/api/ApiArticleFeedback.php 'Update ArticleFeedback to master'
* 06:53 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/
* 06:43 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
* 06:40 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
* 06:28 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
* 06:26 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
* 06:21 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
* 03:10 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
* 03:06 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
* 03:04 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
* 03:03 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
* 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 26 02:52:01 UTC 2013
* 02:46 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
* 02:44 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
* 02:41 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
* 02:37 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
* 02:29 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
* 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Feb 26 02:28:06 UTC 2013
* 02:11 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
* 01:59 Ryan_Lane: test
* 01:59 Ryan_Lane: test
* 01:58 Ryan_Lane: test
* 01:40 K4-713: updated payments cluster to ce28b210374d8
* 00:20 mutante: mc1001,ms1004,mw1029,mw1059,mw1070,mw1157 - fixed puppet freshness
* 00:03 mutante: amssq33,amssq44 - fix more puppet runs
== February 25 ==
* 23:54 mutante: knsq26,knsq28 - deleting puppet lock file, fix puppet runs
* 23:45 logmsgbot: reedy synchronized wmf-config/
* 23:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable SearchExtraNS on commonswiki'
* 23:34 mutante: re-enabling disabled puppet on srv253,srv235
* 22:16 Ryan_Lane: restarting opendj/pdns on virt0
* 22:02 Reedy: Ran namespaceDupes.php on huwikt
* 21:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 20:28 logmsgbot: reedy synchronized docroot
* 20:27 logmsgbot: reedy synchronized wmf-config/
* 19:50 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuilding message cache to update WikimediaMessages
* 19:40 mutante: powercycling frozen niobium
* 19:32 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuilding message cache to update WikimediaMessages
* 19:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf10
* 18:54 mutante: wikivoyager.de and wikivoyager.org transferred and both redirect to wikivoyage.org
* 18:50 mutante: deleting foundation-news-l (deprecated, replaced by WikimediaAnnounce-l)
* 18:49 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMessages
* 18:31 AaronSchulz: Copying all captchas into ceph
* 18:11 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Wikibase
* 06:38 apergos: restarted lucene search on search1016
* 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 25 02:52:32 UTC 2013
* 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Mon Feb 25 02:28:55 UTC 2013
* 01:15 mark: Powercycled strontium
* 01:07 paravoid: powercycling niobium (both unresponsive from the load)
* 01:06 paravoid: powercycling arsenic
== February 24 ==
* 06:28 apergos: restarted search1016 lucene
* 02:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 24 02:54:30 UTC 2013
* 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sun Feb 24 02:29:29 UTC 2013
== February 23 ==
* 21:45 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb
* 21:44 logmsgbot: reedy synchronized php-1.21wmf10/cache/interwiki.cdb
* 21:43 Reedy: mw110 is asking for a password
* 06:13 apergos: restarted lucene search on search1016
* 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 23 02:52:10 UTC 2013
* 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sat Feb 23 02:27:17 UTC 2013
* 00:43 RobH: lvs1003 had a ton of hung puppet processes (no output on them when tracing, so dunno whats up). killed them and kicked manual run
== February 22 ==
* 23:59 RobH: argon is shut down now
* 23:55 RobH: argon is going offline, killing our limesurvey install
* 23:29 paravoid: deploying squid config for upload's new transcoded container
* 23:26 paravoid: deploying squid
* 22:55 Ryan_Lane: changing dns entry for wikitech-static
* 22:41 RobH: mw110 throwing odd puppet errors, will come back to it later
* 22:41 RobH: sq41 reinstalled, throws endless puppet errors during redeployment
* 22:07 MaxSem: reindexing complete
* 21:39 RobH: harmon and copper online with OS installed for datadumps misc host support
* 21:39 MaxSem: Nuking GeoData Solr index and reindexing
* 21:38 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/GeoData/solrupdate.php 'https://gerrit.wikimedia.org/r/#/c/50388/'
* 21:35 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/GeoData/solrupdate.php 'https://gerrit.wikimedia.org/r/#/c/50388/'
* 21:05 RobH: installing harmon and copper into dump generation misc use
* 20:17 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/GuidedTour/modules/tours/gettingstarted.js 'Resume split test (3/3)'
* 20:16 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/GettingStarted/resources/ext.gettingstarted.js 'Resume split test (2/3)'
* 20:16 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/E3Experiments/experiments/openTask.js 'Resume split test (1/3)'
* 20:14 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Math 'Updating Math extension to fix db caching behavior (Change-Id: I9b690e55001859c97fd40330272791d49ec6de75)'
* 20:13 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/Math 'Updating Math extension to fix db caching behavior (Change-Id: I9b690e55001859c97fd40330272791d49ec6de75)'
* 20:09 RobH: authdns-update
* 19:58 RobH: sq41 boot order was fubar, fixed, reinstalling (as reboot into installer wiped mbr)
* 19:56 RobH: wtp1002-1004 os installed
* 19:34 Jeff_Green: start mysqldump of otrs database on db49
* 19:18 RobH: kicked ntpd manually on mw1045
* 19:09 RobH: sq41 reinstalling
* 18:57 RobH: mw112-125 back into pybal api pool
* 18:56 RobH: mw86-99 & mw111 back into pybal pool
* 18:49 logmsgbot: robh Finished syncing Wikimedia installation... :
* 18:37 logmsgbot: robh Started syncing Wikimedia installation... :
* 17:53 sbernardin: taking down sq41 to replace controller card.... https://rt.wikimedia.org/Ticket/Display.html?id=4550
* 16:24 mutante: tartupeedia.ee now redirects to et.wp portal page (bug 44893)
* 16:12 mutante: gracefulling eqiad Apaches via dsh to push tartupeedia.ee redirect (bug 44893)
* 16:11 logmsgbot: dzahn gracefulled all apaches
* 16:07 mutante: mw110 - sits at partioning screen in installer
* 15:47 hashar: gallium removed live hack in /srv/org/wikimedia/doc/puppetsource/manifests/role/labsmediawiki.pp
* 14:52 ^demon: restarted gerrit on manganese, jetty flapping again
* 14:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 13:14 mark: Halted ms-be3001-3003 for controller replacement
* 13:11 mark: Halted ms-be3004 for controller replacement
* 13:07 mark: Halted cp3009 for controller replacement
* 12:20 mark: Halted cp3010 for controller replacement
* 11:23 mutante: DNS update - add indiawikipedia.com
* 07:33 apergos: restarted lucene search on search1016
* 06:49 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
* 05:33 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
* 05:33 logmsgbot: tstarling synchronized docroot/bits/favicon/wikipedia.ico
* 05:20 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
* 05:18 logmsgbot: tstarling synchronized docroot/bits
* 02:53 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 22 02:53:05 UTC 2013
* 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Fri Feb 22 02:29:09 UTC 2013
* 01:57 binasher: migrating wikivoyage and wikidata external blob stores to innodb
* 01:08 logmsgbot: aaron synchronized wmf-config/filebackend.php 'deployed f699c5a9ad2f780dc18863592dcae4a51ba6d841'
== February 21 ==
* 23:21 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/CodeEditor 'Syncing patch that disables background linting'
* 23:18 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, and GuidedTour
* 23:02 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, and GuidedTour
* 22:35 logmsgbot: awjrichards synchronized php-1.21wmf9/extensions/MobileFrontend/
* 22:33 logmsgbot: awjrichards synchronized php-1.21wmf10/extensions/MobileFrontend
* 22:12 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Add photo upload schema for event logging'
* 21:56 notpeter: running do-release-upgrade on dbs 52, 39, 51, and 35
* 21:49 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/EventLogging
* 21:48 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing 1 slave for s2-5 from db-secondary for upgrades and maria'
* 21:48 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
* 21:47 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/CodeEditor
* 21:37 notpeter: restarting mysql on db1047 to pick up conf changes
* 21:31 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/SyntaxHighlight_GeSHi 'Disable highlighting of very large files.'
* 21:23 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/SyntaxHighlight_GeSHi 'Disable highlighting of very large files.'
* 21:05 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php Enabling CodeEditor for Schema namespace on test2 and metawiki.
* 20:02 binasher: some wikis have myisam page_props tables. converting all to innodb via osc
* 18:39 Ryan_Lane: enabled Extension:OpenID as a provider on labsconsole
* 11:04 hashar: Ariel saved Gerrit by restarting it!
* 10:51 hashar: huge CPU spike on manganese (gerrit host)
* 10:06 hashar: Jenkins: reseting git branch specifier :-/
* 09:32 hashar: Jenkins: removing git branch specifier from all mediawiki-core jobs.
* 05:50 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'shell cgroup'
* 02:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 21 02:54:17 UTC 2013
* 02:30 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'new extension1 shard (as an externalload config) - initially for AFTv5'
* 02:30 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'new extension1 shard (as an externalload config) - initially for AFTv5'
* 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Thu Feb 21 02:28:20 UTC 2013
* 00:53 RobH: issues with new mw servers 86+, took back out of pybal until i can troubleshoot
* 00:49 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance/
* 00:48 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMaintenance/
* 00:34 binasher: moved wikidatawiki tables on s3 to wikidatawiki_old to keep around for a time just in case; then "drop database wikidatawiki"
* 00:20 logmsgbot: asher synchronized wmf-config 'enabling wikidatawiki on shard s5'
* 00:18 logmsgbot: asher synchronized s5.dblist 'adding wikidatawiki'
* 00:17 logmsgbot: asher synchronized s3.dblist 'removing wikidatawiki'
== February 20 ==
* 22:48 RobH: authdns-update
* 22:35 binasher: importing wikidatawiki with schema changes into the s5 master, with sql_log_bin disabled
* 22:24 binasher: importing wikidatawiki with schema changes into the inactive pmtpa side of the s5 shard, with replication of the import just in pmtpa. ignore repl warnings for db{35,44,45,55}
* 21:48 RobH: authdns-update for wtp1002-1004
* 21:45 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'Take pmtpa Solr servers out of rotation, increase load on master: https://gerrit.wikimedia.org/r/#/c/50056/'
* 21:41 RobH: ignore db66 errors, asher is working on it
* 21:25 binasher: running wikidata wb_items / wb_items_per_site schema migrations in parallel on the db66 slave (creating primary keys, cannot be done via online-schema-change)
* 21:24 RobH: kicked knsq23 to clear puppet freshness alarm
* 21:24 RobH: maerlant still offline (200 days?). dropped esams ticket for onsite troubleshooting, as mgmt is offline.
* 21:20 logmsgbot: asher synchronized wmf-config/InitialiseSettings.php 'setting wgReadOnly for wikidatawiki durring maintenance'
* 21:18 RobH: forced puppet run on lardner, was tired of it showing error for no runs
* 21:13 RobH: mw112-125 now in service as api apaches in tampa
* 20:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 20:38 RobH: mw86-mw111 tested ok, pushing into apache service in pmtpa
* 20:32 RobH: mw86-mw111 being added to nodelists, ignore errors on them for next few minutes
* 19:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks to 1.21wmf10
* 19:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, wikisource and wikiquote to 1.21wmf10
* 19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage, wikinews and wiktionary to 1.21wmf10
* 19:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, special, wikimedia, private and fishbowl to 1.21wmf10
* 05:34 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 're-enable Scribunto profiler'
* 04:32 Tim: updating luasandbox to version 1.2, with profiler bug fixes
* 02:53 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Wed Feb 20 02:53:32 UTC 2013
* 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 20 02:29:11 UTC 2013
* 01:28 RobH: mw1161-1208 have wrong power setting, changing all to system.power.hotspare.enable 0
* 01:00 notpeter: do-release-upgrade-ing on db58
* 00:57 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing db58 from db-secondary for upgrades and maria, replacing db43'
== February 19 ==
* 23:47 notpeter: do-release-upgrade-ing on db43
* 23:38 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing db43 from db-secondary for upgrades and maria'
* 23:21 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/49946/'
* 23:04 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
* 23:03 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/
* 22:49 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
* 22:41 logmsgbot: preilly synchronized php-1.21wmf9/extensions/ArticleFeedback 'fix MAX call with GREATEST'
* 22:29 RobH: sq41 offline, rt 4550
* 22:20 RobH: sq41 puppet freshness bad, wont accept ssh connections, serial redirection results in no output. will need to reboot it
* 22:17 RobH: sq41 not allowing ssh, drac resetting
* 22:15 sbernardin: taking down ms-be12 for hardware replacement per: https://rt.wikimedia.org/Ticket/Display.html?id=4546
* 22:15 RobH: sq79/80 puppet in locked state, manually restarted puppet, removed lock file, ran manually
* 22:11 logmsgbot: preilly synchronized php-1.21wmf9/extensions/ArticleFeedback 'fix mysql 5.5 / ansi sql incompatibility'
* 21:54 RobH: mw48 was set to puppet agent disabled, re-enabled and fired puppet run
* 20:04 logmsgbot: mflaschen synchronized . 'Syncing GuidedTour submodule omitted when new branch cut was first cloned.'
* 19:59 Ryan_Lane: disabling project storage volumes for all labs projects with no instances
* 15:02 hashar: restarting jenkins to enable some startup setting
* 14:58 hashar: manganese CPU spike ongoing. Started roughly at 14:40 UTC
* 13:29 mutante: wikimedia.jp.net now redirects to wikimedia.org | wikimediacommons.jp.net now redirects to commons.wm
* 13:27 mutante: DNS update - adding wikimedia.jp.net
* 13:25 logmsgbot: dzahn gracefulled all apaches
* 13:20 mutante: gracefulling eqiad Apaches via dsh to push redirect change for .jp.net domains
* 12:39 apergos: powercycling ms-be7 (unresponsive to ping or from console)
* 11:29 mutante: installing package upgrads on neon/icinga
* 02:49 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Feb 19 02:49:32 UTC 2013
* 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 19 02:28:25 UTC 2013
* 01:12 logmsgbot: anomie synchronized php-1.21wmf10/includes/api/ApiQuerySiteinfo.php 'Backport fix for ugly API bug'
* 00:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 00:30 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
== February 18 ==
* 23:51 logmsgbot: tstarling synchronized php-1.21wmf9/extensions/Scribunto
* 23:51 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Scribunto
* 23:46 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
* 23:25 Tim: on snapshot1002 -- fixed incorrect owner on some MW files, killed broken apt-get and upgraded wikimedia-task-appserver
* 23:24 Tim: fixed rsyncd.conf on scap proxies to allow connections from new apaches
* 23:14 logmsgbot: tstarling Finished syncing Wikimedia installation... :
* 23:11 logmsgbot: tstarling Started syncing Wikimedia installation... :
* 22:45 logmsgbot: tstarling Finished syncing Wikimedia installation... :
* 22:42 logmsgbot: tstarling Started syncing Wikimedia installation... :
* 22:37 logmsgbot: tstarling synchronized php-1.21wmf9/extensions/Scribunto
* 22:37 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Scribunto
* 22:20 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 20:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.21wmf10
* 20:37 logmsgbot: reedy synchronized php-1.21wmf10/cache/l10n/ 'Scap is a liar'
* 20:31 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild l10n cache for wikidata related extensions
* 20:15 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild l10n cache for wikidata related extensions
* 19:59 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Wikibase
* 19:59 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Diff
* 19:58 logmsgbot: reedy synchronized php-1.21wmf10/extensions/DataValues
* 19:58 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMessages
* 19:44 logmsgbot: reedy synchronized php-1.21wmf10/cache/l10n/
* 19:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf10
* 19:03 logmsgbot: reedy Finished syncing Wikimedia installation... : Build 1.21wmf10 message cache, and push it this time
* 18:45 logmsgbot: reedy Started syncing Wikimedia installation... : Build 1.21wmf10 message cache, and push it this time
* 17:54 logmsgbot: reedy Finished syncing Wikimedia installation... : Build 1.21wmf10 message cache
* 17:53 logmsgbot: reedy Started syncing Wikimedia installation... : Build 1.21wmf10 message cache
* 17:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf10
* 17:28 logmsgbot: reedy synchronized php-1.21wmf10 'Initial sync out'
* 17:03 logmsgbot: reedy synchronized wmf-config/
* 17:02 logmsgbot: reedy synchronized docroot
* 17:01 logmsgbot: reedy synchronized live-1.5/
* 12:33 mutante: wikiartpedia.(biz|co|info|me|mobi|net) activated - redirect to wikipedia.org (RT-4240)
* 12:30 mutante: adding wikiartpedia domains to DNS / authdns-update
* 12:28 logmsgbot: dzahn gracefulled all apaches
* 12:24 mutante: gracefulling eqiad Apaches via dsh for redirect.conf change
* 12:22 mutante: mw1045 - eth0 NO-CARRIER - check cable (RT-4545)
* 12:13 mutante: mw1041 - down - CPU Machine Chk error on mgmt - removing from dsh groups (RT-4517)
* 05:45 logmsgbot: tstarling synchronized wmf-config/db-eqiad.php
* 05:16 logmsgbot: tstarling synchronized wmf-config/db-eqiad.php
* 05:10 Tim: on db1043: running alter table to convert the relevant field to signed, since I figure this is less disruptive than sql_mode = 'NO_UNSIGNED_SUBTRACTION' and less likely to break again than skipping the query
* 04:58 Tim: replication broken on db1043, possibly others, due to update of article_feedback_revisions with negative value for afr_total
* 03:40 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
* 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 18 02:29:47 UTC 2013
== February 17 ==
* 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 17 02:29:31 UTC 2013
== February 16 ==
* 13:38 apergos: restarted lucene-search on search1015 about 20 minutes ago
* 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 16 02:28:26 UTC 2013
* 01:47 logmsgbot: reedy synchronized php-1.21wmf9/
* 01:11 logmsgbot: reedy synchronized wmf-config/filebackend.php
== February 15 ==
* 23:56 RobH: authdns-update to correct mw110 reverse entry
* 23:28 Ryan_Lane: *modified them in LDAP
* 23:28 Ryan_Lane: modified project objects to indicate whether or not they are using gluster volumes for project and/or home storage
* 23:04 RobH: disregard mw1161 flaps, im working on it
* 20:15 Ryan_Lane: restarting opendj and pdns on virt0 and virt1000
* 20:15 Ryan_Lane: removed opendj from nfs1/2
* 19:07 RobH: no one sign mw86-mw125 puppet certs, im still working on them
* 18:23 RobH: mw87-mw125 installing
* 17:33 binasher: reimaging db1029-1031
* 16:55 mutante: removing mw1085 from dsh groups - broken hardware (RT-4453)
* 16:00 hashar: Enabled a linter for operations/apache-config . Any apache typo in operations/apache-config will publicly write your name on the typo wall of shame. See for implementation details and https://integration.mediawiki.org/ci/job/operations-apache-config-lint/ for some build examples.
* 14:31 mutante: wikimediafoundation.info activated to redirect .org
* 14:28 mutante: gracefulling Apaches in eqiad using dsh
* 14:25 logmsgbot: dzahn gracefulled all apaches
* 14:15 ^demon: [gerrit] updated replication plugin to 20c6fc7e (1.1-SNAPSHOT) which is back in line with upstream
* 08:02 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Echo 'Syncing Iab077a77aa75a6ff0cf03acecca38403595dda93 to fix exception 640c60c4'
* 02:30 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 15 02:30:05 UTC 2013
* 01:19 paravoid: deploying squid changes for s/knams/esams/ for knsq* squids (see git log for rationale)
* 01:09 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Enabling Schema NS on test2wiki'
== February 14 ==
* 22:49 RobH: mw1194 online and now in service as api apache
* 22:48 RobH: mw1200 added to api pool, online and in service
* 22:44 RobH: mw1189-1199 (sans 1194) now online as eqiad api apaches
* 22:37 logmsgbot: mflaschen Finished syncing Wikimedia installation... :
* 22:29 logmsgbot: mflaschen Started syncing Wikimedia installation... :
* 22:21 sbernardin: taking storage1 offline for decommissioning per: https://rt.wikimedia.org/Ticket/Display.html?id=4529
* 22:18 RobH: pushing mw1161-1188 into apache service (sans 1161/1164) from false to true in pybal, otherwise all online
* 22:16 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Updating for test2wiki'
* 22:14 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Math
* 22:03 Ryan_Lane: replacing gluster upstarts on all labstore nodes with a regular init script
* 22:01 logmsgbot: reedy synchronized docroot
* 22:00 logmsgbot: olivneh synchronized wmf-config
* 21:49 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling PostEdit on {be|mediawiki|outreach}wikis'
* 21:45 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
* 21:39 sbernardin: taking down srv266 and srv268 for decommissioning ... https://rt.wikimedia.org/Ticket/Display.html?id=4534
* 21:31 ottomata1: restarted solr jetty on solr2
* 21:30 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update ArticleFeedbackv5, Echo, PageTriage to master
* 21:25 ottomata: restarted solr jetty on solr3 and solr1003 for MaxSem
* 21:03 logmsgbot: bsitu Started syncing Wikimedia installation... : Update ArticleFeedbackv5, Echo, PageTriage to master
* 20:24 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'updating CommonSettings.php for Echo'
* 20:15 apergos: rebooting mw1010 after some issues with OOM
* 19:22 RobH: new servers are NOT in pybal yet, and thus not getting traffic
* 19:22 RobH: just added mw1161-1200 into node lists for eventual service, some may throw sync script errors as I track them down
* 19:15 Ryan_Lane: rebooting labstore3
* 19:08 Ryan_Lane: rebooting labstore4
* 19:01 Ryan_Lane: rebooting labstore2
* 18:52 Ryan_Lane: rebooting labstore1
* 15:22 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Updating wikibase sort order'
* 15:22 logmsgbot: demon synchronized php-1.21wmf9/extensions/Wikibase/client/includes/SortUtils.php 'Updating wikibase sort order'
* 14:52 mutante: re-enabled cp1044 in pybal - mobile varnish cache servers all on precise now
* 14:39 hashar: Jenkins: adding in several extension to Zuul configuration 
* 14:30 hashar: jenkins : updated all jobs using latest version jjb-config
* 13:35 mark: Fixed fixing
* 13:27 mutante: disabling cp1044 in pybal, reinstalling with precise
* 13:22 mutante: re-enabling cp1043 in pybal
* 13:11 mark: Upgraded varnish to latest version with updated range patch on cp1021-1028
* 12:16 mark: Inserted varnish 3.0.3plus~rc1-wm7 into the precise-wikimedia APT repository
* 12:14 mutante: stopping varnish backend on cp1043 and starting reinstall
* 12:10 mutante: disabling cp1043 in pybal
* 11:31 mutante: ms1004 - full disk, deleted large HTCPpurger.log file, fixed puppet runs
* 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 14 02:29:05 UTC 2013
* 01:17 andrewbogott: updated OpenStackManager to git head on virt0. Previous state is preserved (temporarily) in the local 'premerge' branch.
* 01:13 paravoid: removing mysql & apache from emery(!)
* 00:03 RobH: db29 os installed, handed off to peter
== February 13 ==
* 23:50 RobH: db29 reinstalling
* 23:48 paravoid: reinstalling sq48
* 23:22 RobH: db27 and db29 will be up and down for reinstallations
* 23:17 sbernardin: rebooting db29 per https://rt.wikimedia.org/Ticket/Display.html?id=4528
* 23:17 sbernardin: rebooting db27 per https://rt.wikimedia.org/Ticket/Display.html?id=4528
* 23:11 RobH: db1013 online and puppet updated, needs to be pushed into some db cluster service
* 22:45 RobH: mw27 locked up, powercycling
* 22:36 notpeter: powercycling all eqiad jobrunners
* 22:13 RobH: adding mw1201-1208 to eqiad apache pool
* 21:44 Reedy: Running sync-common on mw1203
* 21:43 logmsgbot: reedy synchronized php-1.21wmf9 'resync to fix file permission errors'
* 21:32 Reedy: Running sync-common on mw1204
* 21:30 Reedy: Running sync-common on mw1202
* 21:27 RobH: db27 rebooting for reinstallation
* 21:27 RobH: db29 offline due to hardware issue, related ticket 4526
* 21:19 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 21:11 logmsgbot: csteipp synchronized wmf-config
* 21:09 logmsgbot: reedy cleared profiling data
* 20:25 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Uncommenting enwiki from localClientDatabases'
* 19:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Wikibase Client on enwiki'
* 19:47 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase/
* 19:30 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved remaning wikis to 1.21wmf9
* 19:21 logmsgbot: aaron synchronized php-1.21wmf9/maintenance/runJobs.php 'deployed b03384f0ebd95e7f79638fb14ccd55da9c186d97'
* 19:12 logmsgbot: py gracefulled all apaches
* 18:59 cmjohnson1: mw1041 is borked and will be powered down and removed from network for h/w checks
* 18:27 logmsgbot: aaron synchronized php-1.21wmf9/extensions/TranslationNotifications 'deployed d7bf60b57f472e5f75101f58505f61842848f961'
* 16:26 Nemo_bis: Job queue weirdness and TranslationNotifications duplicates again noticed on Meta (bug 42715)
* 13:33 logmsgbot: reedy synchronized php-1.21wmf9/resources/mediawiki
* 09:17 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48804/'
* 09:10 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48804/'
* 08:52 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48797/'
* 08:25 hashar: jenkins: changing encryption key and regenerating secrets. See 
* 05:50 logmsgbot: tstarling synchronized multiversion/MWScript.php
* 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Feb 13 02:52:39 UTC 2013
* 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 13 02:28:06 UTC 2013
* 00:42 notpeter: restarting varnishncsa on cp1043
* 00:03 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend 'Touchhhhh'
== February 12 ==
* 23:54 logmsgbot: awjrichards synchronized php-1.21wmf9/extensions/MobileFrontend/javascripts/specials/donateimage.js 'touch file'
* 22:58 logmsgbot: maxsem Finished syncing Wikimedia installation... :
* 22:46 notpeter: disabling innodb-flush-log-at-trx-commit=2 on db1047 w/asher's blessing. will schedule downtime to apply
* 22:38 logmsgbot: maxsem Started syncing Wikimedia installation... :
* 22:04 notpeter: creating metrics user on all s1-7 masters
* 20:25 logmsgbot: reedy synchronized php-1.21wmf9/resources/mediawiki
* 20:22 binasher: flushed mobile varnish
* 20:21 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend
* 20:19 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend
* 20:06 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 20:05 binasher: flushed mobile varnish cache �:(
* 19:59 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend
* 19:57 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend
* 19:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable WikibaseClient on enwiki'
* 19:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable WikibaseClient on enwiki'
* 18:49 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
* 18:41 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
* 18:36 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
* 18:25 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
* 18:20 cmjohnson1: nas1001-a swapping mainboard
* 18:10 logmsgbot: aaron synchronized php-1.21wmf9/extensions/UploadWizard 'deployed 4548f9817d3d8025f4c19e279e9fb1f05b178087'
* 18:09 logmsgbot: catrope synchronized php-1.21wmf9/extensions/Wikibase 'Un-rollback Wikibase'
* 18:06 RoanKattouw: Reverted Wikibase update earlier, now reapplying
* 17:53 paravoid: powercycling niobium, SSH & console unresponsive
* 17:50 mark: Blocking modules=mobile.startup requests with 503 on bits varnish (earlier)
* 17:50 mark: Forced mw1149-1150 to healthy in bits.eqiad varnish
* 17:30 logmsgbot: catrope synchronized php-1.21wmf9/extensions/Wikibase 'Rolling back Wikibase'
* 16:18 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 12 16:18:31 UTC 2013
* 16:13 Jeff_Green: restarted apache on mw1152
* 16:01 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
* 16:00 Reedy: mw1161 and mw1165 are both missing mediawiki deployment files
* 15:58 logmsgbot: reedy synchronized php-1.21wmf9/includes/site/
* 15:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 12 15:54:11 UTC 2013
* 14:25 mutante: ran puppet on professor
* 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 12 02:52:42 UTC 2013
* 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 12 02:28:53 UTC 2013
* 01:01 ^demon: stopped puppet and gerrit on manganese and formey
* 00:28 logmsgbot: aaron synchronized php-1.21wmf9/includes/db/Database.php 'deployed d8705542627f006a7ec9f81a9fb488fcc9a367bd'
== February 11 ==
* 23:19 RobH: unpooled mw1161-1188 from pybal config
* 22:03 logmsgbot: demon synchronized php-1.21wmf9/extensions/VisualEditor/modules/ve/init/mw/ve.init.mw.Platform.js 'Touching file'
* 21:08 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Wikibase client throwing fatals, disabling on enwiki pending fix'
* 21:04 cmjohnson1: auth-dns update
* 21:03 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'wikidata client on enwiki'
* 21:03 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'wikidata client on enwiki'
* 20:30 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: huwiki and itwiki to 1.21wmf9
* 20:27 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: hewiki and itwiki to 1.21wmf9
* 20:22 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf9
* 20:11 logmsgbot: demon synchronized php-1.21wmf9/includes/site 'Updating sites code to pull in I8964a03c, I42221832'
* 18:39 RobH: disregard, pushing one to test, mw1161
* 18:36 RobH: mw1161-mw1188 being pushed into pybal pool as general apaches
* 16:58 logmsgbot: demon synchronized php-1.21wmf9/extensions/Wikibase/lib/includes/modules/SitesModule.php 'Touching file'
* 16:50 logmsgbot: demon synchronized php-1.21wmf9/extensions/TorBlock 'TorBlock to master, now with less annoying logs'
* 16:41 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Enable logging group for torblock'
* 16:40 logmsgbot: demon synchronized php-1.21wmf9/extensions/TorBlock 'Updating TorBlock to master'
* 16:27 cmjohnson1: commenting out srv226-234 on pybal/pmtpa
* 14:59 logmsgbot: hashar synchronized wmf-config/CommonSettings.php ' hardcode $wgDBuser = "wikiuser" '
* 13:55 hashar: jenkins: regenerating all phpcs-HEAD jobs so they stop failing when no files need to be processed 
* 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Mon Feb 11 02:52:39 UTC 2013
* 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 11 02:27:44 UTC 2013
== February 10 ==
* 22:16 Tim: fixed syslog.eqiad.wmnet DNS entry, restarting rsyslogd on the eqiad apaches (RT 4500)
* 19:20 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/48015/'
* 02:53 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sun Feb 10 02:53:56 UTC 2013
* 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 10 02:28:46 UTC 2013
== February 9 ==
* 12:52 Reedy: mw1161 is missing /usr/local/apache/common-local
* 12:51 logmsgbot: reedy synchronized php-1.21wmf8/resources/
* 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sat Feb 9 02:52:45 UTC 2013
* 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 9 02:28:28 UTC 2013
== February 8 ==
* 23:48 logmsgbot: anomie synchronized php-1.21wmf8/extensions/Scribunto/engines/LuaCommon/LuaCommon.php
* 23:48 logmsgbot: anomie synchronized php-1.21wmf9/extensions/Scribunto/engines/LuaCommon/LuaCommon.php
* 23:09 RoanKattouw: and on cerium too
* 23:08 RoanKattouw: Running puppet on titanium
* 20:20 RobH: disregard errors for mw1161-1164
* 20:13 RobH: mw1161-1164 offline for reallocation from job to general apache
* 20:10 cmjohnson1: decommissioning srvs226-234
* 19:43 RobH: mw1165-1188 os installed for apache use, initial puppet runs in progress
* 19:33 RobH: celsus back online
* 19:23 RobH: celsus rebooting to fix power mgmt cpu spike issue
* 19:02 RobH: mw1161-1164 in service as eqiad jobrunners
* 16:32 ottomata: updated vumi packages in apt to vumi_0.5.0~a+143-0~ppa3 and vumi-wikipedia_0.1~a+14-0~ppa3
* 16:09 mark: Upgraded and rebooted cr2-eqiad (both REs)
* 15:11 mark: Set cr2-eqiad VRRP prio to 50 to transition to backup role
* 15:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 15:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: srwikibooks back to 1.21wmf9 to test variant fixes
* 14:39 hashar: gallium installed php-luasandbox manually. Gerrit change is .
* 05:47 logmsgbot: tstarling synchronized php-1.21wmf9/includes/context/RequestContext.php
* 02:54 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Fri Feb 8 02:54:49 UTC 2013
* 02:30 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 8 02:30:42 UTC 2013
* 02:02 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
* 01:48 LeslieCarr: cr2-knams is back up on the new code
* 01:36 LeslieCarr: cr2-knams reloading after upgrade
* 01:34 LeslieCarr: deactivating european bgp sessions - may cause some connectivity issues as connections reset
* 01:24 LeslieCarr: disabling notifications on icinga and nagios
* 01:03 logmsgbot: reedy synchronized php-1.21wmf9/languages/classes/
* 00:25 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/GettingStarted
== February 7 ==
* 23:31 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling GuidedTour on additional wikis'
* 23:25 LeslieCarr: manually cleared arp and turned managmeent interface off and on of asw-c-eqiad to solve reachability issue
* 23:02 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploy GuidedTour
* 22:47 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy GuidedTour
* 22:39 RobH: mw82 pooled into tampa apaches
* 22:04 ottomata: pulled power and rebooted amaranth toolserver for multichill. Its back online. multichill will notify whomever needs to do more work.
* 21:58 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/RelatedArticles/ 'Fixes 44761'
* 21:58 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/RelatedArticles/ 'Fixes 44761'
* 21:50 RobH: mw81-mw85 in pmtpa apache service
* 21:39 cmjohnson1: performing a hard reboot on amaranth
* 21:34 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/E3Experiments
* 21:33 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/E3Experiments
* 21:12 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/PostEdit
* 21:11 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/PostEdit
* 21:10 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/EventLogging
* 21:09 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
* 20:29 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Being paranoid, node lists were in flux during last sync
* 20:23 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: All ku/sr wikis back to 1.21wmf8 - lang converter breakage
* 20:20 Nemo_bis: !Wikipedia and !Wikimedia projects are/were down, issue being investigated
* 20:14 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: kuwiktionary back to 1.21wmf8
* 20:07 RobH: removing srv219-224 from rendering config in pmtpa
* 20:06 mark: Rebooting cr1-sdtpa
* 19:56 RobH: adding mw75-mw80 into tampa rendering pool
* 18:42 mark: Rebooted re0.cr1-eqiad
* 18:05 mark: Rebooting re1.cr1-eqiad
* 16:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 15:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 15:44 mark: Rebooting cr1-esams
* 15:40 logmsgbot: reedy synchronized php-1.21wmf8/cache/interwiki.cdb 'Updating 1.21wmf8 interwiki cache'
* 15:39 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb 'Updating 1.21wmf9 interwiki cache'
* 14:54 mutante: granting RT group 'toolserver admins' the 'modify ticket' permission in queue esams
* 14:38 mutante: planet now updating hourly instead of daily
* 14:13 mutante: planet speaks SSL now for all languages. (it does not force http->https but it can if there is a demand)
* 12:00 Nemo_bis: amaranth dead/unreachable from Europe, ping dies at xe-5-3-1.cr2-eqiad.wikimedia.org (173.241.131.218)
* 10:08 hashar: jenkins : regenerating jobs to make phpcs to show the sniff codes being used. 
* 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Thu Feb 7 02:52:26 UTC 2013
* 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 7 02:27:52 UTC 2013
* 02:23 logmsgbot: spage synchronized php-1.21wmf9/extensions/GuidedTour/modules/externals/mediawiki.libs.guiders/mediawiki.libs.guiders.submodule 'sync GuidedTour submodule omitted from 1.21wmf9'
* 01:57 RoanKattouw: Force-running puppet on wtp1 to figure out why it's broken
* 01:53 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'I9fd98309'
* 00:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 00:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 00:20 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance
* 00:00 logmsgbot: reedy synchronized langlist
== February 6 ==
* 23:59 RobH: authdns-update to add min.wikipedia.org via addwiki automation
* 23:56 logmsgbot: reedy synchronized php-1.21wmf8/cache/interwiki.cdb 'Updating 1.21wmf8 interwiki cache'
* 23:55 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb 'Updating 1.21wmf9 interwiki cache'
* 23:50 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 23:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 23:43 logmsgbot: reedy synchronized langlist
* 23:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 23:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 23:28 RobH: srv190-srv223 & srv225 should be offline, they are being decommissioned, can ignore nagios errors
* 23:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 23:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 23:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 23:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 23:13 RobH: specifically (i can spell, maybe)
* 23:13 RobH: removed from pybal specifially
* 23:12 cmjohnson1: removing srv192-218 and srv225 per rt 4438
* 23:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 23:05 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance/
* 21:36 cmjohnson1: srv193 is back online
* 21:21 cmjohnson1: srv193 power down to relocate to different rack
* 19:47 RobH: authdns-update
* 19:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikibooks to 1.21wmf9
* 19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiquote to 1.21wmf9
* 19:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews, wikivoyage and wikiversity to 1.21wmf9
* 19:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special and wikimedia wikis to 1.21mwf9
* 19:11 cmjohnson1: powerdowing mw1 to reseat DIMM B1
* 19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, private and fishbowl wikis to 1.21wmf9
* 18:43 ^demon: running puppet on manganese, restarting gerrit
* 18:41 RobH: authdns-update
* 17:48 LeslieCarr: restarted morebots
* 02:51 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 6 02:51:36 UTC 2013
* 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Feb 6 02:28:05 UTC 2013
* 02:05 logmsgbot: reedy synchronized wmf-config/
* 00:42 RobH: mailing lists returned to normal
* 00:41 RobH: done with data removal on mailing list server, rebuilding mbox
* 00:37 RobH: stopping mailman to scrub an entry out of archives, will restart shortly
== February 5 ==
* 23:29 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend/javascripts/modules/mf-photo.js 'https://gerrit.wikimedia.org/r/#/c/47670/'
* 23:26 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/javascripts/modules/mf-photo.js 'https://gerrit.wikimedia.org/r/#/c/47670/'
* 22:44 logmsgbot: maxsem Finished syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-02-05
* 22:28 logmsgbot: maxsem Started syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-02-05
* 22:08 RobH: analytics1007 coming down for troubleshooting, disregard nagios errors
* 22:01 RobH: analytics1001 reinstalled, not puppet signed, back online
* 22:00 RobH: analytics1001 respects my authority.
* 21:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 44688 - Please disable MoodBar extension on English Wikipedia'
* 21:28 logmsgbot: robh synchronized wmf-config/InitialiseSettings.php
* 21:12 RobH: analytics1001 going down for troubleshooting/reinstallation
* 20:21 logmsgbot: kaldari Finished syncing Wikimedia installation... :
* 20:12 RobH: blog update sucessful, caching plugin update nonsuccessful, rolling it back to older plugin
* 20:07 logmsgbot: kaldari Started syncing Wikimedia installation... :
* 20:04 RobH: updating the blog software, here goes nothing
* 19:17 hashar: gallium : Zuul updated!
* 19:13 hashar: gallium : updating Zuul by cherry-picking a set of changes that let us fetch tags. ff79197..1854e32
* 17:05 cmjohnson1: authdns update
* 16:37 LeslieCarr: copied all equinix ashburn IPv4 peering sessions to their new /22 on cr2-eqiad
* 16:37 Jeff_Green: dist-upgrades for payments systems
* 16:36 Jeff_Green: restarted morebots :-P
* 05:37 Ryan_Lane: restarting opendj on all hosts
* 05:36 Ryan_Lane: remove virt1 replication agreement from all ldap servers
* 02:51 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 5 02:51:10 UTC 2013
* 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 5 02:29:02 UTC 2013
* 01:07 logmsgbot: reedy synchronized php-1.21wmf9/extensions/TimedMediaHandler
* 00:51 logmsgbot: reedy synchronized php-1.21wmf8/extensions/ZeroRatedMobileAccess
* 00:39 logmsgbot: reedy synchronized php-1.21wmf8/extensions/ZeroRatedMobileAccess
* 00:29 logmsgbot: reedy synchronized php-1.21wmf9/extensions/ZeroRatedMobileAccess
== February 4 ==
* 23:49 logmsgbot: reedy synchronized multiversion/
* 23:46 logmsgbot: reedy synchronized multiversion/
* 23:45 logmsgbot: reedy synchronized wmf-config/
* 23:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
* 21:02 hashar: gallium moved a git extension : mv /var/lib/git/rt-authen-externalauth.git /var/lib/git/operations/debs/
* 19:44 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
* 19:39 ottomata: manually gracefully reloaded eqiad apaches. See RT 4449
* 19:37 logmsgbot: root gracefulled all apaches
* 19:20 logmsgbot: reedy synchronized wmf-config/
* 19:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, mediawikiwiki and wikidatawiki to 1.21wmf9
* 18:35 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
* 17:56 logmsgbot: reedy Finished syncing Wikimedia installation... : Build messages for 1.21wmf9
* 17:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 17:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 17:24 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
* 17:14 logmsgbot: reedy synchronized php-1.21wmf9/extensions/GuidedTour/
* 17:09 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
* 17:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
* 17:06 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
* 16:45 logmsgbot: reedy synchronized php-1.21wmf9/ 'Initial sync'
* 16:22 logmsgbot: reedy synchronized wmf-config/
* 16:06 Ryan_Lane: force starting any glusterfs volume that has a downed process
* 15:12 Ryan_Lane: rebooting labstore1
* 15:11 Ryan_Lane: upgrading labstore1 to precise
* 14:33 apergos: powercycled ms-be3
* 14:32 Ryan_Lane: make that rebooting labstore3
* 14:32 Ryan_Lane: rebooting labstore2
* 14:07 Ryan_Lane: upgrading labstore3 to precise
* 11:14 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'cbd432f..82ed797 Vector micro design on mlwiki, wikipedia translation on pawiki'
* 07:37 Ryan_Lane: upgrading labstore2 to precise
* 02:25 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Mon Feb 4 02:25:58 UTC 2013
* 00:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 00:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
== February 3 ==
* 20:58 hashar: jenkins : generating mwext-**-phpcs-HEAD jobs
* 17:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 17:42 logmsgbot: reedy synchronized wmf-config/
* 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sun Feb 3 02:28:29 UTC 2013
* 02:03 logmsgbot: reedy synchronized wmf-config/
* 02:02 logmsgbot: reedy synchronized images/sul/foundation.png
* 01:14 logmsgbot: reedy synchronized wmf-config/
* 00:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 00:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
== February 2 ==
* 23:59 logmsgbot: reedy synchronized wmf-config/
* 23:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 23:34 logmsgbot: reedy synchronized wmf-config/
* 23:30 logmsgbot: reedy synchronized php-1.21wmf8/extensions/SearchExtraNS
* 23:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 22:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 19:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 16:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 16:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 09:23 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/PageImages/initImageData.php 'https://gerrit.wikimedia.org/r/#/c/44556/'
* 08:45 MaxSem: Running foreachwikiindblist wikipedia.dblist extensions/PageImages/initImageData.php --earlier-than=20121229000000
* 03:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sat Feb 2 02:28:18 UTC 2013
== February 1 ==
* 23:02 K4-713: synchronized payments cluster to b421871b43822
* 20:00 RobH: authdns-update
* 19:43 RobH: authdns-update
* 14:45 cmjohnson1: authdns update "adding mw1161-1200 to eqiad mgmt and production zone files
* 12:04 cmjohnson1: authdns update adding db1051-60 to zone files
* 02:27 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Fri Feb 1 02:27:09 UTC 2013
* 01:50 K4-713: updated payments cluster to 45f8a6c6009a4bd7
== January 31 ==
* 23:14 andrewbogott: rebooting virt0 in a fit of optimism and/or desperation
* 22:44 logmsgbot: spage Finished syncing Wikimedia installation... : take 2: Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
* 22:37 logmsgbot: spage Started syncing Wikimedia installation... : take 2: Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
* 22:26 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
* 22:20 ottomata: deployed tab delimiter log format change on squids, varnishes and nginxes
* 21:40 ottomata: reverted tab separartor change to squid
* 21:37 ottomata: deloyed tab separator log format changes to squids, merged the corresponding puppet changes to varnish and nginx
* 20:47 paravoid: deploying patched ganglia and removing .htaccess. Ganglia is public again!
* 14:59 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase
* 14:58 notpeter: taking down db1047 for upgrade to precise
* 13:21 hashar: gallium: installed mercurial manually (puppet change is PS2)
* 13:19 hashar: gallium manually removed bzr: apt-get remove bzr python-keyring python-httplib2 python-launchpadlib python-zope.interface python-oauth python-bzrlib bzr python-simplejson python-configobj python-lazr.uri python-lazr.restfulclient python-wadllib
* 13:19 hashar: gallium: manually installed "bzr"
* 12:12 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase
* 10:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Remove strategyappswiki
* 02:29 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Thu Jan 31 02:29:47 UTC 2013
* 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Thu Jan 31 02:28:18 UTC 2013
== January 30 ==
* 22:40 binasher: deployed new text/upload squid confs with eqiad in proxySubnets
* 22:38 logmsgbot: reedy synchronized wmf-config/
* 22:23 logmsgbot: tstarling cleared profiling data
* 22:12 logmsgbot: aaron synchronized php-1.21wmf8/includes/filerepo/file/LocalFile.php 'deployed f884911f5562fc41980664c424e7037bde6ba110'
* 22:11 logmsgbot: aaron synchronized php-1.21wmf8/includes/filerepo/file/OldLocalFile.php 'deployed f884911f5562fc41980664c424e7037bde6ba110'
* 21:59 LeslieCarr: killing varnishhtcpd on cp3009 for realzies this time
* 21:58 RobH: mw23 offline, reinstalled with precise and signed into puppet, puppet throwing errors on initial run.
* 21:52 binasher: started udplog with correct mw config on fluorine
* 21:49 LeslieCarr: restarting varnishhtcpd on cp3010 - it appears to be locked up , netstat -l shows a huge Recv-Q for port 4827 and appears to be the previously discovered issue
* 21:43 binasher: restarting udp2log on fluorine
* 20:59 logmsgbot: reedy cleared profiling data
* 20:50 RobH: authdns-update
* 20:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 20:20 RobH: mw23 was stuck in installer, restarting and setting up.
* 20:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 19:45 logmsgbot: reedy synchronized wmf-config/ 'Enabling wikidata client on itwiki and hewiki'
* 19:42 ottomata: added updated udplog_1.8-5 .debs to apt.wikimedia.org. No real changes, these now include the packet-loss program in the package.
* 19:22 logmsgbot: reedy synchronized wmf-config/
* 19:12 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase/
* 19:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf8
* 18:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
* 18:48 Ryan_Lane: downgrading memcached to ubuntu version on virt0
* 17:18 Reedy: Created wb_changes_dispatch on wikidatawiki
* 16:55 Reedy: Created sites/site_identifier tables on hewiki and itwiki
* 16:34 cmjohnson1: mw1072 going to down to check h/w state currently failed rt4381
* 15:38 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 44395) Allow bureaucrats to remove the translateadmin group on wikidatawiki '
* 15:37 logmsgbot: hashar synchronized wmf-config/throttle.php '(bug 44506) raise throttle for an Israel editthon '
* 12:54 logmsgbot: maxsem synchronized php-1.21wmf7/extensions/GeoData
* 12:52 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/GeoData
* 05:55 Tim: rebooted kaulen, was not responding on network or serial console
* 03:11 mutante: wikimania.asia redirect now works, it did not before because of a bug in apache-graceful-all not actually restarting Apaches in eqiad
* 03:07 mutante: reloading Apaches in eqiad using manual dsh (because apache-graceful-all sanity check has a bug)
* 02:55 logmsgbot: catrope gracefulled all apaches
* 02:52 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Wed Jan 30 02:52:03 UTC 2013
* 02:39 logmsgbot: dzahn gracefulled all apaches
* 02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Jan 30 02:28:22 UTC 2013
* 00:44 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
* 00:43 mutante: sync InitialiseSettings.php for bug 44473
* 00:19 binasher: restarted pdns-recursor on dobson
* 00:13 logmsgbot: reedy synchronized wmf-config/
== January 29 ==
* 23:54 logmsgbot: reedy synchronized wmf-config/
* 23:21 logmsgbot: pgehres synchronized php-1.21wmf8/extensions/CentralNotice/ 'Updating CentralNotice to master, resolving namespace issues'
* 23:17 logmsgbot: pgehres synchronized php-1.21wmf7/extensions/CentralNotice/ 'Updating CentralNotice to master, resolving namespace issues'
* 23:00 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Enable Special:Nearby everywhere'
* 22:51 logmsgbot: maxsem synchronized php-1.21wmf7/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/46630'
* 22:48 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/46630'
* 22:47 Tim: rotated SAL wikitech page - test logging
Category:SAL