Server Admin Log/Archive 14

== December 15 ==
* 01:09 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'adding CN banner Jimmy1'
* 01:04 logmsgbot: !log tfinc synchronized php-1.5/wmf-config/CommonSettings.php 
* 23:29 logmsgbot: !log tfinc synchronized php-1.5/extensions/GeoLite/GeoLite_body.php 
* 23:29 logmsgbot: !log tfinc synchronized php-1.5/extensions/GeoLite/GeoLite.php 
* 21:30 Mark: !log Stopped backend squid on sq20; broken disk drive /dev/sda
* 20:55 Roan: !log Running namespaceDupes on kowiki for bug 20863
* 18:44 logmsgbot: !log andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Add DPL to strategywiki'
* 17:35 Mark: !log Shutdown will for decommissioning
* 14:23 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21439: Add autopatrolled group on simplewiktionary'
* 14:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21571: Allow sysops to add/remove flood flag for all users on simplewiki'
* 14:13 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21560: Add rollbacker group on cawiki'
* 11:47 RoanKattouw: Fixed three more broken renames on arwiki about 15 mins ago, but morebots doesn't like Arabic usernames :(
* 11:36 RoanKattouw: Fixing broken rename from Mohammed Khalil to MK on arwiki
* 10:23 tomaszf: outage window closed early for payments server. cc pipeline back up.
* 09:00 tomasz: taking outage on cc payments server.
* 03:22 tomaszf: bouncing mobile1 through cluster stop/start script
* 00:16 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Adding Craigs banner and landing page to tracking.'
== December 14 ==
* 22:47 mark: Migrated smokeping from will to streber
* 22:29 mark: Migrated torrus data from will to streber and moved the torrus service IP
* 20:08 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21601: space -> _ in ckbwiki namespace alias'
* 19:27 mark: Moved torrus from will to streber, but pending RRD migration
* 17:04 mark: Cleaned up configuration of br1-knams
* 16:54 mark: Moved Rancid from will to streber
* 16:15 mark: OS-installed Karmic on streber
* 14:33 RoanKattouw: Running namespaceDupes on enwikinews for bug 21428
* 14:33 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21428: Add WN: alias on enwikinews'
* 14:29 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21516: Enable subpages in Template namespace on eswikibooks'
* 14:24 RoanKattouw: Running namespaceDupes on zhwiki for bug 20641
* 14:24 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20641: namespace aliases for zhwiki'
* 14:18 RoanKattouw: Running namespaceDupes on kowiki for bug 20863
* 14:16 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20863: Add Portal namespace on kowiki'
* 11:47 logmsgbot: midom synchronized php-1.5/languages/classes/LanguageEt.php 
== December 13 ==
* 20:42 RoanKattouw_away: Corrected enwiki arbcom end time from 00:00 UTC on 14 Dec to 23:59 UTC on 14 Dec
* 19:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaGrammarForms.php 'r60010'
* 19:24 logmsgbot: catrope synchronized php-1.5/languages/classes/LanguageEt.php 'r60010'
* 19:23 RoanKattouw: Updating Estonian grammar forms for bug 20332
* 19:08 RoanKattouw: Running cleanupTitles on frwikisource for bug 20741
* 19:04 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20332: new logos, sitenames for etwiktionary, etwikibooks, etwikiquote'
* 18:38 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20280: add de, es, fr, it, pl as import sources on enwiki'
* 10:07 hcatlin: Loss of several thins on mobile1, required cluster restart
== December 12 ==
* 22:41 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20666: disallow anonymous page creation on idwiki'
* 22:26 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20442: rename Wikipedia namespace on mhrwiki'
* 22:24 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20442: change sitename for mhrwiki'
* 22:20 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20280: add meta and nostalgia as import sources on enwiki'
* 21:50 RoanKattouw: DB trouble on enwiki is over, thanks Domas
* 21:49 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 21:38 logmsgbot: catrope synchronized php-1.5/wmf-config/db.php 
* 21:38 RoanKattouw: Trying master switch from db22 to db12 (s1) again
* 21:36 RoanKattouw: Master switch failed because db22 is unresponsive
* 21:36 logmsgbot: catrope synchronized php-1.5/wmf-config/db.php 
* 21:35 RoanKattouw: Switching master on enwiki from db22 to db12
* 21:33 RoanKattouw: db22 (enwiki master) is down
* 20:35 RoanKattouw: Clearing out some ghost entries in categorylinks on {de,en,fr,it,ru}wiki per bug 15152
* 16:59 RoanKattouw: Strike my last, will do it later
* 16:58 RoanKattouw: Clearing out some ghost entires in categorylinks on {en,de,fr,ru,it}wiki per bug 15152
* 16:20 RoanKattouw: Fixing another rename (English peasant -> King of the North East) on enwiki
* 15:05 RoanKattouw: Cleaning up another incomplete rename (Lucasbfr -> Luk) on enwiki. Running this one in batches because it's larger
* 14:49 RoanKattouw: Cleaning up incomplete rename RS2007 -> RS1900 on enwiki (bug 12969)
* 14:33 logmsgbot: midom synchronized php-1.5/includes/User.php 'livehack to check who and when invalidates caches'
* 14:13 RoanKattouw: Running namespaceDupes on ptwiktionary
* 00:02 logmsgbot: fred synchronized php-1.5/wmf-config/CommonSettings.php 'Adding 2009_Notice_18_g-22_g-21_g.'
== December 11 ==
* 23:55 logmsgbot: fred synchronized php-1.5/wmf-config/CommonSettings.php 'Adding 2009_Notice46 to tracking.'
* 21:17 Rob: srv217 mainboard replaced, needs OS reinstallation, then its good to go
* 21:17 Rob: fred fixed morebots, yay
* 19:17 Rob: db27 back online, restarting mysql.
* 18:38 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.i18n.php 
* 18:33 Rob: cleared errors on ms2 by reseating the power connections
* 18:33 Rob: cleared errors on ms2
* 18:29 Rob: db27 is down for hardware replacement
* 18:26 RoanKattouw: Running namespaceDupes on ckbwiki
* 18:25 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21601, bug 21415: Create Portal namespace and rename Wikipedia namespace on ckbwiki'
* 18:23 Rob: shutting down myswl on db27, as it has to come down for hardware replacement
* 18:12 logmsgbot: catrope synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics_body.php 'Deploying r59957'
* 18:12 logmsgbot: catrope synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.i18n.php 'Deploying r59957'
* 18:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20996: import sources for pLwikt, not pTwikt'
* 17:01 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21783: Only allow sysops to upload on svwiktionary'
* 16:45 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21794: new logo for ruwikiversity'
* 16:42 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21792: add autoreviewer group on ptwiki'
* 16:36 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21783: Activate file uploads on svwiktionary'
* 16:17 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'actually enable patrolling on ruwikiversity'
* 16:15 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix broken apcond'
* 16:14 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21629: automatically grant patrol, autopatrol to users with 1000+ edits on ruwikiversity'
* 16:04 RoanKattouw: Running namespaceDupes on plwikisource
* 16:04 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21034: WS: alias, change name of user talk namespace on plwikisource'
* 15:58 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20996: import sources for plwikisource'
* 12:39 logmsgbot: midom synchronized php-1.5/includes/api/ApiQueryUserContributions.php 
* 12:06 logmsgbot: midom synchronized php-1.5/includes/api/ApiQueryUserContributions.php 
* 08:21 logmsgbot: tstarling synchronized php-1.5/includes/filerepo/LocalRepo.php 'reverted profiling'
* 08:17 logmsgbot: tstarling synchronized php-1.5/includes/filerepo/LocalRepo.php 'profiling file redirects'
* 01:26 logmsgbot: midom synchronized php-1.5/includes/api/ApiQueryUserContributions.php 
* 00:37 Tim: killed long-running (34ks) API contributions query on db12, apparently filled up /a/tmp
* 00:02 logmsgbot: aZaFred synchronized php-1.5/wmf-config/CommonSettings.php 'Adding 2009_Notice_47-48 to tracking.'
== December 10 ==
* 23:03 domas: enwiki is on 5.1
* 23:03 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 18:49 Fred: ro.planet.w.o is now working
* 17:05 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 16:05 Rob: srv217 is powered on, but not in LVS pool and is just running dell diagnostics, please do not touch.
* 15:47 Rob: ms8 required controller board assembly to be reseated, system LOM now can control it properly.
* 15:39 domas: btw, side effect of my maintenance is that job queue was unclogged, yay.
* 15:19 Rob: bringing down srv217 to run diagnostics
* 14:38 domas: s1 master switch to db22-bin.000001:106
* 14:37 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 14:36 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 14:36 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 14:29 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 14:15 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 14:12 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 12:41 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 12:33 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 11:38 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 11:32 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'taking down one more server oh noes'
* 11:30 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 11:29 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 11:18 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 04:55 atglenn: unpacking the last three months of incrementals on ms7
* 02:19 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 00:34 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'stress testing hehehehehe'
* 00:27 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 00:26 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'increasing db12 load'
* 00:24 Andrew: db12 recovered, db26 still barely coping with the load
* 00:17 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'unloading general load from db12, excessive lag >800s'
* 00:09 Andrew: db12 seems not to be replicating at all.
** Note that it actually was replicating, it was just slow due to excessive read load, added by domas at 23:06 when he balanced the loads incorrectly in db.php.
* 00:09 Andrew: make that db12, not db22
* 00:06 Andrew: s1 lagged, db12 282s, db26 33s -- no slave servers up-to-date enough to serve pages
== December 9 ==
* 23:53 logmsgbot: fred synchronized php-1.5/wmf-config/CommonSettings.php 'Adding 2009_Notice_43-44-45 to tracking.'
* 23:06 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'taking down one more server oh noes'
* 22:59 Fred: fixed noc.wikimedia.org
* 21:08 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'db14 offline'
* 20:54 atglenn: open support call with sun for ms8, won't power up (no POST), even after resetting ILOM
* 17:27 logmsgbot: andrew ran sync-common-all 
* 17:18 Andrew: Updating production LiquidThreads installations to new beta version, branch of current alpha.
* 17:08 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20253 Install Extension:NewUserMessage on strategy wiki'
* 13:39 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js 'Merge r59886'
* 13:38 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r59885'
* 13:08 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r59883'
* 13:05 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/jquery/js2.combined.js 'merge r59882'
* 12:50 logmsgbot: andrew ran sync-common-all 
* 12:39 Andrew: Updating LiquidThreads to trunk state, using sync-common-all to deploy to apaches (since scap crashes the cluster)
* 00:04 RoanKattouw: Manually reallocating edits from User:Until It Sleeps to User:The Thing That Should Not Be on enwiki
== December 8 ==
* 23:51 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21718: Allow sysops to add/remove abusefilter too on arwiki'
* 23:46 logmsgbot: Big_Fred synchronized php-1.5/wmf-config/CommonSettings.php 'Added tracking for Notice_42'
* 23:31 logmsgbot: catrope synchronized php-1.5/includes/DefaultSettings.php 'Deploying r59858'
* 23:31 logmsgbot: catrope synchronized php-1.5/includes/Skin.php 'Deploying r59858'
* 23:31 logmsgbot: catrope synchronized php-1.5/skins/vector/main-rtl.css 'Deploying r59858'
* 23:31 logmsgbot: catrope synchronized php-1.5/skins/vector/main-ltr.css 'Deploying r59858'
* 23:31 logmsgbot: catrope synchronized php-1.5/skins/common/ajaxwatch.js 'Deploying r59858'
* 23:26 TrevorParscal: Running sync-file repeatedly to deploy r59858 (usability fixes)
* 23:16 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21718: allow bureaucrats to add&remove abusefilter group on arwiki'
* 23:12 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'bug 21718: Restrict abusefilter-modify to abusefilter group on arwiki'
* 22:13 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialNoticeText.php 'picking up r59852'
* 19:09 logmsgbot: Big_Fred synchronized php-1.5/wmf-config/CommonSettings.php 'added geo-location for Italie.'
* 16:43 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20216 AbuseFilter for itwikiquote'
* 16:43 logmsgbot: andrew synchronized php-1.5/wmf-config/abusefilter.php 'Bug 20216 AbuseFilter for itwikiquote'
* 16:20 Rob: running diagnostics on srv217, please do not try to put it into service.
* 16:06 Rob: srv229 memory replaced, system restarted.
* 15:58 Rob: shutting down srv229 to swap out faulty dimm
* 15:46 Rob: sq43 disk replaced, system back online and in rotation
* 15:31 Rob: replaced bad disk in sq43
* 15:23 Rob: replaced bad disk in sq37, detected, restarted squid backend
* 15:18 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 15:09 Rob: db27 back up with mysql running.
* 15:07 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 15:03 Rob: mysql stopped normally on db27, restarting it.
* 15:02 Rob: was able to hotswap the fan, shutdown was not needed
* 14:58 Rob: stopping mysql on db27 to shut it down and swap out a bad fan assembly.
* 14:56 Rob: disregard, did not receive part for db28, just db27
* 14:52 Rob: db28 powered down already, pulling to replace bad fan
* 11:12 rainman-sr: searchidx1 rebooted by domas, restarting indexing
* 11:00 rainman-sr: searchidx1 has a "cp -lr" process stuck in I/O, cannot be killed, dmesg says "aacraid: Host adapter reset request. SCSI hang ?". Stopping indexing until someone sorts this out. Hardware check, reboot?
* 01:38 domas: new s2 position db30-bin.000001:106 
* 01:28 domas: s2 is pure 5.1 now
* 01:27 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 01:22 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 01:22 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 01:21 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'up up up'
* 00:37 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 00:36 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Added Site_Notice41 to tracking.'
* 00:19 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
== December 7 ==
* 22:16 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'taking out one more server for a clone'
* 21:47 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Lower $wgClickTrackThrottle to 1 on strategywiki, requested by Eugene and Nimish'
* 19:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Lower $wgClickTrackThrottle from 1000 to 100'
* 19:35 RoanKattouw: Cranking up click tracking ratio from 1:1000 to 1:100 now that we're only tracking toolbars
* 19:34 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/ClickTracking/ClickTracking.php 'Deploy r59793, r59794: disable clicktracking for left navigation'
* 19:33 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/ClickTracking/ClickTracking.js 'Deploy r59793, r59794: disable clicktracking for left navigation
* 19:29 RoanKattouw: Issues mentioned by Rob probably caused by running svn up as root. Please don't do that again
* 19:26 Rob: spoke with Roan and Brion and updated the permissions of the wmf-deployment to put all files to wikidev group and all writable by that group
* 19:24 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php '... and add w:en'
* 19:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20552: Remove b: prefix from import sources on ptwikibooks'
* 18:41 RoanKattouw: Running namespaceDupes on ttwiki for bug 21656
* 18:40 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21656: Add Portal namespace on ttwiki'
* 18:25 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21506: add import sources for plwiki'
* 18:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21755: add commons as import source on testwiki'
* 17:09 Fred: removed locke from wmnet zonefile since locke doesn't have internal address.
* 17:04 logmsgbot: aaron ran sync-common-all 
* 16:27 logmsgbot: aaron ran sync-common-all 
* 16:10 aaron: nvm, autopromote was fine since the duplicate setting was supposed to be the new plwiktionary addition
* 16:17 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'flaggedrevs for plwiktionary'
* 16:10 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed duplicate ptwikisource config (autopromote setting must have been broken)'
* 14:37 mark: Drive 0 SMART predicted failure on fuchsia
* 14:36 mark: Power cycling fuchsia
* 14:31 mark: Enabled knsq7 frontend squid in pybal
* 14:31 RoanKattouw: Running namespaceDupes on nlwiki for bug 21722
* 14:30 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21722: Add WP, H, P aliases for nlwiki'
* 13:59 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Removed decommissioned servers, added in a few more spares'
* 13:57 mark: Updated memcached list in puppet, wielded out old/down/decommissioned servers
* 13:23 mark: Removed memcached from srv151-153, 156
* 13:19 logmsgbot: root synchronized php-1.5/wmf-config/mc.php 'Replaced srv151,152,153,156 by new memcached nodes, added ~30 new spares'
* 13:04 mark: Installed memcached on srv226..253
* 11:42 Andrew: Ran sync-common on srv125, srv156 as they missed some of the most recent updates.
* 11:37 logmsgbot: andrew synchronized php-1.5/wmf-config/ExtensionMessages.php 'Updating LiquidThreads alpha to trunk state (r59781)'
* 11:36 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/pages/ThreadPermalinkView.php 'Updating LiquidThreads alpha to trunk state (r59781)'
* 11:36 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/pages/TalkpageView.php 'Updating LiquidThreads alpha to trunk state (r59781)'
* 11:33 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/pages/SpecialNewMessages.php 'Updating LiquidThreads alpha to trunk state (r59781)'
* 11:33 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js 'Updating LiquidThreads alpha to trunk state (r59781)'
* 11:30 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/i18n/Lqt.i18n.php 'Updating LiquidThreads alpha to trunk state (r59781)'
* 11:29 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Updating LiquidThreads alpha to trunk state (r59781)'
* 11:28 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/ThreadRevision.php 'Updating LiquidThreads alpha to trunk state (r59781)'
* 11:24 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Thread.php 'Updating LiquidThreads alpha to trunk state (r59781)'
* 11:22 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/api/ApiThreadAction.php 'Updating LiquidThreads alpha to trunk state (r59781)'
* 11:22 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/i18n/Lqt.i18n.php 'Updating LiquidThreads alpha to trunk state (r59781)'
* 11:16 Andrew: Some root-owned .svn directories spread throughout wmf-deployment.
* 11:13 Andrew: Updating LiquidThreads alpha to trunk state, this time by quickly sync-file-ing each individual file in a for loop so I don't bring down the site with scap
* 10:12 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'full load on db30'
* 00:51 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'reducing load for some inspection'
== December 6 ==
* 23:39 domas: s2 slave load is being handled by single 5.1-wm-3139 server
* 23:23 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'wheeeeeeeeee'
* 23:12 domas: srv83 was not in sync groups, but alive and serving load..
* 22:50 mark: ...on db30 :P
* 22:50 domas: and yes, it was db30
* 22:17 domas: s/bank8/bank9/
* 22:15 domas: bank13 & bank8 MCE warnings ( http://p.defau.lt/?KWMB35Z13ysXpN6IHcca9A )
* 22:13 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 20:22 mark: Puppetized NTP configuration on the Solaris servers
* 19:37 mark: Installed CSW pkgutil and puppet on ms4, updated it on ms5
* 18:00 apergos: resending incrementals for last three months from ms1 to ms7 with -I to get the intermediate snaps. using netcat, running in screen as root on both hosts
* 15:54 apergos: cleaned up / on ms1, was out of space (tossed some old files from /root)
* 01:20 mark: Disabled xinetd and extdist crontab on zwinger
* 00:40 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Moved svn-invoker (ExtensionDistributor) from zwinger to fenari'
* 00:27 mark: sq27 is flooding syslog; placed temporary firewall entry for syslog packets on nfs1
== December 5 ==
* 03:26 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionStatistics_body.php 'picking up bugfix from r59753'
* 00:46 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'adding CN Notice 22'
* 00:44 atglenn: start transfer of incremental via zfs send (600gb?) from ms1 to file on ms4, in prep for nc to ms7 later, running in screen as root on ms1
* 00:14 logmsgbot: Fred synchronized php-1.5/wmf-config/InitialiseSettings.php 'changed logo for usabilitywiki.'
* 00:11 logmsgbot: Fred synchronized php-1.5/wmf-config/InitialiseSettings.php 'changed logo for usabilitywiki.'
== December 4 ==
* 23:30 atglenn: started netcat of the bulk of the data from ms5 to ms7. running in screen as root on both hosts. 
* 23:21 atglenn: started ncat of (small piece of) image date from ms5 to ms7, running in screen as root on both hosts
* 20:47 Rob: which doesnt work, damn.
* 20:47 Rob: got sick of racktables.wikimedia.org not redirecting correctly, put in a rewrite for non ssl connections to ssl
* 20:24 Fred: fixed nrpe on db20 and db7
* 20:13 logmsgbot: root ran sync-common-all 
* 20:12 Rob: running sync-common-all to update configuration for support of flaggedrevs on plwiktionary
* 19:20 Rob: srv144 removed from node groups & pybal, nagios resynced.
* 19:19 Rob: srv144 is out of warranty and rebooting randomly, decommissioning.
* 19:05 Fred: finished setup of srv245.
* 19:02 Rob: srv126 removed from node groups and lvs. nagios restarted to exclude it.
* 19:01 Rob: srv126 refuses to even post when benched, out of warranty, slating for immediate decommissioning
* 19:00 Rob: srv144 reinstalling with a single hard disk, no more raid1
* 18:50 Rob: swapped primary srv144 drive with old decommissioned spare. reinstalling OS, will reinstall packages and get online later.
* 18:45 Rob: sq22 back online, all drives nominal, rebuilding cache and ensuring it is in rotation
* 18:41 Rob: rebooted sq22
* 18:38 Rob: rebooted srv144 and srv126
* 18:36 Rob: srv245 package install failed. I do not have time to tinker with it while in the DC, I have other things that require my physical access to the machines. Leaving it alone for now to work on remotely.
* 18:28 Rob: srv245 OS installed, setting up wikimedia-task-appserver
* 18:06 Rob: srv245 was sitting idle with no OS, depooled from apaches. reinstalling system.
* 17:57 Rob: rebooted srv83 per fred
* 17:35 Fred: removed srv83 from the nodelist since it was causing ddsh to never finish executing.
* 17:26 Fred: fixed broken apache. Seems like there is a machine down that is preventing normal sync-file from finishing... Looking into it.
* 16:50 rainman-sr: stopped logging of search queries on searchidx1 until someone sets up proper log archiving to a different machine
* 16:48 rainman-sr: searchidx1 had full disk, freed some 100gb of space by deleting logs and stuff laying around
* 16:14 Rob: srv245 down and unresponsive, rebooting
* 16:12 Rob: sq43's replacement disk is also bad (talk about bad luck), placing rma with dell. system will remain powered down for now.
* 15:55 Rob: sq43 isn't seeing a replaced disk, rebooting and troubleshooting
* 15:33 domas: 'arcconf setcache 1 logicaldrive 0 roff ' - disabling any read caching on db11-db30 RAIDs
* 15:13 Rob: after tinkering with it with domas, it appears rebuild is indeed automatic. db21 rebuilding raid array
* 15:09 Rob: db21 bad disk swapped out, rebuild should be automatic
* 14:57 Rob: sq14 back up, rebuilding its cache
* 14:54 Rob: sq13 primary disk dead, out of warranty
* 14:53 Rob: swapping sdc in sq13 and sq14 to bring sq14 back online
* 14:53 Rob: sq14 disk sdc dead, out of warranty.
* 05:18 Tim: on fenari: running all pending renameUser jobs from enwiki
* 03:37 Tim: Around 03:12, accidentally renamed enwiki's job table and so renamed it back a second later. This caused all slaves to stop due to a replication bug. Fixed now.
* 03:25 Tim: testing fixJobQueueExplosion.php on commonswiki
* 02:46 Tim: srv156 not responding to ssh, trying reboot
* 01:13 Tim: restarting job runners
* 01:13 logmsgbot: tstarling synchronized php-1.5/includes/HTMLCacheUpdate.php 'patching out all category backlink updates, major bug causing job queue to stall'
* 00:12 Tim: granted access to root@fenari on all servers in the mysql node group
== December 3 ==
* 23:46 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Allow bcrats to add and remove new arbcom group on nlwiki'
* 23:40 RoanKattouw: Synced InitiatiseSettings.php: allow bcrats to add and remove new arbcom group on nlwiki
* 22:49 RoanKattouw: Importing 365 images into Commons as User:GeographBot, requested by Multichill
* 22:39 RoanKattouw: Synced InitialiseSettings.php for bug 21238: self-removal of flood flag on plwiki
* 22:33 RoanKattouw: Synced InitialiseSettings.php for bugs 20775 and 21719. sync-file is stalling on what seems to be an unresponsive server
* 21:35 RoanKattouw: Running namespaceDupes on usabilitywiki for bug 21753
* 21:35 RoanKattouw: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21753 Fix Multimedia talk NS on usabilitywiki'
* 04:20 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionTrackingStatistics_body.php 'fixing conversion rate bugs'
== December 2 ==
* 23:28 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'reenabling db18 and db25, also, attempting to overwrite stale db.php copies'
* 23:25 Fred: massaged mc.php to retrieve working spare, and remove broken memcached nodes. all is now good in the land of memcache
* 22:12 mark: Recovered torrus from deadlock
* 21:00 Fred: rebooted srv194 (hung)
* 20:48 Rob: removed bayle and khaldun from dsh, both are in rack running wipe with network pulled
* 20:38 Fred: bart removed from nagios (well that sounds funny)
* 20:36 Rob: khaldun is down forever! decomissioned and running wipe in rack with the network pulled
* 20:35 Rob: isidore rebooted by accident due to power cable issues
* 20:21 Rob: srv136 crashed with temp warnings, going to decommission it, rebooting to wipe and remove network
* 20:15 Rob: bart decommissioned, unracked, wipe running on testbench with usbcdrom
* 19:49 Rob: decommissioned, unracked srv66, srv51, srv81, srv118 (previously removed from pybal)
* 19:39 Rob: decommissioned srv130, unracked
* 19:20 Rob: srv122 decommissioned, wiped, unracked
* 18:19 Rob: ms7/ms8 racked in sdtpa a2, network wired, dns setup, racktables updated, & LOM online
* 18:18 Rob: serial connection to ps1-a4-sdtpa returned to normal
* 18:05 Rob: ps1-a4-sdtpa temp losing its serial connection, stealing adapter to setup ms7/8
* 18:04 Rob: added ms7/ms8 to dns for wmnet and mgmt nics
* 16:20 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Deploy r59665'
* 16:19 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/jquery/js2.combined.js 'Deploy r59665'
* 15:40 Rob: rebooted the following per domas request: srv101 srv105 srv112 srv117 srv138 srv183 srv89 srv84 srv89 srv91 srv96 srv98 srv99
* 15:02 mark: Shutting down mint for installation of a wifi card
* 14:35 Andrew: Scapping to update LiquidThreads alpha
* 14:34 Andrew: Updates of LiquidThreads alpha to trunk state in progress
* 13:46 RoanKattouw: Remove all LU cache files and rebuild from scratch to mitigate problems with root-owned cache files
* 13:41 RoanKattouw: Purging LocalisationUpdate hashes and running the update script to ensure we didn't miss anything in the Nov 19 - Dec 1 blackout
* 05:28 Tim: edited authz on mediawiki svn to deny write access to root
* 01:17 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'new banner for cn'
* 01:04 Tim: restarted apache on srv211 and srv127
== December 1 ==
* 20:35 logmsgbot: catrope synchronized php-1.5/includes/DefaultSettings.php 'Deploying r59608 (clicktracking for old toolbar)'
* 20:34 logmsgbot: catrope synchronized php-1.5/skins/common/edit.js 'Deploying r59608 (clicktracking for old toolbar)'
* 15:20 apergos: temporarily turned off upload-by-url on test.wikipedia, it interferes with testing the external page retrieval extension for fundraising
* 01:00 Tim: (45 minutes ago) removed non-stewards from the arbcom admin list
* 00:26 logmsgbot: tstarling synchronized php-1.5/extensions/SecurePoll/includes/ballots/RadioRangeBallot.php 
== November 30 ==
* 23:30 brion: restarted parser test runner on wikitech [@brion fixme: set up init script]
* 22:54 Fred: rebooting pascal since it once again went down.
* 21:21 RoanKattouw: Fred moved l10nupdate cronjob on hume to /etc/cron.d/l10nupdate , now runs as brion instead of root
* 21:21 RoanKattouw: Unlocked /h/w/l10n/trunk/extensions , had been locked since Nov 19th blocking L10nUpdates to extensions
* 20:06 rainman-sr: set search limit to 51 per Roan
* 20:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.php 'Raise style version for r59598, r59599'
* 20:05 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.js 'r59599 for real this time'
* 20:04 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.js 'r59599'
* 20:00 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.js 'r59598'
* 19:55 rainman-sr: converting searchidx1 back to indexer, things seem to be back to normal now
* 19:46 atglenn: enabled ExternalPages on test.wikipedia (fundraising extension) after adding it by hand to extension-list and ExtensionMessages.php
* 19:30 rainman-sr: doing a delayed restart of all searchers
* 19:10 mark: Setup puppet to enforce desired JVM alternatives on the search servers
* 17:00 rainman-sr: all search servers (except search11) should be switched back from 64 bit java (java-6-openjdk) to 32 bit java (ia32-java-6-sun)
* 16:33 rainman-sr: temporarely stop indexing and put searchidx1 in en.wp rotation, will back out when peak times are over
* 15:28 mark: Puppet removed syslog-ng on nfs1/2 as well. Restored.
* 14:49 mark: Replacing stock syslogd with rsyslog globally across the cluster (with puppet). rsyslog is the default in Ubuntu 9.10 onwards
* 10:35 mark: Using streber for karmic install testing
* 10:35 mark: Using brewster for karmic install testing
* 10:19 mark: Created karmic-wikimedia APT repository
* 02:27 rainman-sr: added search11 to en.wp search rotation.. we're getting peak-level traffic on 2am on sunday?! 
== November 29 ==
* 21:32 mark: Unmounted /home on ms5
* 20:58 mark: Disabled NIS on ms5
* 20:54 mark: Carefully starting puppet management of ms5 (Solaris)... just ssh host key exchanging for now
* 20:42 mark: Installed Blastwave puppet package (and deps) on ms5
* 20:39 mark: Installed Blastwave pkgutil on ms5
* 20:37 mark: Uninstalled coolstack CSKruntime and CSKruby packages on ms5
* 20:16 mark: Installed coolstack CSKruntime and CSKruby packages on ms5
* 19:32 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 19:25 domas: s3 replication switched from db11-bin.215:36889084 to db17-bin.001:79
* 19:22 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 19:22 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 17:40 mark: Fixed puppet config for searchidx1
* 14:32 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'adding db1 and db7 to s6'
* 14:16 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Per Domas, do bug 21510 a little bit more performance-friendly'
* 14:12 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 21510: Second attempt'
* 14:04 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'backing out last change, breaks'
* 14:03 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 21510: Temp raise acct create throttle on eswiki for UEM IP for wiki week'
* 13:54 mark: Installed puppet on db20, added new db20 node block in site.pp
* 13:37 Tim: loaded arbcom election configuration into SecurePoll on enwiki
* 13:28 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'taking out db1 for db7 init'
* 13:19 RoanKattouw: Running l10nupdate on hume for debugging
* 12:37 domas: s6 start db29-bin.001:10246135 
* 12:32 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'welcome s6'
* 10:54 domas: enwiki db is now served by 32g/16disk servers only... what is our next upgrade? :-)
* 10:53 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'taking out db7 from enwiki duty, it is too tiny...'
* 04:05 logmsgbot: tstarling synchronized wmf-deployment/cache/trusted-xff.cdb 
* 04:03 Tim: updating TrustedXFF to r59540
* 03:20 Fred: distupgrade on filetserv1 (new kernel)
* 03:13 Fred: apt-get upgrade on fileserv1...
== November 28 ==
* 22:36 Fred: customized morebots startup script a tad, restarted it as user werdnum instead of root...
== November 27 ==
* 11:27 Raymond_: Run of ParserTests on mw.o/CodeReview is broken. Last run Nov, 25th
* 06:37 Fred: adjusted site.pp to removed nfs-home stanza for Bayes as it shouldn't be there. 
* 06:02 Tim: on fenari: running SecurePoll/cli/wm-scripts/makeArbcomList.php to generate arbcom election voter list
* 01:39 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'second attempt'
* 01:37 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'removing clicktracking right (which does nothing) from the sysop group'
* 00:29 Tim: fixed catrope's umask and ran: find -user catrope -exec chmod g+w {} \;
== November 26 ==
* 23:54 Tim: removed broken extensions from extension-list: MWSearchUpdateHook.php and LiquidThreads_alpha
* 23:47 RoanKattouw: Scapped, logmsgbot seems to be broken again in that it doesn't report scaps with the !log prefix
* 23:36 Tim: on fenari, did a source install of parsekit using the pecl tool, no package was available
* 23:21 Tim: installing MediaWiki dependencies on fenari
* 23:17 Tim: on fenari, added /home/wikipedia/bin to $PATH using /etc/profile
* 23:14 RoanKattouw: Deploying r59465 to make LocalisationUpdate work again for renamed usability messages
* 17:55 mark: Removed NIS on yongle
* 17:23 mark: Removing wikimedia-nis-client on all search servers
* 17:01 mark: Running apt-get upgrade on all search servers
* 16:43 mark: Removed wikimedia-nis-client on srv225
* 16:41 mark: Removed wikimedia-nis-client on spence
* 16:38 mark: Removed wikimedia-nis-client on hume
* 16:33 mark: Removed wikimedia-nis-client on fenari
* 16:32 mark: Removed wikimedia-nis-client on amane
* 16:27 mark: Removed wikimedia-nis-client on srv124
* 16:27 mark: Replacing all NIS clients by puppet managed users
* 15:08 mark: Set domain of fenari back to 'wikimedia.org' from 'pmtpa.wmnet'. Why was it changed? Quite a few things got confused...
* 15:04 mark: Made NFS /home mounting puppet managed on all relevant servers
* 13:18 mark: Running apt-get dist-upgrade on pdf1, in part to upgrade libpoppler (PDF rendering library)
* 13:18 mark: Removed rogue apt repositories on pdfd1
* 02:24 Tim: started apache on spence
== November 25 ==
* 23:46 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 
* 18:16 Fred: rebooted wikitech linode / added to Nagios for basic monitoring.
* 16:28 mark: Starting NFS failover test on nfs1 & nfs2
* 14:57 mark: Setup syslog-ng server on nfs1/nfs2 with puppet. Flipped syslog-ng.pmtpa.wmnet CNAME to point to the NFS service ip
== November 24 ==
* 23:50 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Adding 2009_Notice30_bold'
* 23:00 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'deebee seventeen and twenteeseven'
* 22:13 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'adding db21 and db29 back'
* 21:20 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'taking out db17 and db29 too'
* 21:16 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'taking out db21 and db27, AND NO PUPPETS'
* 21:14 mark: Setup logrotate for /home/w/logs/ on both nfs1/nfs2 (of course, with puppet ;-)
* 19:07 mark: Resized JFS filesystem of /home on nfs2 to fit the new LV size on nfs1/nfs2
* 19:03 mark: Moved cronned rsync of /home from db20 to nfs1 and nfs2, managed by puppet
* 18:46 mark: MediaWiki logging, broken by yesterday's NFS service IP migration, restored (using puppet)
* 18:15 mark: Puppetised rsyncd setup on nfs1/nfs2
* 18:00 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Flood flag for strategywiki bug 20347'
* 01:43 Tim: enabled write-back caching on db14 despite broken controller battery, due to excessive lag and MASTER_POS_WAIT()
* 00:24 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'It turns out that this time FiveFacts is actually referenced as Five_Facts_About_Wikipedia... who knew...'
* 00:10 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'It turns out that this time FiveFacts is actually referenced as Five_Facts_About_Wikipedia... who knew...'
* 00:01 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Adding Notice_[36,37,28] and FiveFacts landing page'
== November 23 ==
* 21:42 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Adding RU and IL for GEOIP location.'
* 21:27 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Adding RU and IL for GEOIP location.'
* 20:53 river: moving ptolemy to TS vlan to reinstall it as the TS OSM db
* 20:11 mark: Finished migrating NFS off db20 to a DRBD cluster of nfs2 (primary) and nfs1 (secondary)
* 19:14 RoanKattouw: prototype got stuck, rebooted it. Linode sucks and we should not use it ever again
* 16:53 mark: Re-signed puppet cert for fenari... why was it gone?
* 12:39 domas: spence was taken down by runaway rrdtool process, http://p.defau.lt/?8FAf9FtfQTbkSQKCcP_qmw
* 11:00 Andrew: Scapping to prod message updates
* 10:49 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/pages/NewUserMessagesView.php 'Merge r59353'
* 10:35 Andrew: Scapping
* 10:34 Andrew: Updating LiquidThreads to trunk state on alpha only
* 08:28 domas: sq14 has failed sdc
* 02:05 Tim: fixed clear-profile so that it works on fenari and other ubuntu hosts
* 01:54 logmsgbot: tstarling synchronized php-1.5/index.php 'test'
* 01:54 Tim: set up fenari's /root/.ssh/config like the one on zwinger, with its user known hosts file pointing to the global one
* 01:41 Tim: Moved logmsgbot to fenari. Opened up its UDP port to zwinger and added firewall rules to /etc/rc.local for honour-system access control.
* 01:21 Tim: commented out the nagios-specific nickserv stuff in ircecho, so that logmsgbot doesn't ghost nagios-wm
* 01:11 Tim: installed udprec at fenari:/usr/local/bin/udprec
* 00:55 Tim: on fenari, symlinked ddsh -> dsh (dancer's) so that the sync scripts can work
== November 22 ==
* 14:15 domas: powercycled locke after 10h downtime
* 10:45 RoanKattouw: Reports that file deletion and undeletion on Commons is broken
* 03:35 rainman-sr: rebuilding mwsuggest index for en.wp which throws strange lucene exceptions, hopefully fresh built is going to solve it
* 01:24 rainman-sr: deployed latest lucene-search to test liquidthreads integration
== November 21 ==
* 21:03 domas: sq13 needs reboot and probably hardware maintenance
* 21:03 domas: sq43 and sq37 need hdd replacements.
* 20:46 domas: squid peermonitor is receiving 504s from peers... 
* 20:45 domas: firewalled too.
* 20:44 domas: cleaned up sq13 crap in remote logs on db20, was full / fs
== November 20 ==
* 22:11 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Adding 2009_Notice35 to whitelist'
* 22:10 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Adding Notice30_EML to whitelist'
* 19:13 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Added Help_Us_Change_the_World to allowed targets'
* 18:44 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Added Help_Us_Change_the_World to allowed targets'
* 18:41 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling tracking of 2009_Notice30_EML'
* 13:56 mark: OS-installed nfs1
* 13:09 mark: Renamed auth1 to nfs1 in DNS
* 13:00 mark: apt-get upgrade and reboot on fenari
* 12:48 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Re-enabling ClickTracking with throttle 1:1000'
* 12:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/ClickTracking/ClickTracking.php 'Deploying r59284'
* 12:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/ClickTracking/ClickTracking.js 'Deploying r59284'
* 04:51 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'load test'
* 04:49 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'load test'
== November 19 ==
* 23:56 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'whitelisting 2009_Notice_30_EML'
* 22:56 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling GeoIP for HU.'
* 21:08 mark: Shutting down bayle for decommissioning
* 21:04 mark: Stopped pdns recursor on bayle
* 20:20 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix error by Aaron allowing anybody to edit foundationwiki, which has wgRawHTML on...'
* 19:57 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'NS_MEDIAWIKI edit rights for all users on foundationwiki'
* 17:35 mark: Upgrading pdns-recursor on lily to 3.1.7.1
* 17:33 mark: Moved service ip 208.80.152.131 (first DNS resolver ip) from bayle to dobson
* 17:25 mark: Deployed new PowerDNS recursor on mchenry, with puppet
* 15:14 mark: Deploying new sudoers file on search servers with puppet, correcting sudo access to /usr/bin/jstack for rainman
* 15:08 mark: Removed stale spamassassin lock files on lily
* 14:58 mark: Rebooting lily
* 14:41 mark: Finished upgrade on lily
* 14:34 mark: Running apt-get dist-upgrade on lily
* 14:13 mark: Stopped authoritative PowerDNS server on bayle
* 13:58 mark: Rebooting dobson
* 13:56 mark: Moved ns0.wikimedia.org service ip from bayle to dobson
* 13:29 mark: Removed firewall entry on ns1
* 13:29 mark: Rolled out new PowerDNS and packages on linne with puppet
* 13:22 mark: Temporarily filtered all incoming DNS packets on ns1 for upgrade
* 11:57 tomaszf: flipping payments back on .. all is well after updates
* 10:42 tomaszf: taking outage on payments for updates
* 10:39 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable ClickTracking again'
* 10:38 tomaszf: upgrading activemq version on erzurumi to 5.3
* 10:37 Andrew: Some JS failures reported, because of ClickTracking extension, which seems to think it is a good idea to hijack all links to API redirects, which doesn't work so great for links with href=javascript:doSomething()
* 00:25 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 're-enabling ClickTracking hooks'
* 00:24 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'disabled Special:ClickTracking'
* 00:23 logmsgbot: tstarling synchronized php-1.5/extensions/UsabilityInitiative/ClickTracking/ClickTracking.hooks.php 'r59230'
* 00:06 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Whitelisted new banners.'
== November 18 ==
* 22:44 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php 
* 22:00 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Deploy r59222'
* 21:59 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/jquery/js2.combined.js 'Deploy r59222'
* 21:41 Andrew: Scapping to deploy updates.
* 21:41 Andrew: ... and merging r59218 into LiquidThreads
* 21:40 Andrew: Updating LiquidThreads_alpha to trunk state, and merging rrm: cannot remove directory `.svn/tmp/prop-base': Permission denied
* 21:14 logmsgbot: catrope ran sync-common-all 
* 21:12 RoanKattouw: Deploying r59214, r59217
* 20:57 RoanKattouw: Deploying r59212
* 20:33 logmsgbot: catrope ran sync-common-all 
* 20:31 RoanKattouw: sync-common-all throwing lots of rsync errors "failed to set times on dirname/.svn : Operation not permitted"
* 20:30 RoanKattouw: Deploying usability fixes for real this time
* 20:04 RoanKattouw: Deploying usability fixes
* 20:03 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics_body.php 
* 20:02 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics.css 
* 19:50 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php 'Remove debugging code'
* 19:31 RoanKattouw: Non-roots can't compile texvc on at least srv124, which breaks sync-common on that server and scap for non-roots
* 19:28 RoanKattouw: Syncing srv124 (test.wikipedia.org)
* 19:12 RoanKattouw: Running svn up on test for real this time, in /home instead of /apache
* 18:56 RoanKattouw: Running svn up on test for usability deployment
* 17:59 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php 'Debugging'
* 17:58 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'More debugging'
* 17:23 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php '-m Debugging code'
* 06:24 Tim: running logrotate -f on db20
* 05:39 Tim: logrotate on db20 has been broken since October 23, now the root partition is full. Fixing.
* 02:55 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabling the ClickTracking extension due to scary SQL escaping practices'
* 00:01 Tim: on ms1, changed the max-age for centralnotice/images to 3600, max-age for the rest of centralnotice remains at 300.
* 00:01 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/lqt.js 'Merged r59190'
== November 17 ==
* 23:59 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js 'Merged r59189'
* 23:36 Tim: deploying squid configuration (second attempt)
* 23:32 Tim: squid conf had a syntax error in it, all upload squids crashed, mostly recovered now
* 23:21 Tim: deploying an aggressive squid refresh_pattern (ignore-reload) for http://upload.wikimedia.org/centralnotice/images
* 20:50 Andrew: Scapping to deploy LiquidThreads alpha updates, let us hope the site does not go downa gain :)
* 19:58 domas: frontend squids occasionally have 100% cpu usage
* 19:57 domas: http://p.defau.lt/?QsA_sTZ_ykf0R_t0dnNtuQ <--- ipvs state during the overload
* 19:55 domas: hawthorn hitting %si capacity limits after +50% fundraiser image surge
* 18:01 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 're-adding db26, ixia'
* 02:13 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 
* 01:16 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.php 'picking up 2009 time range'
== November 16 ==
* 23:27 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'updating for 2009 figures'
* 22:44 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 22:02 Fred: enabled raw data gathering from the squids on Support pages (on Locke)
* 19:41 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'taking out db18 and db27 for reimaging'
* 19:03 Fred: root=fred above
* 19:03 logmsgbot: root synchronized php-1.5/wmf-config/CommonSettings.php 'Added CH and NL landing page for geo-location'
* 17:24 domas: increased file/connection limits on ms3
* 14:39 Andrew: To clarify, software update was NOT reverted, as the issue was due to the scap itself.
* 14:27 Andrew: Site seems to be mostly back up by now.
* 14:24 Andrew: Site seems to be up again, mostly.
* 14:18 Andrew: Looks like the error was caused by the scap pushing 4-cpu apaches into swap, causing memcached nodes to fall over, resulting in higher than normal database traffic
* 14:12 Andrew: Reverting software update
* 14:12 Andrew: Reports of slowness and down-ness
* 14:07 Andrew: Scapping to apply r59136, r59135, r59133, r59127
* 12:24 logmsgbot: andrew synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php 'Merge r58865 and r59070 from trunk -- fixes for deferred updates.'
* 12:12 Andrew: scapping to update LiquidThreads alpha
* 07:35 tomaszf: adding backup cron on db10 for civi and grosley for /srv
== November 15 ==
* 12:41 mark: Restarted NIS client on fenari
== November 14 ==
* 16:13 river: removed db26 from rotation for toolserver dump
* 16:12 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 
* 02:49 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionTrackingStatistics_body.php 
* 02:14 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionTrackingStatistics_body.php 
* 01:59 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 
* 01:58 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.i18n.php 
* 01:54 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionTrackingStatistics_body.php 
== November 13 ==
* 23:21 logmsgbot: tfinc synchronized php-1.5/extensions/GeoLite/GeoLite.php 
* 22:55 logmsgbot: tfinc synchronized php-1.5/extensions/GeoLite/GeoLite_body.php 
* 21:45 Andrew: Scapping to update LiquidThreads alpha to LiquidThreads trunk state
* 17:06 mark: Deployed new sudoers file on the search servers
* 15:53 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php 'Updating alpha version to trunk state'
* 15:52 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/ParserFunctions.php 'Updating alpha version to trunk state'
* 12:48 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Use the newly setup proxy for upload-by-url'
* 00:31 logmsgbot: robh synchronized php-1.5/cache/interwiki.cdb 'added outreach to the interwiki map'
== November 12 ==
* 23:45 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'GB not UK for GeoLite'
* 23:36 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php 'Adjustment'
* 23:31 Andrew: Trying LiquidThreads alpha fix again, scapping.
* 20:30 logmsgbot: tfinc synchronized php-1.5/extensions/GeoLite/GeoLite.php 
* 20:10 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 
* 20:09 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'tinkering with skwikiquote logo settings to force change'
* 20:06 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'tinkering with skwikiquote logo settings to force change'
* 19:55 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '21431 Set logo for Bengali Wikibooks'
* 19:10 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21262 Set q:sk: logo'
* 19:03 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'setting ru.wikiversity to the default engligh logo until they have a localized version'
* 18:57 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding back in ImportSources for new ru.wikiversity'
* 18:39 logmsgbot: midom synchronized php/cache/interwiki.cdb 
* 18:28 Rob: bad php error pushed by me, rolled it back, things should come back up now
* 18:28 logmsgbot: root ran sync-common-all 
* 18:21 logmsgbot: root ran sync-common-all 
* 16:59 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'LiquidThreads_alpha stuff doesn't seem to have taken effect, turned it off for lqt labs for now, will have another stab later'
* 16:50 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php 'Fix name of LiquidThreads alpha variable (wmgLiquidThreadsAlpha)'
* 16:49 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix name of LiquidThreads alpha variable (wmgLiquidThreadsAlpha)'
* 16:47 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'commit live hack in r58954'
* 16:29 Andrew: Re-scapping to update localisation for LiquidThreads alpha (which is technically a "new extension")
* 16:26 Andrew: tomasz has left some non-group-writeable dirs around in our checkout, please change your umask drwxr-xr-x 3 tfinc wikidev 32 Nov 9 06:44 GeoLite
* 16:21 Andrew: Which one is used is controlled by a new variable, wmgLiquidThreads_alpha -- true to use the alpha. Only activated on liquidthreads_labswikimedia. Scapping to apply updates.
* 16:20 Andrew: Split LiquidThreads to LiquidThreads_alpha and LiquidThreads, LiquidThreads_alpha being the most recent version.
* 15:37 Rob: bayes back online, let the statistic-crunching resume =]
* 15:29 Rob: updating dns to reflect new mgmt ip for bayes
* 03:29 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'turning $wgHtml5 back off again due to bugs'
* 01:37 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling $wgHtml5'
* 00:15 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php 
== November 10 ==
* 23:16 Tim: fixed DB errors in ContributionTrackingStatistics by granting SELECT access on drupal.contribution_tracking and civicrm.public_reporting to the contrib_tracking user
* 19:02 Rob: renamed auth1 to nfs1, replaced disks with 1tb drives, connected eth2 on nfs1 to eth2 on nfs2
* 17:00 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'adding db27'
* 02:52 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionHistory_body.php 
== November 9 ==
* 23:15 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disabling GlobalUsage due to major scaling issues, discussion on private-l'
* 22:56 logmsgbot: tstarling synchronized docroot/secure/keys.html 
* 22:55 logmsgbot: tstarling synchronized docroot/secure/keys.txt 
* 20:57 Rob: wordpress upgraded for techblog. by upgraded we mean new install and manual copy of existing uplaoded files and customizations.
* 20:51 Rob: techblog accessibility still spotty. Wordpress upgrades are sometimes painful.
* 20:12 Rob: messing with updating wordpress for techblog. It failed before so techblog will be up and down during the next few minutes
* 20:11 Fred: raised the number of apache threads on Singer to accommodate secure traffic.
* 19:29 mark: Disconnected drbd resource nfshome on nfs2: RAID1 resync was still running, waiting for it to finish before attempting DRBD resync
* 19:15 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionHistory_body.php 
* 18:59 mark: Stopped NFS kernel server on db20 (yes, that's /home)
* 18:34 mark: Turned off swap on db20: block device needed for DRBD external metadata
* 17:04 mark: OS-installed nfs2
* 16:40 mark: Running restart of puppetd across the cluster
* 16:35 mark: Running backup of db20 /home
* 16:28 mark: Running dist-upgrade on locke
* 16:00 mark: fsck repaired /var/backup on mchenry, filesystem remounted
* 15:55 mark: Added tridge to the ALL node group
* 15:49 mark: dist-upgrade and reboot of tridge
* 15:42 mark: Running new fsck on mchenry while the system is up
* 15:41 mark: /var/backups filesystem on mchenry is corrupt
* 15:22 mark: Reset DRAC password of sanger
* 15:18 mark: Rebooting mchenry
* 15:16 mark: Running apt-get dist-upgrade on mchenry
* 15:06 mark: Rebooting sanger
* 15:04 mark: Running apt-get dist-upgrade on sanger
* 14:41 mark: Shutdown bart for decommissioning
* 14:36 mark: Deployed cron job to restart puppet daily
* 14:35 mark: Deployed automatic updating of GeoIP database across the cluster
* 08:14 brion: killed duplicate wikibugs irc bot connection, stopping double bug reporting in #mediawiki
* 03:17 Tim: updating SecurePoll to r58802
== November 8 ==
* 22:10 mark: Added the esams ganglia clusters to the Florida grid (why the hierarchy?)
* 21:52 domas: dropped one snapshot on ixia, other still up. 
* 21:52 Andrew: domas says he did some magic with i/o on ixia, replag back to zero.
* 21:51 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turning GlobalUsage back on'
* 21:41 mark: Experimentally installed unattended-upgrades on server sockpuppet
* 21:41 mark: Reboot srv156
* 21:38 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Deactivating GlobalUsage to check impact on ixia replag.'
* 21:24 Andrew: ixia still has huge replication lag, refreshGlobalImageLinks definitely not still running.
* 18:31 mark: Unmounted /home on thistle, ixia, lomaria, db1 and db11
* 17:26 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/specialpages/Stabilization_body.php 'deployed r58786'
* 16:51 brion: taking load off ixia to let it catch up; it's still lagged on GlobalUsage inserts
* 16:51 logmsgbot: brion synchronized php-1.5/wmf-config/db.php 
* 16:09 Bryan: Correction: --start-image=Symbol_neutral_vote.svg --start-page=15088846 --maxlag=5
* 16:04 Bryan: Parameters to restart refreshGlobalimagelinks.php with on enwiki: --start-image Symbol_neutral_vote.svg --start-page=15088846 --maxlag 5
* 16:02 brion: stopped andrew's refreshGlobalimagelinks.php job on hume pending fixes and someone to babysit it
* 15:58 brion: ixia lag from GlobalUsage population script; Bryan poking at it to make it behave nicer
* 15:55 brion: commonswiki | Connect | 510 | NULL | INSERT /* GlobalUsage::setUsage */ INTO `globalimagelinks` (gil_wiki,gil_page,gil_page_namespace,g |
* 15:50 brion: circa 8 minutes replag on ixia :(
* 15:30 brion: adding missing 'zh' variant URL alias (used for default or something? there's a tab for it)
* 15:25 brion: updating all main.conf wiki entries w/ zh variant alias entries (most missing from all but wikipedia.org and wikisource.org) for bugzilla:19019
* 13:32 mark: Shutdown khaldun
* 13:04 brion_: restarted parser test loop on wikitech [fixme: make an init script]
* 13:01 brion_: parser tests not running since oct 27; looks like wikitech was rebooted then. checking....
* 10:24 logmsgbot: andrew synchronized php-1.5/extensions/GlobalUsage/SpecialGlobalUsage.php 'Deploy r58752'
* 10:21 logmsgbot: andrew synchronized php-1.5/extensions/GlobalUsage/GlobalUsage_body.php 'Deploy r58752'
* 09:50 Andrew: Set up globalusage for all wikis
* 09:48 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 09:28 apergos: cleaned out some space from /tmp and ran apt-get clean on srv218, these should probably become a cron job
* 09:27 logmsgbot: andrew synchronized php-1.5/includes/ImagePage.php 'Merge r58692'
* 09:23 Andrew: scapping to update GlobalUsage
* 09:20 brion: srv218: rsync: write failed on "/apache/common/php-1.5/wmf-config/InitialiseSettings.php": No space left on device (28)
* 09:19 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'adding TimedText namespace on Commons for mdale subtitle awesomeness'
* 08:43 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Activated LiquidThreads on MediaWiki.org'
* 08:39 Andrew: Sticking LiquidThreads in opt-in mode on mediawiki.org
* 05:54 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'putting db1 and db21 live with jafrru dataset'
* 03:58 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'taking out db1 and db21 for new image load'
* 01:04 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'putting db21 back live'
== November 7 ==
* 22:14 mark: Started puppet on sq44
* 22:12 mark: Unstuck puppet on srv149
* 22:02 mark: Fixed puppet on ixia
* 21:48 mark: Installed puppet on yongle
* 21:26 mark: Rolling dist-upgrade finished
* 21:12 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'removing db21 from rotation - will be used for migration source'
* 18:35 mark: Moved planet.wikimedia.org DNS CNAMEs to singer
* 18:35 mark: Setup planet.wikimedia.org instance on singer
* 17:09 mark: Moved noc.wikimedia.org CNAME to fenari
* 17:09 mark: Setup noc.wikimedia.org instance on fenari, using puppet
* 14:57 mark: Doing rolling dist-upgrade of all application servers
* 14:13 mark: Removed optsview APT repository on srv225
* 14:12 mark: Removed optsview APT repository on srv224
* 13:36 mark: Manually upgrading puppet across the cluster to help itself get past a catalog bug
* 13:03 mark: Setup puppet to automatically upgrade wikimedia-task-appserver on all application servers; in this case it will roll out package php5-geoip across the cluster
* 09:12 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'enable Collection on strategy wiki per bug 21361'
== November 6 ==
* 23:45 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php 'dropping 2008 theme'
* 23:39 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionHistory_body.php 'dropping 2008 theme'
* 23:19 mark: Replaced the Webrick puppetmaster setup on sockpuppet by an Apache/Mongrel setup with 4 children to deal with scalability problems
* 21:29 mark: Increased active record DB connection pool size from 5 to 50... by just editing the ruby library file
* 18:54 mark: db10 replication caught up with db9
* 18:08 mark: Set up puppet to run "apt-get update" on every run (for now, may make that somewhat less aggressive if needed)
* 17:10 logmsgbot: andrew synchronized php-1.5/wmf-config/ExtensionMessages.php 
* 17:08 mark: Setup puppet to exchange all hostkeys between all servers
* 17:07 logmsgbot: andrew synchronized php-1.5/includes/LinksUpdate.php 
* 17:05 Andrew: Scapping
* 17:05 Andrew: Set up GlobalUsage on test.wikipedia.org
* 16:55 mark: Set up puppet to upgrade itself everywhere
* 16:40 mark: Fixed permission problem in the MySQL data dir, restarted replication on db10
* 16:37 mark: Fixed LVM problem on db10
* 16:01 mark: Rebooting db10: OOM errors
* 15:59 mark: Replication db9->db10 broken again
* 15:56 mark: Set up MySQL database 'puppet' on db9/db10
== November 5 ==
* 19:59 logmsgbot: tfinc synchronized php-1.5/extensions/DonationInterface/donate_interface/donate_interface.i18n.php 
* 19:51 logmsgbot: tfinc synchronized php-1.5/extensions/DonationInterface/donate_interface/donate_interface.php 
* 18:48 mark: Setup puppet Stored Configurations with Stomp queuing; made a custom /usr/local/sbin/puppetqd since it was missing in the puppetmaster package.
* 17:58 Rob: transcode1-3 drac setup, servers racked, not in racktables yet, no os yet.
* 17:42 mark: Upgraded distribution of sockpuppet to 9.10 (Karmic)
* 17:35 Rob: pushing dns changes for mgmt on transcode1-3
* 16:54 mark: Upgraded puppetmaster on sockpuppet to version 0.25.1
* 15:35 mark: Killed all Nagios processes on bart
* 15:28 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disabled InitialiseSettings.php again on all wikis, except testwiki. Pending HTTP proxy setup and security review.'
* 15:19 mark: Moved secure.wikimedia.org service IP from bart to singer
* 15:13 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling upload by URL for sysops on all wikis'
* 15:09 mark: Moved NameVirtualHost entries to /etc/apache2/conf.d/namevirtualhost on singer, to stop Apache from complaining
* 15:06 mark: Set up secure.wikimedia.org vhost on singer, SSL proxying to the apaches
* 14:49 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Add singer to the SquidServers list'
== November 4 ==
* 23:03 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php 
* 21:09 mark: Stopped most services on khaldun, to prepare for decommissioning
* 21:06 mark: Setup csw5-pmtpa to use brewster as dhcp ip helper
* 21:05 mark: Stopped and deinstalled dhcp3-server on khaldun
* 21:04 mark: Setup new dhcpd on brewster using puppet. Host entries are now factored out to files /etc/dhcp3/linux-host-entries*
* 18:48 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Disabling paypal portion until contrib tracking works correctly'
* 18:46 logmsgbot: tfinc synchronized php-1.5/extensions/DonationInterface/payflowpro_gateway/payflowpro_gateway.body.php 
* 17:21 RoanKattouw: Running cleanupTitles on frwikisource
* 14:50 logmsgbot: tfinc synchronized php-1.5/extensions/DonationInterface/donate_interface/donate_interface.php 
* 14:31 logmsgbot: tfinc synchronized php-1.5/extensions/DonationInterface/payflowpro_gateway/payflowpro_gateway.php 
* 13:54 logmsgbot: tfinc synchronized php-1.5/extensions/DonationInterface/payflowpro_gateway/payflowpro_gateway.php 
* 13:54 RoanKattouw: Updating Special:BrokenRedirects, Special:DoubleRedirects and Special:OrphanedPages on frwikisource; community request related to mass cleanup
* 13:47 logmsgbot: tfinc synchronized php-1.5/extensions/DonationInterface/paypal_gateway/paypal_gateway.php 'trying again as cluster didn't pickup change'
* 13:38 logmsgbot: tfinc synchronized php-1.5/extensions/DonationInterface/paypal_gateway/paypal_gateway.php 
* 13:38 logmsgbot: tfinc synchronized php-1.5/extensions/DonationInterface/payflowpro_gateway/payflowpro_gateway.body.php 
* 13:24 RoanKattouw: tomaszf scapped
== November 3 ==
* 23:47 logmsgbot: tstarling synchronized wmf-deployment/cache/trusted-xff.cdb 
* 23:23 Tim: deploying trusted XFF update from r58506
* 23:12 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'removing db28'
* 23:10 Tim: restarted udp2log on locke, sampled-1000.log was empty
* 19:32 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/NewMessagesController.php 'Deploy r58494'
* 16:12 mark: Recovered torrus from its deadlocked state
* 15:24 mark: Setup APT repository and preseeding repository on brewster, using puppet
* 15:21 mark: Moved apt.wikimedia.org DNS CNAME from khaldun to brewster
* 15:17 mark: Shutdown BGP session to AS 16150; no connectivity to wikitech.wikimedia.org
* 15:14 mark: Added --timeout=30 to the wget invocations in the post-commit script on svn.wikimedia.org
* 14:17 Andrew: Scapping
* 14:09 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc.php 'swapped dead memcache from srv217 to srv156'
* 14:06 Andrew: Will scap to update LiquidThreads in a few minutes
== November 2 ==
* 19:30 logmsgbot: aaron synchronized php-1.5/includes/FileDeleteForm.php 'deployed r58429'
* 14:12 Fred: We are running out of memcache spare boxes... Need some cleanup of the "downned memcached"
* 14:12 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc.php 'swapped dead memcache from srv146 to srv217'
== November 1 ==
* 19:41 rainman-sr: started search indexer on searchidx1
* 19:21 mark: Renamed db6 to locke in DNS and racktables
* 19:04 mark: Powercycled frozen server searchidx1
* 18:59 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Changed proxy host for file-by-url testing on test.wikipedia.org from khaldun to brewster'
* 18:59 mark: Stopped oprofile on srv218 and removed all samples; its disk was full
* 18:54 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 18:53 mark: Setup Squid proxy on brewster, and setup selective proxying for security.ubuntu.com through it on all hosts, using puppet
* 17:44 mark: Set up tftp server on brewster (using puppet)
* 16:54 mark: Added ubuntu.wikimedia.org DNS entry
* 16:45 mark: Set up Ubuntu mirror on brewster using puppet
* 16:45 mark: Fixed Domas's IPC cleanup cron job to not run if there are no stale ipc semaphores
* 12:48 river: allocated vlan 301 to the toolserver, 10.23.1.0/24 (we're out of public IPs)
== October 31 ==
* 11:18 logmsgbot: catrope synchronized php-1.5/includes/MagicWord.php 'Removing MagicWordArray live hack; problem was found and fixed yesterday'
* 09:47 Raymond_afk: ParserTests for CodeReview does not run. Last run for r58231 2009-10-27
* 00:48 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'deployed r58380'
== October 30 ==
* 23:11 RoanKattouw: Rebuilt l10ncache on srv194, seems to have fixed it. Presumably caused by an incomplete run of a sync script caused by permissions errors (see private-l)
* 23:00 Fred: depooled srv194 while testing / fixing
* 22:08 Fred: actually stopped apache on srv194 since the restart did not help. Investigating....
* 22:07 Fred: restarted apache on srv194 since it was throwing a lot of ""is not a valid magic thingie for xxxx"
* 20:05 Fred: scaped as root to resolve cache problems on some boxes.
* 20:02 logmsgbot: fvassard ran sync-common-all 
* 19:50 RoanKattouw: Scapping in the hopes of resolving the MagicWordsArray exception; looks like it could be a problem with localisation cache
* 19:37 logmsgbot: catrope synchronized php-1.5/includes/MagicWord.php 'Live hack for debugging: make MagicWordArray::parseMatch parameter not found error more verbose'
* 18:36 logmsgbot: catrope synchronized php-1.5/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'Deploy r58359 (LocalisationUpdate fix for languages with a '-' in their name)'
* 18:33 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Per bug 19865, enable the enhanced toolbar by default on enwikinews and disable the OptIn extension'
* 16:39 RoanKattouw: sync-common-all threw lots of errors like srv175: rsync: failed to set times on "/usr/local/apache/common-local/wmf-deployment/includes/.svn": Operation not permitted (1)
* 16:38 logmsgbot: catrope ran sync-common-all 
* 16:36 RoanKattouw: Scapping so srv194 will receive updates
* 15:34 apergos: cleared out some files in /tmp and ran apt-get clean on srv194 to get back about 1gb space on /
* 14:08 RoanKattouw: @wikimediatech feed on identi.ca has been broken for the past 3 days
* 14:01 RoanKattouw: srv194 has full disk, causing syncs to fail; throws PHP exception "MagicWordArray::parseMatch: parameter not found"
* 11:47 rainman-sr: searchidx1 ssh dead, shows 100% I/O, disk dead?
== October 29 ==
* 22:49 logmsgbot: catrope synchronized php-1.5/includes/upload/UploadFromUrl.php 'Re-sync for srv226'
* 22:48 RoanKattouw: Scapping again for previous deployment because srv226 wouldn't let me log in at that time
* 20:27 logmsgbot: catrope synchronized php-1.5/includes/upload/UploadFromUrl.php 'Deploy r58339 (upload API fix)'
* 18:51 mark: Cleaned up lighttpd.conf on khaldun
* 17:27 mark: OS-installed brewster.wikimedia.org
* 07:53 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'removing tidy from wmf wiki since its really in CommonSettings.php'
* 07:51 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'testing disabling tidy on wmfwiki .. now one level higher '
* 07:44 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'testing disabling tidy on wmf'
* 06:15 domas: deployed ipc semaphore cleanup cron via puppet
* 05:51 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'adding new donate skin for wmf wiki'
* 05:44 logmsgbot: tfinc synchronized php-1.5/extensions/skins/Donate/Donate.class.php 
== October 28 ==
* 20:58 Tim: deploying r58282, SecurePoll update
* 18:17 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'adding vector as defaul skin for outreachwiki'
* 18:00 mark: Found an upload squid in the text squid configuration list, causing these 404s - removed
* 17:56 mark: Removed stripping of caching headers from frontend squids - needed for debugging
* 17:50 logmsgbot: mark synchronized live-1.5/404.php 
* 17:46 logmsgbot: mark synchronized docroot/wikipedia.org/w/404.php 
* 17:28 Fred: (re)started apache on srv219,234
* 16:44 rainman-sr: revert r58235 on search1/4 because it seems to confuse pybal and make is depool then randomly
* 16:33 mark: Increased memory cache of text frontend squids from 10 to 50 MB
* 02:26 Tim: deploying r58239, LiquidThreads revert
* 01:44 Tim: trying to use debugging symbols on srv189, shutting down apache there temporarily
== October 27 ==
* 21:14 logmsgbot: tstarling synchronized wmf-deployment/cache/interwiki.cdb 
* 21:13 Tim: fixed outreach.wikimedia.org
* 21:12 logmsgbot: tstarling ran sync-common-all 
* 21:10 hcatlin: pushed fix to native-iphone-app html... cleared memcached and.... a *momentary meltdown* occured
* 21:07 RoanKattouw: Last scap deployed r58216
* 21:06 logmsgbot: tstarling ran sync-common-all 
* 20:59 RoanKattouw: Running LocalisationUpdate/update.php on test
* 20:58 RoanKattouw: running svn up on test
* 20:54 Tim: deployed r58215
* 20:46 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Fix config for strategywiki and usabilitywiki so the enhanced toolbar is enabled by default again'
* 20:18 logmsgbot: midom synchronized php-1.5/includes/memcached-client.php 'bumping up timeout to 0.1s up from 0.05s - as we do have some megabyte sized objects....'
* 19:32 mark: Ran "deluser catrope" across the cluster to prompt puppet to recreate
* 19:30 mark: Fixed admins.pp in puppet, "managehome" attribute had disappeared
* 17:16 logmsgbot: midom synchronized php-1.5/languages/LanguageConverter.php 
* 10:14 logmsgbot: midom synchronized php-1.5/StartProfiler.php 
* 10:08 logmsgbot: midom synchronized php-1.5/languages/LanguageConverter.php 'oops, this is not entirely right, livehacking for now'
* 09:58 logmsgbot: midom synchronized php-1.5/languages/LanguageConverter.php 'push locking change live'
* 07:45 domas: rolled live memcached changes, read/write timeouts down from 1s to 50ms, connect timeouts from 3x10ms with backoff to 2x10ms with no backoff, and fixed some host blacklist bug. 
* 07:42 logmsgbot: midom synchronized php-1.5/includes/memcached-client.php 'HERE WE GO MEMCACHED FIXES'
* 06:05 domas: fixed perms in survey.wikimedia.org's /srv/org/wikimedia/survey/tmp/ , as well as set display_errors to off, in case there's more incompetence around ;-)
* 01:39 rainman-sr: turned back on highlighting on en/de/fr, turned off interwiki search on smaller wikis ... we need more servers to cope with increase in traffic on large wikis
* 01:11 atglenn: disabled search2 from lvs3 pybal config at rainman's request (it had load 21)
* 01:01 rainman-sr: could someone please remove search2 from lsv3 search group ASAP
* 00:15 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/pages/SpecialNewMessages.php 'Deploy r58176'
== October 26 ==
* 20:45 Andrew: scapping to update LiquidThreads
* 20:16 Andrew: Going to update LiquidThreads to trunk state in a few minutes
* 16:08 rainman-sr: overloads all around, turned off en/de/fr wiki highlighting so that searchs don't time out
* 11:10 hcatlin: reworked mobile1's config so that its more standardized and more of the config is in the repo
* 08:53 domas: updated nagios to reflect changed server roles
* 08:43 domas: dewiki is now separate cluster, s5, replication switch over done at http://p.defau.lt/?kfvvlNOc4TkJ_6SCAVe6mg
* 08:42 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'dewiki readwrite'
* 08:40 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'restructuring s2dewiki into s5'
* 08:38 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'dewiki read-only'
* 07:57 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'entirely separating dewiki slaves'
* 06:54 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'taking out db4 for copy to db23'
* 05:45 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
== October 25 ==
* 15:23 domas: converting usability initiative tables to InnoDB...
* 13:23 domas: set up snapshot rotation on db10
* 12:36 hcatlin: mobile1: created init.d/cluster to correct USR1 sig problem, fully updated sys ops on wikitech
* 12:03 domas: Mark, I'm sure you'll like that! ;-p~
* 12:02 domas: started sq43 without /dev/sdd COSS store (manual conf hack)
* 11:54 domas: removed ns3 from nagios, added ns1
* 11:45 domas: bounced ns1 too, was affected by selective-answer leak (same number as ns0, btw, 507!) ages ago, just not noticed by nagios. this seem to resolve some slowness I noticed few times. 
* 11:41 domas: bounced pdns on ns0, was affected by selective-answer leak 
== October 24 ==
* 16:49 rainman-sr: decreasing maximal number of search hits per request (e.g. page) to 50
* 16:40 apergos: re-enabled zfs replication from ms1 to ms5, set to 20 minute intervals now, keeping an eye on it to see if we have failures in running to completion
* 13:28 rainman-sr: finished restructuring en.wp, continuing with normal incremental search updates
* 11:50 domas: removed hardy-backports from fenari sources.list, added bzr ppa to sources.list.d/bzrppa.list 
== October 23 ==
* 23:37 logmsgbot: tstarling synchronized wmf-deployment/cache/trusted-xff.cdb 
* 23:31 logmsgbot: tstarling synchronized wmf-deployment/cache/trusted-xff.cdb 
* 23:24 Tim: updating TrustedXFF (bolt browser)
* 22:36 domas: db28 has multiple fan failures (LOM is finally able to do something :) - still needs datacenter ops
* 22:20 domas: db28 is a toast, needs cold restart by datacenter ops, LOM not able to do anything 
* 22:20 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'db28 dead'
* 11:17 domas: Fixed skip-list of cached query pages, was broken for past two months :)
* 10:54 logmsgbot: midom synchronized php-1.5/thumb.php 'removing livehack'
* 10:52 domas: rotating logs becomes difficult when they become too big, so they continue to grow indefinitely! db20 / nearly full, loooots of /var/log/remote ;-)
* 10:39 domas: who watches the watchers? :) rrdtool process on spence was using 8G of memory. :-) http://p.defau.lt/?NOGuiw1ht_9_4KmtD3r4lA
* 10:24 domas: semaphore leaks made some apaches fail, failed apache in rendering farm was not depooled, thus having 404 handler serve plenty of "can't connect to host" broken thumbs. 
* 10:12 domas: apparently there're intermittent connection failures from ms4 to scalers
* 09:56 logmsgbot: midom synchronized php-1.5/thumb.php 'error header livehack'
* 04:04 domas: noticed intermittent network failure inside pmtpa - most of input on this has been one stalling SSH session and spence failure
* 04:01 domas: switched jobs table on db22 with an empty one, old one was having just few noop entries and five million invalidated rows... hit interesting (but probably easy to fix) performance problem at mtr_memo_release/mtr_commit code inside MySQL :)
* 03:16 Fred: restarted powerdns on ns2 to kill some zombies with a double tap :p
== October 22 ==
* 21:48 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/language/FlaggedRevs.i18n.php 'deploy r58038'
* 16:25 Andrew: Updating LiquidThreads to trunk state again
* 13:34 Andrew: Updating LiquidThreads to trunk state, scapping.
== October 21 ==
* 22:30 Tim: upgraded libpoppler2 on all apaches
* 21:20 Tim: updating ubuntu mirror
* 20:50 Tim: apt-get upgrade on pdf1 for USN-850-1
* 19:59 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 13:37 rainman-sr: restarting search incremental update on all wikis, will sync to search servers when updates catch up
== October 20 ==
* 20:16 RoanKattouw: Brion synced r57957
* 19:59 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '21023 Allow bureaucrats to remove sysop flag on Simple English WIkiquote'
* 15:54 hcatlin: Deployed changes to S60. Stopping redirect for Nokia Series60, because it needs more work.
* 14:38 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/api/ApiThreadAction.php 'Merge r57945'
* 14:32 Andrew: Updating LiquidThreads again, scapping
* 14:03 Andrew: Updating LiquidThreads, scapping
* 12:20 hcatlin: Updated Common.js on en.wiki to add in redirects for the new batch of supported devices
* 11:57 hcatlin: deploying updated software to mobile1
* 04:39 Tim: fixing static.wikipedia.org, broken due to lack of DumpHTML extension in wmf-deployment
* 00:42 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '19865 en.Wikinews would like to make Vector the default skin'
* 00:37 logmsgbot: robh synchronized php-1.5/wmf-config/CommonSettings.php 'Bug 20361'
== October 19 ==
* 23:12 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/api/ApiThreadAction.php 'Merge r57930'
* 23:05 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/lqt.js 'Merge r57928'
* 23:05 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/api/ApiThreadAction.php 'Merge r57928'
* 23:04 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php 'Merge r57928'
* 22:52 rainman-sr: starting complete rebuild of en.wp index
* 22:31 atglenn: turned off replication on ms1 to ms5 (it was failing), running manually to catch up
* 22:27 Fred: restarted apache on srv[105,117,173]
* 22:19 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/lqt.js 'Merge r57924'
* 21:56 Andrew: Scapping
* 21:44 Andrew: Updating LiquidThreads to trunk state again for bugfixes and ajax improvements.
* 19:50 rainman-sr: stopping search incremental updates to rebuild en.wp index for increased efficiency
* 19:39 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 19:37 logmsgbot: tstarling synchronized php-1.5/includes/User.php 'r57910'
* 19:36 logmsgbot: tstarling synchronized php-1.5/includes/DefaultSettings.php 'r57910'
* 18:54 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 18:47 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'rate limit exclusion for seminar'
* 14:50 logmsgbot: andrew synchronized php-1.5/wmf-config/mc.php 'Swap out srv156'
* 14:29 Andrew: Wikitech is insanely slow :)
* 14:27 Andrew: Scapping
* 14:25 Andrew: Updating LiquidThreads installation to trunk state.
* 13:33 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Deployed AbuseFilter to LiquidThreads Labs Site'
== October 17 ==
* 06:45 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Deploy r57856'
* 05:12 logmsgbot: aaron synchronized php-1.5/includes/api/ApiQueryWatchlist.php 'deployed r57852'
* 04:57 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'deployed r57850'
* 04:57 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.class.php 'deployed r57850'
* 04:27 logmsgbot: aaron synchronized php-1.5/extensions/CodeReview/backend/CodeTestSuite.php 'deployed r57848'
* 04:22 logmsgbot: aaron synchronized php-1.5/includes/api/ApiQueryWatchlist.php 'deployed r57846'
* 03:58 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/specialpages/RevisionReview_body.php 'deployed r57843'
== October 16 ==
* 19:41 Andrew: Scapping
* 19:38 Andrew: updating LiquidThreads to trunk state (r57832)
* 16:55 logmsgbot: andrew synchronized php-1.5/extensions/UsabilityInitiative/js/js2.combined.min.js 
* 05:53 Tim: disabled password auth for sshd on mayflower, to simplify debugging of authentication failures for new users
* 01:05 domas: fixed locke, brought udp2log up in screen, and collector somewhere else :) 
* 00:59 domas: next time someone leaves NIC above HDD in boot order.... 
* 00:47 domas: locke stalled, connected to DRAC via db6.mgmt, runs installer after reboot...
== October 15 ==
* 20:53 tomaszf: adding payments.wikimedia.org to loudon
* 17:04 RoanKattouw: Getting LQT-related DB errors on prototype, trying update.php
* 16:02 Tim: updated extdist snapshot
* 10:50 logmsgbot: aaron synchronized php-1.5/includes/LogEventsList.php 'deployed r57767'
* 09:29 logmsgbot: andrew synchronized php-1.5/includes/filerepo/ArchivedFile.php 'Deployed r57755, fixes for fatal errors introduced in r57602 by blindly copy/pasting code.'
* 09:27 logmsgbot: andrew synchronized php-1.5/includes/filerepo/OldLocalFile.php 
* 07:12 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/specialpages/Stabilization_body.php 'done, reverted'
* 06:39 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/specialpages/Stabilization_body.php 
* 06:39 aaron: investigating small flaggedpages tracking bug
* 04:43 apergos: remove manually oldest dumps for fr, ru, en, es, pt wikis to get a bit more room on storage2 (til monitor catches up with the config change on new dumps)
* 04:08 Tim: running apt-get upgrade on mayflower
* 04:01 apergos: changed number of dumps we keep from 10 to 9 since storage2 was low on space (99% = 75G left) 
* 03:07 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticle.php 
* 03:04 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/specialpages/StablePages_body.php 'deployed r57735, r57736'
* 03:03 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticle.php 'deployed r57735, r57736'
* 03:03 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevsXML.php 'deployed r57735, r57736'
* 03:03 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.class.php 'deployed r57735, r57736'
== October 14 ==
* 23:26 Fred: created http://wikitech-static.wikimedia.org/ as a static dump of wikitech (read faster, but read-only)
* 21:27 brion: r57727 wmf-deployment loaded but scap delayed due to usability test (?!)
* 20:56 logmsgbot: tfinc synchronized php-1.5/extensions/FundraiserPortal/FundraiserPortal.php 'pushing changes from r57629'
* 20:54 logmsgbot: tfinc synchronized php-1.5/extensions/FundraiserPortal/Templates/Tourmaline.css 
* 20:54 logmsgbot: tfinc synchronized php-1.5/extensions/FundraiserPortal/Templates/Sapphire.css 
* 20:54 logmsgbot: tfinc synchronized php-1.5/extensions/FundraiserPortal/Templates/RubyText.css 
* 20:54 logmsgbot: tfinc synchronized php-1.5/extensions/FundraiserPortal/Templates/Ruby.css 
* 19:46 mark: Deployed squid config to sq43 which was running with an outdated squid.conf. This would explain the weird 404 thumbnail problem, because sq43 was contacting eiximenis which was an upload squid but is now a text squid...
* 18:00 Fred: rebooting sq43. 
* 16:49 Fred: sq47 has a bad sdd drive. Adjusted the squid config to reflect.
* 16:17 Fred: restarted a couple of downed apaches and Squids.
* 16:08 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc.php 'removed srv144 and replaced it with srv215.'
* 12:20 hcatlin: deploying WAP support (thedj) and Nokia Series60 support to mobile1
* 12:17 mark: Removed full copy snapshot 'copy-snap' on ms3
* 09:26 Andrew: Scapping to deploy LiquidThreads updates
== October 13 ==
* 22:48 rainman-sr: turn off interwiki search for "other wikis" to help out with en.wp overload during peak times
* 16:32 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/LiquidThreads.php 'Merge r57673'
* 16:29 Andrew: (Note: In opt-in mode per-page.)
* 16:29 Andrew: Deployed LiquidThreads to the strategy wiki
* 16:27 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 16:23 Andrew: Preparing to deploy LiquidThreads to strategy wiki, in opt-in mode per page.
* 16:15 Andrew: Enabled LiquidThreads on test.wikipedia.org, came across some fun interactions with drafts and turned it off again.
* 15:53 logmsgbot: andrew synchronized php-1.5/extensions/OAI/OAIRepo_body.php '-m Merge r57672'
* 15:52 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/LiquidThreads.php '-m Merge r57672'
* 14:52 logmsgbot: andrew synchronized php-1.5/extensions/OAI/OAIRepo_body.php 'Merge r57666'
* 14:51 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/LiquidThreads.php 'Merge r57667'
* 14:50 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Hooks.php 'Merge r57667'
* 13:38 mark: Removed yaseo from Squid configs, and set CARP weight of eiximenis to 40
* 11:41 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Forgot the talk namespace'
* 11:40 Andrew: Added 'Multimedia' namespace to usability wiki due to private request from somebody working on the Ford grant.
* 11:39 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 11:10 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php 'Fix for dumping/search'
* 06:53 domas: off to Japan (until Oct22) \o/
== October 12 ==
* 23:19 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'flaggedrevs lab config'
* 16:39 hcatlin: ...to mobile1
* 16:38 hcatlin: Pushed out Akan language support.
* 16:09 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/pages/TalkpageView.php 'Deployed r57656'
* 16:08 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Deployed r57655'
* 16:07 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/i18n/Lqt.i18n.php 'Deployed r57655'
* 14:24 hcatlin: updated code on mobile1. a couple minutes of downtime due to mailer config errors.
* 14:22 Andrew: Deployed r57651 (LiquidThreads)
* 14:22 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php 
== October 11 ==
* 20:17 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'patrol for autopatrolled group on frwiki'
* 14:02 mark: Cleaned up BGP sessions on br1-knams
* 13:06 hcatlin: updated mobile1 with one-hour expiration of cache to see if that helps anything.
* 03:23 Tim: restarted thin on mobile1, apparently it didn't reconnect to memcached at all when memcached came back up
* 03:10 Tim: in light of bug 20653 (complete lack of cache invalidation), reduced memcached memory on mobile1 to 5GB, to reduce the effective expiry time. Will monitor CPU but past cache clear events suggest it won't be a problem.
== October 10 ==
* 21:40 domas: nuking htmlCacheUpdate jobs for Biography_articles_with_listas_parameter (a hidden category)...
* 20:39 logmsgbot: midom synchronized php-1.5/includes/LogEventsList.php 
* 18:16 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'deployed r57620'
* 18:15 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticle.php 'deployed r57620'
* 09:05 domas: resolving job queue constipation: alter table job drop key job_cmd, add key job_cmd (job_cmd,job_namespace,job_title,job_params(128));
* 02:18 logmsgbot: aaron synchronized php-1.5/includes/LogEventsList.php 'deployed r57602'
* 02:17 logmsgbot: aaron synchronized php-1.5/includes/filerepo/OldLocalFile.php 'deployed r57602'
* 02:17 logmsgbot: aaron synchronized php-1.5/includes/filerepo/ArchivedFile.php 'deployed r57602'
== October 9 ==
* 21:55 hcatlin_: restarted the mobile1 cluster
* 21:37 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'crank wgClickTrackThrottle to 1:1 to see what happens'
* 21:18 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'bump ClickTracking throttle to 1:100 edit reqs'
* 21:10 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 21:10 brion: turning on ClickTracking at 1:1000 sampling sitewide
* 21:03 logmsgbot: andrew synchronized php-1.5/includes/ChangesList.php 
* 20:44 rainman-sr: running /home/rainman/scripts/build-new liquidthreads_labswikimedia on searchidx1
* 17:57 brion: installing DB tables for ClickTracking ext on all wikis
* 17:15 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Deploy r57572'
* 15:58 Andrew: Updating LiquidThreads to trunk state @r57567
* 14:12 Rob: pushed updates to some blog software on blog and techblog
* 05:43 Fred: restarted thin on mobile1.
* 05:39 apergos: restarted memcached on mobile1 (it had died), ... thin server is being an expletive deleted though
* 05:16 Fred: mobile1 doesn't have a swap partition for some reason... 
* 05:15 Fred: mobile1 went OOM
* 05:07 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'autopromote logging in RC for plwiki'
* 00:04 brion: cranked ClickTracking to 1:1 on test
* 00:03 brion: testing ClickTracking on testwiki: 1/1000 sampling
== October 8 ==
* 23:52 logmsgbot: tstarling synchronized wmf-deployment/cache/trusted-xff.cdb 
* 23:50 Tim: updating TrustedXFF
* 23:35 logmsgbot: brion synchronized php-1.5/extensions/UsabilityInitiative/ClickTracking/ClickTracking.php 
* 20:03 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'profiling host change'
* 19:56 logmsgbot: andrew synchronized php-1.5/includes/Revision.php 'Deploy r57530, fix for leaking of oversighted data'
* 19:46 logmsgbot: andrew synchronized php-1.5/includes/Revision.php 'Deploy r57530, fix for leaking of oversighted data'
* 18:55 brion: fixed thumb rendering on test.wikipedia; caveat that thumb.php reqs are sent to synced copy, not nfs master
* 18:54 logmsgbot: brion ran sync-common-all 
* 18:50 brion: sync-common-all to update MWVersion.php w/ hackaround for test thumbs.php
* 18:50 logmsgbot: brion ran sync-common-all 
* 18:45 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '21059 Enable patrol rights on commons now with autopatrolled'
* 18:40 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 18:39 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 18:39 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 
* 18:25 mark: Installed server dobson
* 17:55 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '21059 Enable patrol rights on commons'
* 16:31 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php 'Deployed r57525'
* 16:14 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php 'Deploy r57520-3'
* 15:15 Fred: rebooting srv156 
* 15:14 Andrew: 4-cpu apaches seem to be out of memory and swapping
* 15:04 Andrew: Scapping
* 15:03 Andrew: Applying LiquidThreads schema change to liquidthreads_labswikimedia and updating/deploying LiquidThreads to r57518 state.
* 14:52 mark: Redeployed ntp server on linne using puppet
* 14:32 mark: DNS cleanup: removed all traces of yaseo, and 66.230.200.0/24
* 14:26 mark: Unbound 66.230.200.10/24 range from csw5-pmtpa interface ve5. 
* 14:24 mark: Removed ip 66.230.200.234 from zwinger
* 13:38 mark: Restarting all ServerTech CDUs for NTP configuration change to take effect
* 07:16 apergos: shot the pr_query_count queries on dbs 17 18 19 21 25 as domas did his sync, they are all cleaned up now
* 07:02 logmsgbot: midom synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php 'wmf:r57507'
* 06:00 Fred: restarted srv100-139-150 for good measure since apache was not running.
== October 7 ==
* 22:51 AaronSchulz: running populateLogSearch.php to fill suppression log search filter data
* 22:39 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticle.php 'deployed r57501'
* 21:30 Fred: powercycling pdf1 -> c�gone unresponsive even from serial console
* 21:29 logmsgbot: brion synchronized php-1.5/includes/specials/SpecialContributions.php 
* 21:27 brion: pdf1 unreachable
* 20:12 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'deleted content rights for checkusers on frwiki'
* 20:11 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'deleted content rights for checkusers on frwiki'
* 19:41 Fred: ganglia 3.1.2 has been installed on spence. Configuration still in progress, but mostly done. http://ganglia3.wikimedia.org
* 19:28 Rob: resynced nagios for all the removed and decommissioned servers
* 19:28 Rob: decommissioned srv142 & srv143. wiped, unracked
* 19:12 Rob: srv85 decommissioned
* 19:09 Rob: srv148 dead, decommissioned, removed from dsh, lvs, and server roles
* 19:04 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed second level for now'
* 19:00 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'update flaggedrevs_labs wiki config'
* 18:59 Rob: decommissioning srv51, removed from server roles, node groups, lvs, and wiping in rack
* 18:56 Rob: decommissioned srv48, srv49, srv50. removed from node groups, server roles, wiped, unracked.
* 18:50 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'deployed r57480'
* 18:40 mark: Installed puppet on grosley
* 18:39 logmsgbot: brion synchronized php-1.5/includes/specials/SpecialContributions.php 'sync r57479 spacing fix for contribs tools links'
* 18:34 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'update flaggedrevs_labs wiki config'
* 18:33 brion: cleared flaggedrevs, flaggedpage_config tables on flaggedrevs_labwikimedia: prep for new config test
* 18:28 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'deployed r57475'
* 18:28 mark: Manually updated ntp.conf on isidore
* 18:18 logmsgbot: brion synchronized php-1.5/includes/OutputPage.php 'r57473 fix for fatals in edge case'
* 18:12 mark: Manually updated ntp.conf on srv1, srv2 and bart
* 18:06 logmsgbot: brion synchronized php-1.5/includes/HistoryPage.php 'r57471 perm fix for show/hide revs box on history page'
* 17:46 logmsgbot: brion synchronized php-1.5/includes/OutputPage.php 'sync r57468 fix -- mdale changes to script loading broke wgStylePath again'
* 17:46 mark: Installed puppet on loudon and locke
* 17:43 mark: Installed puppet on sq1
* 17:41 mark: Installed puppet on tridge
* 17:28 brion: js2-borne code is breaking wikibits.js again; mdale please make sure $wgStylePath is respected
* 17:22 brion: during update some brief borkage with missing stylesheets. inconsistency in file update order? seems ok now
* 16:58 Rob: decommissioning srv48, srv49, srv50. removed from server_roles, node groups, running wipe in rack
* 16:52 Rob: srv81 crashed, decommissioning, removed from node groups, server roles, lvs, wiping in rack
* 16:35 Rob: pulled srv44-srv47 from rack, decommissioned
* 16:17 Rob: srv31 already wiped, unracked
* 16:15 mark: Installed puppet on singer
* 16:14 Rob: srv31 still racked, but is crashed, decommissioning.
* 16:12 mark: Installed puppet on streber
* 16:10 Rob: decommissioning srv137, pulled from rack, wiped, pulled from nodegroups and lvs
* 16:09 mark: Installed puppet on mchenry and sanger
* 16:08 mark: Removed puppet again on isidore; it's ubuntu 7.04 and really needs to be replaced soon
* 16:06 mark: Installed puppet on isidore
* 16:05 mark: Installed puppet on pdf1
* 16:00 mark: Installed puppet on bayle and bayes
* 15:59 mark: Installed puppet on db20
* 15:58 Rob: negate that, not depooling.
* 15:58 Rob: depooling sq43 until later today when i can poke at it.
* 15:55 Rob: srv131 decommissioning, removed from lvs, node groups and server roles, wiping in rack.
* 15:49 mark: Installed puppet on spence and williams
* 15:47 mark: Installed puppet on lvs2-4
* 15:42 Rob: redeployed sq43
* 15:41 mark: Installed puppet on storage1-3
* 15:35 Rob: srv130 was down, out of warranty, decommissioned, removed from node groups and lvs, wiping in rack
* 15:32 mark: Shutdown srv9 for decommissioning
* 15:31 mark: Installed puppet on amane and erzurumi
* 15:25 mark: Installed puppet on fenari and mobile1
* 15:20 mark: Installed puppet on ms2 and ms3
* 15:19 Rob: srv122 decommissioning, pulled from nodes and lvs, wiping in rack
* 15:10 mark: Installed puppet on hume
* 15:10 Rob: srv90 pulled, decommissioned, wiped, and out of node groups and nagios
* 15:07 Rob: lucene group in nagios sync was halting it, since no hosts exist in that group anymore. commented out group in nagios conf.php and synced.
* 15:00 Rob: server rose decommissioning, removed from nodegroups, pulled from rack, removed from server roles
* 14:57 Rob: srv118 running wipe, removed network connection.
* 14:56 Rob: srv118 down, reboots for no good reason, out of warranty, decommissioning, removed from LVS, nodegroups, wiping in rack
* 14:43 Rob: decommissioned coronelli, pulled from rack, removed from server roles, removed from node groups
* 14:39 Rob: decommissioned maurus, pulled from rack, removed from server roles, removed from node groups
* 02:20 Tim: hard reset of mobile1, was not responding on ssh or serial
* 02:18 brion: mobile1 been down for a bit; Tim is poking at remote console to reboot it
* 01:54 brion: testing svn up on test 
== October 6 ==
* 23:24 logmsgbot: brion synchronized php-1.5/extensions/OggHandler/OggHandler.php 'update defualt search path for oggThumb'
* 22:49 logmsgbot: brion synchronized php-1.5/extensions/OggHandler/OggHandler_body.php 'update with oggThumb support'
* 22:49 logmsgbot: brion synchronized php-1.5/extensions/OggHandler/OggHandler.php 
* 17:24 Fred: Loudon.w.o has been commissioned for use by the Usability team for a couple days/weeks. 
* 15:47 logmsgbot: robh ran sync-common-all 
* 15:11 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '20973 Additional transwiki import sources on Incubator'
* 15:06 Rob: updating dns for ro.planet
* 14:49 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '20986 Please change the Bengali Wikibooks sitename'
* 14:39 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling uploads on vewikimedia'
* 14:34 logmsgbot: robh ran sync-common-all 
* 14:31 logmsgbot: robh ran sync-common-all 
* 13:53 mark: Fixed srv237
* 12:42 mark: Fixed situation for home-mounted apaches in puppet
* 11:57 mark: scap
* 11:53 mark: Shut down coronelli, rose, maurus: these should all be decommissioned
* 11:44 mark: Fixed up search node group
* 11:31 mark: Made puppet manage ganglia on all core dbs
* 11:16 mark: Installed puppet on adler as well ;)
* 11:10 mark: Installed puppet on all core databases
== October 5 ==
* 23:20 mark: Deployed net NTP client configuration on all application servers and all squids (via puppet)
* 20:21 Andrew: Deploying LiquidThreads updates
* 19:50 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'boring'
* 18:58 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'load test'
* 17:37 domas: fixed the slack at db10. what were sysadmins doing?!
* 17:36 mark: Set up NTP server on linne, to replace zwinger
== October 4 ==
* 18:35ish aaron: deployed r57261
* 15:31 domas: implemented a mutex for update-special-pages jobs
== October 3 ==
* 13:29 hcatlin: cleared out cache and logs on mobile1
== October 2 ==
* 20:36 logmsgbot: jeluf synchronized php-1.5/wmf-config/db.php 'adding db22 back to the pool'
* 20:34 logmsgbot: jeluf synchronized php-1.5/wmf-config/db.php 'take db22 out of the pool for a few minutes'
* 20:26 JeLuF: killed two long running sql queries on db22
* 17:19 atglenn: updated puppet files: don't manage homedirs on test.wikipedia
* 14:57 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/pages/NewUserMessagesView.php 
* 14:57 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/pages/SpecialNewMessages.php 
== October 1 ==
* 20:33 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20926'
* 19:13 Andrew: Handy hint: To recache extension messages, you have to scap with the extension in extension-list *before* you turn it on on aawiki. Otherwise the require_once() fails because it's already been included.
* 19:12 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 
* 19:10 Andrew: Fixing localisation NavigableTOC bug
* 18:32 logmsgbot: brion synchronized php-1.5/wmf-config/extension-list 
* 16:03 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php 'Merge r57230'
* 15:53 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php 'Fix infinite loop, deploy r57227'
* 15:18 mark: Made www.nl.wikimedia.org work as a redirect
* 15:18 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Add missing / :))'
* 15:13 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php 'Deploy r57225'
* 14:47 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/DeletionController.php 'Deploy r57224.'
* 14:23 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php 'Deploy r57220'
* 14:23 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/DeletionController.php 'Deploy r57220'
* 14:22 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Deploy r57220'
* 13:00 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'r57215'
* 12:55 logmsgbot: andrew synchronized php-1.5/includes/db/Database.php 
* 12:47 domas: andrew keeps failing!!!! :-)
* 12:45 logmsgbot: andrew synchronized php-1.5/includes/db/Database.php 
* 12:42 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php 
* 12:40 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php 
* 12:40 logmsgbot: andrew synchronized php-1.5/includes/db/Database.php 
* 12:38 logmsgbot: andrew synchronized php-1.5/includes/db/Database.php 
* 10:59 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/pages/NewUserMessagesView.php 'Deploy 57207'
* 10:58 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php 'Deploy 57207'
* 10:29 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/pages/TalkpageView.php 'Deploy 57202'
* 10:28 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Hooks.php 'Deploy 57202'
* 10:20 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php 'Deploy r57200'
* 10:19 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Deploy r57200'
* 09:26 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php 
* 09:04 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/NewMessagesController.php 'Deploy r57193'
* 08:51 domas: mod_rewrite's shm-based mutex consumed all shm semaphore array slots on multiple apache servers, causing them all to die
* 08:23 Andrew: Scapping again with fixed scap script.
* 08:13 Andrew: scap script invocation of mergeMessageFileList.php was broken, was trying to invoke mergeMessageList.php (which doesn't exist). Fixed.
* 07:52 Andrew: Scapping to fix LiquidThreads missing messages. Solution was in a post to private-l by Tim, subject "[Private-l] Increased bandwidth usage on internal network since yesterday's scap", (add the new extension to $IP/wmf-config/extension-list)
* 02:59 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.class.php 'deployed r57186'
* 02:12 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.class.php 'deployed r57183'
== September 30 ==
* 23:06 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Deploy r57169'
* 21:35 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc.php 'depooled srv123 and replaced it with srv214'
* 21:26 brion: memcached 10.0.2.123:11000 down
* 21:23 atglenn: hard restart of apache on srv115, fred shot the localisation cache rebuilds also.. it's happier now
* 21:12 brion: apche2ctl via dsh
* 21:10 brion: apache-restart-all is still broken for puppet hosts
* 20:59 tomaszf: running static html dump on wikitech.linode
* 20:50 Andrew: Uh, make that [andrew@zwinger ~]$ FANOUT=16 dsh -f -N mediawiki-installation php /apache/common/wmf-deployment/maintenance/rebuildLocalisationCache.php --force --threads=8
* 20:50 Andrew: Running to fix LiquidThreads localisation issue: srv167: 	bash: /apache/common/wmf-deployment/maintenance/rebuildLocalisationCache.php: Permission denied
* 19:48 Andrew: Scapping to deploy updated LiquidThreads localisations
* 19:43 Andrew: Running l10nupdate on zwinger to update for LiquidThreads localisation.
* 19:37 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/pages/TalkpageView.php 'Deploy r57145'
* 19:36 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/pages/ThreadPermalinkView.php 'Deploy r57145'
* 19:31 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Turning on LQT on liquidthreads.labs.wikimedia.org'
* 19:21 logmsgbot: andrew ran sync-common-all 
* 19:18 Andrew: scapping to create liquidthreads_labswikimedia
* 19:10 Andrew: Setting up LiquidThreads labs wiki liquidthreads.labs.wikimedia.org
* 18:58 logmsgbot: aaron synchronized php-1.5/includes/specials/SpecialRevisiondelete.php 'deployed r57131'
* 17:27 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'setting duallicense on strategyappswiki'
* 16:27 Fred: restarting pdns on Lilly (too many defunct processes)
* 16:10 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticle.php 'deployed r57112'
* 15:32 Rob: fixed the issue regarding centralauth, first spotted by inablility to create new accounts by email
* 15:32 logmsgbot: robh ran sync-common-all 
* 14:30 Rob: had a duplicate 'strategyappwiki' database for the strategyappswiki by mistake, backed it up and dropped it, as strategyappSwiki is the correct database already in place.
* 13:45 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Duplicate namespace id 110 for Participants/Participans_talk, only search the former'
* 13:41 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable default searching of Participants namespaces on strategyappswiki'
== September 29 ==
* 21:33 tomaszf: added ActiveAbstract to wmf-deployment and ran sync-common on srv225. xml snapshots are now generating abstracts properly
* 19:34 logmsgbot: fvassard ran sync-common-all 
* 19:09 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'enable MWReleases on mediawiki.org'
* 19:09 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 
* 19:06 brion: cli scripts now loading AdminSettings.php again. Ideally should be restructured a bit
* 19:05 logmsgbot: brion synchronized php-1.5/wmf-config/PrivateSettings.php 
* 18:28 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding config parameters for fi.wikimedia.org'
* 18:28 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 18:14 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'enable UserDailyContribs'
* 17:47 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'db25 back'
* 17:29 logmsgbot: brion synchronized php-1.5/includes/specials/SpecialUserrights.php 'regression fix'
* 17:09 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'deployed r57062'
* 16:55 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/specialpages/UnreviewedPages_body.php 'deployed r57058'
* 16:54 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/specialpages/OldReviewedPages_body.php 'deployed r57058'
* 16:18 Fred: restarted apache on srv114-168. It appears the servers went OOM
* 14:29 mark: reloading asw-a5-sdtpa
* 14:16 mark: reloading asw-a5-sdtpa
* 14:04 mark: asw-a5-sdtpa switch reload went wrong, was asking for some license PROM. Reloaded previous software
* 13:40 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'wait for db25 resync'
* 13:38 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'wait for db25 resync'
* 13:18 logmsgbot: robh synchronized php-1.5/wmf-config/flaggedrevs.php 'enabling a few other items on flaggedrevstestwiki weeee'
* 13:11 logmsgbot: robh synchronized php-1.5/wmf-config/flaggedrevs.php 'enabling deleterevision for sysops on flaggedrevslabswiki'
== September 28 ==
* 23:38 brion: UserDailyContribs needs fixing for deployment on mysql 4.0 :( easy fix :)
* 23:35 brion: testing UserDailyContribs on test
* 22:55 brion: setting up UserDailyContribs tables...
* 21:43 atglenn: moved default kernel to the last entry in grub on ms6, see if that fixes the "boot in failsafe mode cause we feel like it" problem. 
* 21:41 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/maintenance/updateStats.inc 'deploy 57027'
* 21:14 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/maintenance/updateStats.inc 'deployed r56315'
* 20:39 domas: mobile1 was by the way serving only 404s, service was down
* 20:32 domas: mobile1 was out of disk space, fixed/cleaned up, disabled access log, etc. it is really really sad when services are in such shape...
== September 27 ==
* 16:07 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'deployed r56982'
* 16:07 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.class.php 'deployed r56982'
* 04:54 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.class.php 'deployed r56980'
* 03:13 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/specialpages/RevisionReview_body.php 
* 03:12 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticle.php 
* 03:12 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.class.php 
* 03:12 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'deploy r56890, r56892, r56972'
* 00:57 logmsgbot: midom synchronized php-1.5/includes/specials/SpecialMostlinked.php 
* 00:35 logmsgbot: midom synchronized php-1.5/includes/specials/SpecialMostlinked.php 
== September 26 ==
* 23:51 domas: db3 just... rebooted?
* 21:35 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'deploy r56958, r56959'
== September 25 ==
* 22:03 Fred: double-escaped command for image_scaler cleanup in puppet.
* 17:35 Fred: hard drive sdd on sq27 has gone bad, causing issues with 2nd Tier squid.
* 17:16 Fred: puppetized cron job on the appserver cluster for LocalisationUpdate.
* 15:16 logmsgbot: tstarling synchronized php-1.5/includes/Title.php 'r56919'
* 15:05 Tim: automated the task, ran fixCleanupTitles/revertCleanupTitles.php to revert another 558 page moves
* 13:19 Tim: repairing some more cleanupTitles.php damage from yesterday, using manual (text processed) queries to revert followed by namespaceDupes.php to fix
== September 24 ==
* 23:38 brion: restarting rebuildLocalisationCache job @ 8 threads per box / 16 boxes at a time, with --force
* 23:29 logmsgbot: brion synchronized php-1.5/extensions/LocalisationUpdate/LocalisationUpdate.class.php 
* 23:27 brion: running rebuildLocalisationCache on all boxen, dsh 8 at a time to avoid killing things
* 23:05 brion: tested rebuildLocalisationCache on srv235
* 23:02 logmsgbot: brion synchronized php-1.5/maintenance/Maintenance.php 'fix for maint scritps memory_limit'
* 22:39 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'try general rollout of LU, see what happens'
* 22:20 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabled LU on incubator, wikinews/books/tionary/versity/source'
* 21:49 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 'removing stub LUDependency def'
* 21:39 atglenn: upgraded ilom firmware on ms6, rebooted again, updated racktables with sn and ip addrs, back to normal now
* 19:04 logmsgbot: brion synchronized php-1.5/extensions/LocalisationUpdate/update.php 'rem mem limit'
* 18:57 brion: attempting a run of LocalisationUpdate updater script...
* 18:56 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'LU on on test & aa'
* 18:55 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 'update LU config'
* 18:29 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '19707 Set up a logo for mhr.wikipedia'
* 18:25 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20552 Add sources and enable transwiki import for pt.wikibooks'
* 17:17 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 15829 Better Malayalam translation for Namespace'
* 16:55 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20761 Set Portal namespace alias to NS_Portal in Bengali wikipedia'
* 16:51 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20271 Give bureaucrats the ability to remove sysop rights at simplewiki'
* 16:45 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20513 enable email on notify for watchlist on strategywiki'
* 16:38 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20367 Set WP namespace alias to NS_PROJECT in Bengali wikipedia'
* 08:05 logmsgbot: midom synchronized php-1.5/extensions/ConfirmEdit/FancyCaptcha.class.php 
* 05:42 Tim: [[Talk:Portal:...]] pages on bnwiki were corrupted due to half-completed namespace change (bug 20314) and cleanupTitles.php run. Fixed and deployed r56868.
* 04:30 Tim: running cleanupTitles.php on all wikis
* 04:25 Tim: running cleanupTitles.php on frwikinews
* 04:23 Tim: updated maintenance directory to r56864.
* 03:59 logmsgbot: aaron synchronized php-1.5/wmf-config/CommonSettings.php 
* 03:54 logmsgbot: aaron synchronized php-1.5/wmf-config/CommonSettings.php 'added feedback size'
* 03:48 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'set feedback size default'
* 03:45 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'cleaned up feedback vars a bit'
* 03:41 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'cleaned up feedback vars a bit'
== September 23 ==
* 22:11 tomaszf: copying zwinger ssh keys to fenari
* 21:09 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'strategyappswiki readerfeedback threshhold set'
* 19:39 Rob: slated erzurumi for internal use as it was only running pdf export, which now exists on pdf1
* 19:36 Rob: updated dns for erzurumi internal ip
* 16:35 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'reverted hack'
* 16:30 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'Created aliases for RUWIKINEWS for Project and Portal.'
* 16:24 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'zhwikisource test'
* 16:15 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'Reducing $wgFeedDiffCutoff due to reports of OOM'
* 10:18 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'adding ixia'
* 00:59 logmsgbot: aaron synchronized php-1.5/extensions/ReaderFeedback/specialpages/RatedPages_body.php 'deployed r56490, r56496'
== September 22 ==
* 23:37 logmsgbot: brion synchronized php-1.5/includes/specials/SpecialUpload.php 'testing second fix to commonist upload bug'
* 22:59 logmsgbot: brion synchronized php-1.5/includes/specials/SpecialUpload.php 'testing possible upload bot workaround'
* 22:11 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 21:39 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'disable l10nupdate on test as well'
* 21:36 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 'emergency hack -- empty LUDependency class'
* 21:34 brion: hitting rebuildLocalisationCache
* 21:33 brion: LocalisationUpdate pollutes the localisation cache and breaks when you disable it
* 21:31 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'LU back off to confirm cpu usage bug'
* 21:20 brion: attempting to run updates on LocalisationUpdate
* 21:18 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling LocalisationUpdate pre-update; make sure it dont explode'
* 21:01 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 'l10nupdate settings'
* 21:00 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'l10nupdate settings'
* 20:50 brion: setting up l10nwiki databases on each DB cluster to hold LocalisationUpdate tables
* 20:30 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'testing config var for db stuff'
* 19:58 apergos: rebooting ms6, try to unstick ipmi interface
* 18:23 brion: running LocalisationUpdate from current trunk for testwiki
* 18:18 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'Testing LocalisationUpdate on test.wikipedia again...'
* 18:15 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'thanks brion'
* 18:06 brion: ok, bad proofread queries killed on most/all s3 boxen
* 17:53 logmsgbot: brion synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php 
* 17:37 brion: removing db query fix for ProofreadPage; runs too slow. may need version revert
* 17:33 logmsgbot: brion synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php 'testing db query fix, fixed'
* 17:32 logmsgbot: brion synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php 'testing db query fix'
* 17:31 brion: we found a buglet in ProofreadPage (ambiguous join in query), should be resolved shortly
* 16:46 brion: setting up pr_index tables for ProofreadPage update
* 16:44 brion: noting that DB admin credentials aren't currently configured right on live deployment. (AdminSettings needs migration?)
* 16:42 logmsgbot: brion synchronized php-1.5/maintenance/sql.php 
* 12:48 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'taking out ixia for hardy upgrade'
* 10:27 mark: Switched back traffic, so far so good
* 08:41 mark: Capacity test, DNS scenario knams-down
* 06:48 mark: Redistributed LVS weights of upload squids
* 06:39 mark: Redistributed LVS weights of text squids
* 00:39 logmsgbot: brion synchronized php-1.5/includes/specials/SpecialMovepage.php 'testing fix for image redir bug'
== September 21 ==
* 21:20 logmsgbot: andrew synchronized php-1.5/includes/upload/UploadBase.php 'Deploy r56734'
* 21:02 Andrew: Deployed r56631 on test, waiting for test to sync it
* 19:44 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'removed deleterevision ability on sysops on simplewiki due to issues listed in 20186 and 18780'
* 19:40 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'removed deleterevision ability on sysops due to issues listed in 20186 and 18780'
* 19:04 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'wgAllowImageMoving on by default'
* 18:57 logmsgbot: andrew synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionView.php 'Deploy r56728, blocking/codereview interaction bugfix'
* 18:36 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '20632 Activate collection extension on itwiki'
* 18:34 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '19819 Activate revisiondelete for sysops on simple wiki'
* 18:32 Fred: srv217 disabled in LVS since it is unstable.
* 18:06 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19697 Activate RevisionDelete for sysops on dewiki'
* 17:47 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20331 Enable RevisionDelete on Polish Wikipedia'
* 17:38 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'updates for bug 20314 adding portal namespace to bnwiki, 20322 setting local time on vecwiki, and 20324 setting some namespacealiases on vecwiki'
* 16:48 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20505 added namespaces to trwikinews'
* 16:44 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'upated logo for ptwiki per bug 20659'
* 16:35 Rob: thats arwikiquote, hit enter too soon.
* 16:35 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19060 logochange on arwiki'
* 16:33 Rob: spoke too soon, forgot to sync some files, restarting apache proceesess once more
* 16:33 Rob: Fred found a number of apaches not in node lists, added them back in, then i ran updates so hopefully strategyappswiki will work properly without redirections
* 16:21 logmsgbot: andrew synchronized php-1.5/maintenance/cleanupTable.inc 
* 16:20 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20334 corrrection'
* 16:18 mark: Installed puppet on sq16-40, which increases maximum FDs from 32k to 64k
* 16:15 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20527 Adding Project namespace in eowiki'
* 16:12 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 'bug 20291 disable obsolete MakeSysop and MakeBot extesions'
* 16:06 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20334 namespace additions and aliases for viwikisource'
* 15:57 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'adding ixia again'
* 15:47 Rob: done resyncing that file over and over for this moment, stratappswiki has proper settings now
* 15:47 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'adding all three had an issue for stratappswiki, adding htem back one at a time'
* 15:45 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'adding all three had an issue for stratappswiki, adding htem back one at a time'
* 15:44 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'adding all three had an issue for stratappswiki, adding htem back one at a time'
* 15:43 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'adding three new readerfeedback rating options for strategyappswiki'
* 15:42 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'adding three new readerfeedback rating options for strategyappswiki'
* 11:36 Andrew: cleanupTitles.php stopped because I installed new internet, rerunning, this time in a screen on zwinger.
* 10:15 Andrew: Running cleanupTitles.php on all wikis
* 07:06 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'removing ixia, it got lagged as wasnt part of lag tests'
* 02:58 tomaszf: decreasing retention rate on storage2 to 10 instead of 15 for snapshots
== September 20 ==
* 20:44 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 're-adding ixia'
* 16:13 domas: apparently LVM doesn't like snapshot volumes getting full - either wrong policy set, or a bug
* 16:08 domas: LVM snapshots on ixia deadlocked i/o: "[5284280.064801] device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception " - meminfo: http://p.defau.lt/?eytJOh1FNy_Fnf9dwyQyjg
* 16:06 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 01:49 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'removing feasibility and impact readerfeedback categories from strategyappswiki'
== September 19 ==
* 16:29 logmsgbot: aaron synchronized php-1.5/includes/filerepo/ArchivedFile.php 'deployed r56641'
* 15:00 domas: saper is doing package installs on ptolemy, consult apt logs for details
* 13:57 mark: Installed puppet on sq16
* 13:56 mark: Fixed puppet site.pp
* 13:42 hcatlin: mobile stats redeployed and page-error fixed: http://stats.m.wikipedia.org/
* 12:29 hcatlin: reconfigured nginx on mobile1 to use the new cap sources as the static file source instead of the deprecated /srv/wikimedia folder
* 12:20 hcatlin: pushing up a fix to mobile1 to correct mw-headline changes made to skin.php that broke sectioning on mobile
== September 18 ==
* 17:57 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'updated logo location for specieswiki.'
* 17:32 Rob: replaced bad hdd/3 in sq43
* 17:32 Rob: replaced bad hdd in db18
* 17:32 Rob: replaced bad hdd in srv148
* 17:32 Rob: replaced bad powersupply on srv 245 powersupply
* 13:57 logmsgbot: andrew synchronized php-1.5/includes/HTMLForm.php 'Merge r56184, bugfix for preferences regression caused by new Html class with different parameter ordering.'
* 11:42 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'adding reinstalled db5'
* 10:39 Andrew: (UsabilityInitiative toolbar brokenness). Also, turning the toolbar back on.
* 10:39 Andrew: Scapping to deploy r56592.
* 10:02 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'ixia readded, copy is done from snapshot anyway hehe'
* 09:26 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'taking out ixia for db5 copy'
* 09:11 domas: nuked db5 disk array, rebuilding system with raid10
* 08:56 domas: (threshold for autocommit is 30 minutes)
* 08:54 domas: wmf-config autocommit is in midom@fenari crontab
* 05:30 logmsgbot: tstarling synchronized php-1.5/includes/LocalisationCache.php 'added profiling sections for isExpired and initLanguage'
* 05:12 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'fixed /tmp/mediawiki references'
* 05:10 Tim: removed the old /tmp/mediawiki cache directories, were using about 1.4 GB per server
* 05:08 Tim: remerged the l10n cache directories back into a single directory, /tmp/mw-cache. Fixed lack of local message cache (probably caused huge memcached traffic).
* 05:06 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 05:06 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 
* 01:50 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/api/ApiReview.php 'deployed r56576'
* 01:49 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/specialpages/RevisionReview_body.php 'deployed r56576'
== September 17 ==
* 21:37 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 'Disabling new edit toolbar until problems resolved'
* 20:58 brion: removing software update sitenotice; most problems cleared out by now
* 20:58 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 
* 20:21 logmsgbot: aaron synchronized php-1.5/includes/LogPage.php 'deployed r56564'
* 20:07 logmsgbot: andrew synchronized php-1.5/includes/upload/UploadFromStash.php 
* 19:17 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.php 'deployed r56553'
* 19:16 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.class.php 'deployed r56553'
* 19:16 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'deployed r56553'
* 17:35 logmsgbot: aaron synchronized php-1.5/extensions/ReaderFeedback/specialpages/RatingHistory_body.php 'deploy r56545'
* 17:24 logmsgbot: brion synchronized php-1.5/includes/specials/SpecialSearch.php 'fix ordering of search creation link'
* 17:08 mark: Increased uplink capacity of asw-a5-sdtpa to csw1-sdtpa to 2x 1 Gbps trunk
* 15:38 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticle.php 'deployed 56529'
* 15:29 logmsgbot: andrew synchronized php-1.5/includes/Skin.php '-m Deploying r56523, fix for hidden category issue'
* 15:12 logmsgbot: andrew synchronized php-1.5/includes/Title.php 'Deploy r56521, fix for user css/js permissions check'
* 15:02 logmsgbot: andrew synchronized php-1.5/languages/LanguageConverter.php 'Deploy r56520, fix for unconverted text causing blank pages'
* 15:02 logmsgbot: aaron synchronized php-1.5/languages/LanguageConverter.php 'NOCC bug fig'
* 14:36 Andrew: also deploying r56517, tooltip fixes
* 14:35 Andrew: Scapping to deploy r56515 and r56516, fixes to CentralAuth global groups broken by a userrights API module.
* 14:07 logmsgbot: andrew synchronized php-1.5/includes/EditPage.php 'Merge r56478, which we forgot to deploy'
* 14:02 logmsgbot: andrew synchronized php-1.5/includes/parser/Parser.php 
* 13:01 logmsgbot: andrew synchronized php-1.5/extensions/CentralAuth/CentralAuthHooks.php 'Deploy r56509, regression spewing exceptions in CentralAuth'
* 12:32 domas: db5 RAID has issues (1s await to start with)
* 12:32 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 12:02 domas: mark confirmed network issue!!!!! I am happy!!!!!
* 09:55 logmsgbot: andrew synchronized php-1.5/extensions/AbuseFilter/AbuseFilter.hooks.php 'Deploy r56500, fix for AbuseFilter regression'
* 09:37 logmsgbot: andrew synchronized php-1.5/includes/Xml.php 'Deploy r56497'
* 02:25 logmsgbot: aaron synchronized php-1.5/includes/specials/SpecialUserrights.php 'fix fatal - declare $wgOut'
* 02:16 logmsgbot: aaron synchronized php-1.5/includes/templates/Userlogin.php 
* 02:08 logmsgbot: aaron synchronized php-1.5/includes/templates/Userlogin.php 
* 01:53 logmsgbot: aaron synchronized php-1.5/extensions/ReaderFeedback/specialpages/RatedPages_body.php 'Fixed usage of wrong messages'
* 01:50 logmsgbot: aaron synchronized php-1.5/extensions/ReaderFeedback/specialpages/RatingHistory_body.php 'Fixed usage of wrong messages'
* 01:45 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/specialpages/ProblemChanges_body.php 'query tweaks'
* 01:25 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/specialpages/ProblemChanges_body.php 'fix index name'
* 01:24 logmsgbot: brion synchronized php-1.5/extensions/FlaggedRevs/specialpages/ProblemChanges_body.php 
* 01:19 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 
* 01:18 logmsgbot: brion synchronized php-1.5/includes/api/ApiUpload.php 
* 01:08 logmsgbot: brion synchronized php-1.5/extensions/FlaggedRevs/specialpages/ProblemChanges_body.php 'index fix'
* 01:08 logmsgbot: brion synchronized php-1.5/includes/Linker.php 'fix HTML validity ("Array" in portal output)'
* 01:01 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'wgHtml5 off for now to reduce new vars'
* 01:01 logmsgbot: andrew synchronized php-1.5/includes/specials/SpecialContributions.php 'Deploy r56469'
* 01:00 logmsgbot: andrew synchronized php-1.5/includes/User.php 'Deploying r56473'
* 00:49 logmsgbot: brion synchronized php-1.5/includes/parser/Parser.php 'fix missing page problem'
* 00:39 logmsgbot: brion synchronized php-1.5/includes/specials/SpecialContributions.php 
* 00:28 atglenn: tossed old oprofiled.log from srv187, back to 3gb free now
* 00:25 logmsgbot: brion synchronized php-1.5/languages/LanguageConverter.php 
* 00:22 atglenn: apt-get clean on srv187, got back .5gb, / was full 
* 00:10 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 'update sitenotice'
== September 16 ==
* 23:53 logmsgbot: brion synchronized php-1.5/languages/classes/LanguageZh.php 
* 23:50 logmsgbot: brion synchronized php-1.5/languages/classes/LanguageZh.php 
* 23:41 logmsgbot: midom synchronized php-1.5/includes/LocalisationCache.php 'profiling'
* 23:40 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'tweaking wgCachedirectory to /tmp/mediawiki/$site-$lang'
* 23:32 brion: running a manual apache2ctl restart batch as apache-restart scripts are broken on many hosts (sudo)
* 23:32 domas: where is my attribution :-)
* 23:30 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'setting wgCacheDirectory to /tmp/mediawiki'
* 23:28 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'adding wgCacheDirectory'
* 23:19 brion: starting scap to wmf-deployment r56456
* 23:07 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 'updating with sitenotice for soft updates'
* 22:03 brion: running cleanupTitles on testwiki
* 21:53 brion: ok fixed the readerfeedback config i think
* 21:08 brion: svn up'ing -- do not scap until things are confirmed working
* 20:22 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'stratwiki api final fix i hope'
* 20:21 logmsgbot: robh synchronized php-1.5/wmf-config/CommonSettings.php 'removing private wiki overrides for api use'
* 20:03 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'stratwiki api tinkering'
* 19:57 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling writeapi on stratappswiki'
* 18:14 domas: set up 5xx logging (without upload and old query interface) at locke:/a/squid/5xx.log
* 17:59 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 
* 17:57 aZaFred_: sapshot[1..3] have been puppetized
* 17:56 brion: mediawiki-installation group troubles have been worked out. thx rob & fred!
* 17:56 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'updating w/ config preps for code update'
* 17:39 brion: sync system is currently broken. bogus digits (9, 7, 6, 8) and not-quite-set-up snapshot* machines in mediawiki-installation group
* 17:10 Rob: removed some outdated security plugins on blogs, updated some others
* 15:51 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 15:44 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Updating logo for stratappswiki'
* 14:54 mark: Moving traffic back to Europe - florida squids overloaded
* 14:39 mark: Capacity test, DNS scenario knams-down
* 14:37 Rob: moved masters from db13 to db15 with some major assistance (basically did it himself ;) from tim
* 14:34 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 14:22 Rob: script had some issues, Tim is debugging
* 14:22 Rob: yep, switching masters because db13 raid battery is dead.
* 14:20 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 'switching masters from db13 to db15'
* 14:18 logmsgbot: robh synchronized php-1.5/wmf-config/db.php 
== September 15 ==
* 23:13 brion: applying patch-log_user_text.sql to newly created wikis: mhrwiki strategywiki uawikimedia cowikimedia ckbwiki pnbwiki mwlwiki acewiki trwikinews flaggedrevs_labswikimedia readerfeedback_labswikimedia strategyappswiki strategyappwiki
* 23:10 brion: adding stub l10n_cache table to all wikis
* 23:02 brion: checking to confirm log_page/log_user_text update is applied on all wikis
* 23:01 tomaszf: installed memcache on sage.knams
* 21:31 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'changing settings for readerfeedback on stratapps'
* 21:12 atglenn: /home nfs mounted and added to fstab on srv124, new test.wikipedia
* 21:09 domas: where is my attribution ;-D
* 21:08 Rob: test.wikipedia.org fixed by mounting nfs
* 20:55 Rob: setup new private wiki, added to dns as well as configuration files
* 20:38 logmsgbot: robh ran sync-common-all 
* 19:58 Rob: servers running wipe were burdening the logging host. added drop rules to iptables on db20 to refuse those servers access since ssh doesnt work with wipe destorying things
* 19:24 Rob: depooled srv124 to use as test.wikipedia.org, then updated squid config and pushed
* 19:09 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'oh well'
* 17:14 Rob: db12 is back online with mysql running
* 16:41 Rob: installed python-pyfribidi on pdf1
* 16:06 aZaFred_: snapshot[1..3] wikimedia-task-appserver install completed. Added hosts to dsh nodegroup for mediawiki-installation so common updates get pushed to them
* 16:04 Rob: srv245 bad powersupply, swapped with on site spare
* 15:59 Rob: rebooting srv245 to fix its drac access
* 15:33 Rob: some odd invalid entries in dsh nodes, removed.
* 15:24 Rob: running wipe on srv35, srv52, srv54, srv56
* 15:20 Rob: srv66 running wipe
* 15:12 mark: Started MySQL on ms2 and restarted replication
* 15:11 mark: Redistributed the spare drives on ms2 back into the spare pool (/dev/md1)
* 15:08 Rob: shutting down db12 for raid battery swap
* 15:04 mark: Swapped drive c3t6d0 in ms2, readded it to /dev/md14 and moved back the spare /dev/sdao into the spare pool (/dev/md1)
* 14:42 mark: Shutting down MySQL on ms2
* 14:33 Rob: removed a number of decommissioned servers from nagios
* 14:30 Rob: wipe running on srv44, srv45, srv47
* 14:26 Rob: srv31, srv32, srv33 running wipe in screen sessions, do not try to use them ;]
* 14:22 Rob: srv30=srv80 to be decommissioned, wiping the drives with them in rack now. mark already depooled from apache and memcached
* 14:19 Rob: srv145 is back up and ok
* 14:07 Rob: srv145 coming back up, my bad
* 13:53 Rob: srv52 hdd died.
* 13:51 domas: cron jobs work way better, if one figures how to set permissions right (like, executable? :)
* 00:03 atglenn: grrr.. that would be ms4. 
* 00:03 atglenn: testing mailiferr (workaround for no MAILTO on solaris) on ms5 for hourly snaps
== September 14 ==
* 23:26 atglenn: rerunning the rsync list of changed files again on ms6 (last run was borked). it's in screen as root
* 23:23 aZaFred_: added nagios/ganglia monitoring for snapshot[1..3]
* 22:32 atglenn: created a directory /export/upload/wikipedia/common/thumb with no write perms on ms1 so that static html dump ext doesn't try to write in it
* 21:39 aZaFred_: upgrading Ubuntu on Sage to 8.04 LTS
* 21:12 tomaszf: running timings for en static html snapshot from hume
* 19:50 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 19:47 Rob: left out the virtual host, resyncing and restarting apaches
* 19:44 logmsgbot: robh ran sync-common-all 
* 18:21 Rob: added volunteer.wikimedia.org to dns and pushed authdns-update
* 18:03 Rob: added flagrevs stats update to hume crontab
* 16:24 brion: restarted apache on wikitech
== September 13 ==
* 21:03 mark: Converted aufs cache dir into COSS on eiximenis
* 20:41 mark: Increased cache dir object size split from 512kb to 1MB
* 17:45 domas: readded srv211 to pybal apaches list, either someone removed it by mistake, or didn't document the reason of removal
* 16:57 domas: rolling out php5 5.2.4-2ubuntu5.7wm1 and php5-apc 3.1.3p1-1wm1 live on all apaches
* 15:32 domas: rolling out limited php 5.2.4-2ubuntu5.7wm1 testing
* 15:07 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'show draft by default on iawiki'
* 08:58 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Set $wgFlaggedRevsAutoconfirm on dewiki'
== September 12 ==
* 11:08 _3^3: srv230-srv250 are running now APC-3.1.3p1-1wm1
* 08:53 _3^3: APC 3.1 is ~3-5% faster than 3.0 under load. Will roll out limited deployment for longer-term stability tests. 
* 08:29 mark: Rerouted pmtpa->esams traffic via hgtn-leaseweb
== September 11 ==
* 19:14 domas: xcache was both slower and less stable (got corrupted cache within seconds, can we blame that on -O3? :))
* 18:56 domas: (xcache is configured to have ttl of 600 seconds on all php objects.. ;-)
* 18:55 domas: installed php5-xcache on srv250 (manual comment in apc.ini though...)
* 18:39 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling Collection on cswiki per Bug 20436'
* 06:25 domas: added api.php to wpstatic acl
== September 10 ==
* 21:36 tomaszf: starting [a-z]* sync of commons images from ms5->belgrade
* 19:30 tomaszf: starting [0-9]* sync of commons images from ms5->belgrade
== September 9 ==
* 20:16 domas: unmounted /mnt/scratch on amane, /dev/sdb is gone long time ago. 
* 19:05 aZaFred_: modified puppetized cron job on the scalers to escape parenthesis in the find command
* 18:14 domas: restarted bunch of evil apaches
* 10:04 domas: noticed hawthorn flapping at http://torrus.wikimedia.org/torrus/Network?token=T0484
* 05:00 Tim: restarted apache on srv108, APC cache corruption
== September 8 ==
* 18:25 atglenn: made sure all scalers have /a/magick-tmp and are using it
* 15:49 logmsgbot: midom synchronized php-1.5/includes/db/Database.php 'removing the verbose comment'
* 12:25 logmsgbot: midom synchronized php-1.5/includes/db/Database.php 'verbosity hook yet again'
* 10:42 logmsgbot: midom synchronized php-1.5/includes/Linker.php 'mft:55983 wmf:56025'
* 10:32 logmsgbot: midom synchronized php-1.5/extensions/MWSearch/MWSearch_body.php 'wmf56024, trunk55392'
* 10:21 logmsgbot: midom synchronized php-1.5/includes/db/Database.php 
* 10:09 logmsgbot: midom synchronized php-1.5/includes/db/Database.php 'sql comment hack'
== September 7 ==
* 15:10 mark: Set AS 16265 weight to 100 on csw1-esams to balance traffic better
* 14:27 mark: Restarted apache on srv171
* 14:26 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace sub-srv81 nodes by spares'
* 14:17 mark: Depooled all apaches below srv81 in LVS
* 13:55 mark: Prepared srv43-srv47 for decommissioning
* 13:51 mark: Converted srv100 to /home-less puppet image scaler
* 13:44 mark: Depooled srv43-srv47 as image scalers in LVS
* 13:32 mark: Converted srv221-srv224 to /home-less with puppet
* 13:23 mark: Converted srv220 to /home-less with puppet
* 13:14 mark: Installed cron script to remove gs temp files on srv219, using puppet
* 13:14 mark: Converted srv219 to /home-less with puppet
* 12:34 mark: Fixed ganglia configuration in puppet
== September 6 ==
* 21:33 mark: Puppetised srv171-180
* 21:17 mark: Puppetised srv161-170
* 21:12 mark: Puppetised srv151-160
* 20:54 mark: Restarted all job runners to use the local mediawiki copy instead of /home
* 20:53 logmsgbot: mark synchronized php-1.5/maintenance/jobs-loop.sh 'Update path to use wmf-deployment branch'
* 20:41 mark: Puppetised srv211-218
* 20:38 mark: Puppetised srv201-srv210
* 20:34 mark: Puppetised srv191-srv200
* 20:30 mark: Puppetised srv181-srv190
* 20:22 mark: Not puppetising srv151-160 because of job queue runners
* 20:15 mark: Puppetised srv141-srv150
* 20:11 mark: Puppetised srv131-srv140
* 20:07 mark: Puppetised srv121-srv130
* 20:02 mark: Puppetised srv111-srv120
* 19:57 mark: Puppetised srv103-srv110
* 19:46 mark: Puppetised srv102
* 19:45 mark: Puppetised srv101
* 19:13 mark: Converted eiximenis into a text squid
* 16:38 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Added eiximenis to the Squid list, and removed obsolete entries'
* 01:05 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'rollback group for simplewiktionary'
== September 5 ==
* 23:47 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'desirability rating on strategywiki'
* 23:36 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'rollback group for simplewiktionary'
* 21:10 domas: amane has I/O failures
* 20:13 mark: Pooled application servers srv226-253
* 17:41 domas: svn switch --relocate http://svn.wikimedia.org/ svn+ssh://svn.wikimedia.org/
* 16:34 apergos: cleared out some stuff in /tmp on srv220-224 to free up more space. this is only a TEMPORARY FIX
* 16:29 hcatlin: Setup autoclearing logs on mobile1
* 15:46 logmsgbot: midom synchronized php-1.5/includes/GlobalFunctions.php 'removing extended profiling'
* 15:31 apergos: apt-get clean on a few (srv220-224) hosts to buy us a tiny bit of disk space
* 13:13 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'added Proposal NS to default search on strategywiki'
* 13:00 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'fixed ruwikiquote config'
* 12:59 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'fixed ruwikiquote config'
* 12:32 mark: Ganglia tmpfs filesystem on zwinger ran out of memory, removed some obsolete RRDs in there, and in the on-disk copy
* 11:18 mark: Restarted confused gmetad_pmtpa on zwinger
* 11:15 mark: Rebooting all new apache servers
* 11:15 mark: Updating Nagios with the new servers failed. Could someone please update the docs?
* 11:06 mark: Running apt-get dist-upgrade on all new servers
* 10:54 mark: scap
* 10:53 mark: Added new servers to dsh node groups
* 10:34 mark: Installed puppet on srv227-srv253, signed all certificates
* 10:30 mark: OS-installed srv227-srv253, except broken srv245
* 08:27 mark: Installed srv226 as mediawiki application server
* 08:15 mark: Removed 29 GB of access.log in /usr/logs/ on mobile1 (why is it there and not logrotated?). Restarted thin, memcached. Test URL does not work, but the site itself does seem to work. Please check/update the docs!
== September 4 ==
* 22:27 brion: rerunning parser tests for revs that had negatives to clear out the false failures
* 22:09 brion: added 1gig swap file to wikitech to see if that helps let the parser tests complete
* 17:22 brion: started rsyncd on searchidx1 per robert's note on broken updates
* 03:24 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'fixed wiki name typo'
* 01:30 atglenn: turned on automatic zfs replication from ms1 to ms5; copying over incrementals every ten minutes out of cron
== September 3 ==
* 21:35 atglenn: huh guess I should log this, gmond installed on ms5 (from old ts repo spec, v3.1 can't coexist with v3.0x), fred started collecting stats on it (yesterday)
* 17:48 atglenn: first incremental zfs rep from ms1 to ms5, still using cat on the receiving end, mbuffer + zfs recv is just too slow
* 16:43 mark: Installed gmond on mobile1
* 16:39 mark: Power cycled mobile1
* 16:37 brion: mobile1 not responding to ping, needs a reboot
* 02:34 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'removed PovWatch'
* 02:12 rainman-sr: rsyncd died for some misterious reason on searchidx1, needs restarting
== September 2 ==
* 22:01 atglenn: started zfs recv on ms5 from local copy of images data (local copy arrived complete yesterday pm)
* 21:22 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'let bureaucrats remove Confirmed group'
* 17:32 aZaFred_: fixed a couple problems with planet on bart (fr and zh causing the python script to crash and stop updating the other "planets"
== September 1 ==
* 22:42 logmsgbot: aaron synchronized php-1.5/wmf-config/abusefilter.php 'log detail rights for autoconfirmed on dewiki'
* 21:31 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'removed unused flood flag added to eswikinews; eswikibooks was the correct wiki'
* 20:54 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Re-sync for srv220'
* 20:16 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php
* 20:12 aaron: also renamed huwiki 'confirmed' group to 'trusted' (bug 19885)
* 20:12 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Cleaned-up redundant settings and a few instances of improper setting of autopromote array'
* 18:39 Fred: Added Ganglia stats for /export usage on export2
* 18:39 Fred: Added Ganglia stats for /export usage
* 18:27 Fred: rebooted srv156 after it went OOM
* 13:56 mark: Power cycled mayflower
* 09:07 domas: lily pdns was dead probably because of my tampering with defunct processes
* 02:27 river: restarted pdns_recursor on lily as it had died somehow
== August 31 ==
* 23:32 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'rollback group for arwiki'
* 23:24 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'autopatrol for rollbackers on svwiki'
* 22:02 logmsgbot: aaron synchronized php-1.5/wmf-config/abusefilter.php 'log detail rights for autoconfirmed on zhwiki'
* 21:55 logmsgbot: aaron synchronized php-1.5/wmf-config/abusefilter.php 'itwiki abusefilter settings'
* 20:26 logmsgbot: aaron synchronized php-1.5/wmf-config/abusefilter.php 'itwiki abusefilter settings'
* 20:08 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'rollback, autoreviewer, and confirmed groups for zh_yuewiki'
* 19:49 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'flood flag for eswikibooks'
* 12:18 domas: restarted srv183 via drac
== August 30 ==
* 19:49 aZaFred_: rebooting srv159
* 19:47 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc.php 'srv159 is dead. replacing with srv218'
* 19:44 aZaFred_: srv159 <-> srv218 in mc.php
== August 28 ==
* 19:09 brion: raising upload file size limit on wikitech wiki so my presentation uploads :P
* 17:41 logmsgbot: midom synchronized php-1.5/includes/OutputHandler.php 'per Tim and tests, set gz level to 6'
* 17:39 domas: TimStarling was nearly right - compression level impact on mediawiki output: http://spreadsheets.google.com/pub?key=t-EjyzEfh0t39hoQbf-a5fw&output=html
* 17:12 domas: comparison of failing vs working APC oprofiles: http://p.defau.lt/?1BWDb3VOalQ7d9zMaKhWcg
* 17:11 domas: srv194 seems to be deciding not to use APC, somewhere at apc_cache_busy() or somewhere near - same problem seen few times before on other servers
* 15:23 domas: compression that is, not encoding ;-)
* 15:22 logmsgbot: midom synchronized php-1.5/includes/OutputHandler.php 'bumped up encoding from 3 to 9'
* 15:12 Tim: added mobile1 to the tiertwo ACL so that it gets X-Cache headers
* 15:09 logmsgbot: midom synchronized php-1.5/includes/AutoLoader.php 'removing profiling hook'
* 14:50 logmsgbot: midom synchronized php-1.5/includes/OutputHandler.php 'ooooops'
* 14:49 logmsgbot: midom synchronized php-1.5/includes/OutputHandler.php 'compression profiling hook'
* 14:33 mark: Set up blocking of unnecessary http headers in squid
== August 27 ==
* 20:50 brion: rerunning past commit parser tests as i deleted them all by mistake :P
* 20:22 brion: re-disabling pdfhandler on wikitech, something ain't rendering right yet
* 20:15 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 20:14 rainman_: move eswiki highlighting from search4 to search11 to balance out the load a bit better
* 20:01 brion: installing PdfHandler on wikitech; udpating to current wmf-deployment
* 19:52 brion: re-running parsertests for rev 55603 and above after updating php packages
* 19:29 brion_: attempting to apt-get upgrade on wikitech, see if that helps crashy tests
* 18:25 mark: Restarted squid frontend on eiximenis
* 17:53 Tim: fixing missing blobs tables on ms3 cluster rc1
* 17:44 Tim: running recompressTracked.sh on hume
* 17:40 mark: Shutdown MySQL on ms2 for reboot
* 17:28 mark: Replication on ms2 caught up with master ms3
* 17:08 mark: Recovered data on ms2 by rebuilding raid-arrays, started mysql, started replication
* 16:44 logmsgbot: midom synchronized php-1.5/wmf-config/InitialiseSettings.php 'remove optin for performance reasons'
* 16:43 logmsgbot: midom synchronized php-1.5/wmf-config/InitialiseSettings.php 'remove optin for performance reasons'
* 16:28 mark: Installed gmond on mayflower
* 16:20 mark: Power cycled mayflower again
* 16:06 mark: Recovering RAID arrays on ms2
* 15:43 brion: updating to wmf-deployment r55629 -- fixes to api, (un)delete storage, and collection
* 15:33 logmsgbot: tstarling synchronized php-1.5/includes/Revision.php 'r55628'
* 15:26 logmsgbot: midom synchronized php-1.5/includes/AutoLoader.php 'profiling'
* 15:10 domas: bunch of servers in srv190+ range have intermittent network problems, leading to connection failures, etc
* 14:58 logmsgbot: tstarling synchronized php-1.5/includes/Revision.php 'testing archive issue pre-commit'
* 10:03 mark: Power cycled mayflower (svn)
== August 26 ==
* 21:56 Tim: observing overload on s4 slaves due to long-running ApiQueryBacklinks queries, will kill
* 21:29 brion: update to r55616 -- fixlets and api limit tweak
* 18:11 logmsgbot: midom synchronized php-1.5/includes/AutoLoader.php 'profiling hook'
* 17:08 brion: rerunning parser tests for r55603; temporary failure on the test box
== August 25 ==
* 15:52 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'mwahahaha'
* 15:47 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 15:44 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'load testing, muhaha'
* 15:40 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'putting storage1 to cluster22'
* 14:21 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'enable PdfHandler sitewide; forgot to do this earlier :D'
== August 24 ==
* 00:45 domas: CPU 100% on apache cluster for few minutes, reason unknown :) (thats what 'unbalanced load' probably means, hehe)
* 00:12 brion: unbalanced load; db16 reporting too many connected threads
== August 23 ==
* 23:32 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20363: namespace fixes'
* 23:14 Fred: HD in db18 Enclosure 0 Slot 21 failed and needs to be replaced. 
* 23:12 Fred: HD in db21 - Enclosure0 Slot 7 needs to be replaced.
* 22:55 Fred: restarted apache on srv140
* 22:55 Fred: restarted memcached on srv197
* 16:43 Fred: restarted memcached on srv211
* 00:50 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php '20338 Enable Collection on Bengali Wikipedia'
* 00:18 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix eswikibooks english import source (bug 20352)'
== August 22 ==
* 00:48 atglenn: next approach: zfs send (ms1) -> cat > filename (ms5). running in screen.
* 00:43 atglenn: stopped zfs send/receive from ms1 -> ms5: 3 GB in two hours = fail :-P
== August 21 ==
* 22:16 atglenn: starting copy of image data to ms5 (consider this timing tests, not the real thing): zfs send running in screen session
* 20:51 aZaFred_: restarted memcached on srv212
* 20:36 logmsgbot: fvassard synchronized php-1.5/wmf-config/db.php 'commenting out the rc2 cluster since it doesnt seem to be used anymore...'
* 20:01 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc.php 'Removed srv52 and added srv219 in its place since srv52 is not behaving...'
* 17:48 aZaFred_: fixed nrpe config on adler as it was set not to accept any connection not coming form 127.0.0.1
* 17:37 aZaFred_: switching Nagios from bart to spence (its new home) and decomissioning Nagios on Bart.
* 16:34 aZaFred_: changed settings for sq43 since /dev/sdd disappeared.
* 16:15 aZaFred_: restarted memcached on srv89
* 15:09 hcatlin: Mobile1 just had af, az, bn, ca, bg, and drb languages deployed.
== August 20 ==
* 23:02 domas: opensearch is causing most of 'revision' SQL load (probably text too). search overall accounts for more than 25% of database::query time :-)
* 19:48 logmsgbot: midom synchronized php-1.5/includes/Revision.php 'removing profiling hooks'
* 19:07 logmsgbot: midom synchronized php-1.5/includes/Revision.php 'profiling hooks'
* 18:42 brion: running featured article imports on readerfeedback & flaggedrevs test wikis
* 18:25 brion: pulling enwiki featured articles to copy into readerfeedback & flaggrevs test wikis
* 18:18 domas: Just logging, that SUL and job queue was broken because of bogus entry in dblist
* 18:18 domas: DNS was unhappy!
* 18:09 brion: nuked bogus 'en_flaggedrevs_labswikimedia' entry from all.dblist; per domas it broke job queue
* 18:00 logmsgbot: brion synchronized php-1.5/wmf-config/flaggedrevs.php 'adding flaggedrevs_wikimedia config'
* 17:41 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'Added default languages for flaggedrevs.labs and readerfeedback.labs'
* 17:22 Fred: synched db-list file for readerfeedback/flaggedrevs test wikis
* 17:22 Fred: removed srv52 from dsh nodelist as it was causing sync* to fail (amongst other things)
* 16:58 brion: poking at readerfeedback/flaggedrevs test wikis setup with fred
* 16:42 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '20300 - Fix arwikisource namespaces'
* 12:25 logmsgbot: andrew synchronized php-1.5/wmf-config/abusefilter.php 'Remove abusefilter-private right from abuse filter users on nlwiki, it leaks IP addresses to unidentified users'
== August 19 ==
* 21:00 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'labs settings'
* 20:53 Rob: added dns for readerfeedback and flaggedrevs lab sites
* 20:47 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'flaggedrevslabswiki'
* 18:57 Rob: and bug 19019
* 18:57 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'setting wgVariantArticlePath for bug 8532'
* 17:29 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 17:14 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'rob undoing changes he made'
* 17:08 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 17:05 logmsgbot: robh synchronized wmf-deployment/wmf-config/InitialiseSettings.php 
* 15:39 mark: Shutting down MySQL on ms2, rebooting the machine
* 15:13 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'db/mc file renames'
* 15:12 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 15:12 logmsgbot: midom synchronized php-1.5/wmf-config/mc.php 
* 15:07 logmsgbot: midom synchronized php-1.5/wmf-config/db-pmtpa.php 
* 14:27 apergos: snapshots enabled on ms1 about ten hours ago after a small disaster with image deletion affecting wps: ch, cdo, ce, ceb (restored from ms6 and elsewhere, thumbs not impacted) 
* 12:11 mark: Repeated cache dir changes for SSD testing on sq49... they disappeared last week, how come? Did someone overwrite the upload-settings squid conf file?
* 10:40 rainman-sr: updated to latest mwdumper on searchidx1 to fix en.wp index update breakage
* 04:38 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Give unreviewedpages right to Coders'
== August 18 ==
* 23:16 logmsgbot: brion synchronized php-1.5/languages/messages/MessagesBg.php 'touching for cache test'
* 22:24 brion: configured wikitech-l and mediawiki-l to accept jpeg, png, and gif attachments
* 20:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#20300 arwikisource namespaces'
* 19:34 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 'bump style version'
* 18:31 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#20301 bnwiki sitename'
* 12:12 domas: ruby processes were running out of FDs because of new logging code - fds weren't closed, leading to 1024 limit hit in ~16 hours :)
* 11:19 domas: mobile1 was serving 502's. mobile1 is a fucked up mess too. restarted merb via /etc/init.d/thin start, seems to have brought mobile1 up
* 00:44 atglenn: removing the rest of the (now unused) thumb data from ms1. yay! 
== August 17 ==
* 23:26 atglenn: add fred to root alias on mchenry 
* 23:04 atglenn: enabling zfs snapshots for thumbs on ms4 (hourly/daily/weekly/monthly)
* 21:40 logmsgbot: brion synchronized php-1.5/extensions/ContributionTracking/ContributionTracking_body.php 'update r55206 contrib tracking timestamp'
* 20:14 atglenn: moving thumbs out of the way on ms1 (here we go again)... nothing should be using these any more...right?
* 20:04 atglenn: move outdated stuff from /export/upload/scripts on ms1 to /root/export-scripts-old (including rcs dir; we have current stuff in svn now)
* 19:33 Fred: enabled ganglia data gathering of export/thumbs .
* 18:52 atglenn: tweak thumbhandler on ms4 to handle multiple / after the hostname
* 18:11 atglenn: remove cache server stuff from thumbhandler.php on ms4, we don't have thumb cache servers any more
* 17:54 atglenn: change error handler on ms1 from thumbhandler to 404.php since we don't serve thumbs from there now
* 15:25 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Increment wgStyleVersion to placate Simetrical'
* 15:16 logmsgbot: tstarling synchronized php-1.5/skins/monobook/KHTMLFixes.css 'r55187'
* 13:37 Tim: removed the /wmf-config/ from sync-file so that you don't have to prefix every filename with ".."
* 13:33 logmsgbot: tstarling synchronized php-1.5/wmf-config/../includes/RawPage.php 'r55180'
* 13:33 logmsgbot: tstarling synchronized php-1.5/wmf-config/../api.php 'r55180'
* 13:32 logmsgbot: tstarling synchronized php-1.5/wmf-config/../index.php 'r55180'
* 13:32 logmsgbot: tstarling synchronized php-1.5/wmf-config/../includes/WebRequest.php 'r55180'
== August 16 ==
* 21:08 domas: set db8 read_only=0 :) 
* 21:06 domas: commonswiki split off to separate database cluster s4 (db8,db3,db5,ixia), split off log position: db8-bin.005:650239113
* 21:06 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'commons ro end'
* 21:05 logmsgbot: midom synchronized php-1.5/wmf-config/db-pmtpa.php 
* 21:04 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'commons going readonly'
* 11:02 domas: sq45 I/O errors from start: http://p.defau.lt/?87cfQAK1ybCk8SALdYPV7g
* 10:53 domas: sq45 sdd was misbehaving, got better after restart
* 10:47 domas: powercycled sq45
== August 15 ==
* 20:27 mark: Started MySQL on ms2
* 19:18 logmsgbot: root ran sync-common-all 
* 19:08 JeLuF: closed advisorywiki (bug 19855)
* 19:08 logmsgbot: jeluf ran sync-common-all 
* 19:03 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#19777 tkwiktionary and trwiktionary namespace settings'
* 17:18 JeLuF: Added a check to sync-common-file and scap that stops the script if SSH_AUTH_SOCKET is not set (missing ssh-agent). See bug 20080
* 17:03 mark: Set up udev rules for Solaris like hard drive symlinks, Partitioned unused drives in ms2, set up RAID-1 arrays over unused drives (now syncing)
* 17:01 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#20194 favicon for strategywiki'
* 16:58 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#20232 arcwiki sitename change'
* 16:51 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#20240 mwlwiki logo'
* 16:41 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#19661 Namespace alias'
* 14:38 mark: Rebooted sq1
* 14:37 mark: Deinstalled squid on sq1, it's for varnish testing
* 12:39 mark: Shutdown MySQL on ms2
* 07:38 domas: verified that morebots is functional :)
* 07:36 domas: fixed main instance replication for ms3->storage1
* 07:36 domas: restarted morebots, was hanging somewhere in the ether
== August 14 ==
* 04:13 aaron: synchronized php-1.5/wmf-config/flaggedrevs.php 'Disabled flaggedrevs autopromote on ukwiktionary'
* 18:29 atglenn: tweak thumb url regexp for upload squids and make live
* 17:50 atglenn: turn logging back off on sq9, found em 
* 17:42 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc-pmtpa.php 'Replaced srv122 with srv212 since srv122 is down.'
* 17:42 atglenn: turn on access logging temporarily on sq9 to track down some thumb requests to ms1
* 09:55 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable optin on strategy, usability. There is no point.'
== August 13 ==
* 22:20 atglenn: redeploy sq1 change, since its files don't get refreshed by the generate script :-P
* 21:45 Rob: sq43 depooled due to bad hdd
* 21:40 Rob: rebooting sq43 for possible bad hdd check
* 20:52 atglenn: sq43 looking for a cache dir on /dev/sdd (which it doesn't have), leaving nagios to whine while I look into it
* 20:41 atglenn: restarted backend squid on sq43
* 20:32 atglenn: pushed out ms4 change to all pmtpa upload squids 
* 20:23 aZaFred_: rebooted clematis after it became partially unresponsive.
* 19:55 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Big 20226 set logo for pnbwiki'
* 19:37 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20094 fixed'
* 19:07 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '19777 Set sitename of Turkmen wiktionary'
* 19:04 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '20191 Remove autopatrol usergroup from Arabic Wikipedia'
* 19:01 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '20222 Turkish Wikinews Translations of Site Variables'
* 18:58 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '19923 Flood flag for es.wikibooks'
* 18:48 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20094 please create Auteur namespace at fr.wikisource'
* 18:40 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'enable upload on trwikinews per bug 20215'
* 18:35 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fixing language of ua.wikimedia.org'
* 18:32 atglenn: deploy ms4 update to sq13 (live squid)... see how it goes
* 17:32 brion: set usermatchmode=search in Bugzilla config, should help in assigning bugs/CCs
* 15:32 mark: Rebooting ms2
* 12:22 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'Limit patrolling to $wgFlaggedRevsPatrolNamespaces on huwiki'
* 11:13 Andrew: Scapping to get a consistent state.
* 11:13 Andrew: turned HoneypotIntegration off, Somebody went and removed useful options from HttpFunctions, which are needed to maintain the data.
* 10:50 logmsgbot: andrew synchronized php-1.5/wmf-config/secret-projects.php 'HoneypotIntegration secret data'
* 10:42 Andrew: Setting up a test setup of HoneypotIntegration extension on testwiki
* 07:48 logmsgbot: tstarling synchronized php-1.5/wmf-config/db-pmtpa.php 'removed ms2 from cluster22'
* 00:50 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 'adding width and height to wikimedia-project footer icon per bug 20203'
== August 12 ==
* 23:56 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'ptwiki 500k celebration logo per bug 20207'
* 22:19 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'Configured reader feedback on strategy wiki'
* 22:10 logmsgbot: aaron synchronized php-1.5/wmf-config/CommonSettings.php 'Configured reader feedback on strategy wiki'
* 22:09 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'Configured reader feedback on strategy wiki'
* 22:00 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'Configured reader feedback & Proposal namespace for strategywiki'
* 20:50 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix wgFeedbackNamespaces'
* 20:49 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Fix wgFeedbackNamespaces'
* 20:48 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix wgFeedbackNamespaces'
* 20:46 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Activating ReaderFeedback on strategywiki'
* 20:45 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Activating ReaderFeedback on strategywiki'
* 20:33 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'rollback for enwikinews Editors'
* 19:38 Rob: running authdns-update for all the new project langcodes added just now
* 19:33 logmsgbot: robh ran sync-common-all 
* 19:06 logmsgbot: robh ran sync-common-all 
* 18:31 logmsgbot: robh ran sync-common-all 
* 16:26 aZaFred_: upgraded wordpress to 2.8.4 on techblog and blog to address newly released security vulnerability. 
* 14:38 logmsgbot: andrew synchronized php-1.5/skins/vector/main-ltr.css '-m Updated skins/vector/main-?t?.css to r54861, fixes for r53975 which broke because it was only partially merged into wmf-deployment.'
* 14:37 logmsgbot: andrew synchronized php-1.5/skins/vector/main-rtl.css '-m Updated skins/vector/main-?t?.css to r54861, fixes for r53975 which broke because it was only partially merged into wmf-deployment.'
* 13:36 logmsgbot: mark synchronized php-1.5/wmf-config/db-pmtpa.php 'Depool ms2'
* 13:33 mark: Shutting down MySQL on ms2
* 06:26 logmsgbot: midom synchronized php/cache/trusted-xff.cdb 'adding UAE proxies'
* 02:48 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'Allow blocked users to edit own talk page on es.wikibooks'
* 02:27 Aaron|notebook: Note: Confirmed group is completely add/removeable on arwiki by sysops, not just self-add/removeable; this is how it should be
* 02:19 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'made Confirmed group self-add/removeable on arwiki for sysops'
* 02:07 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'add Portal to reviewable namespaces for enwikinews'
* 01:55 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'Sync the file that actually changed'
* 01:51 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'flood flag for plwiki'
* 01:26 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'rollback for editors on plwiki'
== August 11 ==
* 23:38 atglenn: new backend cache file on sq1 for testing
* 22:52 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 'reenable EditWarning now bug is fixed'
* 22:28 atglenn: sq4 back in 
* 21:56 atglenn: take sq4 out of upload pool temporarily for thumb testing
* 20:56 AaronSchulz: Set $wgFlaggedRevTags/$wgFlaggedRevValues on dewikiqoute per bug 19179
* 20:44 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 
* 18:53 tomaszf: pulling optin_survey table from enwiki on db28 for Roan
* 18:12 domas: started rsync to storage1 from ms3, in root@storage1's screen
* 18:04 mark: Prepared storage1 for emergency copy of ms3 data
* 18:01 RobH_A90: srv122 shutdown due to broken cooling fan
* 16:30 RobH_A90: db12 back up, restarted mysql
* 16:13 mark: Shutting down MySQL on ms2
* 16:03 domas: db12 is now down for raid battery replacement (thanks cobi for noticing that)
* 16:01 RobH_A90: shutting down db12 for raid battery replacement
* 15:59 RobH_A90: shutting down mysql on db12 for raid battery replacement
* 15:58 logmsgbot: robh synchronized php-1.5/wmf-config/db-pmtpa.php 'depooling db12 to replace raid controller battery'
* 15:53 RobH_A90: sq49 shutdown and disks 3/4 replaced with ssd for mark
* 12:39 domas: powercycled srv156/srv159, seems like job memory use skyrocketed, and conflicted with apache/memcached/puppet/etc 
* 07:39 domas: db3 and db8 had same server_id, so, replication worked, just with reconnect after every event.. :) 
* 07:25 Domas: db3 and db8 had the same server ID, fixed.
* 06:59 logmsgbot: tstarling synchronized php-1.5/wmf-config/db-pmtpa.php 'repooling db8, not as broken as I thought'
* 06:41 Tim: restarting the slave thread on db3 since the error log on db13 shows an error flood due to that slave "Start client, asynchronous binlog_dump to slave_server(100236)"
* 06:27 Tim: depooled db8 due to broken replication
* 06:27 logmsgbot: tstarling synchronized php-1.5/wmf-config/db-pmtpa.php 
* 06:20 Tim: investigating replication issues on db8, tried restarting slave thread, no improvement. 
** Shows "Slave: received 0 length packet from server, apparent master shutdown"
* 05:48 Tim: disabled EditWarning due to bug 20171
* 05:47 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php
== August 10 ==
* 23:46 logmsgbot: fvassard synchronized php-1.5/wmf-config/db-pmtpa.php 'Adding db8 to commonwiki pool.'
* 23:43 logmsgbot: fvassard synchronized php-1.5/wmf-config/db-pmtpa.php 're-pooled db3'
* 23:11 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 
* 22:37 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20001 -- activate fancy edit toolbar by default on strategy wiki'
* 22:33 Andrew: Updated interwiki cache for strategy wiki interwiki prefix.
* 22:32 logmsgbot: andrew ran sync-common-all 
* 20:59 Fred: shutting down db3 for replication to db8.
* 20:58 logmsgbot: fvassard synchronized php-1.5/wmf-config/db-pmtpa.php 
* 14:37 Tim: deployed r54721
* 14:35 logmsgbot: tstarling synchronized wmf-deployment/cache/trusted-xff.cdb 
== August 9 ==
* 12:42 logmsgbot: midom synchronized php-1.5/wmf-config/db-pmtpa.php 
* 07:03 logmsgbot: midom synchronized php-1.5/wmf-config/db-pmtpa.php 
* 06:07 domas: restarted few segfaulting apaches......... ... ..... . .. 
== August 7 ==
* 23:20 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'updating sr wikinews logo for 10k event per dungodung'
* 22:09 domas: brought up db18 after some maintenance hybernation
* 22:09 domas: db8 raid toasted
* 22:01 logmsgbot: brion synchronized php-1.5/wmf-config/flaggedrevs.php 'add unreviewedpages, autoreview to coders on MWW'
* 21:54 logmsgbot: brion synchronized php-1.5/wmf-config/flaggedrevs.php 'let coders review and validate on FR pages on mediawikiwiki'
* 21:33 logmsgbot: brion synchronized php-1.5/wmf-config/flaggedrevs.php 
* 21:30 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 21:27 brion: syncing dblist (sync-dblist now fixed... i hope)
* 21:21 logmsgbot: brion synchronized php-1.5/wmf-config/flaggedrevs.php 'testing flaggedrevs limited config on MW wiki'
* 20:11 Rob: removed UsernameBlacklist from CommonSettings.
* 20:09 logmsgbot: robh synchronized php-1.5/wmf-config/CommonSettings.php 
* 19:59 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 19152 Enable flood flag on Simple English Wikiquote'
* 19:55 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '15685 Enable Extension:Nuke on all but largest wikis - cept its already on enwiki, so now its on EVERYWHERE'
* 19:16 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix _ missing from wgMetaNamespace on strategy wiki'
* 19:14 Rob: changed wording in flaggedrevs for dewikiquote since the non-english name was crashing the project.
* 19:12 logmsgbot: robh synchronized php-1.5/wmf-config/flaggedrevs.php 
* 19:09 Andrew: Started spamming board elections notices to eligible voters in a screen on zwinger.
* 19:01 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '19521 Enable the flood flag on the English Wikinews'
* 18:50 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '20034 Reconfiguration of flagged revs on en.Wikibooks'
* 18:50 logmsgbot: robh synchronized php-1.5/wmf-config/flaggedrevs.php '20034 Reconfiguration of flagged revs on en.Wikibooks'
* 18:49 brion: updated vector & usability skin per trevor's recent updates
* 18:48 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 
* 18:46 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 
* 18:22 brion: some problems with upload serving being unusually slow. we're looking into it
* 18:16 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '19813 Enable the rollback usergroup at Simple English Wikiquote'
* 18:11 logmsgbot: robh synchronized php-1.5/wmf-config/abusefilter.php 'Updates to bug 19772'
* 17:26 brion: added some left padding to header and mid-content sections on techblog so it doesn't look like crap
* 17:11 logmsgbot: robh ran sync-common-all 
* 17:11 Rob: enabled abuse filter on itwiki per bug 19772 Activate abuse filter on itwiki
* 17:04 logmsgbot: robh ran sync-common-all 
* 16:56 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '19401 please rename WikiSaurus namespace on de.wikt'
* 16:48 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20110 suppressredirect right gone missing on en.wikibooks'
* 07:32 domas: restarted puppetd (forgot yesterday) everywhere
* 07:07 domas: cleaned up/signed certs for srv101 and srv122
* 05:56 logmsgbot: midom synchronized php-1.5/wmf-config/db-pmtpa.php 'db11 load wasnt 0 for frja'
* 02:26 brion: added a news item on our old sourceforge project directing folks to mediawiki.org. Might help discourage people from discovering and installing 1.9.3 :)
* 00:24 logmsgbot: brion synchronized wmf-deployment/extensions/UsabilityInitiative/OptIn/OptIn.hooks.php 
* 00:20 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 'adding usebetatoolbar to pref stats tracking'
== August 6 ==
* 22:00 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 'enable EditWarning sitewide'
* 21:41 logmsgbot: brion synchronized php-1.5/wmf-config/InitialiseSettings.php 'enable beta ui opt-in sitewide'
* 21:21 brion: setting up tables for opt-out stats and prefstats on all wikis...
* 20:31 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling Abusefilter on fawiki per bug 19642'
* 20:28 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '19636 Enable for gdwiki'
* 20:16 Rob: added more blogs to planet per requests.
* 19:57 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '19993 Change autoconfirmed settings on plwiki'
* 19:56 Rob: rolled back autoconfirmed change to huwiki until consensus is reached.
* 19:56 logmsgbot: robh synchronized php-1.5/wmf-config/flaggedrevs.php 
* 19:52 logmsgbot: robh synchronized php-1.5/wmf-config/CommonSettings.php 'Bug 19941 Hungarian Wikinews Licence Change'
* 19:45 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '20066 Logo change for li.wiktionary'
* 19:43 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'REVERTING Bug 20045 Logo change'
* 19:42 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20045 Logo change for pih.wikipedia'
* 19:39 Rob: enabled flaggedrevs on arwiki per bug 19332, kicked over to aaronbecause the auto promote scripts no longer function.
* 19:36 logmsgbot: robh ran sync-common-all 
* 19:34 logmsgbot: robh ran sync-common-all 
* 19:18 Fred: added regex match for to all http checks on Nagios. (getting a better idea of when things are actually partially broken)
* 18:52 logmsgbot: ariel synchronized wmf-deployment/wmf-config/CommonSettings.php 'switch to ms4 for thumbs everywhere'
* 18:47 Rob: gave autoconfirmed users the same rights as confirmed on flaggedrevs in huwiki (fixing the issue of the new confirmed group overriding the old settings for huwiki.)
* 18:46 logmsgbot: robh ran sync-common-all 
* 18:41 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 19315 Set =false on Hungarian Wikipedia'
* 18:37 Rob: Enabled abusefilter with default settings on huwiki per bug 19109 Enable AbuseFilter in Hungarian Wikipedia
* 18:37 logmsgbot: robh ran sync-common-all 
* 18:18 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php '14716 Grant noratelimit right to the editor group in the Hungarian Wikipedia'
* 17:41 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 'ok now actually correct order. lucene, then titlekey'
* 16:34 brion: moving TitleKey initialization before Lucene per robert's rec (fixes lucene prefix search for enwiki)
* 16:34 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 
* 16:28 brion: reenabled TitleKey on enwiki (disabled by hidden hack in lucene config). rebuilding index
* 16:27 logmsgbot: brion synchronized php-1.5/wmf-config/lucene.php 
* 15:27 Rob: correction, flashing the raid bios, not system bios
* 15:27 logmsgbot: robh synchronized php-1.5/wmf-config/db-pmtpa.php 'db18 down for flashing bios'
* 12:53 domas: cluster is full of hanging brion's rsyncs back from Jul17 :) 
* 12:49 domas: puppetd memleaking memory in hundreds of megabytes...
* 12:04 domas: srv101 had corrupted PHP tree, was giving out 'Override this function.' message to everyone. sync-common'ed it, needs sysadmin investigation.
== August 5 ==
* 21:59 brion: prepping wmf-deployment update on test, includes updates to vector & usabilityinitiative ext
* 15:02 domas: cleaned up space on db20, filtered out db8 syslog stream
== August 4 ==
* 22:23 atglenn: testing move of thumb repo to ms4, on test.wikipedia 
* 21:48 atglenn: shut down hadoop on hume, no longer in use
* 19:32 brion: scapping wmf-deployment r54386 (adds wgMainPageTitle, needed for mobile redir)
* 19:07 logmsgbot: midom synchronized php-1.5/wmf-config/db-pmtpa.php 'disable db8'
* 15:52 Rob: updated software for blogs
* 14:15 RobH_A90: dobson drives swapped back to original for comparsion ssd testing
== August 3 ==
* 23:28 Fred: updated wikimedia-task-appserver on srv[92,110,130,131] as they were running old versions.
* 23:25 brion: provisional setup for auto-triggering of parser tests on trunk commits up. (needs init scripts etc)
* 23:11 Fred: removed db4 from mysql group since mysql is not installed on it. 
* 23:09 Fred: fixed RAID check on db2
* 22:59 Fred: added nagios (web) and ganglia monitoring to Grosley
* 22:49 Fred: reconciled the LVS config and what is in the node-list for the Apaches group. 
* 21:31 domas: addded db17 and db18 back to s3 rotation
* 21:31 logmsgbot: midom synchronized php-1.5/wmf-config/db-pmtpa.php 
* 19:36 Rob: pushed updated file for flaggedrevs, if there are issues, old file is named flaggedrevs.messy.bak in the wmf-config. If not needed within a few hours, it will be archived.
* 19:35 logmsgbot: robh synchronized php-1.5/wmf-config/flaggedrevs.php 'pushing aarons updated file'
* 16:16 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19639 import sources for vecwiki'
* 15:29 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Logo updates on bugs 19675 19898 20016'
* 15:24 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19884 dewikisource alias'
* 15:22 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 16919'
* 15:19 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 19400'
* 15:18 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 19709 rolling back'
* 15:17 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 19709'
* 08:54 logmsgbot: midom synchronized php-1.5/wmf-config/db-pmtpa.php 'enabling db26...'
== August 2 ==
* 20:01 mark: Started frontend squid on sq23
* 16:12 hcatlin: mobile1 ran out of disk b/c of log file size. Fixed now.
* 15:44 mark: Eiximenis frontend seems limited at 26 k FDs somehow, reduced its LVS load again
* 14:23 mark: Installed gmond on sq50
* 14:09 mark: Increased COSS cache dir sizes on sq50 from 15 to 35 GB, aufs from 20 to 24 GB
* 13:47 mark: Increased LVS weight on eiximenis from 45 to 60 (4x other squids)
* 13:47 mark: Increased COSS cache dir size on eiximenis from 65 GB to 128 GB
* 07:41 brion: running parsertests on trunk checkins from r54052 on (not yet auto triggered)
* 07:20 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 
* 07:20 logmsgbot: brion synchronized php-1.5/wmf-config/secret-projects.php 
* 07:07 logmsgbot: brion synchronized php-1.5/wmf-config/secret-projects.php 'add key for codereview test uploads'
* 07:04 brion: updating CodeReview extension, starting test of the testing integration :)
== August 1 ==
* 20:17 mark: Added DNS entries for <language>.m.wikipedia.org (autogenerated from langlist)
* 20:11 mark: Running apt-get upgrade on bayle
* 14:56 mark: Increased LVS weight of eiximenis from 30 to 45 (3x the load of other squids)
* 04:54 Rob: updated blogs software with more plugins for security
* 00:44 logmsgbot: kate synchronized php-1.5/wmf-config/db-pmtpa.php 'removing db30 for commons dump for TS'
== July 31 ==
* 22:46 tomaszf: running log splitter on searchidx1 to cleanup udplogger searchqueries
* 21:08 Fred: bounced squid on sq23
* 21:08 Fred: bounced squid on sq43
* 20:50 Fred: and finally srv174
* 20:49 Fred: and srv32 / srv97
* 20:48 Fred: and srv207
* 20:47 Fred: bounced apache on srv129
* 20:46 Fred: added missing srv130 to apaches nodelist and synched common on it.
* 18:46 Fred: re-synched nagios to get rid of 'false' mysql alerts
* 18:42 Fred: deployed wikimedia-base (0.20) everywhere.
* 17:40 Fred: updated wikimedia-base to include acct and enable SAR in order to do post-crash analysis when necessary. New version: 0.20. (Also changed dependency from ntp-simple to ntp as the ntp-simple package does not exist anymore)
* 15:17 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Reenabled CentralNotice'
* 15:15 logmsgbot: kate synchronized php-1.5/wmf-config/db-pmtpa.php 'remove db26 to dump s1 for TS'
* 14:15 brion: removing stray readonly message from db.php on s3 (but not on s3frja... god our config's weird)
* 14:15 logmsgbot: brion synchronized php-1.5/wmf-config/db-pmtpa.php 
* 14:14 brion: noting for toolserver repl fix: old s3 pos was db18-bin.090 454738665
* 14:10 brion: disabling general readonly
* 14:10 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 
* 14:09 brion: s3 reset to master_host='db11', master_log_file='db11-bin.001', master_log_pos=79
* 14:06 brion: setting db11 up as temp master on s3
* 14:04 mark: Restarted MySQL on db18, running recovery
* 14:01 brion: prepping a manual master switch
* 13:50 logmsgbot: brion synchronized php-1.5/wmf-config/db-pmtpa.php 'disabled down db18'
* 13:49 brion: removing down db18
* 13:47 logmsgbot: brion synchronized php-1.5/wmf-config/CommonSettings.php 
* 13:28 logmsgbot: mark synchronized php-1.5/wmf-config/db-pmtpa.php 
* 13:27 Andrew: Note that sync-file wmf-config/db-pmtpa.php does not work, you use sync-file db-pmtpa.php
* 13:27 Andrew: Updated SwitchSettings.php for new location of db.php (wmf-config/db-pmtpa.php)
* 13:26 logmsgbot: andrew synchronized php-1.5/wmf-config/db-pmtpa.php 'test'
* 12:40 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'temporarily disabled CentralNotice'
* 12:32 Rob: scheduled reboot of csw5-pmtpa took place at 8am, traffic between esams and pmtpa is very high since as traffic and squid cache misses normalize out.
== July 30 ==
* 20:26 mark: Pooled eiximenis as frontend upload squid
* 19:10 logmsgbot: brion ran sync-common-all 
* 19:10 brion: fred fixed login perms on 101, 122. running a sync-common-all to resync deployment
* 19:02 brion: srv101, srv122 serving HTTP but not taking updates via ssh
* 18:29 brion: installing texvc build & run deps on wikitech box for parser testing
* 16:07 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 20015'
* 00:12 logmsgbot: brion synchronized wmf-deployment/wmf-config/CommonSettings.php 'disable temp req logging on en'
* 00:11 logmsgbot: brion synchronized wmf-deployment/api.php 'reenable api to test'
* 00:08 logmsgbot: brion synchronized wmf-deployment/api.php 'temp disable api to test'
* 00:08 brion: connections still fill up with api disabled
* 00:07 logmsgbot: brion synchronized wmf-deployment/wmf-config/CommonSettings.php 
* 00:07 logmsgbot: brion synchronized wmf-deployment/wmf-config/InitialiseSettings.php 
* 00:04 brion: setting up some experimental req url logging on backend for enwiki issues
== July 29 ==
* 23:43 brion: temporarily bumping max connections to 6k on db16
* 23:39 brion: new boxes need to be fixed so apache-restart family of scripts work, or else the scripts replaced
* 18:17 Rob: srv131 back online
* 18:17 Rob: srv118 and srv130 back online
* 18:15 Rob: rebooted srv131 for lockup
* 18:13 Rob: rebooting srv118 & srv130, locked up
* 18:11 Rob: srv110 and srv113 back online
* 18:08 Rob: rebooted srv110, srv113, both locked up
* 16:18 river: removed stale fingerprint for ns1.wikimedia.org on bayle
* 14:01 Rob: shutting down mysql on db12, when done will shut down system and replace bad battery on controller
* 13:30 Rob: upgraded dobson memory for squid ssh testing
* 13:24 Rob: shutdown eiximenis, upgraded ram
* 04:36 Tim: running svn cleanup on locked directories as indicated by find -name lock
* 04:19 Tim: resharing ms1:/export/upload with root allowed, so that the above command actually works
* 04:15 Tim: fixing ExtensionDistributor permissions with: find -not -user extdist -exec chown extdist {} \;
* 04:11 Tim: reduced the number of svn-invoker threads on zwinger to 1, to hopefully avoid permanent "lock held" errors (bug 19889)
== July 28 ==
* 21:35 brion: updating to wmf-deployment branch r53906 - adds search API output details
* 18:06 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'setting skin to vector on strategywiki'
* 17:58 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'setting subpages and lang for strategywiki'
* 16:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#19908 - remove some redundant settings for wmgUseCollection'
* 16:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#19908 - remove some redundant settings for wmgUseCollection'
* 16:54 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#19908 - remove some redundant settings for wmgUseCollection'
* 09:40 Tim: updating SecurePoll via branch
== July 27 ==
* 23:03 atglenn: installed sudo and added george wilson (sun guy) to sudoers on ms1
* 20:41 mark: Shut down BGP transit session to AS 13680 on seemingly faulty blade 8 in csw5-pmtpa
* 18:05 atglenn: added account for george wilson (sun zks dev) on spence and ms1
* 17:52 tomaszf: adding one more worker to snapshots taking total to eight
* 17:05 mark: Shutdown BGP peer 8342 (RTComm) to see if it resolves the reachability issues from Russia
== July 26 ==
* 21:54 hcatlin_: just pushed UI tweaks to the mobile server
== July 25 ==
* 21:17 RoanKattouw: Reenabled Apache on prototype with a disallow-all robots.txt; it seems GoogleBot was DoSing us
* 21:12 RoanKattouw: Stopping Apache on prototype, because it gets stuck in OOM death or whatever it is after about 5 minutes every time
* 20:46 mark: Preffed traffic to 15169 (Google) via 13680
* 20:05 RoanKattouw: Rebooted prototype
== July 24 ==
* 21:51 hcatlin: pushed new mobile code to support multiple notices
* 20:08 brion: added $wgUseXVO = true to default config in InitialiseSettings so it'll be there when bug 19845 fix goes live
* 19:11 tomaszf: starting three xml snapshot threads
* 18:59 tomaszf: pointing storage2 to /export/dumps instead of /export/archive
* 17:06 domas: removed henbane & amaryllis from nagios
* 10:54 logmsgbot: midom synchronized php-1.5/wmf-config/../includes/specials/SpecialSearch.php 'livehack per rainmans suggestion'
* 05:31 Tim: download.wikimedia.org was totally broken, just showing a few empty directories. Changed the document root on storage2 to /export/archive/public, where there appears to be some useful files
== July 23 ==
* 22:16 mark: Decommissioned all yaseo servers, wiped their disks
* 20:35 mark: Updated the glue record for ns1.wikimedia.org
* 20:20 mark: Changed IP of ns1.wikimedia.org to 208.80.152.142 (a svc ip on linne)
* 19:58 mark: Installed linne.wikimedia.org as auth DNS server
* 19:12 logmsgbot: brion synchronized wmf-deployment/wmf-config/CommonSettings.php 'restore FancyCaptcha now that image crisis is diverted'
* 15:38 Rob: setup strategywiki for the strategy planning whatever
* 15:37 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 15:32 logmsgbot: robh ran sync-common-all 
== July 22 ==
* 21:20 Fred: installing wikimedia-task-appserver on srv122. Incoming reboot
* 15:11 Tim: fixed firewall on browne to deny RC->IRC UDP packets from outside the local network
* 09:57 logmsgbot: midom synchronized php-1.5/wmf-config/../StartProfiler.php 
* 09:51 domas: apparently someone decided that our profiling is not useful and should be disabled? :)
== July 21 ==
* 23:56 Fred: rebooted pascal (for realz this time)
* 23:15 tomaszf: fred is pulling backups from ms4 onto storage2.
* 23:07 Fred: rebooting pascal as he fell over again
* 22:45 tomaszf: adding snapshot1,2,3 to DHCP
* 22:03 mark: Increased large object cache dir size to 120 GB on eiximenis
* 18:28 domas: srv122 booted into netinstall, apparently
* 17:39 Rob: updated both blog and techblog to newest stable release of wordpress
* 16:36 brion: internal UDP logging broken since 17 July; looks like udp2log isn't running on db20 since reboot?
* 16:21 logmsgbot: robh synchronized php-1.5/wmf-config/CommonSettings.php 'death to captcha'
* 16:15 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'death to the uplaoder group'
* 16:06 logmsgbot: robh synchronized php-1.5/wmf-config/CommonSettings.php 
* 16:00 Rob: updated sync-fiel script for new file locations
* 16:00 logmsgbot: robh synchronized php-1.5/wmf-config/CommonSettings.php 
* 15:50 Rob: removing old whygive blog data from dns and archiving the database.
* 15:07 Rob: updated planet with http://meta.wikimedia.org/wiki/Planet_Wikimedia#Requests_for_inclusion
* 14:43 mark: Increased COSS cache dirs on pmtpa upload squids
* 11:30 domas: for i in $(ssh db20 findevilapaches); do ssh $i invoke-rc.d apache2 restart; done \o/
* 11:29 domas: killed brion's sync processes on zwinger, hanging since July17 :) 
* 09:15 domas: mgmt-restarted srv156
== July 20 ==
* 22:18 mark: Rebooted pascal
* 21:15 mark: Doubled cache dir sizes on eiximenis, upped carp load from 20 to 30
* 18:09 hcatlin: restarted mobile1 cluster to load in new software
* 15:55 Fred: bounced apache on srv193
* 07:18 Tim: re-enabled CentralNotice
* 07:17 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 07:13 apergos: enough data removed from ms1 to feel safe for a few days; started mass copy of remaining thumbs to ms4 in prep for complete repo switchover (running in root screen on ms1)
* 04:48 Tim: copying up all available MW release files from my laptop
* 04:21 Tim: mounted ms4:/export/dumps on zwinger
* 04:16 Tim: changed export options for ms4:/export/dumps to allow root access for the local subnet
* 01:19 hcatlin: On mobile1 we are now gzipping the log files after rotation in /srv/wikimedia-mobile/logs
== July 19 ==
* 22:02 Fred: restarted memcached on srv159
* 20:43 mark: eiximenis backend squid pooled
* 20:10 mark: Restarted deadlocked powerdns on bayle
* 19:14 mark: Installed eiximenis with a Squid OS install
* 18:58 mark: Moved eiximenis to vlan 100 (squids)
* 18:55 mark: Changed eiximenis' IP into 208.80.152.119 for Squid testing
* 17:41 hcatlin: Mobile1's web stack just got switched from Phusion Passenger to Nginx/Thin/Rack.
== July 18 ==
* 15:23 apergos: some thumb directories on ms4 created at request of img scalers were created with owner root and perms 700... fixing
* 03:55 river: ms5 is ready
* 01:20 atglenn: continuing with removals of thumbs on ms1. 789G free now, need to reach about 1450 before we can just "maintain". but we're gaining on it.
* 00:22 brion: set up temporary data dump index, copied the dvd index (it's just offsite links). still need to track some MW releases
* 00:07 brion: recovering MediaWiki 1.6 through 1.10 release files and re-uploading them...
== July 17 ==
* 23:42 brion: added a 404 page and recovered index.php for our temp download.wikimedia.org
* 22:05 brion: set wikitech to use vector skin by default :D
* 22:03 Andrew: Fixed morebots, which was relying on a fragile version check. Just deleted it :)
* 20:43 brion: fixed paths for noc.wikimedia.org/conf file highlighting
* 20:38 domas: ms2 has broken disks..
* 20:31 brion: We're going to see about setting up the previously-idle ms5 so we can get our thumbnailing on
* 20:01 brion: rob's poking raid rebuild on storage2 (dumps server)
* 19:03 RobH_A90: eiximenis and dobson pulled for solid state drive testing, do not use for other tasks
* 18:28 logmsgbot: brion synchronized wmf-deployment/wmf-config/InitialiseSettings.php 'enabling vector for rtl'
* 18:25 atglenn: started mass move out of the way of thumbnail dirs and replacing with symlinks to ms4 
* 18:25 logmsgbot: brion synchronized wmf-deployment/wmf-config/CommonSettings.php 'bump style version'
* 18:25 logmsgbot: brion ran sync-common-all 
* 18:24 brion: running sync-common-all for UI updates. need to poke the style ver too :)
* 18:08 brion: svn up'ing wmf-deployment for test.wikipedia.org. Merged UI fixes from usability team
* 18:03 Fred: spun a couple more apache server into image scalers: srv219..srv224.
* 17:28 rainman-sr: putting new location of initialisesettings to lsearch-global-2.1.conf so the incremental updater works again
* 17:20 Fred: srv224 is now an image_scaler. Adjusted on lvs3, ganglia and dsh's node_list. 
* 17:14 Fred: db20 back online
* 16:50 Fred: rebooting db20 as it is in a "state"
* 16:45 brion: looks like we've lost internal /home NFS, which makes some of our internal services very unhappy. investigating...
* 16:43 brion: ganglia out.
* 13:44 apergos1: doing next round of removals on ms1 (/export/upload/wikipedia/en/thumb/2) to keep ahead of the game
* 04:15 apergos: starting removal of /export/upload/wikipedia/en/thumb/1 on ms1 (moved away and symlink to ms4 done already) for more space
* 03:54 logmsgbot: brion synchronized wmf-deployment/wmf-config/CommonSettings.php 'Disabling sitenotice from maintenance'
* 03:29 brion: reenabling uploads & image deletion/undeletion
* 03:29 logmsgbot: brion synchronized wmf-deployment/wmf-config/CommonSettings.php 
* 03:28 brion: remounting ms1 on apaches
* 00:49 atglenn: only about 1gb gain on each so doing all of /export/upload/wikipedia/en/thumb/0
* 00:39 atglenn: removing more directories in /export/upload/wikipedia/en/thumb/0 on ms1 and replacing with symlinks to ms4
* 00:30 logmsgbot: brion synchronized wmf-deployment/includes/specials/SpecialUpload.php 
* 00:30 logmsgbot: brion synchronized wmf-deployment/includes/ImagePage.php 
* 00:27 logmsgbot: brion synchronized wmf-deployment/wmf-config/CommonSettings.php 
* 00:20 brion: temporarily disabling image delete/rename during maintenance
== July 16 ==
* 23:56 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Disabling uploads and setting captcha to not-fancy.'
* 23:29 atglenn: removing the images in /export/upload/wikipedia/en/thumb/0/00 on ms1 (real dir is a symlink to ms4) to get back some space
* 22:51 atglenn: sym link back in place, let's see what happens
* 22:47 atglenn: reverting temporarily while we resolve mount issues for the ms4 share
* 22:40 atglenn: ...whether the image scalers will fall over if we force them to do (some) regeneration. 
* 22:37 atglenn: on ms1, /export/upload/wikipedia/en/thumb/0/00 symlinked to (shared from ms4) /mnt/thumbs/wikipedia/en/thumb/0/00 to test
* 21:29 brion: robots.php for robots.txt generation now also working. yay!
* 21:28 logmsgbot: brion synchronized live-1.5/robots.php 
* 21:28 brion: extract2.php now fixed up for new deployment; portal pages ok (www.wikipedia.org)
* 21:27 logmsgbot: brion synchronized live-1.5/robots.php 
* 21:26 logmsgbot: brion synchronized extract2.php 
* 21:26 logmsgbot: brion synchronized extract2.php 
* 21:22 logmsgbot: brion synchronized extract2.php 
* 21:18 logmsgbot: brion ran sync-common-all 
* 21:18 brion: rsync messed up the php-1.5 directory to symlink translation. retrying as root
* 21:14 logmsgbot: brion synchronized extract2.php 
* 21:13 logmsgbot: brion synchronized extract2.php 
* 21:13 atglenn: started copy of thumbnails to ms4, symlinks going in on ms1 (but no data removal yet)
* 21:11 logmsgbot: brion synchronized live-1.5/extract2.php 
* 21:10 logmsgbot: brion synchronized live-1.5/robots.php 
* 21:09 logmsgbot: brion ran sync-common-all 
* 21:08 brion: attempting to replace the old php-1.5 dir with wmf-deployment symlink
* 21:02 logmsgbot: brion synchronized wmf-deployment/wmf-config/InitialiseSettings.php 'I think touching the new master InitialiseSettings will fix it'
* 21:01 logmsgbot: brion synchronized wmf-deployment/includes/GlobalFunctions.php 'mkdir error trackdown hack'
* 20:54 logmsgbot: brion synchronized wmf-deployment/wmf-config/missing.php 
* 20:52 logmsgbot: brion synchronized wmf-deployment/wmf-config/CommonSettings.php 
* 20:52 logmsgbot: brion synchronized wmf-deployment/wmf-config/reporting-setup.php 
* 20:48 brion: switching all sites to wmf-deployment branch
* 20:48 logmsgbot: brion synchronized live-1.5/MWVersion.php 
* 19:06 Tim: copying ExtensionDistributor stuff to ms4:/export/ext-dist, from root screen on ms1
* 19:01 brion: Now running test.wikipedia.org, www.mediawiki.org, and meta.wikimedia.org on new deployment checkout
* 19:01 logmsgbot: brion synchronized live-1.5/MWVersion.php 
* 18:58 logmsgbot: brion ran sync-common-all 
* 18:39 Tim: restarted xinetd on zwinger
* 18:24 logmsgbot: tstarling synchronized php-1.5/CommonSettings.php 
* 17:57 brion: also restarted 186, 196 which had some funkiness in php err log
* 17:56 brion: srv186 also bad sudo
* 17:55 brion: srv171 has some borkage; sudo config is broken can't run apache-restart as user
* 17:52 logmsgbot: brion ran sync-common-all 
* 17:51 brion: running updated sync-common-all friendly to non-NFS boxes
* 17:49 brion: swapped private SVN-managed /home/wikipedia/bin into place
* 15:09 apergos: removing the last of our snapshots on ms1 :-( getting us a little more space
* 14:47 apergos: disabled snapshots on ms1 in preparation for move of thumbnails to ms4
* 14:38 brion: updated wikibugs-l list config to allow bugzilla-daemon@wikimedia.org to post
* 14:34 brion: restarted wikibugs bot
* 14:27 brion: ms1 performance seems to be sucking again
* 14:17 logmsgbot: brion synchronized php-1.5/InitialiseSettings.php 'adjusting throttle temporarily for outreach event'
* 11:55 RoanKattouw: ExtensionDistributor repeatedly reported broken in the past 48 hrs
* 07:08 Fred: traffic profile switched back to normal. Esams is back to normal.
* 06:11 hcatlin: Mobile1 has returned to normal function.
* 05:58 hcatlin: Error after restarting mobile1 stopped stats logging from working. Stats will be low for July 15th and higher for July 16th. Parsing of the 6 hour log file (about 1GB) might slow server for next few minutes until caught up.
* 04:24 Rob: outage for esams servers started at approx 3:20 gmt
* 04:15 Rob: still waiting on esams to update us about the rack(s), moving traffic to pmtpa
* 00:59 tomaszf: started backup for latest xml snapshots from storage2 to ms4
== July 15 ==
* 22:30 Rob: updated dns for new snapshot servers becasue tomasz did not want to be in charge of dump servers.
* 22:10 brion: brion checking around for 0-byte files (not thumbs) to see if we can recover
* 21:33 atglenn: verified that zfs patch is in place on ms4 (it got sucked in during river's update yesterday)
* 21:26 logmsgbot: brion synchronized php-1.5/CommonSettings.php 'Restore fancy captcha mode'
* 21:16 logmsgbot: I_am_not_root synchronized php-1.5/CommonSettings.php 're-enabling Uploads and removing site notice.'
* 21:01 atglenn: rebooting ms1 after applying zfs patch. *cross fingers*
* 20:51 logmsgbot: brion synchronized php-1.5/CommonSettings.php 
* 20:51 logmsgbot: brion synchronized php-1.5/InitialiseSettings.php 
* 20:42 brion: reenabled captcha in simple mode (no images; math q)
* 20:37 brion: captcha system broken while images are offline, need to disable it temporarily
* 20:18 brion: updated http://en.wikipedia.org/wiki/MediaWiki:Uploaddisabledtext & http://commons.wikimedia.org/wiki/MediaWiki:Uploaddisabledtext
* 19:43 logmsgbot: fvassard synchronized php-1.5/CommonSettings.php 'Disabling Uploads while ms1 gets fixed (again with an s after upload).'
* 19:40 logmsgbot: fvassard synchronized php-1.5/CommonSettings.php 'Disabling Uploads while ms1 gets fixed.'
* 19:40 atglenn: bringing solaris up to current patch level on ms1
* 19:34 brion: Ok, we're going to temporarily shut off uploading and unmount the uploads dir while we muck about with ms1.
* 19:14 brion: dropping export/upload@daily-2009-07-11_03:10:00
* 19:08 brion: restarting web server on ms1, see if that resets some connections to the backend scalers
* 19:05 brion: restarting nfsd on ms1
* 18:58 brion: dropping zfs snapshot export/upload@daily-2009-07-09_03:10:00
* 18:25 RobH_A90: drac and physical setup done for dump1,2,3, will install remotely
* 17:52 RobH_A90: updated dns for new dump processing servers public and management ips
* 17:41 Fred: bounced apache on srv45
* 17:37 Fred: bounced apache on srv47
* 17:09 RobH_A90: pdf1 is not coming back, working on it
* 16:56 RobH_A90: shutting down pdf1 and mobile1 to move their power too, weee
* 16:55 RobH_A90: shutting down spence to move
* 16:50 RobH_A90: shutting down singer to move its power, blogs and other associated services will be offline for approx. 5 minutes
* 16:47 Andrew: Restarting apache on prototype
* 16:46 RobH_A90: shutting down grosley for power move
* 16:45 RobH_A90: all these power moves are to add the new dump processing servers to the rack
* 16:45 RobH_A90: shutting down fenari for power move
* 16:43 RobH_A90: shut down eiximenis and erzurumi to move their power
* 16:34 RobH_A90: shutting down some servers and moving power around in a4-sdtpa
* 16:17 Andrew: Changed morebots to tell you through a channel message instead of a private notice when the logging is successful.
* 15:54 Fred: kernel updated on wikitech from 2.6.18.8 to 2.6.29 (latest available on linode)
* 15:49 Andrew: Fixed auto-submission of honeypot data, was broken because it needed my perl include path.
* 15:40 Fred: rebooting wikitech to install new kernel
* 14:04 Ariel: stopped apaches on image scalers, stopped nfs on ms1, restarting nfs and apaches...
* 13:52 Ariel: removing more snapshots on ms1 (lockstat showed it hung up in metaslab_alloc again)
== July 14 ==
* 23:15 logmsgbot: brion synchronized php-1.5/InitialiseSettings.php 'fix fix to enwiki confirmed gruop :D'
* 22:24 logmsgbot: brion synchronized php-1.5/InitialiseSettings.php 'fix to confirmed group for en'
* 20:44 logmsgbot: brion synchronized wmf-deployment/cache/trusted-xff.cdb 
* 20:41 logmsgbot: brion synchronized wmf-deployment/cache/trusted-xff.cdb 
* 20:40 logmsgbot: brion synchronized wmf-deployment/AdminSettings.php 
* 20:22 Fred: restarted a bunch of dead apaches
* 20:10 brion: doing a sync-common-all w/ attempt to put test.wikipedia on wmf-deployment branch
* 19:50 logmsgbot: robh synchronized php-1.5/InitialiseSettings.php 
* 19:11 logmsgbot: robh synchronized php-1.5/InitialiseSettings.php 'bug 19611 forgot one thing'
* 19:09 logmsgbot: robh synchronized php-1.5/InitialiseSettings.php 'bug 19611'
* 19:08 logmsgbot: robh synchronized php-1.5/InitialiseSettings.php 'bug 19611'
* 14:16 domas: dropped all june snapshots on ms1, thus providing some relief 
* 01:52 river: patched ms4 in preperation for upload copy
== July 13 ==
* 21:31 Rob: pushing dns update to fix management ips for new apaches
* 19:05 Fred: added storage3 to ganglia monitor.
* 18:50 logmsgbot: brion synchronized php-1.5/abusefilter.php 'Disable dewiki missingsummary, mysteriously in abusefilter section. Per bug 19208'
* 16:30 Fred: install wikimedia-nis-client to srv66 and mounted /home.
* 16:28 logmsgbot: brion synchronized php-1.5/InitialiseSettings.php 'fixing wikispecies RC-IRC prefix to species.wikimedia'
* 16:27 brion: test wiki was apparently moved from dead srv35 to srv66, which has new NFS-less config. thus fail since test runs from nfs
* 16:24 brion: test wiki borked; reported down for several days now :) investigating
* 15:12 logmsgbot: midom synchronized php-1.5/db.php 'db26 raid issues'
* 14:55 logmsgbot: midom synchronized php-1.5/db.php 'db3 and db5 coming live as commons servers'
* 14:13 domas: dropped few more snapshots, as %sys was increasing on ms1...
* 11:16 domas: manually restarted plethora of failing apaches (direct segfaults and other possible APC corruptions, leading to php OOM errors)
* 09:50 logmsgbot: tstarling synchronized php-1.5/includes/specials/SpecialBlockip.php 
* 09:00 Tim: restarted apache2 on image scalers
* 08:39 logmsgbot: tstarling synchronized php-1.5/includes/Math.php 'statless render hack'
* 08:05 Tim: killed all image scalers to see if that helps with ms1 load
* 08:00 Tim: killed waiting apache processes
* 07:35 logmsgbot: midom synchronized php-1.5/mc-pmtpa.php 
* 07:24 logmsgbot: midom synchronized php-1.5/mc-pmtpa.php 'swapping out srv81'
* 04:11 Tim: fixed /opt/local/bin/zfs-replicate on ms1 to write the snapshot number before starting replication, to avoid permanent error "dataset already exists" after failure
* 02:16 brion: -> https://bugzilla.wikimedia.org/show_bug.cgi?id=19683
* 02:12 brion: sync-common script doesn't work on nfs-free apaches; language lists etc not being updated. Deployment scripts need to be fixed?
* 02:03 brion: srv159 is absurdly loaded/lagged wtf? 
* 01:58 brion: reports of servers with old config, seeing "doesn't exist" for new mhr.wikipedia. checking...
* 01:16 brion: so far so good; CPU graphs on image scalers and ms1 look clean, and I can purge thumbs on commons ok
* 01:10 brion: trying switching image scalers back in for a few, see if they go right back to old pattern or not
* 01:03 brion: load on ms1 has fallen hugely; outgoing network is way up. looks like we're serving out http images fine... of course scaling's dead :P
* 00:59 brion: stopping apache on image scaler boxes, see what that does
* 00:49 brion: attempting to replicate domas's earlier temp success dropping oldest snapshot (last was 4/13): zfs destroy export/upload@weekly-2009-04-20_03:30:00
* 00:45 brion: restarting nfs server
* 00:44 brion: stopping nfs server, restarting web server
* 00:40 brion: restarting nfs server on ms1
* 00:36 brion: doesn't seem so far to have changed the NFS access delays on image scalers.
* 00:31 brion: shutting down webserver7 on ms1
* 00:23 brion: investigating site problem reports. image server stack seems overloaded, so intermittent timeouts on nfs to apaches or http/squid to outside
== July 12 ==
* 20:30 domas: dropped few snapshots on ms1, observed sharp %sys decrease and much better nfs properties immediately 
* 20:05 domas: we seem to be hitting issue similar to http://www.opensolaris.org/jive/thread.jspa?messageID=64379 on ms1
* 18:55 domas: zil_disable=1 on ms1
* 18:34 mark: Upgraded pybal on lvs3
* 18:16 mark: Hacked in configurable timeout support for the ProxyFetch monitor of PyBal, set the renderers timeout at 60s
* 17:58 domas: scaler stampedes caused scalers to be depooled by pybal, thus directing stampede to other server in round-robin fashion, all blocking and consuming ms1 SJSWS slots. of course, high I/O load contributed to this.
* 17:55 domas: investigating LVS-based rolling scaler overload issue, Mark and Tim heading the effort now ;-) 
* 17:54 domas: bumped up ms1 SJSWS thread count
* 11:00 domas: hehehehehe, disabled peer verification on zwinger for now:
 Issuer: C=US, ST=Florida, L=Tampa, O=Wikimedia Foundation Inc., OU=Operations, CN=srv1.pmtpa.wmnet
 Validity
 Not Before: Jul 8 08:03:52 2006 GMT
 Not After : Jul 12 08:03:52 2009 GMT
* 08:43 tomaszf: rebooted wikitech due to out of memory
*Jul 12 14:17:32 <TimStarling>	!log reduced MaxClients on wikitech.wikimedia.org from 150 to 5
*Jul 12 14:06:33 <domas>	!log srv1 certificate expired
*Jul 12 11:31:58 <tomaszf>	!log rebooted wikitech due to out of memory
*Jul 12 11:07:58 <tomaszf>	!log rebooting wikitech
*Jul 12 08:41:30 <logmsgbot>	!log tstarling synchronized php-1.5/InitialiseSettings.php 
*Jul 12 08:40:31 <logmsgbot>	!log tstarling synchronized php-1.5/includes/ImagePage.php 
*Jul 12 08:40:15 <logmsgbot>	!log tstarling synchronized php-1.5/includes/DefaultSettings.php 
*Jul 12 08:39:55 <TimStarling>	!log merging and deploying r53130, will disable archive thumbnails and see if it has an impact on ms1 load
*Jul 12 00:31:07 <logmsgbot>	!log midom synchronized php-1.5/db.php 
*Jul 11 22:17:15 <logmsgbot>	!log andrew synchronized php-1.5/InitialiseSettings.php 
*Jul 11 22:15:46 <werdna>	!log Still very slow, going to disable CentralNotice again
*Jul 11 22:07:30 <RoanKattouw>	!log wikitech.wikimedia.org is down
*Jul 11 20:40:26 <logmsgbot>	!log tstarling synchronized php-1.5/InitialiseSettings.php 're-enabling CentralNotice'
*Jul 11 19:32:06 <TimStarling>	!log killed waiting processes again
*Jul 11 19:24:11 <TimStarling>	!log killed all processes in the rpc_wait state, to buy us some time
*Jul 11 19:12:06 <mark>	!log Reverted cache_mem reduction on upload squids; the cause of memory pressure is a memleak
*Jul 11 19:07:47 <TimStarling>	!log apaches took a while to restart due to some shell processes hanging on to listening *:80 filehandles while waiting for NFS, *should be fixed now
*Jul 11 19:03:02 <mark>	!log Restarting memory leaking frontend squids in upload pmtpa cluster
*Jul 11 18:57:48 <TimStarling>	!log restarting apaches
*Jul 11 18:56:15 <mark>	!log Reduced cache_mem from 3000 to 2000 MB on pmtpa upload cache squids
== July 11 ==
* 15:45 mark: Rebooting sq1
* 15:31 Tim: rebooting ms1
* 14:54 Tim: disabled CentralNotice temporarily
* 14:54 logmsgbot: tstarling synchronized php-1.5/InitialiseSettings.php 'disabling CentralNotice'
* 14:53 logmsgbot: tstarling synchronized php-1.5/InitialiseSettings.php 'disabling CentralAuth'
* 14:36 Tim: restarted webserver7 on ms1
* 14:22 Tim: some kind of overload, seems to be image related
* 10:09 logmsgbot: midom synchronized php-1.5/db.php 'db8 doing commons read load, full write though'
* 09:22 domas: restarted job queue with externallinks purging code, <3 
* 09:22 domas: installed nrpe on db2 :) 
* 09:22 logmsgbot: midom synchronized php-1.5/db.php 'giving db24 just negligible load for now'
* 08:38 logmsgbot: midom synchronized php-1.5/includes/parser/ParserOutput.php 'livemerging r53103:53105'
* 08:37 logmsgbot: midom synchronized php-1.5/includes/DefaultSettings.php 
== July 10 ==
* 21:21 Fred: added ganglia to db20
* 19:58 logmsgbot: azafred synchronized php-1.5/CommonSettings.php 'removed border=0 from wgCopyrightIcon'
* 18:58 Fred: synched nagios config to reflect cleanup. 
* 18:52 Fred: cleaned up the node_files for dsh and removed all decommissioned hosts.
* 18:36 mark: Added DNS entries for srv251-500
* 18:18 logmsgbot: fvassard synchronized php-1.5/mc-pmtpa.php 'Added a couple spare memcache hosts.'
* 18:16 RobH_DC: moved test to srv66 instead.
* 18:08 RobH_DC: turning srv210 into test.wikipedia.org
* 17:56 Andrew: Reactivating UsabilityInitiative globally, too.
* 17:55 Andrew: Scapping, back-out diff is in /home/andrew/usability-diff
* 17:43 Andrew: Apply r52926, r52930, and update Resources and EditToolbar/images
* 16:44 Fred: reinstalled and configured gmond on storage1.
* 15:08 Rob: upgraded blog and techblog to wordpress 2.8.1
* 13:58 logmsgbot: midom synchronized php-1.5/includes/api/ApiQueryCategoryMembers.php 'hello, fix\!'
* 12:40 Tim: prototype.wikimedia.org is in OOM death, nagios reports down 3 hours, still responsive on shell so I will try a light touch
* 11:07 logmsgbot: tstarling synchronized php-1.5/mc-pmtpa.php 'more'
* 10:58 Tim: installed memcached on srv200-srv209
* 10:51 logmsgbot: tstarling synchronized php-1.5/mc-pmtpa.php 'deployed the 11 available spares, will make some more'
* 10:48 Tim: mctest.php reports 17 servers down out of 78, most from the range that Rob decommissioned
* 10:37 Tim: installed memcached on srv120, srv121, srv122, srv123
* 10:32 Tim: found rogue server srv101, missing puppet configuration and so skipping syncs. Uninstalled apache on it.
== July 9 ==
* 23:56 RoanKattouw: Rebooted prototype around 16:30, got stuck around 15:30
* 21:43 Rob: srv35 (test.wikipedia.org) is not posting, i think its dead jim.
* 21:35 Rob: decommissioned srv55 and put srv35 in its place in C4, test.wikipedia.org should be back online shortly
* 20:04 Rob: removed decommissioned servers from node groups, getting error on syncing up nagios.
* 20:03 Rob: updated dns for new apache servers
* 19:54 Rob: decommissioned all old apaches in rack pmtpa b2
* 16:22 Tim: creating mhrwiki (bug 19515)
* 13:27 domas: db13 controller battery failed, s2 needs master switch eventually
== July 8 ==
* 15:48 domas: frontend.conf changes: fixed cache-control headers for /w/extensions/ assets, did some RE optimizations %)
* 13:31 logmsgbot: midom synchronized php-1.5/InitialiseSettings.php 'disabling usability initiative on all wikis, except test and usability. someone who enabled this and left at this state should be shot'
== July 7 ==
* 19:06 Fred: adjusted www.wikipedia.org apache conf file to remove a redirect-loop to www.wikibooks.org. (bug #19460)
* 17:34 Fred: found the cause of Ganglia issues: Puppet. Seems like the configuration of the master hosts gets reverted to being deaf automagically... 
* 17:05 Fred: ganglia fixed. For some reason the master cluster nodes were set to Deaf mode... (ie the aggregator couldn't gather data from them).
* 15:02 logmsgbot: robh synchronized php-1.5/InitialiseSettings.php '19470 Rollback on pt.wikipedia'
* 03:37 Fred: fixing ganglia. Expect disruption
* 00:27 tomaszf: starting six worker threads for xml snapshots
* 00:12 Fred: srv142 and srv55 will need manual power-cycle. 
* 00:10 Fred: Rolling reboot has finally completed. 
== July 6 ==
* 23:57 Fred: restarted ganglia since it is acting up... 
* 23:54 tomaszf: restarting all xml snapshots due to kernel upgrades
* 18:49 Rob: upgraded spam detection plugins on blog and techblog
* 18:47 Fred: starting rolling reboot of servers in Apaches cluster. 
* 17:53 tomaszf: cleaning out space on storage2. lowering retention for xml snapshots to 10
* 17:53 Fred: upgrading kernel on cluster. This will take a while! 
* 17:46 Fred: rebooting srv220 to test kernel update.
== July 3 ==
* 12:51 logmsgbot: andrew synchronized php-1.5/extensions/AbuseFilter/Views/AbuseFilterViewEdit.php 'Re-activating abuse filter public logging in the logging table now that log_type and log_action have been expanded.'
* 11:45 mark: Kicked iris so it would boot
* 10:11 logmsgbot: andrew synchronized php-1.5/skins/common/htmlform.js 'IE7 fixes for new preference system
* 05:51 Tim: restarted squid instances on sq28
* 05:47 Tim: restarted squid instances on sq2
* 05:46 Tim: started squid backend on sq10 and sq23, sq24, sq31, restarted frontend on most of those to reduce memory usage
* 05:35 Tim: restarted squid backend on sq16, was reporting "gateway timeout" apparently for all requests. Seemed to fix it. Will try that for a few more that nagios is complaining about.
== July 2 ==
* 21:38 Rob: sq24 wont accept ssh, depooling.
* 21:34 Rob: rebooting sq21
* 21:26 Rob: ran changes to push dns back to normal scenario
* 19:52 mark: Power outage at esams, moving traffic
* 19:44 Andrew: Knams down, Rob is looking into it
* 19:41 Andrew: Reports of problems from Europe
* 19:25 Andrew: running sync-common-all to deploy mobileRedirect.php to fix hcatlin's mobile redirect/cookie bug
* 19:22 logmsgbot: andrew synchronized live-1.5/mobileRedirect.php 
* 17:15 mark: Rebooted srv159
* 16:13 Fred: shutting 217 back down as it is not supposed to be up due to faulty timer causing issues.
* 16:12 Fred: rebooted srv217. Was unpingable. 
* 14:09 Andrew: Started sending updates of spam.log to Project Honeypot folks every 5 minutes, in my crontab on hume.
* 11:20 logmsgbot: andrew synchronized php-1.5/skins/common/shared.css 'Live-merging r52669, r52684 at rainman's request, search fixes.'
* 11:18 logmsgbot: andrew synchronized php-1.5/includes/specials/SpecialSearch.php 'Live-merging r52669, r52684 at rainman's request, search fixes.'
* 00:03 logmsgbot: brion synchronized php-1.5/CommonSettings.php 
* 00:02 logmsgbot: brion synchronized php-1.5/extensions/MWSearch/MWSearch_body.php 'de-merge broken r52664'
== July 1 ==
* 23:40 brion: poking in tweaks to search and updates to vector
* 23:22 logmsgbot: brion synchronized php-1.5/CommonSettings.php 'bump wgStyleVersion'
* 23:21 logmsgbot: brion synchronized php-1.5/skins/vector/main-rtl.css 
* 23:21 logmsgbot: brion synchronized php-1.5/skins/vector/main-ltr.css 
* 23:10 logmsgbot: brion synchronized php-1.5/InitialiseSettings.php 'set vector skin, new toolbar on for usability wiki'
* 23:07 mark: Kicked pascal
* 23:05 logmsgbot: brion synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.php 'bumping the js ver no'
* 23:01 logmsgbot: brion synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.js 
* 22:59 logmsgbot: brion synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'to 52659'
* 22:57 logmsgbot: brion synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.i18n.php 
* 22:44 logmsgbot: brion synchronized php-1.5/InitialiseSettings.php 'enabling UsabilityInitiative (for optional EditToolbar)'
* 22:43 logmsgbot: brion synchronized php-1.5/CommonSettings.php 'disabling EditWarning pending addl talk'
* 22:40 brion-codereview: updating UsabilityInitiative ext to r52657 in prep for enabling new toolbar option
* 22:10 logmsgbot: brion synchronized php-1.5/InitialiseSettings.php 'Enabling new search UI formatting sitewide'
* 22:02 logmsgbot: brion synchronized php-1.5/InitialiseSettings.php 'fixing the RTL disable for vector'
* 21:58 logmsgbot: brion synchronized php-1.5/InitialiseSettings.php 'Vector should now be available in prefs for non-RTL sites'
* 21:57 logmsgbot: brion synchronized php-1.5/CommonSettings.php 'vector config tweak'
* 21:42 brion-codereview: updating Vector to current
* 19:38 logmsgbot: midom synchronized php-1.5/db.php 
* 16:13 Fred: bayes is running out of memory on a regular basis. Enabled process accounting / sar to gather more data.
* 15:48 Fred: rebooting Bayes as it locked up again. 
* 11:48 logmsgbot: tstarling synchronized php-1.5/InitialiseSettings.php 'trying a lower value for $wgMaxMsgCacheEntrySize'
* 11:19 domas: cleaned up srv100
* 11:18 domas: noticed that imagemagick tempfiles are currently created in /u/l/a/c-l/p/ :)
* 09:24 domas: pinned mysqlds on half of cores on 8-core boxes: for i in {11..30}; do ssh db$i 'taskset -pc 0-15:2 $(pidof mysqld)' ; done