Server Admin Log/Archive 19

== December 31 ==
* 02:01 logmsgbot: LocalisationUpdate completed (1.18) at Sat Dec 31 02:05:00 UTC 2011
== December 30 ==
* 19:36 LeslieCarr: added 31133 to as path list "AVOID-PATHS" and soft reset AMS-IX route server peerings
* 14:47 Jeff_Green: unbroke replication on fundraisingdb's by skipping a bad query on the slaves
* 02:01 logmsgbot: LocalisationUpdate completed (1.18) at Fri Dec 30 02:05:02 UTC 2011
* 01:00 LeslieCarr: restarted puppetd on ms1002
== December 29 ==
* 18:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 33407 33408 Babel stuff for iswiki'
* 15:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33410 - Allow IP edits on wikimania2012.wikimedia.org'
* 08:29 apergos: live hack on db31: added nagios ALL=NOPASSWD: /usr/bin/arcconf getconfig 1 to sudoers to shut up security warnings, this is a bad approach though
* 08:27 apergos: live hack on hume: added ". /etc/apache2/envvars.default" to /etc/apache2/envars on hume, w/o it apache wouldn't start, which killed static.wp.o
* 07:20 apergos: from 1 hour and 20 mins ago: Ryan_Lane: virt4 is back up. br103 had a bad MAC of 00...
* 05:28 Ryan_Lane: rebooting virt4 again
* 05:27 Ryan_Lane: rebooting all migrated instances
* 05:11 Ryan_Lane: virt4 has some networking issue
* 05:11 Ryan_Lane: migrating all instances off of virt4 onto virt3 and virt2
* 04:29 Ryan_Lane: restarting virt4 because upstart is an unreliable POS
* 02:53 Ryan_Lane: enabled instance live migration on virt2-4
* 02:01 logmsgbot: LocalisationUpdate completed (1.18) at Thu Dec 29 02:05:01 UTC 2011
== December 28 ==
* 23:00 binasher: db31 wasn't puppetized, fixed
* 22:49 logmsgbot: catrope synchronized php-1.18/extensions/MoodBar/ 'updating to trunk state'
* 22:49 logmsgbot: catrope synchronized php-1.18/extensions/MarkAsHelpful/ 'updating to trunk state'
* 22:26 logmsgbot: catrope synchronized php-1.18/extensions/MoodBar 'Forgot to sync this earlier'
* 21:25 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable MarkAsHelpful on enwiki'
* 20:08 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add wmgUseMarkAsHelpful trigger code'
* 19:59 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add wmgUseMarkAsHelpful, default false'
* 19:56 RoanKattouw: Creating DB table for MarkAsHelpful on all wikis
* 19:49 logmsgbot: catrope synchronizing Wikimedia installation... : Pushing out MarkAsHelpful code
* 17:31 Jeff_Green: srv191 shutdown, looks like failed hdd, see RT #2193
* 17:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 33319 bug 33320'
* 16:53 Reedy: srv191 complaining of / being readonly
* 16:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 33319 bug 33320'
* 16:02 Jeff_Green: hume:/usr/local/bin/copy_impression_logs_from_storage3.pl deprecated, t'was an emergency hack deployed when storage3 dropped a disk
* 14:51 mutante: fixed mobile traffic logger checks - what they report is for real now. 2 procs on first server, 4 procs on the other three
* 07:33 apergos: live-hacked /usr/local/bin/copy_impression_logs_from_storage3.pl on hume, it was rsyncing everything into /a/static/uncompressed/2... do we need this job? there is also /usr/local/bin/offhost_backups on storage3 that seems to copy to the same dir, can whoever set this up take a look?
* 07:03 apergos: maybe the cron job on these image scalers should run more often... cleaned up /tmp on srv221
* 02:01 logmsgbot: LocalisationUpdate completed (1.18) at Wed Dec 28 02:04:37 UTC 2011
== December 27 ==
* 21:21 logmsgbot: catrope synchronized php-1.18/extensions/ArticleFeedbackv5/modules/ext.articleFeedbackv5/ext.articleFeedbackv5.startup.js 'r107427'
* 19:39 LeslieCarr: removed puppetdlock on brewster
* 19:04 LeslieCarr: removing manual localhost accept all iptables rule on locke (replacing with puppetized rule)
* 18:39 Ryan_Lane: clearing some logs on brewster and restarting lighttpd and squid
* 18:21 Jeff_Green: powercycled kaulen yet again
* 17:47 LeslieCarr: removed file�s out of srv223's /tmp directory to free up its space
* 17:15 mark: Power cycled amslvs3
* 17:06 apergos: powercycled kaulen, it was (presumably) swapping away. report.cgi was to blame
* 13:45 logmsgbot: nikerabbit synchronizing Wikimedia installation... : I18ndeploy r107378
* 13:07 logmsgbot: nikerabbit synchronized wmf-config/ 'cs gender aliases bug 33367'
* 02:01 logmsgbot: LocalisationUpdate completed (1.18) at Tue Dec 27 02:04:40 UTC 2011
== December 26 ==
* 19:59 logmsgbot: reedy synchronized php-1.18/extensions/ConfirmEdit/FancyCaptcha.class.php 'r107340'
* 18:08 logmsgbot: reedy synchronized php-1.18/extensions/ArticleFeedback/SpecialArticleFeedback.php 'r107332 To fix fatal in prod'
* 02:02 logmsgbot: LocalisationUpdate completed (1.18) at Mon Dec 26 02:05:32 UTC 2011
== December 25 ==
* 19:32 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php 'bug 29742 Some tweaks to FR config for fiwiki'
* 02:01 logmsgbot: LocalisationUpdate completed (1.18) at Sun Dec 25 02:04:23 UTC 2011
== December 24 ==
* 22:29 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33200 - Enable Autopatrol user group and RC patol for sa wiki'
* 22:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33291 Add autopatrolled flag right on sa.wikipedia'
* 22:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 33137 Add transwiki namespace'
* 22:15 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 33137 adding import sources for ptwikinews'
* 22:00 Reedy: Created flaggedrevs tables on ukwiki
* 21:59 logmsgbot: reedy synchronized docroot/wikipedia.org/favicon-si.ico 
* 21:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'kill tenwiki setting'
* 17:04 logmsgbot: reedy synchronized closed.dblist 'Add tenwiki to closed.dblist per bug 33185'
* 17:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 32641'
* 16:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Part of bug 33137 Enable GoogleNewsSitemap on ptwikinews'
* 16:46 logmsgbot: reedy synchronized closed.dblist 'Locking readerfeedback.l.w.o per bg 33229'
* 16:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33042 - Set wgBabelMainCategory to false for nlwiki'
* 07:51 apergos: removed manually log files olders than 14 days from /a/static/uncompressed on hume; looks like there's a problem with the script /usr/local/bin/offhost_backups on storage3 which was modified recently, guess the rsync delete option isn't being picked up
* 02:02 logmsgbot: LocalisationUpdate completed (1.18) at Sat Dec 24 02:06:03 UTC 2011
== December 23 ==
* 16:34 mutante: kswapd crashed, there is a Call Trace, and there was a load spike before, guess it is https://bugs.launchpad.net/ubuntu/+bug/721896 or similar
* 16:27 mutante: first interesting syslog line when it started: formey kernel: [39665413.570024] INFO: task kswapd0:36 blocked for more than 120 seconds.
* 16:10 mutante: gerrit and svn back up
* 16:07 mutante: gerrit stopped working and formey would still ping but no ssh connect and no mgmt output, powercycling formey
* 02:07 binasher: db9 maintenance completed
* 02:02 logmsgbot: LocalisationUpdate completed (1.18) at Fri Dec 23 02:05:19 UTC 2011
* 01:56 binasher: starting db9 maintenance - services will be unavailable for approx 15 minutes
* 00:08 logmsgbot: hashar synchronized php-1.18/extensions/WikimediaIncubator/IncubatorTest.php 'deploy r107120 - (bug 32772) fix up MoodBar and WikiLove on WIkimedia Incubator'
== December 22 ==
* 23:43 maplebed: put owa1-3 in as container servers, took ms1-3 out for pmtpa test swift cluster
* 20:56 logmsgbot: neilk synchronized p/resources/mediawiki/mediawiki.feedback.js 
* 20:45 Ryan_Lane: *date
* 20:45 Ryan_Lane: bringing mediawiki on virt1 up to daye
* 20:24 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Re-enable Contest extension'
* 20:10 mark: Turned off puppet dashboard reporting
* 15:07 apergos: er, on srv219, that is.
* 15:07 apergos: cleaned out tmp but... see, there really was only today's stuff in there so it's making me nervous
* 14:15 31NAAAO2O: hashar: testswarm: deleted mobile jobs requests since they are now disabled
* 12:30 mark: Made swift thumb seeder fetch from the squids instead of ms5, as a performance test
* 05:10 Ryan_Lane: finished creating all puppet configuration groups, classes, and variables. It's safe to configure and create instances again.
* 04:40 Ryan_Lane: upping version of OpenStackManager on virt1 to match development. configuration and creation of instances should be avoided.
* 02:04 binasher: db9 is writable again 
* 02:00 logmsgbot: LocalisationUpdate completed (1.18) at Thu Dec 22 02:04:05 UTC 2011
* 01:59 binasher: started db9 maintenance phase 1 (getting it replicated to db10 again)
* 00:15 K4-713: synchronized payments cluster to r107018
* 00:07 logmsgbot: awjrichards synchronizing Wikimedia installation... : r107015
== December 21 ==
* 22:44 mark: proxy worker processes increased from 8 to 24 on owa1-2, 48 on owa3
* 22:28 mark: Unmounted /srv/swift-storage/sdab1 on ms2 (borken filesystem)
* 22:16 mark: Flushed all iptables rules down the drain on ms1-3 (live hack, puppet will restore)
* 21:54 mark: Ran swift-init all restart on ms3
* 21:41 mark: Ran swift-init all restart on ms2
* 21:33 mark: Running ben's swift thumb loader script in a screen on hume
* 21:05 apergos: three more bin logs tossed from ds9
* 20:31 mark: Restarted swift-container on ms1 with higher worker count (4 instead of 2)
* 20:31 Jeff_Green: power cycled kaulen because it's deathswapped and unresponsive
* 19:41 mark: Ended oprofile run on ms1
* 19:33 logmsgbot: catrope synchronized wmf-config/missing.php 'Update missing.php from trunk, see bug 30206'
* 19:24 mark: Started oprofile run on ms1
* 19:20 mark: Migrated DRBD sync between nfs1 and nfs2 from protocol C (sync) to A (async)
* 17:48 RoanKattouw: srv224 has a full disk
* 17:48 logmsgbot: catrope synchronized php-1.18/extensions/ArticleFeedbackv5/modules/jquery.articleFeedbackv5/jquery.articleFeedbackv5.js 'r106959'
* 17:28 maplebed: ran apt-get clean on hume to clear out ~600M space on the / partition
* 16:18 apergos: so that was fast. barf from scp, nice call trace etc, shot the process on ds2, will email the vendor
* 15:26 apergos: and starting another huge copy from ds2 to ds1, let's see what happens...
* 15:18 apergos: reboot dataset1 with new kernel
* 15:14 apergos: installing 2.6.38 from natty backports on ds1 for further testing
* 13:55 apergos: powering on and off ds1 the hard way via the pdu. 
* 11:14 apergos: rebooting ds1 as it's got the one cpu tied up with a hung scp process and continual spewing to syslog...
* 10:23 apergos: s/lgo/log/ as in syslog. saving a copy of the bad log in fenari:/home/ariel/dataset1-syslog-dec-20-2012
* 10:09 apergos: dataset1 kernel panics in lgo during copy :-( :-( 
* 09:27 apergos: a few more binlogs deleted on db9...
* 03:53 logmsgbot: LocalisationUpdate completed (1.18) at Wed Dec 21 03:56:58 UTC 2011
* 03:48 Tim: doing a manual run of l10nupdate to check recache timings
* 03:26 logmsgbot: tstarling synchronized php-1.18/includes/LocalisationCache.php 'r106927'
* 02:40 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'LC recache log'
* 02:38 logmsgbot: tstarling synchronized php-1.18/includes/LocalisationCache.php 'r106922'
* 02:03 logmsgbot: LocalisationUpdate completed (1.18) at Wed Dec 21 02:06:08 UTC 2011
* 01:51 logmsgbot: reedy synchronized php-1.18/resources/mediawiki 'creating empty mediawiki.debug.css/js'
* 01:50 K4-713: synchronized payments cluster to r106917
* 01:16 K4-713: synchronized payments cluster to r106909
== December 20 ==
* 23:57 Ryan_Lane: readded /dev/sda2 partition on streber, it was somehow deleted, borking the raidset
* 23:20 Ryan_Lane: rebooting streber
* 23:00 LeslieCarr: creating a new logical volume on streber called syslog for syslog-ng purposes
* 21:08 awjr: synchronizing CiviCRM instance on grosley and aluminium to r1037
* 19:23 logmsgbot: reedy synchronized php-1.18/extensions/CentralAuth/ 'r106840'
* 19:14 logmsgbot: reedy synchronized php-1.18/extensions/Contest/ 'r106838'
* 17:05 mutante: spence: according to we should even double that if we have "high latency values (> 10 or 15 seconds)" and we have like > 1000
* 17:04 mutante: spence: check out "nagios -s /etc/nagios/nagios.cfg" for performance data - it suggests "Value for 'max_concurrent_checks' option should be >= 1231"
* 16:55 Jeff_Green: manually rotated spence:/var/log/nagios/nagios.log because nagios log rotation appears broken and the file is ~2.6G
* 16:43 logmsgbot: catrope synchronized php-1.18/resources/startup.js 'touch'
* 16:32 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Underscores -> spaces in wmgArticleFeedbackBlacklistCategories'
* 16:14 apergos: restarting scp on ds2, seems that it renegotiates after 64GB and that was failing, fixed
* 15:25 apergos: thumbs cleaner on ms5 complete. (don't worry, a new job will start up tomorrow)
* 15:16 mutante: installing security upgrades on tarin (includes perl and php)
* 14:10 apergos: another couple binlogs gone on ds9
* 13:41 mutante: added testswarm package to repo and installed it on gallium
* 13:15 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Use the correct interwiki prefix'
* 13:13 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Configure $wgImportSources on en_labswikimedia'
* 12:59 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Whitelist Category:Article_Feedback_5_Additional_Articles for AFTv5 and blacklist it for AFTv4 on enwiki and en_labswikimedia'
* 12:57 logmsgbot: catrope synchronized php-1.18/extensions/ArticleFeedbackv5/ 'r106794'
* 11:21 logmsgbot: nikerabbit synchronized php-1.18/extensions/WebFonts/resources/ext.webfonts.css 'bugfix r106781'
* 11:20 logmsgbot: nikerabbit synchronized php-1.18/extensions/Narayam/resources/ext.narayam.core.css 'bugfix r106781'
* 10:46 apergos: ds2 scp to ds1 stalled in the same place, looking into it
* 09:31 apergos: should have logged this earlier, prolly about 2 hours ago removed 3 more bin logs from db9, we were getting crowded again. 
* 09:29 apergos: first attempt at scp from ds2 to ds1 failed after 64gb, nothing useful in log, process on ds1 was hung at "restarting system call"... shot it and running again, from screen as root on ds2. 
* 07:53 apergos: after some playing around on ms5 (whichis responsible for the little io utilization spikes, but I'm done now), thumb cleaner is back at work for what should be its last day
* 06:07 Tim: removed mobile1, srv124, srv159, srv183, srv186 from /etc/dsh/group/apaches: not in mediawiki-installation
* 05:59 Tim: removed srv162, srv174 from /etc/dsh/group/job-runners: not in puppet jobrunners class
* 05:57 Tim: removed srv159, srv183 and srv186 from /etc/dsh/group/job-runners on fenari and stopped mw-job-runner on them, see https://bugzilla.wikimedia.org/show_bug.cgi?id=31576#c23
* 02:22 Ryan_Lane: fixed labsconsole. reverted aws-sdk to 1.4
* 02:04 logmsgbot: LocalisationUpdate completed (1.18) at Tue Dec 20 02:07:54 UTC 2011
* 00:52 Ryan_Lane: seems I broke labsconsole :(
* 00:52 Ryan_Lane: ran svn up on openstackmanager on virt1
* 00:05 logmsgbot: asher synchronized wmf-config/db.php 'putting db50 into rotation for s6'
== December 19 ==
* 23:59 binasher: started replicating db50 from db47
* 22:56 binasher: resolved apt issues on db13,17
* 22:50 binasher: running a hot xtrabackup of db47 to db50
* 22:17 RobH: db50/db51 online for asher to deploy into s6
* 22:12 Jeff_Green: deployed two new cron jobs on hume via /etc/cron.d/mw-fundraising-stats, temporary, will puppetize once we see that they script is working properly
* 21:41 logmsgbot: awjrichards synchronizing Wikimedia installation... : Deploying ContributionReporting fixes to use summary tables (r106696), disabling ContributionReporting everywhere except test and foundationwikis
* 21:25 logmsgbot: asher synchronized wmf-config/db.php 'pulling db43 which died'
* 21:12 apergos: note that dataset1 appears to be keeping time ok
* 21:12 apergos: started an scp (to make ds1 work a tiny bit harder) of some files from ds2. running on ds2 in screen session as root. 
* 21:01 logmsgbot: catrope synchronized php-1.18/extensions/ArticleFeedbackv5/ 
* 20:45 RoanKattouw: ...on enwiki that is
* 20:45 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Re-enable ArticleFeedbackv5, hopefully fixed now'
* 20:43 logmsgbot: catrope synchronized php-1.18/extensions/ArticleFeedbackv5/ 
* 20:36 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php '...and disable for now'
* 20:34 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable ArticleFeedbackv5 on enwiki'
* 20:22 logmsgbot: catrope synchronized php-1.18/extensions/ArticleFeedbackv5/ 
* 20:08 logmsgbot: catrope synchronized php-1.18/extensions/ClickTracking/ApiClickTracking.php 'r106681'
* 19:41 Jeff_Green: dropping several db's from db9 which have already been migrated to fundraisingdb cluster
* 19:40 notpeter: powercycling maerlant
* 19:05 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Translate with tables'
* 18:34 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Translate needs tables'
* 18:28 logmsgbot: nikerabbit synchronizing Wikimedia installation... : I18ndeploy r106667 and new extensions on mediawiki.org
* 16:42 RobH: dataset1 new data partition ready and setup to automount
* 15:49 RobH: dataset1 reinstalled and has had puppet run. Now to see if it can keep time
* 15:46 RoanKattouw: maerlant is fried, load avg is 500+, linearly increasing since Friday. Rejects SSH login attempts
* 15:45 notpeter: restarting indexer on searchidx2
* 14:16 apergos: thumb cleaner to bed for the night... for the last time? 
* 13:15 mutante: truncated spence.cfg in ./puppet_checks.d/ - it had multiple dupe service definitions for all checks on spence
* 13:11 mutante: commented check_job_queue stuff from non-puppetized files on spence (hosts.cfg, conf.php) to get rid of "duplicate definition" now that it's been pupptized
* 12:35 mutante: deleted snapshot4 files from /var/lib/puppet/yaml/node and ./yaml/facts on sockpuppet and stafford, they got recreated and fixed puppet run on sn4
* 10:08 apergos: a few more binlogs on db9 gone. eeking out another 12 hours or so
* 06:57 apergos: thumb cleaner awake for the day. poor thing, slaving away but soon it will be able to retire
* 01:57 logmsgbot: LocalisationUpdate failed (1.18) at Mon Dec 19 02:00:11 UTC 2011
== December 18 ==
* 16:41 notpeter: removing about 4G of binlogs from db9. everything more than 24 hours old.
* 15:12 apergos: thumb cleaner sleeping it off for the night
* 07:34 apergos: thumb cleaner to work for the day
* 01:57 logmsgbot: LocalisationUpdate failed (1.18) at Sun Dec 18 02:00:04 UTC 2011
== December 17 ==
* 22:49 RobH: Anytime db9 hits 98 or 99% someone needs to remove binlogs to bring it back down to 94 or 95%
* 22:48 RobH: removed older binlogs on db9 again to kick it back to a bit more free space to last the weekend.
* 17:53 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Remove SVN dir setting, this is now passed in on the command line'
* 16:43 RoanKattouw: Found out why LocalisationUpdate was failing. Would have been fixed already if puppet had been running on fenari, but it's throwing errors. See r1617 and my comment on r1558
* 14:32 apergos: thumb cleaner to bed for the night... about 2 days left I think
* 07:25 apergos: thumb cleaner started up for the day
* 01:57 logmsgbot: LocalisationUpdate failed (1.18) at Sat Dec 17 02:00:18 UTC 2011
== December 16 ==
* 22:30 RobH: reclaimed space on db9, restarted mysql, services seem to be recovering
* 22:24 maplebed: restarting mysql on db9; brief downtime for a number of apps (bugzilla, blog, etc.) expected.
* 22:03 RobH: db9 space reclaimed back to 94% full, related services should start recovering
* 21:57 RobH: db9 disk full, related services are messing up, fixing
* 21:56 RobH: kicking apache for bz related issues on kaulen
* 19:14 logmsgbot: catrope synchronized php-1.18/resources/startup.js 'touch'
* 19:07 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Set AFTv4 lottery odds to 100% on en_labswikimedia'
* 18:48 LeslieCarr: removed the ssl* yaml logs on stafford to fix the puppet not running error
* 16:13 apergos: thumb cleaner to bed for the night. definitely need an alarm clock for this... good thing it's only got about 4 days of backlog left
* 15:41 RobH: es1002 being actively worked on for hdd controller testing
* 15:39 RobH: lvs1003 disk dead per RT 1549, will torubleshoot on site later today or Monday
* 15:32 RobH: lvs1003 unresponsive to serial console, rebooting
* 15:18 RobH: reinstalling dataset1
* 14:45 mutante: puppet was broken on all servers including "nrpe" due to package conflict with nagios-plugins-basic i added to base, revert+fix
* 13:29 RoanKattouw: Dropping and recreating AFTv5 tables on en_labswikimedia and enwiki
* 13:26 logmsgbot: catrope synchronized php-1.18/extensions/ArticleFeedbackv5/ 'Updating to trunk state'
* 13:25 mutante: tweaked Nagios earlier today: external command_check_interval & event_broker_options (see comments in gerrit Id3b4a458)
* 13:01 mark: Found lvs5 and lvs6 with offload-gro enabled, even though it's set disabled in /etc/network/interfaces... corrected
* 09:21 apergos: restarted lighthttpd on ds2, it had stopped (and why didn't nagios tell us? )
* 08:38 mutante: spence - had killed additional notifications.cgi and history.cgi procs, waited 5 minutes, load went down a lot, restarting nagios
* 08:22 mutante: spence - almost unusable, Nagios notifications.cgi and history.cgi use a lot of memory, stopping Nagios, watching swap
* 08:15 mutante: spence slow again, side-note: tried to use "sar" to investigate but "Please check if data collecting is enabled in /etc/default/sysstat" (want to?)
* 07:54 logmsgbot: nikerabbit synchronized php-1.18/extensions/WebFonts/resources/ext.webfonts.js 'JS fix r106418'
* 07:09 apergos: thumbs cleaner awake for the day
* 01:57 logmsgbot: LocalisationUpdate failed (1.18) at Fri Dec 16 02:00:14 UTC 2011
== December 15 ==
* 23:19 LeslieCarr: pushing rule to planet.wikimedia.org which should redirect all https to http
* 23:00 LeslieCarr: puppetized planet.wikimedia.org on singer
* 22:41 LeslieCarr: removing https support from planet.wikimedia.org
* 21:43 logmsgbot: awjrichards synchronized php/extensions/LandingCheck/SpecialLandingCheck.php 'r106377'
* 21:42 logmsgbot: awjrichards synchronized php/extensions/LandingCheck/LandingCheck.php 'r106377'
* 21:34 logmsgbot: awjrichards synchronized php/extensions/ContributionTracking/ContributionTracking_body.php 'r106375'
* 21:34 logmsgbot: awjrichards synchronized php/extensions/ContributionTracking/ContributionTracking.processor.php 'r106375'
* 21:33 logmsgbot: awjrichards synchronized php/extensions/ContributionTracking/ContributionTracking.php 'r106375'
* 20:21 K4-713: synchronized payments cluster to r106360
* 19:16 LeslieCarr: removed mw* and virt* yaml files from stafford in order to clear up broken files and make puppet run again
* 18:59 logmsgbot: LocalisationUpdate completed (1.18) at Thu Dec 15 19:02:58 UTC 2011
* 18:58 RobH: cp1018 also offline, stealing cables from it for testing
* 18:52 logmsgbot: reedy synchronized php-1.18/includes/Block.php 'Testing fix for bug 33101'
* 18:43 logmsgbot: catrope synchronized php-1.18/extensions/LocalisationUpdate/ 'r106352'
* 18:25 RobH: es1002 and cp1019 offline for harddisk controller testing
* 18:07 RoanKattouw: Running LU /again/ to hopefully fix issues
* 16:31 mark: Fixed database entries in william's exim.conf
* 16:24 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Disabling Contest extension because of XSS'
* 15:31 apergos: thumb cleaner to bed for the night 
* 14:59 logmsgbot: LocalisationUpdate completed (1.18) at Thu Dec 15 15:02:05 UTC 2011
* 14:52 RoanKattouw: Running l10nupdate by hand
* 14:37 RoanKattouw: That InitialiseSettings.php change enabled ArticleFeedback (v4) on en_labswikimedia
* 14:36 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 
* 14:28 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Disable ReaderFeedback on en_labswikimedia'
* 14:26 logmsgbot: catrope synchronized php-1.18/extensions/ArticleFeedbackv5/ 'Updating AFTv5 to trunk staet'
* 13:56 Reedy: FixExtLinksProtocolRelative run now finished on all wikis
* 07:04 apergos: thumb cleaner awake for the day
* 04:00 Tim: restarted all job runners to get Roan's jobs-loop.sh change
* 02:13 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
* 01:59 logmsgbot: LocalisationUpdate failed (1.18) at Thu Dec 15 02:02:13 UTC 2011
* 01:53 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
* 00:28 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
* 00:25 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
== December 14 ==
* 23:54 logmsgbot: catrope synchronized php-1.18/maintenance/jobs-loop.sh 'r106284'
* 23:54 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
* 23:48 logmsgbot: catrope synchronized php-1.18/resources/startup.js 'touch'
* 23:47 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable ClickTracking on en_labswikimedia so AFTv5 will work'
* 23:29 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable AFTv5 on en_labswikimedia'
* 23:22 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying MoodBar changes for feedback dashboard. Along for the ride: VisualEditor changes by Neil and a Badtitle bug fix by Tim
* 23:14 RoanKattouw: Created AFTv5 tables on en_labswikimedia and enwiki
* 23:01 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add trigger code for AFTv5'
* 22:59 binasher: restarted nagios with enable_environment_macros = 0
* 22:57 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Disable AFTv5 for now though'
* 22:56 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add wmg variables for upcoming ArticleFeedbackv5 deployment'
* 22:53 RoanKattouw: Prepping ArticleFeedbackv5 deployment to enlabs; please do not scap while I'm working on this
* 22:34 binasher: ibdata1 on db9 now has 208GB of free space (InnoDB free: 218228736 kB)
* 22:32 binasher: dropping otrs from db9
* 22:21 RoanKattouw: Deployed MoodBar changes to testwiki
* 22:10 Reedy: fixExtLinksProtocolRelative is onto jawiki
* 22:10 binasher: otrs on williams migrated to db48
* 22:01 binasher: exim on mchenry is now using db48/49 for otrs mail verification
* 21:55 awjr: synced DonationInterface on payments cluster to r106247
* 21:26 logmsgbot: awjrichards synchronized php/extensions/DonationInterface/gateway_common/interface.i18n.php 'r106245'
* 21:25 logmsgbot: awjrichards synchronized php/extensions/DonationInterface/gateway_common/countries.i18n.php 'r106245'
* 21:25 logmsgbot: awjrichards synchronized php/extensions/DonationInterface/globalcollect_gateway/globalcollect_gateway.i18n.php 'r106245'
* 21:25 logmsgbot: awjrichards synchronized php/extensions/DonationInterface/payflowpro_gateway/payflowpro_gateway.i18n.php 'r106245'
* 18:19 Jeff_Green: enfired exim's DKIM signing feature on aluminium and grosley
* 17:13 RobH: jetpack extension, already disabled and not in use, deleted off wp blog install on hooper
* 15:52 logmsgbot: catrope synchronized php-1.18/includes/OutputPage.php 'r106206'
* 15:50 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Disable DB logging for ClickTracking'
* 15:45 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Send all ClickTracking logs to emery over UDP, including non-enwiki wikis with ClickTracking enabled'
* 15:44 logmsgbot: LocalisationUpdate failed
* 15:42 RoanKattouw: Aborted LU run based on /home . Will restart once puppet has moved the LU checkout to /var/lib
* 15:26 logmsgbot: nikerabbit synchronized php-1.18/extensions/WebFonts/resources/ 'WebFonts bug fixes r106204'
* 15:14 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Send all ClickTracking logs to emery over UDP, including non-enwiki wikis with ClickTracking enabled'
* 14:53 RoanKattouw: Running l10nupdate one more time, this time with the 'old' script, pulling from the SVN checkout on /home (this'll be slow)
* 14:36 mutante: fixed screen permissions on hume earlier today per RT:2133 - Ubuntu bug #390405
* 14:33 mark: /var/lib/puppet on stafford filled up with puppet reports; cleaned up and installed a cron job to prevent that from happening again
* 14:31 logmsgbot: LocalisationUpdate completed (1.18) at Wed Dec 14 14:34:27 UTC 2011
* 14:26 logmsgbot: catrope synchronized php-1.18/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'r106200'
* 14:20 apergos: thumb cleaner to bed for the night
* 14:00 logmsgbot: catrope synchronized php-1.18/extensions/LocalisationUpdate/LocalisationUpdate.class.php 
* 13:59 RoanKattouw: Fixed ownership of /home/wikipedia/common/php-1.18/cache/message-timestamp from catrope:wikidev to l10nupdate:wikidev
* 13:58 logmsgbot: LocalisationUpdate completed (1.18) at Wed Dec 14 14:01:36 UTC 2011
* 13:49 logmsgbot: catrope synchronized php-1.18/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'r106193'
* 13:43 RoanKattouw: Changing ownership of /home/wikipedia/common/php-1.18/cache/l10n to l10nupdate:wikidev
* 13:40 logmsgbot: LocalisationUpdate failed (1.18) at Wed Dec 14 13:44:01 UTC 2011
* 13:39 RoanKattouw: Doing a trial run of my new LU script on fenari, from my home directory
* 13:34 logmsgbot: catrope synchronized php-1.18/extensions/LocalisationUpdate/ 'Update LU to trunk state'
* 13:31 mark: Moved ms4 to internal
* 12:08 RoanKattouw: Changing ownership of fenari:/home/wikipedia/l10n to l10nupdate:wikidev , recursively (was: 500:wikidev). TODO: move this off of /home and puppetize it
* 09:34 p858snake|l: [19:23] <logmsgbot> \!log Reedy fixExtLinksProtocolRelative.php is still on frwiki at 182\,200
* 06:59 apergos: thumbs cleaner back at work for the day
* 04:24 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
* 01:32 Reedy: fixExtLinksProtocolRelative is on frwiki, 50,500 run in screen on fenari as reedy. Please shoot overnight if needed
* 01:31 Reedy: fixExtLinksProtocolRelative is onrofrwiki, 50,500 run in screen on fenari
* 01:21 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
* 01:14 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
* 01:12 logmsgbot: preilly synchronized php-1.18/includes/OutputPage.php 
* 00:09 logmsgbot: tstarling synchronized php-1.18/includes/Cdb.php 'r106137'
* 00:01 notpeter: restart puppet on brewster
== December 13 ==
* 23:40 binasher: restarted pdns on dobson
* 20:59 logmsgbot: nikerabbit synchronized php-1.18/extensions/WebFonts/resources/ext.webfonts.js 'Updating font urls'
* 20:55 logmsgbot: nikerabbit synchronized php-1.18/extensions/WebFonts/fonts/ 'touching all font files'
* 20:48 logmsgbot: catrope synchronized php-1.18/extensions/WebFonts/fonts/Orya/utkal.ttf 'touch'
* 20:45 logmsgbot: catrope synchronized php-1.18/extensions/WebFonts/fonts/Orya/utkal.ttf 'touch'
* 20:34 RoanKattouw: Actually running Puppet on the bits Apaches now
* 20:28 RoanKattouw: Manually running Puppet on all bits Apaches using dsh
* 19:58 RoanKattouw: Removed spence from the mediawiki-installation node list
* 19:45 logmsgbot: neilk synchronizing Wikimedia installation... : deploying VisualEditor, also deploying some new JS libs and associated tests and messages to core which were taken from UploadWizard
* 19:43 logmsgbot: neilk synchronized wmf-config/extension-list 'added VisualEditor'
* 19:42 logmsgbot: neilk synchronized wmf-config/ExtensionMessages-1.18.php 'added VisualEditor'
* 18:51 Reedy: fixExtLinksProtocolRelative is upto enwiki
* 18:17 notpeter: stopping puppet on brewster again. again, please let me know if you want to restart it...
* 17:15 apergos: thumb cleaner went to bed late... maybe I should set an alarm to remind me :-/ 
* 16:58 Reedy: Running foreachwiki fixExtLinksProtocolRelative in screen on fenari as reedy
* 16:56 Reedy: fixExtLinksProtocolRelative run on meta
* 16:49 Reedy: fixExtLinksProtocolRelative run on mw.org and simplewiki
* 15:07 mutante: spence - copied sudoers file from fenari to allow "admins to run commands as apache, mwdeploy and l10nupdate" and wikidev = admins
* 14:47 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'wgNarayamUseBetaMapping'
* 14:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add wmgNarayamUseBetaMapping'
* 12:38 mutante: rolling kernel and security upgrades on amssq31-33
* 11:52 mutante: spence still slow, restarted nagios (as opposed to just reloading)
* 11:03 mutante: spence changed check_interval for check_job_queue from 1 to 15 (min) as it takes ~5 minutes to complete currently - reloaded nagios
* 10:52 mutante: check_job_queue Nagios check finally works again :) - but why do we check for "3rd biggest" (fix me)
* 10:43 RoanKattouw: Added spence to fenari:/etc/dsh/group/mediawiki-installation
* 10:41 mutante: spence - "chown -R mwdeploy:mwdeploy /usr/local/apache/common-local" per Roan, sync-common again, finished fine..after a while. (spence had really outdated copy of MW source tree)
* 10:31 mutante: ran sync-common from spence and got a ton of permission errors
* 10:29 mutante: spence added root to sudoers to fix sync-common
* 10:25 mutante: spence created mwdeploy user to fix sync-common
* 10:17 mutante: spence - fix comment style in apc.ini (# -> //) to get rid of 'PHP Deprecated' messages - install php-apc for check_job_queue
* 10:11 mutante: copied script files provided by wikimedia-task-appserver from fenari to spence
* 06:59 apergos: thumb cleaner back at work for the day on ms5
* 05:38 logmsgbot: nikerabbit synchronized php-1.18/extensions/WebFonts/resources/ext.webfonts.js 'bugfix for webfonts: r105989'
* 04:34 RobH: blog.wikimedia.org rolled to wp3.3 successfully
* 02:38 notpeter: belay that. running just fine. just hella slow.
* 02:36 notpeter: puppet currently not completing runs on spence. not going to try to divine root cause this late at night
* 02:28 notpeter: restarting puppet on brewster
* 01:29 notpeter: removing a couple of binlogs from db9 to try to get it through until it's replaced....
* 01:12 logmsgbot: tstarling synchronized php-1.18/includes/MagicWord.php 
* 01:04 Ryan_Lane: switching ssl certificate to digicert certificate for wikipedia
* 00:46 Ryan_Lane: enabling https for *.m.wikipedia.org
* 00:46 Ryan_Lane: repooling ssl1001
* 00:28 Ryan_Lane: restarted pybal on lvs1004 for mobile ssl and mobile svc services
* 00:27 Ryan_Lane: make that lvs1001
* 00:27 Ryan_Lane: restarted pybal on ssl1001 for mobile ssl and mobile svc services
* 00:15 Ryan_Lane: depooling ssl1001 for testing
* 00:15 LeslieCarr: depooling and restarting ssl1001 withnew cert
* 00:14 LeslieCarr: pushing out new digicert certificate for *.wikipedia.org
* 00:12 logmsgbot: reedy synchronized php-1.18/extensions/CodeReview/ 'Reverting CR back to previous state'
* 00:02 logmsgbot: reedy synchronized php-1.18/extensions/CodeReview/backend/DiffHighlighter.php 
* 00:00 logmsgbot: reedy synchronized php-1.18/extensions/CodeReview/ui/CodeRevisionAuthorView.php 'fix linker fatal'
== December 12 ==
* 23:57 logmsgbot: reedy synchronized php-1.18/extensions/CodeReview/backend/DiffHighlighter.php 
* 23:42 logmsgbot: reedy synchronized wmf-config/codereview.php 'Add $wgCodeReviewNewPerPath'
* 23:40 logmsgbot: reedy synchronized php-1.18/extensions/CodeReview/ 'syncing'
* 23:03 Ryan_Lane: repooling ssl1001
* 23:00 Ryan_Lane: depooling ssl1001
* 22:22 awjr: syncing CiviCRM on aluminium to r952
* 22:21 logmsgbot: reedy synchronized wmf-config/codereview.php 'Important paths!'
* 22:20 binasher: upgraded xtrabackup in wikimedia-lucid to 1.6.3, changed puppet rule to ensure latest
* 21:39 binasher: stafford:/var/lib/puppet was full, extended fs by 5GB
* 21:16 RoanKattouw: Commented out srv159, srv183, srv186 from /etc/dsh/group/mediawiki-installation : these servers have been decommissioned and don't seem to be picking up SSH key changes from puppet
* 21:11 logmsgbot: catrope synchronized php-1.18/resources/startup.js 'touch'
* 21:11 logmsgbot: catrope synchronized php-1.18/extensions/WebFonts 'Touching all WebFonts JS/CSS files'
* 21:05 notpeter: temp turning off puppet on brewster. feel free to turn it back on, but if you're going to do so, please let me know
* 21:00 Jeff_Green: adding txt records to wikimedia.org zone file for DKIM testing
* 20:48 Jeff_Green: adding txt records to wikimediafoundation.org zone file for DKIM testing
* 20:37 binasher: rebooting db1017 again, this time into new 2.6.32-36
* 20:32 logmsgbot: nikerabbit synchronized php-1.18/resources/startup.js 'Trying to solve caching issues'
* 20:24 binasher: rebooted db1017
* 19:53 RobH: project2 is going down, and hopefully never coming back.
* 18:57 logmsgbot: nikerabbit synchronized php-1.18/includes/DefaultSettings.php 'i18ndeploy: config change needed for Incubator extension'
* 18:57 logmsgbot: nikerabbit synchronized php-1.18/extensions/CodeReview/ui/CodeRevisionView.php 'i18ndeploy: rtl fix for CR'
* 18:56 logmsgbot: nikerabbit synchronized php-1.18/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'i18ndeploy: new group messages'
* 18:55 logmsgbot: nikerabbit synchronized php-1.18/skins/common/shared.css 'i18ndeploy: header css'
* 18:50 logmsgbot: catrope synchronized php-1.18/extensions/Narayam/ 'Trying to reproduce errors reported by Niklas'
* 18:37 LeslieCarr: upgrading racktables
* 18:35 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 
* 18:33 logmsgbot: nikerabbit synchronized php-1.18/extensions/WikimediaIncubator/ 'i18ndeploy: Incubator'
* 18:31 logmsgbot: nikerabbit synchronized php-1.18/extensions/Babel/ 'i18ndeploy: Babel updates'
* 18:26 logmsgbot: nikerabbit synchronized php-1.18/extensions/Narayam/ 'i18ndeploy: Narayam update - for real this time'
* 18:21 logmsgbot: nikerabbit synchronized php-1.18/extensions/Narayam/ 'i18ndeploy: Narayam update'
* 18:08 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Enabling WebFonts on more wikis'
* 15:23 logmsgbot: reedy synchronized php-1.18/includes/specials/SpecialSearch.php 'r105888'
* 15:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 23452 - Change logo on dk.wikimedia.org'
* 14:24 apergos: thumb cleaner suspended for the night
* 13:52 mutante: fixed nagios by manually adjusting out/updated check commands below puppet_checks.d
* 08:14 apergos: chown/grp mwdeploy of /usr/local/apache/common/php/cache on srv150-srv190 to shut up the l10n rsync job
* 07:56 apergos: on hume installed parsekit via pecl, needed for cron jbo mb-dump.sh (mood bar) and for a tor-related cron job
* 07:53 apergos: on hume installed libcrypt-ssleay-perl and (maybe superfluous) libnet-ssleay-perl to fix honeypot cronjob which was relying on an old version in someone's home directory
* 07:02 apergos: also restarted morebots, it was missing from the channel for some reason
* 07:01 apergos: thumb cleaner started up on ms5 for the day
== December 11 ==
* 16:57 apergos: thumb cleaner to bed for the night, let it run a bit long but seems like traffic was ok on a Sunday night
* 16:00 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Set EnableDnsBlacklis for thwiki'
* 15:58 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'swap wgEnableSorbs for wgEnableDnsBlacklist'
* 15:40 logmsgbot_: reedy synchronized php-1.18/resources/startup.js 'touch'
* 15:37 logmsgbot_: reedy synchronized php-1.18/extensions/ApiSandbox/ext.apiSandbox.js 'r105799'
* 07:00 apergos: thumb cleaner back at work on ms5 for the day
== December 10 ==
* 15:24 apergos: thunbs cleaner to bed for the night
* 06:58 apergos: thumb cleaner back at work for the day
* 00:25 Ryan_Lane: fixed security groups in nova, was apparent a changed flag
== December 9 ==
* 22:34 LeslieCarr: manually linked /etc/apache2/sites-enabled/racktables2.wikimedia.org for temporary racktables2 installation
* 17:44 logmsgbot_: nikerabbit synchronizing Wikimedia installation... : WebFonts on testwiki
* 15:24 apergos: thumb cleaner to bed for the night 
* 09:41 logmsgbot_: nikerabbit synchronized php-1.18/extensions/WebFonts/ 'Syncing WebFonts r105661'
* 07:00 apergos: thumb cleaner revived for the day, now skipping (hopefully) images with exif rotation data in them that mediawiki uses
* 01:50 asher: ran "opcontrol --deinit" on db32
* 01:37 logmsgbot_: tstarling synchronized wmf-config/db.php 'r/w on enwiki'
* 01:36 asher: flush tables from set read_only=0 isn't happening, bounced mysqld
* 01:36 asher: removed read-only from my.cnf
* 01:32 asher: ran set global read_only=0 on db32
* 01:32 Tim: on db32 my.cnf reduced the innodb pool size from 55 to 53 GB after OOM
* 01:23 logmsgbot_: tstarling synchronized wmf-config/db.php 'switching s1 to read-only mode'
== December 8 ==
* 23:14 maplebed: nagios and ganglia are back up and alive for now.
* 22:51 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32119 - Change the diacritics in the namespaces at ro.wikinews'
* 22:44 RobH: ganglia and nagios will be down during spence reboot
* 22:43 RobH: rebooting spence
* 22:33 apergos: stopping and restarting nagios on spence... had high load and swap going on 
* 22:26 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32119 - Change the diacritics in the namespaces at ro.wikinews'
* 22:02 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Revert last push due to a lot of page conflicts'
* 21:57 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32119 - Change the diacritics in the namespaces at ro.wikinews'
* 21:50 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgNoticeBanner_Harvard2011 for enwiki'
* 21:36 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'wgNoticeBanner_Harvard2011[salt]'
* 21:26 notpeter: restarting puppet on brewster
* 21:14 logmsgbot_: reedy synchronized php-1.18/extensions/CentralNotice/CentralNotice.php 'remove debugging line'
* 21:10 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Set some centralnotice config for testwiki'
* 21:01 logmsgbot_: reedy synchronized php-1.18/extensions/CentralNotice/CentralNotice.php 'r105588'
* 21:00 logmsgbot_: reedy synchronized php-1.18/extensions/CentralNotice/CentralNotice.db.php 'r105588'
* 20:58 logmsgbot_: reedy synchronized php-1.18/extensions/UserDailyContribs/api/ApiUserDailyContribs.php 'r105587'
* 20:56 logmsgbot_: reedy synchronized php-1.18/extensions/UserDailyContribs/UserDailyContribs.php 'r105587'
* 19:16 logmsgbot_: reedy synchronized php-1.18/resources/jquery.ui/themes/vector/jquery.ui.button.css 
* 19:08 notpeter: temporarily stopping puppet on brewster for prototyping of partman config. feel free to restart if needed :)
* 18:45 logmsgbot_: laner synchronized wmf-config/InitialiseSettings.php 'Enable Emergency Captcha for ptwikinews'
* 18:08 RobH: ns0/1/2 all dig fine now
* 18:07 RobH: restarted pdns on ns1 and ns2, my dns change was an hour ago...
* 17:50 logmsgbot_: reedy synchronized wmf-config/flaggedrevs.php '$wgGroupPermissions[sysop][stablesettings] = true for frwikinews'
* 17:39 logmsgbot_: reedy synchronized wmf-config/flaggedrevs.php 'Tweak fiwiki settings for bug 29742'
* 17:09 RobH: allocating db48/db49 for #1477: move otrs database off of db9/db10
* 17:05 Reedy: That was bug 31947 Namespaces for fawikisource
* 17:05 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 
* 16:50 Reedy: Run namespaceDupes on tlwiki and azwikibooks
* 16:49 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 25651 - Requesting for the namespace Portal for Tagalog Wikipedia'
* 16:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31068 - Add Wikibooks namespace to Azerbaijani Wikibooks'
* 15:49 mark: Restarted formey with lower max clients setting
* 14:40 apergos: thumb cleaner to bed for the night
* 13:55 mark: domas fixed max_open_files issue on db9 (too many connections from mchenry), and fixed replication to db10 which I broke by deleting all binlogs
* 13:37 mark: Removed some binlogs on db9
* 10:06 logmsgbot_: hashar synchronized wmf-config/CommonSettings.php 'bug 32513 : Collection license link: now use upstream URL instead of a license with ton of wikitext'
* 09:51 logmsgbot_: hashar synchronized wmf-config/CommonSettings.php 'GFDL is long gone, we use CC-BY-SA 3.0 nowaday. See bug 32513'
* 09:48 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php '32513 - Collection extension still uses GFDL license'
* 06:57 apergos: thumb cleaner awake and back to work
* 01:36 binasher: installing percona-toolkit on all databases
* 01:09 Ryan_Lane: restarted gmetad on spence
* 01:04 Ryan_Lane: restarting nagios on spence
== December 7 ==
* 23:56 logmsgbot_: awjrichards synchronized wmf-config/CommonSettings.php 'Re-enabling Special:FundraiserStatistics on foundation wiki'
* 23:55 logmsgbot_: awjrichards synchronized php/extensions/ContributionReporting/ContributionReporting.php 'r105494'
* 23:54 logmsgbot_: awjrichards synchronized php/extensions//ContributionReporting/FundraiserStatistics_body.php 'r105494'
* 23:54 logmsgbot_: awjrichards synchronized php/extensions/ContributionReporting/modules/ext.fundraiserstatistics.js 'r105486'
* 23:32 LeslieCarr: on cr1-sdtpa and cr2-pmtpa, routing labs a 208.80.153.192/27 instead of the smaller 208.80.153.192/28 block
* 23:01 LeslieCarr: unpinned php-common in generic::webserver::php5
* 18:35 K4-713: synchronized payments cluster to r105401
* 17:04 apergos: restarted httpd on dataset2
* 15:14 apergos: thumb cleaner to bed for the night (it gets tired so easily :-P)
* 14:51 RobH: rebooting TS-amaranth per DaBPunkt request - unresponsive to ssh 
* 13:16 mark: Fixed package situation on db1048
* 06:58 apergos: thumb cleaner starting its daily run
* 00:35 binasher: rebooting db1041 for new kernel to take effect.. troubleshooting xfs sync kernel panics - probably a failing raid controller
== December 6 ==
* 22:21 K4-713: synchronized payments cluster to r105359
* 19:44 Jeff_Green: temporarily doubling php5/apache2's memory limit on aluminium to see if that helps civicrm
* 18:51 RobH: updated bz per rt2098 with removed_comment template
* 18:36 RobH: applied patch to BZ per RT 2098
* 16:00 Jeff_Green: dist-upgrade and reboot silicon
* 15:05 mark: Setup puppetmaster on stafford, moved puppet DNS aliases for testing. Local git repos need to be manually 'pulled' on stafford:/var/lib/git/operations/puppet for now
* 15:04 mutante: killed java process on search1 and restarted lsearchd, per rainman
* 14:59 rainman-sr: can someone look at search1, it seems to have java process that cannot be stopped with /etc/init.d/lsearchd stop, needs root
* 14:51 apergos: thumb cleaner to bed for the night
* 13:54 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'codereview: auto deferres /trunk/extensions/SemanticMediaWiki'
* 13:47 apergos: very temporarily gained sme space back on srv220 by clearing out temp but it will be tight again in an hour, can't see what's taking up the room besides scaler temp files but there is something
* 11:02 mutante: generated de_DE.UTF-8 locale on singer, fixed de.planet update bug 32798
* 10:36 mark: Temporarily firewalled all eqiad hosts from sockpuppet's puppetmaster port
* 10:04 mark: Reinstalling stafford
* 06:54 apergos: thumb cleaner started up on ms5 for the day
* 00:54 logmsgbot_: awjrichards synchronized php/extensions/CentralNotice/CentralNotice.db.php 'r105263'
* 00:53 logmsgbot_: awjrichards synchronized php/extensions/CentralNotice/special/SpecialBannerAllocation.php 'r105263'
* 00:31 Tim: added docroot file for Yahoo BOSS, bug 32816
== December 5 ==
* 23:59 logmsgbot_: awjrichards synchronized php/extensions/FundraiserLandingPage/FundraiserLandingPage.php 'r105251'
* 23:58 logmsgbot_: awjrichards synchronized php/extensions/FundraiserLandingPage/FundraiserLandingPage.body.php 'r105251'
* 23:58 logmsgbot_: awjrichards synchronized php/extensions/FundraiserLandingPage/FundraiserRedirector.body.php 'r105251'
* 20:42 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
* 20:41 logmsgbot_: reedy synchronized flaggedrevs.dblist 'Bug 28766 - Enable the FlaggedRevs extension on Greek Wikinews'
* 20:40 logmsgbot_: reedy synchronized wmf-config/flaggedrevs.php 'Add elwikinews FR config per bug 28766'
* 20:23 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32357 - Alias for project space in ko.wikiquote'
* 20:16 logmsgbot_: reedy synchronized wmf-config/flaggedrevs.php 'wgSimpleFlaggedRevsUI = false for fiwiki'
* 20:11 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32801 - Create Proofread Page namespaces on Sanskrit Wikisource'
* 20:04 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32641 - Configuration changes for the Babel extension at es.wikibooks'
* 19:59 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32749 - Add user group flooder to vi.wikipedia'
* 19:57 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31801 - Update logo for or.wiktionary'
* 19:53 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31600 - Enable NewUserMessage extension on four th projects'
* 18:51 logmsgbot_: reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
* 18:47 Nikerabbit: hi, that was i18ndeploy
* 18:45 logmsgbot_: nikerabbit synchronizing Wikimedia installation... : 
* 16:56 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32757 - Enable MoodBar on Wikimedia Sverige Wiki'
* 16:28 logmsgbot_: reedy synchronized wmf-config/flaggedrevs.php 'Bug 32804 - mediawiki.org admins have rights to add undefined group "autochecked users"'
* 16:26 logmsgbot_: reedy synchronized wmf-config/flaggedrevs.php 'Bug 32804 - mediawiki.org admins have rights to add undefined group "autochecked users"'
* 16:07 cmjohnson1: shutting down ms2
* 14:31 apergos: thumbs cleaner suspended for the night
* 12:39 mutante: added tarin to site.pp, includes standard class, first puppet run finished
== December 4 ==
* 20:23 RobH: kicked apache on sockpuppet again, seems to have fixed puppet servers timing out on their runs
* 14:56 apergos: thumb cleaner suspended for the night
* 07:04 apergos: thum cleaner back at work on ms5
* 01:57 logmsgbot: LocalisationUpdate failed
== December 3 ==
* 19:11 K4-713: synchronized payments cluster to r105072
* 16:21 apergos: please ignore the spikes behind the ms5 curtain, just checking some space usage in a few directories we haven't looked at yet
* 14:57 apergos: suspending thumb cleaner
* 10:48 apergos: thumb cleaner back in action
* 10:34 apergos: thumb cleaner suspended a few minutes ago, letting things settle down a bit
* 10:23 apergos: test
* 10:20 apergos: morebots dead, tried restarting and it's still refusing to log. bad bot!
* 10:00 apergos: was running some checks on ms5 to see how much gain we were really getting from thumb cleanup, please ignore the little spikes in the graph kthxbye
* 07:01 apergos: thumb cleaner started up again
== December 2 ==
* 20:04 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
* 19:45 apergos: well it was a nice thought but no. suspending til tomorrowmorning
* 19:41 apergos: sneakig in a few more thumb deletions....
* 19:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 're-enabled incubator extension'
* 18:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Temporarily disabling incubator extension on incubator per SPQRobin'
* 16:59 cmjohnson1: disk 11 on storage3 replaced
* 16:13 apergos: suspending again
* 15:50 apergos: and restarting...
* 15:29 apergos: suspending thumb cleaner for a bit again
* 15:17 mutante: sq39 - installing latest kernel and upgrades as a test before upgrading more squids again
* 14:47 mutante: rebooting bayes to finish upgrade
* 14:14 mutante: starting to dist-upgrade bayes (after metrics meeting is over per ezachte)
* 11:03 apergos: er... after suspending the thumbs cleaner about 8 minutes ago :-D
* 11:02 apergos: starting up again
* 08:08 apergos: starting up the thumbs cleaner again, keeping an eye on the stats
* 03:17 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
* 02:20 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
* 02:10 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
* 02:08 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
* 02:04 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
* 01:57 logmsgbot: LocalisationUpdate failed
* 01:08 maplebed: kicked apache on sockpuppet again (puppet was hung again)
* 00:30 logmsgbot: awjrichards synchronized php/resources/startup.js 'touch'
* 00:30 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/modules/ext.fundraiserstatistics.js 'touch'
* 00:16 logmsgbot: awjrichards synchronized php/resources/startup.js 'touch'
== December 1 ==
* 23:53 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/FundraiserStatistics_body.php 'r104921'
* 23:53 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/ContributionTrackingStatistics_body.php 'r104921'
* 23:53 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/ContributionReporting.php 'r104921'
* 23:11 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
* 23:02 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend'
* 21:39 maplebed: restarted apache on sockpuppet at 18:25UTC to repair puppet
* 20:04 apergos: suspending thubm cleaner for a while and watching
* 19:58 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/ContributionStatistics_body.php 'r104890'
* 19:57 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/ContributionHistory_body.php 'r104890'
* 19:57 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/FundraiserStatistics_body.php 'r104890'
* 19:57 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/ContributionTrackingStatistics_body.php 'r104890'
* 19:56 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/ContributionReporting.php 'r104890'
* 19:31 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Un-disabling ContributionReporting special pages on officewiki'
* 19:30 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enabling ContributionReporting extension on officewiki'
* 17:51 ^demon|away: formey: dumping public svn to /svnroot in screen
* 16:28 apergos: cleaning up unused thumbs, slowly, on ms5: running as root in screen session there, from /root directory the command is zcat dirs-no-slashes.gz | python removeThumbDirs.py
* 13:29 RoanKattouw: Updated hume:/etc/cron.d/mw-update-special-pages to refer to /usr/local/bin rather than /home/wikipedia/bin . Now running the special-pages-small script by hand. Turns out the lock file creation wasn't needed after all
* 13:27 mutante: williams - upgrading facter,puppet,php5,.. from wmf repo
* 13:12 logmsgbot: tstarling synchronized php-1.18/extensions/VipsScaler/VipsScaler_body.php 
* 13:12 logmsgbot: tstarling synchronized php-1.18/extensions/VipsScaler/modules/ext.vipsScaler/ext.vipsScaler.js 
* 13:08 logmsgbot: catrope synchronized php-1.18/includes/parser/ParserCache.php 'Respect $wgEnableBug31576Debugging'
* 13:08 logmsgbot: catrope synchronized php-1.18/includes/job/RefreshLinksJob.php 'Respect $wgEnableBug31576Debugging'
* 13:08 logmsgbot: catrope synchronized php-1.18/includes/LinksUpdate.php 'Respect $wgEnableBug31576Debugging'
* 13:07 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add $wgEnableBug31576Debugging and set it to false'
* 11:24 Tim: doing some vips tests on srv224
* 11:23 mutante: done - OTRS is back
* 11:18 mutante: rebooting williams to finish upgrade
* 11:00 mutante: starting to dist-upgrade williams, expect some OTRS downtime (scheduled maintenance)
* 02:30 AaronSchulz: Nuked rows from moodbar_feedback & moodbar_feedback_response tables with user ID 15825391 per erik
* 02:29 AaronSchulz: Added user block status checks to MoodBar APIs
* 02:17 logmsgbot: aaron synchronized php-1.18/extensions/MoodBar/ApiMoodBar.php 
* 02:16 logmsgbot: aaron synchronized php-1.18/extensions/MoodBar/ApiFeedbackDashboardResponse.php 
* 02:14 logmsgbot: aaron synchronized php-1.18/extensions/MoodBar 
* 01:57 logmsgbot: LocalisationUpdate failed
* 00:26 logmsgbot: reedy synchronized php-1.18/includes/User.php 'r104808'
== November 30 ==
* 23:22 logmsgbot: catrope synchronized php-1.18/includes/SkinTemplate.php 'r104783'
* 23:15 logmsgbot: catrope synchronized php-1.18/resources/startup.js 'touch'
* 23:11 logmsgbot: catrope synchronized php-1.18/extensions/MoodBar/modules/ext.moodBar.dashboard/ext.moodBar.dashboard.js 'r104777'
* 23:10 logmsgbot: catrope synchronized php-1.18/extensions/MoodBar/MoodBar.php 'r104777'
* 23:10 logmsgbot: catrope synchronized php-1.18/extensions/MoodBar/MoodBar.i18n.php 'r104777'
* 22:28 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying MoodBar changes
* 22:08 LeslieCarr: powering down asw-b3-eqiad
* 21:27 logmsgbot: reedy synchronized php-1.18/resources/mediawiki/mediawiki.js 'r104754'
* 20:40 mutante: updated the theme of the wikimedia blog per guillaume RT:2033
* 20:27 RobH: updating dns with new servers in b3-eqiad mgmt range
* 20:25 logmsgbot: reedy synchronized php-1.18/extensions/ApiSandbox/ApiSandbox.php 'r104738'
* 20:17 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Disable CustomUserSignup on mediawiki.org'
* 20:11 mutante: started lighttpd on streber (not automatic yet?) - RT is back
* 20:05 mutante: streber - dpkg --configure -a | apt-get dist-upgrade |Setting up rt3.8-db-mysql
* 20:02 logmsgbot: reedy synchronized php-1.18/includes/api/ApiParamInfo.php 'r104731'
* 20:02 logmsgbot: reedy synchronized php-1.18/includes/api/ApiBase.php 'r104731'
* 20:01 mutante: rebooting streber (because RT was broken due to dpkg being stuck due to kernel bug)
* 20:01 logmsgbot: reedy synchronized php-1.18/includes/AutoLoader.php 'r104716'
* 20:00 logmsgbot: reedy synchronized php-1.18/maintenance/Maintenance.php 'r104716'
* 19:49 logmsgbot: reedy synchronizing Wikimedia installation... : Updated ExtensionMessages with Vips and ApiSandbox
* 19:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable ApiSandbox everywhere'
* 19:44 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'resync before apisandbox launch'
* 19:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'resync before apisandbox launch'
* 19:37 K4-713: synchronized payments cluster to r104724
* 19:33 logmsgbot: catrope synchronized php-1.18/includes/LinksUpdate.php 'add wiki name to debugging output'
* 19:23 logmsgbot: catrope synchronized php-1.18/includes/job/RefreshLinksJob.php 'r104722'
* 18:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Group tweak for bug 32637'
* 17:46 mutante: applied hexmode patch to bugzilla RT:2061
* 17:12 RoanKattouw: NFS is hung, expect it to stay that way for at least 15 mins
* 17:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Adding basic wmgUseApiSandbox'
* 17:01 logmsgbot: catrope synchronized php-1.18/includes/LinksUpdate.php 'r104701'
* 17:01 mutante: hume - installed mysql client 5.1, sym link of wikidiff2.so to php_wikidiff2.so in /usr/lib/php5/20090626, changed # to // in /etc/php5/cli/conf.d/fss.ini, changed variables_order in /etc/php5/cli/php.ini to be EGPCS
* 16:57 mutante: applied live hacks apergos did on fenari also on hume (see above), removed php5-wikidiff2 (php-wikidiff2 stays), touch /etc/cluster
* 16:13 RobH: db1025 hdd replaced
* 16:08 RobH: swapping failed drive on db1025 per rt2047
* 16:03 logmsgbot: reedy synchronizing Wikimedia installation... : Pushing ApiSandbox out to the cluster, not enabled anywhere yet
* 15:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32500 - Defining transwiki sources for sa.wikipedia.org'
* 15:49 logmsgbot: catrope synchronized wmf-config/liquidthreads.php 'Fix notice about wmgLQTUserControlNamespaces being undefined'
* 15:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Group tweak for bug 32637'
* 15:37 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Adding wmgBabelUseUserLanguage and making wmgBabelDefaultLevel from InitialiseSettings be respected'
* 15:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Creating wmgBabelUseUserLanguage'
* 14:24 logmsgbot: reedy synchronized php-1.18/extensions/VipsScaler/modules/ext.vipsScaler/ext.vipsScaler.js 'r104663'
* 14:15 cmjohnson1: shutting down ms3
* 13:48 RobH: software raid rebuilding on singer
* 13:45 RobH: singer back up, racktables, outreachcivi and other services back online
* 13:36 cmjohnson1: shutdown singer to replace hard drive
* 13:33 mark: Fixed puppet on the karmic search servers (search13..20)
* 13:27 mark: Imported puppet 2.7.6+2.7.7rc2-1wm1~karmic1 packages into the karmic-wikimedia APT repository
* 13:26 RobH: singer coming down for hard disk replacement. racktables, outreachcivi, and other services will be offline for a short time.
* 12:37 mutante: magnesium - rebooted to finish upgrade, started swift
* 12:22 mutante: torrus threw 500 errors - fixed it by following "deadlock problem" instructions on wikitech
* 11:21 mutante: copper - rebooted to finish upgraded, started swift
* 11:17 mark: Imported puppet 2.7.6+2.7.7rc2-1wm1~hardy1 packages into the hardy-wikimedia APT repository
* 11:16 mutante: copper - installing new kernel and apt,apache2,bind9-host,.. upgrades
* 10:47 mutante: zinc - back up, running swift-init all start
* 10:45 mutante: zinc - installed new kernel,rebooting..
* 10:02 mutante: done, bugzilla back up
* 09:56 mutante: installing apache2,php-wikidiff2,kernel,.. upgrades on kaulen
* 09:50 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 
* 09:43 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 
* 02:07 logmsgbot: tstarling ran sync-common-all 
* 01:57 logmsgbot: LocalisationUpdate failed
* 01:21 binasher: added prod eqiad bits caches to dsh varnish group
* 01:17 Tim: running sync-common on srv260, srv280, srv284
* 01:16 Tim: rebooting srv284, ssh not working
* 01:13 binasher: deployed varnish changes for bits+mobile - serve slightly stale content if backend down + retry backend 503's 4 times
* 01:05 Tim: rebooting srv162, ssh not working
* 01:01 Tim: rebooting srv280, no working ssh
* 00:58 Tim: rebooting srv260, hangs on ssh connections
* 00:43 logmsgbot: tstarling synchronized php-1.18/extensions/VipsScaler/SpecialVipsTest.php 
* 00:42 logmsgbot: tstarling synchronized php-1.18/extensions/VipsScaler/SpecialVipsTest.php 
* 00:39 logmsgbot: tstarling synchronized php-1.18/extensions/VipsScaler/VipsTest.php 
* 00:28 logmsgbot: tstarling synchronized php-1.18/includes/StreamFile.php 'debugging hack'
* 00:21 logmsgbot: tstarling synchronized php-1.18/extensions/VipsScaler/SpecialVipsTest.php 
* 00:19 logmsgbot: tstarling synchronized php-1.18/extensions/VipsScaler/VipsScaler_body.php 
* 00:10 logmsgbot: tstarling synchronized php-1.18/extensions/VipsScaler/SpecialVipsTest.php 
* 00:09 logmsgbot: tstarling synchronized php-1.18/extensions/VipsScaler/VipsTest.php 
== November 29 ==
* 23:34 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'enabling VipsTest on test2wiki instead'
* 22:53 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'enabling VipsTest on testwiki'
* 22:53 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'enabling VipsTest on testwiki'
* 22:35 logmsgbot: tstarling synchronizing Wikimedia installation... : 
* 22:30 Tim: svn up/scap for VipsScaler pre-deployment changes
* 21:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
* 21:53 logmsgbot: reedy synchronized flaggedrevs.dblist 'Bug 29742 - Enable Extension:FlaggedRevs for fiwiki (fi.wikipedia.org)'
* 21:52 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php 'Bug 29742 - Enable Extension:FlaggedRevs for fiwiki (fi.wikipedia.org)'
* 21:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
* 21:48 logmsgbot: reedy synchronized flaggedrevs.dblist 'Bug 26322 - Requesting FlaggedRevs be enabled on ru.wikinews'
* 21:47 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php 'Bug 26322 - Requesting FlaggedRevs be enabled on ru.wikinews'
* 21:43 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php 'Bug 29982 - Enable FlaggedRevs extension on pt.wikibooks.org'
* 21:40 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php 'Bug 29982 - Enable FlaggedRevs extension on pt.wikibooks.org'
* 21:39 logmsgbot: reedy synchronized flaggedrevs.dblist 'Bug 29982 - Enable FlaggedRevs extension on pt.wikibooks.org'
* 21:38 logmsgbot: reedy synchronized flaggedrevs.dblist 'Bug 29982 - Enable FlaggedRevs extension on pt.wikibooks.org'
* 21:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
* 21:37 logmsgbot: reedy synchronized flaggedrevs.dblist 'Bug 29982 - Enable FlaggedRevs extension on pt.wikibooks.org'
* 21:36 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php 'Bug 29982 - Enable FlaggedRevs extension on pt.wikibooks.org'
* 21:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
* 21:30 logmsgbot: reedy synchronized flaggedrevs.dblist 'Bug 28837 - Install FlaggedRevs on VEC Wikipedia'
* 21:29 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php 'Bug 28837 - Install FlaggedRevs on VEC Wikipedia'
* 21:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'touch'
* 21:17 logmsgbot: reedy synchronized flaggedrevs.dblist 'Bug 32591 - Enable Flagged Revisions on Latin Wikisource'
* 21:11 Reedy: Creating flagged revs tables on ruwikinews
* 21:10 Reedy: Creating flagged revs tables on siwiki
* 21:10 Reedy: Creating flagged revs tables on bawiki
* 21:10 Reedy: Creating flagged revs tables on elwikinews
* 21:09 Reedy: Creating flagged revs tables on vecwiki
* 21:09 Reedy: Creating flagged revs tables on fiwiki
* 21:08 Reedy: Creating flagged revs tables on ptwikibooks
* 21:08 Reedy: Creating flagged revs tables on lawikisource
* 20:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32637 - Add Some user group to hindi wikipedia'
* 20:42 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Actually re-enabling ContributionReporting for test, donate, foundation, and meta wikis'
* 20:36 mark: Puppet is currently failing on sanger because it wants to downgrade puppet; I'll fix that tomorrow
* 20:36 logmsgbot: awjrichards synchronizing Wikimedia installation... : Pushing ContributionReporting changes to display a friendlier 'this page temporarily disabled' message
* 19:56 cmjohnson1: msw-c2-sdtpa replacement complete
* 19:50 mark: Upgraded sanger (hardy) to experimental snapshot release of Puppet 2.7.7rc2
* 19:26 cmjohnson1: replacing msw-c2-sdtpa
* 18:55 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32636 - Change UploadNavigationUrl to new link'
* 18:55 notpeter: restarting apache on formey
* 17:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32642 - Enable e-mail notifications for watchlist (EnotifWatchlist) on WMRU wiki'
* 15:26 RoanKattouw: Creating /var/lock/update-special-pages and /var/lock/update-special-pages-small on hume, as empty files owned by apache. Without these lock files, the update-special-pages cron jobs fail immediately
* 13:03 mark: Fixed broken package situation on ms1002
* 13:01 mark: Ran dist-upgrade on streber
* 11:45 logmsgbot: hashar synchronized wmf-config/codereview.php 'CodeReview: auto defer /trunk/extensions/AddThis an extension to add G+, facebook, twitter links to "like" stuff'
* 11:18 mark: Running reboot on streber, stuck on disk
* 11:07 mark: Running apt-get dist-upgrade && reboot on sockpuppet
* 09:47 mutante: transcode1 - installing security updates, new kernel, rebooting
* 09:03 mutante: same on zinc, upgrade apt itself and some libs
* 09:00 mutante: started ntpd and installed some upgrades on magnesium
* 01:57 logmsgbot: LocalisationUpdate failed
== November 28 ==
* 21:24 logmsgbot: reedy synchronizing Wikimedia installation... : Pushing r104486, r104490
* 21:24 mark: Letting Puppet upgrade itself to 2.7.7rc2 on all Lucid servers
* 20:55 Jeff_Green: cycling through payments servers with dist-upgrade/reboot
* 20:20 mutante: installing apt security updates on hooper,spence,fenari,hume
* 20:12 mutante: fenari - removed php5-wikidiff, autoremove unused packages, clean apt cache..
* 20:06 logmsgbot: reedy synchronized php-1.18/includes/api/ApiFormatXml.php 'r104476'
* 19:53 mutante: APT pinned php5-pgsql on gallium - fixed Depends: php5-common (= 5.3.2-2wm1)
* 18:43 logmsgbot: hashar: jenkins backup. The host was missing the SSL cert and apache did not start up. Fixed by Daniel Zahn https://gerrit.wikimedia.org/r/1156
* 17:43 mutante: gallium: installing new kernel and rebooting
* 17:42 logmsgbot: hashar: wrongly stopped jenkins while mutante was doing an upgrade. Jenkins process dead for some minutes :(
* 17:40 logmsgbot: hashar: restarting jenkins
* 17:15 mutante: gallium - installing some upgrades: apache2..,apt,bind9/dnsutils,,php5..,openjdk,puppet..RT 2039
* 16:38 logmsgbot: reedy synchronizing Wikimedia installation... : Pushing r104448
* 16:37 mutante: fenari: re-installed parsekit via pecl, added extension=parsekit.so to php5/cli/php.ini
* 15:22 RoanKattouw: Started logmsgbot on fenari, now running as me. Need to puppetize it with ircecho now that fenari has been upgraded to lucid
* 15:21 logmsgbot: reedy synchronized php-1.18/includes/api/ApiHelp.php 'r104439'
* 15:21 Reedy: sync-file php-1.18/includes/api/ApiMain.php r104439
== November 27 ==
* 23:24 Reedy: Disabled ContributionReportings extension (about 15 minutes ago)
* 22:47 mark: Restarted ganglia on spence
* 22:41 notpeter: restarting nagios on spence
* 22:24 mark: Shutdown mysql on db1025
== November 26 ==
* 22:21 mark: Upgraded puppet client to 2.7.7rc2 on formey (expecting lint breakage)
* 18:25 mark: Let puppet upgrade puppet master and puppet agent to 2.7.7rc2 on sockpuppet
== November 25 ==
* 22:45 apergos: live hacked a few things on fenari so people can use it: installed mysql client 5.1, sym link of wikidiff2.so to php_wikidiff2.so in /usr/lib/php5/20090626, changed # to // in /etc/php5/cli/conf.d/fss.ini, changed variables_order in /etc/php5/cli/php.ini to be EGPCS... this should all be puppetized for bastion hosts
* 18:10 mutante: secure.wm is up again, thanks Rob. (was disk fail. RT to replace hardware created)
* 16:59 mutante: singer down :/ - http://wikitech.wikimedia.org/view/Singer#fail - secure.wm and others affected
* 16:07 mutante: power cycling singer
* 15:48 mutante: restarting singer to finish upgrade
* 15:21 mutante: starting dist-upgrade on singer
* 15:12 mutante: removing rpm and alien package from fenari
* 14:47 mutante: hooper - done, Blog and Etherpad accessible again
* 14:44 mutante: hooper - rebooting for upgrade
* 14:41 mutante: hooper - installing new kernel,apache2 upgrades,dist-upgrade (within lucid)
* 13:41 mutante: rebooting hume to finish upgrade
* 12:57 mutante: starting to dist-upgrade hume
* 12:33 mutante: stopping nagios and rebooting spence to finish kernel upgrade
* 12:26 mutante: reinstalled wikimedia-task-appserver on fenari - fixed dpkg conflict php5-apc vs php-apc by manually installing php-apc first
* 12:23 mutante: dist-upgrading spence (within same release)
* 12:23 mutante: removing php5-apc from spence
* 12:09 mark: Fixed puppet in labs
* 11:36 mutante: fenari back up as lucid :) feel free to use again
* 11:32 mutante: side-note: sh: /home/wikipedia/bin/start-ircbot: not found
* 11:29 mutante: rebooting fenari to finish upgrade
* 10:59 mutante: starting fenari dist-upgrade
* 09:38 mark: whipped aluminium back into submission by killing rfaulk's python processes
* 09:27 apergos: email notification was off momentarily for the second and last batch of bugzilla cleanup. now back on.
* 07:50 apergos: disabled user account creation again on bugzilla for the moment
* 01:57 logmsgbot: LocalisationUpdate failed
* 01:52 Tim: created arbcom 2011 vote on enwiki
* 01:47 Tim: on hume: running makeArbcomList.php in tstarling screen
* 00:06 rainman-sr: set the "last indexed" dates back by 2 days for all wikis so that the wrongly indexed pages get reindexes properly
== November 24 ==
* 20:44 mark: Puppet is broken in labs
* 19:32 mark: Letting puppet upgrade itself on virt1
* 19:04 robla: new account creation on BZ reenabled
* 18:59 apergos: added manually the editbugs right for all users on bugzilla; automatic right via regexp removed by robla 
* 18:12 mark: Temporarily disabled puppet on sockpuppet and virt1
* 18:02 mark: Installed new Puppet 2.7.7rc2 packages into the Wikimedia APT repository (lucid-wikimedia); hardy packages will follow if needed
* 13:50 logmsgbot: catrope synchronized php-1.18/extensions/GlobalUsage/GlobalUsage_body.php 'r104163'
* 11:50 rainman-sr: restarting search indexing on searchidx2 as Bug 32376 is resolved
* 11:47 mutante: activated use_large_installation_tweaks on nagios and restarting
* 10:46 logmsgbot: nikerabbit synchronized php-1.18/resources/startup.js 'Touching startup.js for Narayam'
* 10:42 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Bug 32619 - Narayam on Commons'
* 10:06 apergos: restarted search indexer on searchidx2 (should have also restarted incremental indexer), now that bug 32376 is resolved
* 09:34 logmsgbot: catrope synchronized php-1.18/includes/Export.php 'r104126'
* 09:23 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'bug 31745 disable wgUseRCPatrol on ruwikisource'
* 01:57 logmsgbot: LocalisationUpdate failed
* 00:16 rainman-sr: stopped search indexing on searchidx2 until bug 32376 is resolved to prevent more pages being wrongly indexed
* 00:14 logmsgbot: reedy synchronized php-1.18/extensions/cldr/LanguageNamesCs.php 'Sync for Danny B'
== November 23 ==
* 23:43 Tim: restarted apaches to get new wmerrors
* 23:22 K4-713: synchronized payments cluster to r104100
* 21:34 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying MoodBar updates
* 21:26 logmsgbot: reedy synchronizing Wikimedia installation... : Pushing fr frontend symlink
* 21:21 RoanKattouw: Adding new moodbar_feedback_response table on all wikis
* 21:08 logmsgbot: hashar: Created symlink in php-1.18/extensions/FlaggedRevs for frontend pointing to presentation. Should fix localisation update from twn which use trunk. See r102741
* 20:08 logmsgbot: catrope synchronized php-1.18/resources/jquery/jquery.localize.js 'r104079'
* 19:22 logmsgbot: midom synchronized php/extensions/PrefSwitch/PrefSwitch.classes.php 
* 19:08 domas: added missing columns for PrefSwitch survey on few wikis, was broken for a .. year
* 18:03 logmsgbot: catrope synchronized php-1.18/includes/parser/ParserCache.php 'r104059'
* 17:54 Ryan_Lane: downgrading mysql packages on virt*
* 16:23 Ryan_Lane: shuting down db19
* 16:20 logmsgbot: catrope synchronized php-1.18/extensions/GlobalUsage/GlobalUsageHooks.php 'fix fatal'
* 16:17 logmsgbot: catrope synchronized php-1.18/extensions/GlobalUsage/GlobalUsageHooks.php 'Prospective fix for GlobalUsage OOM issue'
* 16:10 logmsgbot: laner synchronized wmf-config/db.php 'Removing db19, for maintenance'
* 15:56 Ryan_Lane: nova upgrade is finished. there are still a few issues
* 15:36 RoanKattouw: Reason I did that was that I was seeing error messages like proc_open(): fork failed - Cannot allocate memory in /usr/local/apache/common-local/php-1.18/includes/parser/Tidy.php on line 174
* 15:35 RoanKattouw: and on srv256 too
* 15:32 RoanKattouw: Killing some memory-hogging sleeping Apache procs on srv253
* 12:38 mutante: synced InitialiseSettings to set wgBlockDisablesLogin to true for foundationwiki (RT 690)
* 12:37 logmsgbot: root synchronized wmf-config/InitialiseSettings.php 
* 11:51 mark: Creating hardy pbuilder environment on ragweed
* 01:57 logmsgbot: LocalisationUpdate failed
* 01:51 logmsgbot: awjrichards synchronizing Wikimedia installation... : 
* 01:33 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/ContributionStatistics_body.php 'r103999'
* 01:33 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/ContributionTotal_body.php 'r103999'
* 01:33 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/FundraiserStatistics_body.php 'r103999'
* 01:32 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/ContributionReporting.php 'r103999'
* 01:27 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/FundraiserStatistics_body.php 'r103998'
* 01:26 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/ContributionTotal_body.php 'r103998'
* 01:26 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/ContributionStatistics_body.php 'r103998'
* 01:25 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/ContributionReporting.php 'r103998'
* 01:25 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/DailyTotal_body.php 'r103998'
* 01:23 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/YearlyTotal_body.php 'r103998'
* 01:14 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enabling ContributionReporting on metawiki'
* 01:05 K4-713: synchronized payments cluster to r103989
* 00:12 Ryan_Lane: starting gerrit
* 00:04 Ryan_Lane: shutting down gerrit
== November 22 ==
* 23:04 Tim: on kaulen: reverted the firewall rule and createaccount.cgi deny rule from yesterday, replaced the latter by temporarily disabling account creation with BZ "createemailregexp"
* 22:15 Ryan_Lane: started gerrit back up
* 22:10 Ryan_Lane: temporarily shutting down labsconsole
* 22:10 Ryan_Lane: temporarily stopping gerrit
* 21:45 logmsgbot: catrope synchronized php-1.18/resources/mediawiki/mediawiki.util.js 'r103962'
* 21:33 Ryan_Lane: upgrading the nova database
* 21:31 logmsgbot: reedy synchronizing Wikimedia installation... : Stylize. AaronSchulz made me do it
* 21:29 Ryan_Lane: upgrading packages on virt4
* 21:28 Ryan_Lane: upgrading packages on virt3
* 21:26 Ryan_Lane: upgrading packages on virt2
* 21:25 Ryan_Lane: upgrading packages on virt1
* 21:17 Ryan_Lane: stopping all nova services
* 21:10 logmsgbot: reedy synchronized wmf-config/abusefilter.php 'Remove lines of sites setting abusefilter-private to anything'
* 20:18 logmsgbot: catrope synchronized php-1.18/includes/Export.php 'r103954'
* 20:04 Ryan_Lane: upgrading nova from cactus to diablo. during this time period, instances may be occasionally inaccessible, and labsconsole may also be occasionally inaccessible
* 19:45 binasher: increased udp recv buffer on locke to 512M. i think lp-filter needs some work.
* 19:13 binasher: udp2log on emery was broken post upgrade due to files under /var/log/squid being owned by human users. fixed + restarted
* 18:03 logmsgbot: midom synchronized wmf-config/db.php 'readding db19'
* 17:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32521 - Request for new name space on the Japanese Wiktionary'
* 17:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32553 - Rollbacker user group on sk.wikipedia.org'
* 17:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32547 - Add pagetitle-view-mainpage to for zhwiki'
* 17:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32546 - Cleanup interface_editor/interface_editors inconsistency'
* 17:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32587 - Using localized section headings for uploads to Commons through upload-form'
* 17:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32587 - Using localized section headings for uploads to Commons through upload-form'
* 15:35 mark: Ran apt-get dist-upgrade && halt on db17
* 12:34 mutante: sq38 - broken - giving up on it as it now hangs after RAC detection however you boot it
* 12:05 mutante: sq38 - Attempting PXE boot to reinstall (RT 2017)
* 06:40 Ryan_Lane: added a default sudo policy for labs instances. project members automatically have sudo privileges, excluding specific projects.
* 05:03 Tim: on bugzilla: allow reopened -> new transition to make dealing with malicious mass bug closures easier
* 02:54 Tim: fixed resolution and status fields of the rocketmail bugs
* 02:25 logmsgbot: LocalisationUpdate completed (1.18) at Tue Nov 22 02:28:06 UTC 2011
* 01:57 logmsgbot: LocalisationUpdate failed
* 01:57 K4-713: synchronized payments1, 2, and 3 to r103880
* 01:30 awjr: synchronizing thank_you module in CiviCRM on Grosley and Aluminium to r848
* 01:04 notpeter: restzaring apache on formey
* 00:59 awjr: synchronizing thank_you module in CiviCRM on Grosley and Aluminium to r847
* 00:35 Tim: blocked all account creation on bugzilla due to tor abuse
* 00:30 logmsgbot: asher synchronized wmf-config/CommonSettings.php 'changed wgUDPProfilerHost to professor.pmtpa.wmnet'
* 00:24 K4-713: Synchronized payments cluster to r103869
* 00:23 domas: nfs home is fucked up, 1.5s response times on simple ops
* 00:21 logmsgbot: midom synchronized wmf-config/db.php 'taking out db19, was out for past week or so'
* 00:20 Tim: on kaulen: firewalled 46.165.196.182 due to bugzilla trolling
* 00:12 notpeter: upgrading udp2log on emery to udplog_1.8-2 and restarting udp2log
== November 21 ==
* 23:35 logmsgbot: awjrichards synchronized php/extensions/FundraiserLandingPage/FundraiserLandingPage.body.php 'r103849'
* 23:34 logmsgbot: awjrichards synchronized php/extensions/FundraiserLandingPage/FundraiserLandingPage.php 'r103849'
* 23:00 LeslieCarr: updating dns with bast1001
* 22:25 binasher: mediawiki sync scripts under fenari:/usr/local/bin now log deploys to graphite 
* 22:14 LeslieCarr: taking scandium for new bastion host
* 21:39 LeslieCarr: repreffing the tinet route
* 20:01 Ryan_Lane: gerrit is back up
* 19:57 Ryan_Lane: bringing gerrit down again, to move all of /var/lib/gerrit2 to the partition
* 19:49 notpeter: upgrading apache version on stafford to fix sec vuln
* 18:27 mutante: changed default text for new etherpads (bz #29496)
* 17:36 Ryan_Lane: bringing gerrit down, to move its data to a new partition
* 10:42 apergos: shutting down snapshot3 for move to internal network and OS upgrade
* 02:21 logmsgbot: LocalisationUpdate completed (1.18) at Mon Nov 21 02:24:05 UTC 2011
* 01:57 logmsgbot: LocalisationUpdate failed
== November 20 ==
* 17:06 Reedy: Packetloss on emery is averaging 10.74%
* 16:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32435 - Rename namespace and enable on pl.wikinews'
* 16:43 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32488 - on the Thai Wikipedia (th.wikipedia.org)'
* 16:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32491 - Enabling bot admin undeletion on hiwiki'
* 16:32 logmsgbot: reedy synchronized php-1.18/resources/startup.js 'touch'
* 16:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Revert'
* 16:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32491 - Enabling bot admin undeletion on hiwiki'
* 16:24 logmsgbot: reedy synchronized php-1.18/resources/startup.js 'touch'
* 16:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32503 - ArticleFeedback extension on sr.wikipedia'
* 16:21 Reedy: Created articlefeedback tables on srwiki
* 16:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable clicktracking on srwiki for AF'
* 15:58 mutante: sq38 - powered up,but does not allow ssh nor login via mgmt
* 02:23 logmsgbot: LocalisationUpdate completed (1.18) at Sun Nov 20 02:26:06 UTC 2011
* 01:57 logmsgbot: LocalisationUpdate failed
== November 19 ==
* 16:29 logmsgbot: catrope synchronized php-1.18/includes/resourceloader/ResourceLoader.php 'r103687'
* 14:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable title restrictions'
* 14:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable title restrictions'
* 13:51 logmsgbot: catrope synchronized php-1.18/extensions/DoubleWiki/DoubleWiki_body.php 'r103674'
* 13:40 logmsgbot: reedy synchronized php-1.18/extensions/CentralAuth/CentralAuthUser.php 'r103571'
* 11:04 logmsgbot: catrope synchronized php-1.18/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js 'r103459, forgot to sync earlier'
* 11:02 logmsgbot: catrope synchronized php-1.18/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.startup.js 'r103459, forgot to sync earlier'
* 02:21 logmsgbot: LocalisationUpdate completed (1.18) at Sat Nov 19 02:24:04 UTC 2011
* 01:57 logmsgbot: LocalisationUpdate failed
* 01:33 K4-713: synchronized payments cluster to r103630
== November 18 ==
* 23:43 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/FundraiserStatistics_body.php 'r103627'
* 23:43 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/ContributionTrackingStatistics_body.php 'r103627'
* 23:42 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/ContributionReporting.php 'r103627'
* 23:24 LeslieCarr: restarting apache2 on srv288
* 23:23 LeslieCarr: restarted sshd on search14
* 22:59 LeslieCarr: rebooting srv288
* 22:12 Ryan_Lane: rebooting instances on virt3 to bring them back up
* 19:15 LeslieCarr: deactivating as-path TiNet-Telia-esams "3257+ 1299+ 43821+ .*"; on all routers to prefer xo path between US and esams
* 18:11 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 31745 - ruwikisource enable RC patrolling'
* 16:59 binasher: updated gmetad on spence to use db21 as a data source, quieting cron spam
* 11:11 apergos: that message is from much earlier but I forgot to hit "enter" somehow...
* 11:10 apergos: changed smokeping config on streber to check db40 instead of db20 in rack b2 sdtpa
* 02:20 logmsgbot: LocalisationUpdate completed (1.18) at Fri Nov 18 02:23:11 UTC 2011
* 01:57 logmsgbot: LocalisationUpdate failed
* 01:01 binasher: packaged graphite, added to wikimedia-lucid, installed on professor via puppet
* 00:43 logmsgbot: neilk synchronized php-1.18/extensions/MoodBar/FeedbackItem.php 'fix for Moodbar code which apparently cannot deal with empty comment records'
* 00:21 logmsgbot: neilk rebuilt wikiversions.cdb and synchronized wikiversions files: 
* 00:20 logmsgbot: neilk synchronizing Wikimedia installation... : Moodbar phase 1 bugfixes
== November 17 ==
* 23:43 K4-713: Synchronized payments cluster to r103513
* 23:25 notpeter: upgrading and restarting udp2log on emery
* 22:08 binasher: added db20 to decommissioning.pp, renamed server (professor) will be the new mediawiki profiling host
* 20:23 RobH: torrus deadlock, restarting on streber
* 18:27 cmjohnson1: virt13 power inadvertently removed
* 17:23 cmjohnson1: db45 power was removed during new install
* 09:27 mutante: started ActiveMQ on erzurumi (RT#1999)
* 07:43 awjr: activemq disabled on erzurumi
* 07:05 Tim: on fenari: added wmerrors UDP log receiver to /etc/rc.local as a temporary solution pending RT 623
* 06:48 Tim: put the new version of wmerrors into the apt repo
* 06:23 awjr: redisabled donations queue consumer in jenkins on aluminium
* 06:16 awjr: reenabled donations queue consumer in jenkins on aluminium
* 05:56 awjr: disabling donations queue consumer in jenkins on aluminium
* 02:25 logmsgbot: LocalisationUpdate completed (1.18) at Thu Nov 17 02:28:16 UTC 2011
== November 16 ==
* 21:45 cmjohnson1: sq86 shutdown
* 21:16 cmjohnson1: sq85 shutdown
* 20:55 cmjohnson1: shutdown sq84
* 20:26 cmjohnson1: shutdown sq83
* 19:48 Ryan_Lane: shut down sq82
* 19:39 maplebed: grew xfs on ms5 by 2T to get some breathing room
* 19:30 cmjohnson1: sq81 down for rack relocation
* 18:19 awjr: Increasing donatinos queue consumption script frequency to run every 2 mins rather than 3 mins via Jenkins on aluminium
* 16:26 mutante: fixed permissions on nagios external command file (RT #1994)
* 16:11 RobH: dns updated for db48-db58 (both mgmt and production)
* 15:39 mutante: srv180 - powered down, was offline with disk fail anyways, created RT 1993
* 13:36 mutante: argon - dpkg errors processing mysql-client pkg upgrade
* 13:33 mutante: argon - dist-upgrade/kernel,reboot (unused limesurvey box)
* 04:12 awjr: synchronizing payments cluster, r103293
* 03:18 logmsgbot: asher synchronized wmf-config/db.php 'adding db19 to s2, running new mysql build'
* 02:24 logmsgbot: LocalisationUpdate completed (1.18) at Wed Nov 16 02:27:02 UTC 2011
* 01:57 logmsgbot: LocalisationUpdate failed
* 01:48 ^demon: leaving a git svn test running in a screen on formey. Should be fine, but feel free to kill it if svn.wm.o explodes.
* 00:43 logmsgbot: asher synchronized wmf-config/db.php 'putting db11 (s3) into rotation with new mysql build'
* 00:27 Tim: on srv255: ran manual sync-common and puppetd -tv
* 00:19 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: 
* 00:18 Tim: rebooting srv255, has been in swapdeath for a week, no response on serial console
* 00:12 logmsgbot: tstarling synchronizing Wikimedia installation... : 
* 00:05 logmsgbot: tstarling synchronized php-1.18/README 'sync test'
* 00:01 binasher: rebooting db11 to new kernel (it's out of rotation)
== November 15 ==
* 23:50 awjr: synchronizing payments cluster to r103267
* 23:48 logmsgbot: neilk rebuilt wikiversions.cdb and synchronized wikiversions files: 
* 23:35 logmsgbot: neilk synchronizing Wikimedia installation... : renabling multi-file select for UploadWizard on Commons, quashing some bugs with UploadStash
* 23:31 maplebed: imported packages for swift to our repository: swift, swift-*, python-swift, python-eventlet, python-greenlet, python-webob
* 22:44 logmsgbot: reedy synchronizing Wikimedia installation... : 
* 22:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 
* 22:44 logmsgbot: reedy synchronizing Wikimedia installation... : 
* 22:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Test (no changes)'
* 22:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: 
* 22:08 logmsgbot: reedy synchronizing Wikimedia installation... : Pushing r103242 for mathsy updates
* 21:58 notpeter: can't pop a shell on searchidx2 via ssh or ipmi rebooting
* 21:12 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Comment out the account creation throttle hack for the Serbia event. Leaving it in as a comment so it can be used as a boilerplate for a next time'
* 19:39 awjr: updating frequency of donations queue consumption from every 5mins to 3mins in jenkins on aluminium
* 18:35 LeslieCarr: hostway transit moved
* 18:20 LeslieCarr: moving hostway transit from csw5-pmtpa to cr2-pmtpa
* 16:57 mark: shutting down sq80 for relocation
* 16:43 logmsgbot: reedy synchronized php-1.18/resources/jquery/jquery.textSelection.js 'r103192'
* 16:37 logmsgbot: reedy synchronized php-1.18/extensions/FlaggedRevs/presentation/FlaggedPageView.php 'r103191'
* 16:14 Jeff_Green: testing passive-checks config on nagios
* 15:48 mark: Converted eth0 to aggregated bond0 on sq69
* 15:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31842 - Change of namespace names in Old Church Slavonic Wikipedia'
* 14:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32119 - Change the diacritics in the namespaces at ro.wikinews'
* 14:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32145 - Namespaces/Aliases for Wikipedia, Project and Portal namespace in Assamese wikipedia'
* 14:31 mutante: commented cronjob on locke, user nobody, that shipped aggregations to dammit.lt and got connection refused
* 14:07 mark: Shutting down sq75-78 for relocation
* 14:05 mutante: cleared exim paniclog on mchenry,lily & williams - all just temp. spamd failures, mostly from October
* 13:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32340 - Translation of project namespace and sitename for tawikibooks'
* 13:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32357 - Alias for project space in ko.wikiquote'
* 13:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32370 - Enable Rollbacker, Eliminator and Interface_editors on jawiki and grant bureaucrats to add/remove these flags'
* 12:58 apergos: restarted lighttpd on dataset2
* 12:20 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 32418 - Enable Narayam on Incubator'
* 12:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32418 - Enable Narayam on Incubator'
* 12:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32418 - Enable Narayam on Incubator'
* 08:59 logmsgbot: ariel synchronized php-1.18/includes/Export.php 'export of revisions by range, code cleanups'
* 08:23 logmsgbot: ariel synchronized php-1.18/maintenance/dumpTextPass.php 'minor code cleanups'
* 08:18 logmsgbot: ariel synchronized php-1.18/maintenance/dumpBackup.php 'export of revisions by range'
* 08:17 logmsgbot: ariel synchronized php-1.18/maintenance/backup.inc 'export of revisions by range'
* 07:50 apergos: reboot of snapshot2 after kernel update
* 07:39 apergos: reboot of snapshot1 after kernel update
* 07:32 apergos: reboot of dataset2 after kernel update
* 02:51 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/FundraiserStatistics.css 'r103126'
* 02:51 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/ContributionReporting.php 'r103126'
* 02:29 logmsgbot: LocalisationUpdate completed (1.18) at Tue Nov 15 02:32:17 UTC 2011
* 01:57 logmsgbot: LocalisationUpdate failed
* 01:41 awjr: synchronizing payments r103120
* 01:01 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/_search_webkit.html.php 'weekly update to mobile frontend'
* 00:59 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/_search_webkit.html.php 'weekly update to mobile frontend'
* 00:51 RobH: update dns per rt1756
* 00:46 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to mobile frontend'
* 00:45 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/_search_webkit.html.php 'weekly update to mobile frontend'
* 00:18 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Revert Neils changes'
* 00:12 logmsgbot: reedy synchronizing Wikimedia installation... : Pushing r103056
* 00:01 RobH: correction, mw1102, mw1103
* 00:00 RobH: updating firmware on mw1002 & mw1003
== November 14 ==
* 23:51 logmsgbot: ben synchronized wmf-config/db.php 'taking out ms boxes from cluster22, adding es4 so all ES traffic is only on new hosts.'
* 23:50 awjr: synchronizing payments cluster r103082
* 23:43 RobH: ocg3 drac flashed and repaired, installing os now.
* 23:43 maplebed: master switch complete - cluster22 is now writing to es3
* 23:34 logmsgbot: ben synchronized wmf-config/db.php 'making es3 the new master for cluster22'
* 23:32 maplebed: about to switch master for the external store cluster22 from ms3 to es3
* 23:06 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/ContributionReporting.php 'r103065'
* 23:01 Reedy: That wasBug 32417 - Enable MoodBar on Incubator
* 23:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 
* 21:29 ^demon: killed my svn dump from earlier, ate up all the free space on formey
* 21:03 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Enable REL1_18 in ED'
* 21:00 Ryan_Lane: upgraded raskin
* 20:21 ^demon: doing full svn dump in screen on formey, probably will take awhile.
* 18:32 preilly: emergency mobile frontend fix for broken lists 
* 18:31 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'weekly update to mobile frontend'
* 18:27 logmsgbot: nikerabbit synchronized php-1.18/tests/phpunit/languages/LanguageMlTest.php 'r102998 - i18n deploy'
* 18:27 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/stylesheets/common.css 'weekly update to mobile frontend'
* 18:26 logmsgbot: nikerabbit synchronized php-1.18/extensions/CategoryTree/ 'r102998 - i18n deploy'
* 18:25 logmsgbot: nikerabbit synchronized php-1.18/extensions/Babel/ 'r102998 - i18n deploy'
* 18:25 logmsgbot: nikerabbit synchronized php-1.18/languages/ 'r102998 - i18n deploy'
* 17:18 logmsgbot: reedy synchronized php-1.18/extensions/Contest/specials/SpecialContest.php 'r102992'
* 17:14 logmsgbot: reedy synchronized php-1.18/extensions/Contest/Contest.i18n.php 'r102992'
* 16:41 mark: Shutting down sq69 and sq70 for relocation
* 14:56 mark: Migrated sq70 to varnish3; running apt-get dist-upgrade && reboot
* 14:53 mark: Migrated sq69 to varnish3; running apt-get dist-upgrade && reboot
* 14:39 mark: Migrated sq68 to varnish3; running apt-get dist-upgrade && reboot
* 14:29 mark: Migrated sq67 to varnish3; running apt-get dist-upgrade && reboot
* 14:08 mark: Temporarily disabled pmtpa as esams bits backend - preparation for pmtpa varnish upgrade
* 14:04 logmsgbot: catrope synchronized php-1.18/extensions/WikiEditor/modules/jquery.wikiEditor.toolbar.config.js 'r102982'
* 14:04 RoanKattouw: Strike that, it is responding to SSH because neither the connection nor the setup time out. It's just terribly, terribly slow
* 14:01 RoanKattouw: searchidx2 is dead, not responding to SSH. Ganglia shows elevated load
* 13:57 mark: Set all esams text amssq (SSD) squids to CARP weight 30
* 13:46 mark: Depooled knsq1 and knsq2 from esams bits duty, preparing for decommissioning
* 13:26 mark: Ran dist-upgrade and reboot on cp3001 and cp3002
* 13:26 mark: Setup interface aggregates on cp3001 and cp3002
* 12:04 mark: Deploying bits.eqiad
* 11:58 mark: bits-lb.esams record addition wasn't picked up by pdns on reload during the previous update, quickly moving bits-lb back to bits.esams
* 11:53 mark: Added bits-lb.esams.wikimedia.org DNS record, and updated geodns to use that instead of bits.esams
* 11:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31459 - Gender in user namespaces for Portuguese Wikipedia'
* 11:39 logmsgbot: mark synchronized wmf-config/CommonSettings.php 'remove old Squids'
* 11:30 mark: Renamed ganglia clusters/dirs on streber
* 11:22 mark: Renamed ganglia RRD dirs on spence
* 10:34 logmsgbot: catrope synchronized php-1.18/extensions/CentralNotice/special/SpecialBannerController.php 'r102966'
* 10:34 logmsgbot: catrope synchronized php-1.18/extensions/Vector/modules/ext.vector.collapsibleTabs.js 'r102966'
* 09:58 mark: Fixing ganglia's confusedness about $site
* 08:39 logmsgbot: catrope synchronized php-1.18/extensions/Collection/Collection.body.php 'r102950'
* 08:26 RoanKattouw: I meant r102944
* 07:52 logmsgbot: catrope synchronized php-1.18/extensions/Gadgets/Gadgets_body.php 'r102994'
* 05:09 awjr: synchroinzing payments cluster to r102939
* 03:31 awjr: synchronizing payments cluster to r102935
* 02:24 logmsgbot: LocalisationUpdate completed (1.18) at Mon Nov 14 02:27:03 UTC 2011
* 01:57 logmsgbot: LocalisationUpdate failed
== November 13 ==
* 23:58 Ryan_Lane: added updated version of etherpad-lite to wikimedia repo
* 23:39 Tim: installing the new version of lp-filter.c from subversion on locke
* 21:45 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 32399 - [MoodBar] Change [infoUrl] at nlwiki'
* 02:24 logmsgbot: LocalisationUpdate completed (1.18) at Sun Nov 13 02:27:47 UTC 2011
* 01:57 logmsgbot: LocalisationUpdate failed
== November 12 ==
* 15:56 apergos: removed some old snapshot files from ms7 (not actual snaps, but saved copies of them no longer useful), freeing up about 7T of space
* 15:47 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Re-enable TitleBlacklist on kwwiktionary'
* 15:45 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Temp disable TitleBlacklist on kwwiktionary'
* 09:44 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Replace Botswana account creation throttle exemption with one for Serbia'
* 06:04 logmsgbot: neilk synchronized wmf-config/CommonSettings.php 'Goes w/ previous change -- re-enable Special:UploadWizard where it is in use (i.e. Commons)'
* 05:59 logmsgbot: neilk synchronized wmf-config/CommonSettings.php 'Goes w/ previous change -- also need to disable multi-file select when disabling chunks'
* 05:51 logmsgbot: neilk synchronized wmf-config/CommonSettings.php 'Disabling chunked uploads temporarily, according to Erik we seem to have not deployed the API that can handle them'
* 02:38 awjr: synchronizing payments cluster to r102843
* 02:28 logmsgbot: LocalisationUpdate completed (1.18) at Sat Nov 12 02:31:22 UTC 2011
* 02:22 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Set fallbackToAltUploadForm to true per Erik, tempoary site issues'
* 01:57 logmsgbot: LocalisationUpdate failed
* 01:20 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'arwiki: disabled autopromote, added Portal and Annex namespaces to $wgFlaggedRevsNamespaces'
== November 11 ==
* 19:06 awjr: Syncronizing payments r102807
* 12:40 logmsgbot: catrope synchronized php-1.18/languages/messages/MessagesEn.php 'r102777'
* 10:22 mark: Fixing up amssq57
* 10:09 apergos: traffic back to normal, let's see
* 10:06 mark: csw1-esams reloaded, back up
* 09:51 apergos: moved traffic back to tampa
* 09:43 apergos: restored traffic so more tests can be done
* 08:56 apergos: moved traffic to pmtpa, csw1 reboots reported from observium
* 03:43 Ryan_Lane: finished gluster switchover for labs
* 03:32 Ryan_Lane: rebooting virt2 to test gluster switchover
* 03:30 logmsgbot: neilk synchronized php-1.18/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'MFT r102745 - fixes issue where descriptions were accidentally not required for uploadwizard images (accidentally flipped boolean)'
* 03:23 Ryan_Lane: moving instances on virt2 to gluster storager
* 03:19 awjr: Syncing payments boxes to r102740 (1 hour ago, logmsgbot was dead)
* 03:16 Ryan_Lane: rebooting virt4 to test gluster switchover
* 03:03 Ryan_Lane: suspending all instances on virt4 to move instances to gluster
* 02:58 Ryan_Lane: rebooting virt3 to test gluster switchover
* 01:10 LeslieCarr: rebooting csw2-esams
* 00:55 lcarr: installing new os on csw2-esams
* 00:53 lcarr: DNS draining AMS
* 00:28 logmsgbot: ben synchronized wmf-config/db.php 'removing es1 because it has crashed.'
* 00:05 Tim: debugging memory leak on srv189 by attaching to the culprit apache in gdb
* 00:04 logmsgbot: awjrichards synchronized php/extensions/ContributionReporting/arrow.png 'r102721'
== November 10 ==
* 23:50 logmsgbot: awjrichards rebuilt wikiversions.cdb and synchronized wikiversions files: 
* 23:36 logmsgbot: awjrichards synchronizing Wikimedia installation... : 
* 23:31 logmsgbot: awjrichards synchronizing Wikimedia installation... : 
* 23:29 Tim: on fenari: ran puppetd -tv --debug to fix wikimedia-task-appserver
* 23:11 logmsgbot: tstarling synchronizing Wikimedia installation... : 
* 23:10 logmsgbot: neilk synchronized php-1.18/extensions/UploadWizard 
* 22:54 logmsgbot: neilk synchronizing Wikimedia installation... : New version of UploadWizard (r102709), multi-file select, custom licenses (2nd try)
* 22:53 logmsgbot: neilk synchronizing Wikimedia installation... : New version of UploadWizard (r102709), multi-file select, custom licenses
* 22:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix kk logos'
* 22:20 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Adding better config control for ContributionReporting extension'
* 22:20 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Adding better config control for ContributionReporting extension'
* 22:05 Reedy: Run namespaceDupes on aswiki
* 22:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32145 - Namespaces/Aliases for Wikipedia, Project and Portal namespace in Assamese wikipedia'
* 21:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32340 - Translation of project namespace and sitename for tawikibooks'
* 21:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32176 - Create an author namespace on eo.wikisource'
* 21:39 Reedy: Running namespaceDupes on eowikisource
* 21:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32176 - Create an author namespace on eo.wikisource'
* 21:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32048 - Configuration changes for hewiki, add editinterface locally'
* 21:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32048 - Configuration changes for hewiki, add editinterface locally'
* 21:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32191 - Update logo for kk.wiktionary.org'
* 20:40 Reedy: Moodbar tables should now be normalised across WMF wikis. Tables had been created everywhere, but updates not applied afterwards
* 20:31 Reedy: running foreachwiki sql.php php-1.18/extensions/MoodBar/sql/mbf_timestamp_id.sql
* 20:27 Reedy: running foreachwiki sql.php php-1.18/extensions/MoodBar/sql/AddIDToIndexes.sql
* 20:16 Reedy: running foreachwiki sql.php php-1.18/extensions/MoodBar/sql/mbf_hidden_state.sql
* 20:08 Reedy: Run other Moodbar related sql patches on nlwiki
* 20:02 Reedy: Running mbf_hidden_state.sql on nlwiki
* 19:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32202 - Moodbar on nl: wikipedia'
* 19:43 notpeter: restareting nagios
* 19:24 logmsgbot: ben synchronized wmf-config/db.php 'adding es1 and es2 into rotation for all external store clusters'
* 19:07 mark: Repooled knsq24 in pybal, no idea why it was depooled
* 17:26 mark: Increased ms5 nginx worker processes from 4 to 32 (out of puppet)
* 17:25 mark: Increased ms6 worker processes from 4 to 32 (out of puppet)
* 17:06 mark: Moving esams traffic back to esams to seed the upload squid SSD caches
* 14:51 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'resyncing for bug 32309. Some servers did timeout'
* 14:48 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'bug 32309 - Add WMF domains to $wgNoFollowDomainException'
* 12:20 mark: Switching esams traffic to pmtpa
* 11:09 domas: otrs still broken, williams had old passwords, something else is going on with mail delivery
* 10:46 domas: One needs to run 'FLUSH PRIVILEGES' after modifying mysql.* tables directly. OTRS delivery was broken for a day or so?
* 10:22 guillom: OTRS issue: e-mail isn't getting in. No new messages have come through in the last 15h and 40 min. Outgoing e-mail works fine.
* 02:29 logmsgbot: LocalisationUpdate completed (1.18) at Thu Nov 10 02:32:45 UTC 2011
* 02:12 awjr: Updating thank you module code for CiviCRM on aluminium/grosley; r762
* 01:57 logmsgbot: LocalisationUpdate failed
* 01:34 awjr: Updating civicrm on aluminium/grosley to r760
* 01:17 RobH: ocg2 installed per rt1593
* 01:14 Tim: started puppet on fenari
* 01:11 RobH: ocg1 installed, awaiting puppetization (pending labs-ocg1 deployment)
* 00:43 RobH: mobile3 (now ocg1) coming offline for reinstallation, ignore any mobile3 nagios errors
* 00:36 RobH: another dns update, this is me pushing my luck on dns not crashing
* 00:29 logmsgbot: awjrichards synchronized php/extensions/DonationInterface/gateway_common/us-states.i18n.php 'r102610'
* 00:29 logmsgbot: awjrichards synchronized php/extensions/DonationInterface/gateway_common/interface.i18n.php 'r102610'
* 00:28 logmsgbot: awjrichards synchronized php/extensions/DonationInterface/gateway_common/countries.i18n.php 'r102610'
* 00:26 logmsgbot: awjrichards synchronized php/extensions/DonationInterface/globalcollect_gateway/globalcollect_gateway.i18n.php 'r102610'
* 00:25 RobH: ocg dns update, once more with feeling
* 00:25 logmsgbot: awjrichards synchronized php/extensions/DonationInterface/payflowpro_gateway/payflowpro_gateway.i18n.php 'r102610'
* 00:09 binasher: db10 is back up, and caught up on replication
* 00:05 RobH: updating dns to rename mobile1-mobile5 to ocg1-ocg5
== November 9 ==
* 23:40 Tim: running puppetd -tv on fenari to test I6146f4b5
* 23:38 logmsgbot: reedy synchronizing Wikimedia installation... : r102598
* 23:34 logmsgbot: reedy synchronized php-1.18/extensions/AbuseFilter/abusefilter.css 'r102602'
* 23:33 logmsgbot: reedy synchronized php-1.18/extensions/WikiEditor/modules/jquery.wikiEditor.dialogs.config.js 'r102601'
* 23:04 logmsgbot: reedy synchronizing Wikimedia installation... : r102588 r102592 r102593
* 22:25 mark: nagios is back up
* 21:55 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32312 - Enable WikiLove on Japanese Wikipedia'
* 21:54 Reedy: Creating Wikilove tables on jawiki
* 21:00 mark: Running puppetd --test in ddsh, concurrency 8
* 20:29 mark: Undid the sockpuppet firewall rules
* 20:17 mark: Revert nagios changes
* 19:52 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'removed references to patrolother right that does not exist'
* 19:48 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'removed references to autopatrolother right that does not exist'
* 19:36 logmsgbot: ben synchronized wmf-config/db.php 'switching all external store clusters except cluster22 (current active) from srv/ms hosts to new ES hardware es3 and es4.'
* 19:31 mark: Truncated several puppet db tables again
* 19:26 logmsgbot: aaron synchronized php-1.18/extensions/FlaggedRevs/FlaggedRevs.php 'deployed r102559'
* 18:44 notpeter: restarting exim on mchenry
* 18:19 mark: Enabled PassengerGlobalQueueing on sockpuppet
* 17:45 mark: Temporarily firewalled all servers from sockpuppet, except spence
* 17:42 mark: Purged all nagios resources in the puppet db on db9
* 17:31 logmsgbot: catrope synchronized php-1.18/cache/interwiki-pr.cdb 'Rebuilt interwiki cache again for bug 29591'
* 17:31 logmsgbot: catrope synchronized php-1.18/cache/interwiki.cdb 'Rebuilt interwiki cache again for bug 29591'
* 17:20 logmsgbot: catrope synchronized php-1.18/includes/api/ApiQuerySearch.php 'r102538'
* 16:21 mark: Rebooting lvs1001 for kernel upgrade
* 16:12 mark: Failover of lvs1001 to lvs1004, preparing for lvs1001 reboot
* 16:07 mark: Rebooting lvs1004 for kernel upgrade
* 15:45 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Reenable TitleBlacklist on nsowiki'
* 15:43 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Disable TitleBlacklist on nsowiki so I can fix it properly'
* 15:29 RoanKattouw: Deleting and recreating nsowiki per bug 30882
* 15:09 mark: Fixing discrepancy between 10.2.4 and 10.2.2 eqiad service IPs in DNS
* 11:58 Tim-away: on fenari: running puppetd -tv to get the new ED crontab
* 05:35 Ryan_Lane: adding gluster_3.2.4-1 to lucid-wikimedia repo
* 02:52 Tim: on fenari: fixing file permissions after someone ran svn up in the ED snapshot as root
* 02:48 Tim: stopping xinetd on fenari for extdist cleanup
* 02:39 Tim: disabled extdist crontab on fenari for testing of the new cron.php
* 02:32 logmsgbot: LocalisationUpdate completed (1.18) at Wed Nov 9 02:35:19 UTC 2011
* 02:21 logmsgbot: awjrichards synchronized php/extensions/CentralNotice/CentralNotice.db.php 'r102478'
* 02:20 logmsgbot: awjrichards synchronized php/extensions/CentralNotice/CentralNotice.i18n.php 'r102478'
* 02:14 logmsgbot: awjrichards synchronized php/extensions/CentralNotice/CentralNotice.php 'r102478'
* 02:13 logmsgbot: awjrichards synchronized php/extensions/CentralNotice/special/SpecialCentralNotice.php 'r102478'
* 02:13 logmsgbot: awjrichards synchronized php/extensions/CentralNotice/special/SpecialBannerController.php 'r102478'
* 00:12 logmsgbot: aaron synchronized php-1.18/includes/upload/UploadFromStash.php 
* 00:09 logmsgbot: aaron synchronized php-1.18/includes/upload/UploadFromStash.php 
* 00:09 logmsgbot: aaron synchronized php-1.18/includes/upload/UploadFromStash.php 'logging hack'
== November 8 ==
* 23:52 logmsgbot: aaron synchronized php-1.18/includes/api/ApiUpload.php 'removed live hack'
* 23:11 logmsgbot: aaron synchronized php-1.18/includes/upload/UploadFromStash.php 
* 22:46 logmsgbot: aaron synchronized php-1.18/includes/upload/UploadFromStash.php 
* 22:43 AaronSchulz: Doing some api upload debugging via error log
* 22:37 logmsgbot: aaron synchronized php-1.18/includes/api/ApiUpload.php 
* 22:33 logmsgbot: aaron synchronized php-1.18/includes/api/ApiUpload.php 
* 22:30 logmsgbot: aaron synchronized php-1.18/includes/api/ApiUpload.php 
* 22:08 logmsgbot: aaron synchronized php-1.18/includes/api/ApiUpload.php 
* 22:03 logmsgbot: aaron synchronized php-1.18/includes/api/ApiUpload.php 'Made performStash() $stashFile check stronger'
* 21:56 logmsgbot: tfinc synchronized wmf-config/InitialiseSettings.php 'updating aspect ratios for wmf wiki logo'
* 21:33 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'weekly update to mobile frontend'
* 21:22 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'weekly update to mobile frontend'
* 21:21 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/_search_webkit.html.php 'weekly update to mobile frontend'
* 21:12 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'weekly update to mobile frontend'
* 21:07 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to mobile frontend'
* 21:07 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/stylesheets/common.css 'weekly update to mobile frontend'
* 20:50 logmsgbot: aaron synchronized php-1.18/includes 'deployed r102434'
* 20:03 maplebed: changed replication topology for external store cluster; new ES hosts are all now replicating off es3, which is still replicating off ms3.
* 19:52 AaronSchulz: ran fixBug32198.php and confirmed that there are no more affected rows atm
* 19:49 logmsgbot: tfinc synchronized wmf-config/InitialiseSettings.php 'Upping resolution of wmf wiki logo to 35px'
* 19:46 logmsgbot: tfinc synchronized wmf-config/InitialiseSettings.php 'Pushing config change again due to missing ssh agent'
* 19:01 logmsgbot: ben synchronized wmf-config/db.php 'switching external store cluster6 from srv hosts to new ES hardware es3 and es4.'
* 19:00 maplebed: switching external store cluster6 to new ES hardware.
* 18:14 mark: Converted eth0 on sq67 to an aggregated interface of eth0-4
* 17:25 mark: Converted eth0 on sq68 to an aggregated interface of eth0-4
* 15:15 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'bug 28026 trial - Enable $wgEnotifUserTalk on all wikis, but keep $wgShowUpdatedMarker disabled where it was disabled before'
* 15:07 RoanKattouw: ...after fixing permissions on cache/l10n AGAIN
* 15:07 RoanKattouw: Ran sync-l10nupdate
* 12:22 mutante: arsenic - alright known kernel issue with bonding, ack
* 12:17 mutante: arsenic - power cycling
* 12:15 mutante: arsenic - not reachable via ssh, mgmt shows BUG: soft lockup - CPU#6 stuck for 61s!
* 11:08 logmsgbot: catrope synchronized php-1.18/cache/interwiki-pr.cdb 'Update interwiki cache for bug 29591'
* 11:08 logmsgbot: catrope synchronized php-1.18/cache/interwiki.cdb 'Update interwiki cache for bug 29591'
* 11:06 logmsgbot: catrope synchronized special.dblist 'Remove sourceswiki from special.dblist to make r102388 work'
* 05:45 Tim: on fenari: svn up finished, started puppet and xinetd
* 04:46 Tim: increased read timeout on formey apache from 300 to 600 to increase the chances of a successful svn up on fenari extdist
* 02:28 logmsgbot: LocalisationUpdate completed (1.18) at Tue Nov 8 02:31:23 UTC 2011
== November 7 ==
* 23:18 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Removing AU from list of priority countries for LandingCheck'
* 21:30 awjr: reenabling donations queue consumption job in Jenkins on aluminium
* 21:10 mark: Reenabled BGP session to AS1299 on br1-knams
* 20:22 awjr: disabling donations queue consumption in jenkins on aluminium
* 20:17 maplebed: full snapshot of external store data now resides on es1001
* 19:38 awjr: syncing payments cluster to r102314
* 18:37 logmsgbot: nikerabbit synchronized php-1.18/languages/messages/ 'i18ndeploy r102305'
* 18:36 logmsgbot: nikerabbit synchronized php-1.18/skins/common/shared.css 'i18ndeploy r102305'
* 18:35 logmsgbot: nikerabbit synchronized php-1.18/extensions/WikimediaMessages/WikimediaMessages.i18n.php 
* 18:33 logmsgbot: nikerabbit synchronized php-1.18/extensions/Narayam/ 'i18ndeploy r102305'
* 17:36 awjr: pushing FUndraiserLandingPage updates, r102302
* 17:14 awjr: Updating banner hiding configuration for CentralNotice to hide banners until 2012-01-31
* 15:35 apergos: wgContributionTrackingDBserver -> db1008 from db9 in reporting-setup.php, that should be the last of those
* 14:26 apergos: wgContributionReportingDBserver -> db1025 after info from jeff green (but it's still access denied in the same fashion)
* 14:19 mark: Shutdown BGP session to AS1299 on br1-knams
* 13:57 apergos: changed wgContributionReportingDBserver in reporting-setup.php to point to db1008 instead of db10 (following jgreen's aug 18 note in server admin log) but getting Access denied for user 'public_reporting'. Jeff can you have a look?
* 13:26 logmsgbot: hashar: contribution reporting DB is unreacheable: db10
** The ContributionReporting extension still reference db10 as a database server (db: civicrm)
* 09:41 Tim: stopping xinetd, puppet and extdist crontab on fenari to complete ED cleanup
* 06:50 awjr: syncing payments cluster to r102270
* 06:34 awjr: syncing payments cluster to r102268
* 02:19 logmsgbot: LocalisationUpdate completed (1.18) at Mon Nov 7 02:22:25 UTC 2011
== November 6 ==
* 21:28 awjr: "Updating CommongSettings (r2572); Setting fundraising banner cookie expiration for donatewiki and foundationwiki"
* 19:36 awjr: re-enabled CiviCRM cron and Donations queue consumption in jenkins on ALuminium
* 19:13 awjr: Deployed exhange_rate module updates on Aluminium/Grosley (r722)
* 19:02 awjr: disabling CiviCRM cron and Donations queue consumption in jenkins on Aluminium
* 02:24 logmsgbot: LocalisationUpdate completed (1.18) at Sun Nov 6 02:27:34 UTC 2011
== November 5 ==
* 21:29 awjr: Syncing i18n messages for DonationInterface on the cluster, r102122
* 20:27 awjr: i18n messages synched for DonationInterface on the cluster (r102115)
* 12:03 Reedy: Creating moodbar tables on nlwiki
* 12:01 Reedy: Creating ArticleFeedback tables on eswiki
* 02:22 logmsgbot: LocalisationUpdate completed (1.18) at Sat Nov 5 02:25:08 UTC 2011
== November 4 ==
* 21:45 logmsgbot: aaron synchronized php-1.18/includes/Title.php 'deployed r102055'
* 21:00 RobH: updated dns for mgmt on es1-es4
* 20:48 rainman-sr: restarted search11, had lots of stuck threads.. 
* 17:51 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'disabled $wgUseNPPatrol on ruwiki, bug 31650'
* 16:44 maplebed: purged binary logs from >14d ago on db9 to free up disk space, set expire_logs_days to 14.
* 14:08 mark: Labs public NAT ip firewalled to tcp port 25 on lily
* 12:46 mark: Firewalled labs IP addresses to tcp port 25 on mchenry.
* 11:40 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable ArticleFeedback dashboard on meta'
* 04:44 Ryan_Lane: added npm, nodejs, and etherpad-lite to the lucid-wikimedia apt repo
* 02:51 Tim: stopping puppet and xinetd and disabling extdist crontab on fenari to update extensions directory
* 02:32 Tim: on db10: removed the deleted relay logs from mysqld-relay-bin.index with a text editor, then ran "reset slave" and "change master" to get replication working again
* 02:26 logmsgbot: LocalisationUpdate completed (1.18) at Fri Nov 4 02:29:26 UTC 2011
* 02:16 Tim: db10 ran out of disk space on /a with 27GB of relay logs created in the last 2 hours. It wouldn't respond to "stop slave" or shut down gracefully, so I had to delete the relay logs manually to free up enough space for a graceful shutdown
* 02:07 Tim: fixing db10, is out of disk space
* 02:00 awjr: Enabling ParserFunctions string functions on donatewiki (config r2566 and r2567)
* 01:22 awjr: Synchronizing payments r101939
== November 3 ==
* 23:36 logmsgbot: tstarling synchronized docroot/secure/keys.txt 
* 23:31 logmsgbot: tstarling synchronized docroot/secure/keys.html 
* 22:25 LeslieCarr: restarting powerdns on ns2
* 22:11 LeslieCarr: reinstalling mw1019 to test out systems
* 21:56 awjr: Updating payments boxes to r101890
* 21:36 awjr: reenabling Donations queue consume in Jenkins on aluminium
* 21:06 logmsgbot: awjrichards synchronized wmf-config/contribution-tracking-setup.php 
* 20:54 logmsgbot: nikerabbit synchronized php-1.18/extensions/WebFonts/ 'static webfonts test'
* 20:34 notpeter: restarted gammu-smsd on spence
* 19:33 notpeter: putting srv290-srv301 back into api pybal pool
* 19:12 logmsgbot: catrope synchronized php-1.18/README 'Dummy sync to confirm that sync logging is fixed now'
* 18:39 awjr: disabling jenkins Donations queue consume job on grosley
* 17:50 Jeff_Green: flipped DNS for civicrm and other fundraising services from grosley to aluminium
* 17:00 Jeff_Green: stopping apache on grosley and aluminium
* 15:58 notpeter: restarting nagios on spence
* 11:35 Jeff_Green: stopping mysql on storage3
* 10:30 mutante: tried to apache-graceful-all to make them pick up config change, but getting a lot of "Too many authentication failures"
== November 2 ==
* 23:50 Ryan_Lane: reverted Roan's dsh.conf changes
* 23:47 Ryan_Lane: updated Roan's changes to dsh.conf, wrapping the options in quotes
* 22:30 Ryan_Lane: rebooting pad1.pmtpa.wmflabs
* 21:11 RoanKattouw: Added -o SetupTimeout=10 to ssh options in /etc/dsh/dsh.conf
* 19:19 notpeter: depooling srv290-srv301 for actual upgrades to lucid
* 19:14 RoanKattouw: Running scap for real now, was broken. Fixed in puppet
* 19:04 RoanKattouw: Scapping to deploy editor engagement changes
* 18:44 notpeter: reimageing srv290, srv291, srv292 to lucid. this time for real
* 16:56 notpeter: adding new version of php-wikidiff2 package to brewster (that will be pushed out to all apaches...)
* 16:44 notpeter: installing new version of php-wikidiff2 on mw6 for testing
* 14:28 RoanKattouw: (morebots had to be restarted too, got killed in a netsplit earlier)
* 14:28 RoanKattouw: RoanKattouw	!log logmsgbot will be broken until Ryan shows up to save us all. I just e-mailed him about it
* 14:27 RoanKattouw: apergos	!log restarted logmsgbot though it's apparently broken atm
* 02:18 logmsgbot: LocalisationUpdate completed (1.18) at Wed Nov 2 02:21:36 UTC 2011
== November 1 ==
* 21:36 awjr: Deployed thank_you module fixes to CiviCRM on grosley r696
* 21:21 Jeff_Green: changed donate.wikimedia.org from CNAME to A record so we can use MX records
* 21:13 awjr: Configured DonationInterface to only load i18n files on the cluster, enabled on donatewiki and foundation (CommonSettings @ r2560, InitialiseSettings @ r2561)
* 21:10 awjr: Running scap to pick up new DonationInterface set up for the cluster and new message files (r101527)
* 19:55 awjr: Fixed borked config change in CommonSettings for LandingCheck, r2559
* 19:53 awjr: Config change to CommonSettings to make LandingCheck always point to foundationwiki for now r2558
* 19:45 binasher: note, locke and emery changes only made to the running kernel for testing, not in /etc/sysctl*
* 19:43 binasher: made the same kernel changes on emery as locke, udp packet loss dropped from 6% to 0
* 19:32 awjr: deploying LandingCheck updates (@ r101515) via sync-file as well as config change to CommonSettings.php (@ r2557)
* 18:52 notpeter: restarting lighttpd on lily
* 18:43 notpeter: restarting lighttpd on lily
* 18:30 binasher: locke: disabled bbu checks, set per5 to writeback caching
* 18:15 binasher: increased def recv buf on locke to 128M (just experimenting)
* 18:10 RobH: mgmt fixed on mw1027. mw1033, mw1038, mw1039
* 18:03 Ryan_Lane: re-enabling LDAP plugin on labsconsole
* 18:02 Ryan_Lane: temporarily disabling LDAP plugin on labsconsole to give admin rights to Andrew Bogott (stupid bugs)
* 17:49 binasher: on locke: set rmem_default to 64M and rmem_max to 128M, retarted udp2log. packet loss has stopped per /proc/net/udp which is definitive vs. seq numbers
* 17:21 RobH: mw1018 mw1021 firmware updated, mgmt working
* 16:52 RoanKattouw: Synchronized php-1.18/cache/interwiki{,-pr}.cdb , updating interwiki cache for nsowiki addition (bug 30882)
* 16:51 RoanKattouw: sync-file logging is broken due to my changes in puppet, will get Ryan to deploy the fixes
* 15:28 notpeter: scratch that, mw5
* 15:26 notpeter: trying again to replace php-wikidiff2 with php5-wikidiff2 on mw4. then graceful
* 14:32 notpeter: restarting lighttpd on lily for rt 1752
* 03:19 Reedy: Bug 32042 - Narayam for Assamese Wikipedia
* 03:12 Reedy: sync-file is giving "/usr/local/bin/dologmsg: 2: cannot create /var/log/logmsg: Permission denied" and not logging to IRC
* 03:11 Reedy: Sync r101448
* 02:20 logmsgbot: LocalisationUpdate completed (1.18) at Tue Nov 1 02:22:51 UTC 2011
* 01:42 awjr: Oops forgot to log config change - removed Hungary from list of priority countries for LandingCheck extension
* 01:41 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 
== October 31 ==
* 23:35 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/opensearch.js 'weekly update to mobile frontend'
* 23:08 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'weekly update to mobile frontend'
* 23:06 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/opensearch.js 'weekly update to mobile frontend'
* 22:53 logmsgbot: awjrichards synchronizing Wikimedia installation... : Adding FundraiserLandingPage extension and enabling it on donatewiki
* 20:00 binasher: removed deprecated mobile classes and templates from puppetlandia
* 19:43 notpeter: replacing php-wikidiff2 with new php5-wikidiff2 on mw4 for testing
* 19:43 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to mobile frontend'
* 19:41 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to mobile frontend'
* 19:27 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to mobile frontend'
* 18:03 logmsgbot: nikerabbit synchronized php-1.18/languages/messages/MessagesEn.php 'r101349'
* 18:02 logmsgbot: nikerabbit synchronized php-1.18/languages/messages/MessagesStq.php 'r101368'
* 17:42 logmsgbot: nikerabbit synchronized php-1.18/extensions/AbuseFilter/AbuseFilter.i18n.php 'r101368'
* 17:39 logmsgbot: nikerabbit synchronized php-1.18/extensions/Narayam/ 'r101126'
* 16:44 mutante: sq35 - tried to boot up,freezes after DRAC detection, there has been RT 1404 anyways, but now really dead
* 16:28 notpeter: srv193 back up.
* 16:26 RoanKattouw: Gracefulled Apache on srv193 to make it pick up its config
* 15:29 cmjohnson1: db19 is down for memory testing
* 15:02 apergos: restarted lighthttpd on dataset2
* 14:56 notpeter: will be back up for 1700 utc deployment
* 14:55 notpeter: reimaging srv193 (test.wikipedia.org)
* 04:46 logmsgbot: aaron synchronized php-1.18/extensions/CheckUser/CheckUser.hooks.php 'deployed r101337'
* 02:16 logmsgbot: LocalisationUpdate completed (1.18) at Mon Oct 31 02:19:36 UTC 2011
== October 30 ==
* 23:41 Tim: cleanup finished
* 22:52 Tim: on fenari: stopped xinetd and crond for extdist cleanup
* 02:19 logmsgbot: LocalisationUpdate completed (1.18) at Sun Oct 30 02:21:47 UTC 2011
== October 29 ==
* 21:19 logmsgbot: jeluf ran sync-common-all 
* 20:07 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '32035 - ku.wikiquote: change the logo'
* 02:18 logmsgbot: LocalisationUpdate completed (1.18) at Sat Oct 29 02:21:34 UTC 2011
* 00:30 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/opensearch.js 'weekly update to mobile frontend'
== October 28 ==
* 23:00 logmsgbot: ben synchronized wmf-config/db.php 'putting ms1 back in rotation to do its share of the work'
* 22:44 logmsgbot: aaron synchronized php-1.18/includes/filerepo 'deployed r101227'
* 22:14 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/opensearch.js 'weekly update to mobile frontend'
* 22:14 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'weekly update to mobile frontend'
* 22:07 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'weekly update to mobile frontend'
* 22:04 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/stylesheets/common.css 'weekly update to mobile frontend'
* 22:04 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/opensearch.js 'weekly update to mobile frontend'
* 21:57 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/opensearch.js 'weekly update to mobile frontend'
* 21:56 logmsgbot: aaron synchronized php-1.18/extensions/FlaggedRevs/presentation/specialpages/reports/PendingChanges_body.php 'deployed r101213'
* 21:53 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/stylesheets/common.css 'weekly update to mobile frontend'
* 21:53 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/opensearch.js 'weekly update to mobile frontend'
* 21:53 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'weekly update to mobile frontend'
* 21:36 logmsgbot: catrope synchronized php-1.18/extensions/CodeReview/backend/CodeRevision.php 'r101206'
* 21:22 LeslieCarr: normalizing system switch configurations
* 21:11 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'weekly update to mobile frontend'
* 21:11 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/opensearch.js 'weekly update to mobile frontend'
* 20:41 logmsgbot: catrope synchronized php-1.18/extensions/CodeReview/backend/CodeRevision.php 'r101190'
* 19:53 logmsgbot: catrope synchronized php-1.18/extensions/ProofreadPage/ProofreadPage_body.php 'r101183'
* 19:12 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'weekly update to mobile frontend'
* 19:12 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/stylesheets/common.css 'weekly update to mobile frontend'
* 19:12 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/opensearch.js 'weekly update to mobile frontend'
* 18:47 preilly: finish weekly updates for Mobile Frontend
* 18:47 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'weekly update to mobile frontend'
* 18:46 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/stylesheets/common.css 'weekly update to mobile frontend'
* 18:46 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/opensearch.js 'weekly update to mobile frontend'
* 18:46 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to mobile frontend'
* 16:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31831 - Deploy WikiLove on mkwiki'
* 16:57 Reedy: Created WikiLove tables on mkwiki
* 16:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 32014 - Enable Narayam in Meta Wiki'
* 14:50 RoanKattouw: Moved /home/wikipedia/bin/set-group-write2 to sbin and updated set-group-write and fenari:/etc/sudoers accordingly
* 14:27 mutante: srv167 - Nagios reports HTTP down, apache2 running but not on 80, instead on high-port, part of testing?
* 14:13 mutante: srv158 - host up, Apache down - running: bash -c /etc/init.d/apache2 stop; while ps -C apache2; do sleep 1; done; /etc/init.d/apache2 start
* 13:01 RoanKattouw: Manually killing dormant Apache workers with 20%+ mem usage on srv188
* 12:59 RoanKattouw: And starting Apache again on srv189
* 12:56 RoanKattouw: Stopping and restarting Apache on srv189 to get rid of memory-hogging Apache processes
* 12:52 RoanKattouw: Gracefulling Apache on srv189
* 12:21 logmsgbot: catrope synchronizing Wikimedia installation... : r101126
* 05:32 Andrew: srv256: @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
* 05:32 logmsgbot: andrew synchronized wmf-config/CommonSettings.php 'Updated $wgMoodBarConfig[infoUrl]'
* 03:48 Jeff_Green: restarted crashed mysql on payments1, repaired replication
* 02:19 logmsgbot: LocalisationUpdate completed (1.18) at Fri Oct 28 02:22:41 UTC 2011
* 00:53 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to mobile frontend'
* 00:19 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to mobile frontend'
== October 27 ==
* 23:45 Ryan_Lane: setting shell to sillyshell for fvassard and pdhanda, since they no longer need shell access to formey
* 23:41 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to mobile frontend'
* 23:33 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to mobile frontend'
* 23:19 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to mobile frontend'
* 23:16 logmsgbot: reedy synchronized php-1.18/extensions/CodeReview/backend/CodeComment.php 'r10185'
* 23:16 logmsgbot: reedy synchronized php-1.18/extensions/CodeReview/backend/CodeRevision.php 'r10185'
* 22:04 LeslieCarr: the ganglia link on noc.wikimedia.org points to ganglia3 instead of ganglia
* 21:57 RoanKattouw: Fixing LU perms again, hopefully this is the last time with the cron job user fix now in puppet
* 21:41 logmsgbot: reedy synchronizing Wikimedia installation... : Pushing r101067 numerous zh fixes
* 21:32 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/_footmenu_default.html.php 'weekly update to mobile frontend'
* 21:24 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/stylesheets/common.css 'weekly update to mobile frontend'
* 21:00 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to mobile frontend'
* 20:57 logmsgbot: catrope synchronizing Wikimedia installation... : Update MoodBar to trunk state, deploying feedback dashboard
* 20:51 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to mobile frontend'
* 20:47 RoanKattouw: Applying schema changes for MoodBar on enwiki
* 20:36 RoanKattouw: Applying schema changes for MoodBar on testwiki
* 18:30 notpeter: ns2 dead and down. rebooting
* 14:53 RoanKattouw: Restarting lsearch on search1 and search3 because rainman-sr told me to
* 10:19 mutante: Bugzilla back with upgraded kernel and mysql, looks fine
* 10:08 mutante: kaulen (Bugzilla) - dist-upgrade/kernel,reboot (scheduled maintenance)
* 03:16 logmsgbot: LocalisationUpdate completed (1.18) at Thu Oct 27 03:19:13 UTC 2011
* 03:13 Tim: running iperf tests on eqiad internal network to get baseline performance
* 03:07 Tim: taking mw1071 and mw1072 for udp2log testing
* 02:45 logmsgbot: LocalisationUpdate completed (1.18) at Thu Oct 27 02:47:47 UTC 2011
* 02:20 logmsgbot: LocalisationUpdate completed (1.18) at Thu Oct 27 02:22:56 UTC 2011
* 00:34 logmsgbot: neilk synchronizing Wikimedia installation... : MFT @ r99799, to ensure that UploadWizard multi-file select is off by default. Was accidentally on by default in last push, and it still needs work.
* 00:15 logmsgbot: reedy synchronized php-1.18/extensions/CodeReview/ui/CodeRevisionTagView.php 'r100913'
== October 26 ==
* 23:55 logmsgbot: neilk synchronizing Wikimedia installation... : deploying UploadWizard from r99547 trunk (10 Oct 2011) to fix some regressions
* 23:07 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'weekly update to mobile frontend'
* 23:06 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/opensearch.js 'weekly update to mobile frontend'
* 22:46 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'weekly update to mobile frontend'
* 22:40 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/stylesheets/common.css 'weekly update to mobile frontend'
* 22:21 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.i18n.php 'weekly update to mobile frontend'
* 21:47 notpeter: restarting spence to be able to actually run puppet and to slaughter zombies
* 21:12 notpeter: restarting nagios
* 20:11 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'Remove beta logo'
* 19:32 preilly: finish pushing weekly mobile frontend changes 
* 19:32 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/stylesheets/common.css 'weekly update to mobile frontend'
* 19:32 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/opensearch.js 'weekly update to mobile frontend'
* 19:32 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to mobile frontend'
* 19:28 mutante: stopping,starting nagios
* 18:47 logmsgbot: LocalisationUpdate completed (1.18) at Wed Oct 26 18:49:55 UTC 2011
* 18:45 RoanKattouw: Running l10nupdate2 (l10nupdate sans svn up, thanks Reedy) which will hopefully succeed now
* 18:44 RoanKattouw: Fixed ownership of php-1.18/cache/l10n on all servers using dsh
* 18:36 RoanKattouw: Deleting /mnt/upload6/centralnotice completely per Arthur
* 18:31 logmsgbot: catrope synchronized php-1.18/resources/startup.js 'touch'
* 18:29 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable AFT on meta properly. Set AFTNamespaces back to the default, and configure the proper category'
* 17:52 logmsgbot: catrope synchronized php-1.18/extensions/ArticleFeedback/SpecialArticleFeedback.php 'r100832'
* 17:38 mutante: nagios was broken due to undefined check command, up again and commented culprit in puppet
* 17:15 logmsgbot: LocalisationUpdate failed
* 17:15 logmsgbot: LocalisationUpdate failed
* 17:10 Jeff_Green: hooray. EV cert is deployed on payments.wikimedia.org
* 17:03 logmsgbot: reedy synchronized wmf-config/ExtensionMessages.php 
* 15:42 logmsgbot: catrope synchronized php-1.18/resources/startup.js 'touch'
* 15:40 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable ClickTracking on meta so AFT will work'
* 15:34 logmsgbot: catrope synchronized php-1.18/resources/startup.js 'sync'
* 15:33 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Set wmgArticleFeedbackNamespaces to NS_MEDIAWIKI for metawiki'
* 15:20 logmsgbot: catrope synchronized php-1.18/resources/startup.js 'touch'
* 15:17 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable ArticleFeedback on meta, only for a test category for now'
* 15:15 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Code for wmgArticleFeedbackRatingTypes -> wgArticleFeedbackRatingTypes'
* 15:14 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add wmgArticleFeedbackRatingTypes, only set for metawiki'
* 14:56 logmsgbot: catrope synchronizing Wikimedia installation... : ArticleFeedback changes: r100808, r100811, r100814
* 14:29 ^demon: scratch that, svn copy is borked
* 14:26 ^demon: updated users.php on svn.wm.o from svn copy. this should go in puppet anyway.
* 14:19 logmsgbot: catrope synchronized php-1.18/includes/OutputPage.php 'Shut up PHP notice'
* 09:25 mutante: nagios was broken due to services for host knsq14 while the host did not exist anymore, fixed by purge-nagios-resources.py which is called by the init script
* 09:16 logmsgbot: catrope synchronized php-1.18/cache/interwiki-pr.cdb 'bug 31428, again'
* 02:18 logmsgbot: LocalisationUpdate completed (1.18) at Wed Oct 26 02:21:30 UTC 2011
* 00:26 Tim: on locke: set awk to point to mawk instead of gawk and sent a HUP to udp2log
* 00:14 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to mobile frontend'
== October 25 ==
* 23:46 preilly: weekly mobile frontend update
* 23:46 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/opensearch.js 'weekly update to mobile frontend'
* 23:45 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/application.js 'weekly update to mobile frontend'
* 23:45 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/stylesheets/common.css 'weekly update to mobile frontend'
* 23:45 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/_search_webkit.html.php 'weekly update to mobile frontend'
* 23:44 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'weekly update to mobile frontend'
* 23:44 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'weekly update to mobile frontend'
* 23:09 logmsgbot: reedy synchronized php-1.18/extensions/CodeReview/modules/ext.codereview.js 'r100760'
* 23:08 logmsgbot: reedy synchronized php-1.18/extensions/Collection/js/collection.js 'r100760'
* 23:08 logmsgbot: reedy synchronized php-1.18/extensions/AbuseFilter/AbuseFilter.php 'r100760'
* 22:54 logmsgbot: reedy synchronizing Wikimedia installation... : r100759
* 22:06 Tim: rebooting hume, it went OOM and stopped responding
* 21:02 awjr: synced civicrm installs on grosley and aluminium to r671 of the wikimedia repo
* 20:13 LeslieCarr: updating dns
* 18:37 LeslieCarr: puppetizing mw1001-1160 - please ignore all alerts about them for now
* 17:52 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Reenable ClickTracking UDP logger, now pointing to emery'
* 16:52 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Disable ClickTracking UDP logging to offload locke. Collector had already been killed'
* 16:49 RoanKattouw: Killed AFT log collector on locke. Process was a direct child of init (PID 1), command line was /usr/bin/udp2log --config-file=/etc/udp2log/aft -p 842
* 13:11 logmsgbot: catrope synchronized php-1.18/cache/interwiki-pr.cdb 'bug 31428'
* 12:18 mark: All eBGP peerings of br1-knams are off now, no more packet loss
* 12:10 mark: Reenabled some eBGP peerings of br1-knams as cr1-esams transit couldn't handle it alone
* 12:00 mark: Shutting down all BGP peerings of br1-knams
* 11:15 mutante: mw64 - updated APT and libpam, syslog has "nrpe invoked oom-killer","Out of memory: kill process ..(apache2)" etc.,check "free swap" in Ganglia
* 10:39 mutante: mw64 - Apache HTTP flapping OK-CRIT-OK.. - high load avg.,swapping, swap Size/Used 975864
* 09:08 mutante: erzurumi - installed libpam security updates
* 05:59 Tim: on hume: running refreshImageMetadata.php on all wikis
* 02:18 logmsgbot: LocalisationUpdate completed (1.18) at Tue Oct 25 02:20:43 UTC 2011
== October 24 ==
* 22:36 logmsgbot: aaron synchronized php-1.18/includes/User.php 
* 22:31 logmsgbot: aaron synchronized php-1.18/includes 'deployed r100657'
* 22:26 logmsgbot: aaron synchronized php-1.18/includes 'deployed r100655'
* 21:57 LeslieCarr: reformatting all of the wm****.eqiad machines
* 21:54 Ryan_Lane: turning off ci2
* 19:41 cmjohnson1: resetting storage1 to access bios and check setup
* 19:20 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'pre-emptive wg(Add/Remove)Groups settings to account for r100636; should have no visible effect'
* 19:06 logmsgbot: reedy synchronized php-1.18/extensions/FlaggedRevs/FlaggedRevs.php 'r100635'
* 18:59 mutante: sodium - initial mailman setup, created site list, will now notify ops@ via mail, storing pass in private repo ./files/misc
* 18:36 LeslieCarr: restarting dead nagios on spence
* 17:40 logmsgbot: nikerabbit synchronizing Wikimedia installation... : i18ndeploy r100623
* 16:05 mark: restarted pdns on nescio
* 14:59 notpeter: pdns_recursor hung on dobson. restarting
* 14:30 Jeff_Green: installed nagios/nsca on spence
* 14:27 mutante: snapshot4 - dist-upgrade/kernel, reboot
* 14:19 mark: Manually fixed package situation on formey
* 14:10 mutante: ekrem - gzipped access.log.1, can move/delete access.logs older than X months?
* 14:08 mutante: ekrem at 99% disk space ..again
* 13:14 mutante: something fails btw with etherpad using openoffice ([soffice] <defunct>), (was before and after reboot)
* 13:10 mutante: hooper - fsck was forced and OK, back up, blog and etherpad reachable again
* 13:06 mutante: hooper - dist-upgrade/kernel, reboot
* 13:03 mutante: hooper - temp. shutting down blog and etherpad for kernel update/reboot
* 11:34 mutante: added TERM=dumb to sudo line in /home/w/bin/scap to prevent "unknown terminal type" errors (RT 1767)
* 10:52 mutante: gurvin - dist-upgrade/kernel,reboot (is 'False' in pybal https conf)
* 10:52 mutante: fixing date on yvon
* 10:43 mutante: yvon - dist-upgrade/kernel,reboot (is 'False' in pybal https conf)
* 04:22 logmsgbot: tstarling synchronizing Wikimedia installation... : r100578
* 03:48 Tim: gave the whole Engineering group write access to the ops-requests queue
* 03:44 Tim: added myself to the operations group in RT so that I could resolve RT #1789
* 02:53 Tim: re-enabled cron job
* 02:26 logmsgbot: tstarling synchronized php-1.18/extensions/CheckUser/maintenance/purgeOldData.php 'r100574'
* 02:21 logmsgbot: LocalisationUpdate completed (1.18) at Mon Oct 24 02:23:42 UTC 2011
* 02:09 logmsgbot: aaron synchronized /php-1.18/resources/jquery.ui/themes/vector/images 'deployed r100571'
* 02:00 Tim: disabled CheckUser purge cron job due to excessive replication lag
* 01:40 Tim: on hume: installed CheckUser purging cron job
* 01:36 Tim: made a shell script to run extensions/CheckUser/maintenance/purgeOldData.php on all wikis, for use in a cron job. Running it on hume to test it.
== October 23 ==
* 02:24 logmsgbot: demon synchronized php-1.18/extensions/Contest/resources/contest.special.welcome.css 'r100525'
* 02:21 logmsgbot: LocalisationUpdate completed (1.18) at Sun Oct 23 02:24:36 UTC 2011
== October 22 ==
* 17:45 mark: storage1 rebooted, but didn't come back (probably iptables issue)
* 17:21 mark: Moved storage1 to private subnet in SVN, and then reverted that change again
* 17:10 mark: Running dist-upgrade on storage1
* 17:03 logmsgbot: ben synchronized wmf-config/db.php 'putting ms2 back in rotation after its reboot and RAID recovery'
* 16:41 RobH: removed myself from nagios contacts and restarted nagios service, feels awesome.
* 16:34 maplebed: copied mysql-at-facebook to ms2; it's now running that instead of the packaged version
* 16:19 mark: ms2 back up
* 16:18 mark: disk ata28 broken on ms2
* 14:56 maplebed: physically powercycling ms2; it's stuck at grub and the mgmt interface is unavailable.
* 04:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31862 - New logo for Tamil Wikibooks'
* 03:55 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31857 - Narayam for Odia Wiktionary'
* 02:21 logmsgbot: LocalisationUpdate completed (1.18) at Sat Oct 22 02:24:14 UTC 2011
* 02:15 logmsgbot: aaron synchronized php-1.18/extensions/CheckUser 'pushed r100488 code'
* 01:22 logmsgbot: ben synchronized wmf-config/db.php 'taking ms2 out of rotation for a reboot - filesystem errors and an xfs kernel oops'
== October 21 ==
* 22:11 logmsgbot: reedy synchronized php-1.18/extensions/WikiLove/WikiLove.hooks.php 'r100464'
* 20:14 logmsgbot: demon synchronized php-1.18/extensions/Contest/includes/ContestContestant.php 'Push r100446. Shame on Reedy for deploying then going afk'
* 20:08 Jeff_Green: taking payments4 out of pybal pool
* 19:36 logmsgbot: reedy synchronizing Wikimedia installation... : Pushing r100440 for contest
* 18:30 mutante: powercycling snapshot3 - mgmt console just showed numbers !?
* 18:20 mutante: snapshot3 - dist-upgrade/kernel,about to reboot...
* 18:06 mutante: rebooting hume after dist-upgrade
* 15:58 mutante: dataset2 - installed security upgrades/kernel, scheduled for reboot RT 1780
* 15:44 mutante: stafford - dist-upgrade/kernel,reboot
* 14:27 mutante: hume - installed security updates but ask aaron for reboot (long running fixBug28348.php related process?)
* 14:06 mutante: < Jeff_Green> alert: fundraising test is beginning
* 13:27 mutante: srv285 - security updates,rebooting
* 13:01 mutante: ekrem out of disk space (RT 1772), gzipped some apache logs, apache2 graceful,...
* 04:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31815 - Add rollbacker group for orwiki'
* 02:17 logmsgbot: LocalisationUpdate completed (1.18) at Fri Oct 21 02:20:20 UTC 2011
* 02:00 logmsgbot: reedy synchronizing Wikimedia installation... : Pulling in r100403 for contest
* 01:04 Tim: on hume: made the files in /etc/cron.d/* not writable wikidev, since that allows privilege escalation from wikidev to root
== October 20 ==
* 22:29 logmsgbot: reedy synchronizing Wikimedia installation... : Checking all Contest code is up to date
* 22:04 logmsgbot: reedy synchronized php-1.18/extensions/Contest/specials/SpecialContestant.php 'Temporary live hack to attempt to fix rating'
* 19:30 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'removed sysop autoreview settings that actually *are* redundant'
* 19:28 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'reverted r2520'
* 18:18 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'critical update to mobile frontend to support iPhone application'
* 18:04 LeslieCarr: testing log abilities
* 17:57 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'removed redundant config'
* 17:48 LeslieCarr: renamed ruthenium and rhodium to msfe1001 and msfe1002 and put onto public vlan
* 17:46 logmsgbot: reedy synchronized php-1.18/extensions/Contest/includes/ContestContestant.php 'r100355'
* 17:45 logmsgbot: reedy synchronized php-1.18/extensions/Contest/includes/ContestantPager.php 'r100355'
* 17:44 logmsgbot: reedy synchronized php-1.18/extensions/Contest/specials/SpecialContest.php 'r100355'
* 17:42 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'removed obsolete $wgFlaggedRevsReviewForDefault var'
* 17:35 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'critical update to mobile frontend to support iPhone application'
* 16:21 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'restored neverBlocked setting removed by mistake recently'
* 16:16 mutante: bayes - installed security updates, need to ask ezachte for reboot, dont want to kill his perl/R procs
* 15:12 mutante: knsq8,knsq9 - dist-upgrade/kernel,reboot
* 14:38 cmjohnson1: nas1 - removing pcie cable and replacing
* 14:32 mutante: knsq7 - dist-upgrade/kernel, reboot
* 14:32 mutante: knsq4,knsq6 - installed security updates (but dpkg fail on varnish pkg)
* 07:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable af dashboard for zhwiki'
* 05:58 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'removed redundant settings'
* 05:54 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'set excludeLastDays for $wgFlaggedRevsAutoconfirm pre-emptively; w/s tweaks'
* 05:28 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'removed redundant settings'
* 05:07 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'removed unused wgFlaggedRevsAutopromoteInRC'
* 05:06 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'removed wgFeedbackNamespaces from testwiki'
* 04:53 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Config cleanup, remove some entries set the same as the default'
* 04:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 28689 wgNewUserMessageOnAutoCreate to true for thwiki'
* 04:36 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'removed redundant config'
* 04:29 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 29003 - Fix $wgLocaltimezone in projects connected with Poland'
* 04:23 logmsgbot: reedy synchronized php-1.18/cache/interwiki.cdb 'Updating interwiki cache to include mediawikiwiki'
* 04:08 logmsgbot: reedy synchronized php-1.18/cache/interwiki.cdb 'Updating interwiki cache'
* 04:04 Reedy: that was bug 24207
* 04:04 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Please set [math] = 0'
* 03:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31585 - Please enable autopatrolled and patroller groups on sr.wikisource'
* 03:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31646 - Usergroups on ScoWp'
* 03:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31646 - Usergroups on ScoWp'
* 03:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31646 - Usergroups on ScoWp'
* 03:27 Reedy: Imports done
* 03:09 Reedy: Running importImages for bug 31267 (may take a while, also in screen as reedy)
* 03:08 Reedy: Extraction complete
* 03:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix typo from bug 31299'
* 03:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 31267 Enable DoubleWiki on fawiki'
* 02:58 Reedy: Extracting 14GB tar.gz file in /tmp on fenari for bug 31568. Running in a screen session as reedy
* 02:55 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 31578 AF on zhwiki'
* 02:52 Reedy: Creating AF tables on zhwiki
* 02:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31814 - Implementing Narayam for Odia wikipedia'
* 02:22 logmsgbot: LocalisationUpdate completed (1.18) at Thu Oct 20 02:25:33 UTC 2011
* 02:11 logmsgbot: reedy synchronizing Wikimedia installation... : r100322 r100323
* 01:31 logmsgbot: reedy synchronized php-1.18/extensions/Contest/includes/ContestReminderJob.php 'r100319'
* 01:31 logmsgbot: reedy synchronized php-1.18/extensions/Contest/includes/ContestContestant.php 'r100319'
* 01:30 logmsgbot: reedy synchronized php-1.18/extensions/Contest/api/ApiMailContestants.php 'r100319'
* 01:30 logmsgbot: reedy synchronized php-1.18/extensions/Contest/Contest.php 'r100319'
* 01:01 logmsgbot: reedy synchronizing Wikimedia installation... : Next round of Contest bugfixes
* 00:08 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Set contest email customisation again'
* 00:04 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Revert email customisation in CommonSettings'
== October 19 ==
* 23:52 logmsgbot: reedy synchronizing Wikimedia installation... : Syncing out the last of the contest updates
* 23:39 logmsgbot: reedy synchronized php-1.18/extensions/Contest/specials/SpecialMyContests.php 'r100296'
* 23:39 logmsgbot: reedy synchronized php-1.18/extensions/Contest/Contest.php 'r100296'
* 23:38 logmsgbot: reedy synchronized php-1.18/extensions/Contest/includes/ContestContestant.php 'r100296'
* 23:32 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Set Contest Email Sender Address/Name, still needs r100292 merging and pushing for this to take affect'
* 23:15 logmsgbot: reedy synchronized php-1.18/extensions/Contest/includes/Contest.class.php 'Comment out submission_count default to stop it overwriting the database value on edit'
* 22:52 Andrew: test test
* 01:13 logmsgbot: reedy synchronized php-1.18/extensions/Contest/specials/SpecialMyContests.php 'r100199'
* 01:10 logmsgbot: reedy synchronized php-1.18/extensions/Contest/Contest.i18n.php 'r100199'
* 00:40 logmsgbot: reedy synchronizing Wikimedia installation... : r100195
== October 18 ==
* 23:27 logmsgbot: reedy synchronized php-1.18/languages/messages/MessagesEn.php 'r100180'
* 22:27 logmsgbot: reedy synchronizing Wikimedia installation... : Add Contest to extensions-list
* 22:24 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Uncomment require line'
* 22:24 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Uncomment require line'
* 22:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Contest extension on testwiki'
* 22:22 Reedy: Created Contest extension tables on testwiki
* 22:19 logmsgbot: reedy synchronizing Wikimedia installation... : Added contest to ExtensionMessages
* 22:16 logmsgbot: reedy synchronizing Wikimedia installation... : Push out Contest extension, not enabled anywhere as of yet
* 21:45 rainman-sr: restarted search9 which seems to have a stale copy of one of the enwp index parts
* 18:21 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'critical update to mobile frontend'
* 17:55 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'critical update to mobile frontend'
* 17:32 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/_search_webkit.html.php 'critical update to mobile frontend'
* 16:49 logmsgbot: awjrichards synchronizing Wikimedia installation... : Pushing CentralNotice updates from r100100, r100099, r100102
* 16:45 Jeff_Green: switched db1025 to slave off of db1008 as temporary middle-master (db9-->db1008-->db1025)
* 13:48 Jeff_Green: grosley is back online, memory upgrade postponed due to mismatched RAM
* 13:32 Jeff_Green: grosley shut down for RAM upgrade
* 02:18 logmsgbot: LocalisationUpdate completed (1.18) at Tue Oct 18 02:21:02 UTC 2011
== October 17 ==
* 21:26 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Updating list of priority countries for landing check'
* 20:47 rainman-sr: moved around things a bit on the search cluster, search7 seems to have been hitting I/O quite a bit. Relocated eswiki to search14, and "other namespaces" index slices for rest of searchpool3 wikis to search15
* 20:35 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'donate.wikipedia.org -> donate.wikimedia.org in wgServers'
* 20:25 logmsgbot: ariel synchronized wmf-config/lucene.php '...and remove from existing pool, oops'
* 20:21 logmsgbot: ariel synchronized wmf-config/lucene.php 'eswiki searches to search14, thanks rainman-sr'
* 20:21 logmsgbot: awjrichards synchronized php/extensions/Translate/Translate.php 'r100004'
* 20:20 logmsgbot: awjrichards synchronized php/extensions/Translate/_autoload.php 'r100004'
* 20:20 logmsgbot: awjrichards synchronized php/extensions/Translate/api/ApiQueryMessageTranslations.php 'r100004'
* 20:19 logmsgbot: awjrichards synchronized php/extensions/Translate/tag/TPParse.php 'r100004'
* 20:19 logmsgbot: awjrichards synchronized php/extensions/Translate/tag/RenderJob.php 
* 20:19 logmsgbot: awjrichards synchronized php/extensions/Translate/MessageCollection.php 'r100004'
* 19:36 logmsgbot: jgreen synchronized wmf-config/CommonSettings.php 'added settings for donatewiki'
* 19:35 logmsgbot: jgreen synchronized wmf-config/InitialiseSettings.php 'added settings for donatewiki'
* 19:33 logmsgbot: jgreen synchronized wmf-config/InitialiseSettings.php 
* 18:38 apergos: tried increasing ulimit for open files from 8192 to 12288 for lsearch on search7, see /etc/init.d/lsearchd. (lots of "too many open files" in the logs) ... if this kills it please lower it again
* 18:08 preilly: done pushing critical MF fixes
* 18:07 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'critical update to mobile frontend'
* 18:01 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'critical update to mobile frontend'
* 17:51 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'critical update to mobile frontend'
* 17:49 apergos: restarted lsearchd on search7 again
* 17:49 preilly: push critical fix for MF
* 17:49 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/_search_webkit.html.php 'critical update to mobile frontend'
* 17:48 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'critical update to mobile frontend'
* 15:41 mark: restarted lsearchd on search7
* 05:32 Tim: restarted lsearchd on search7
* 04:21 notpeter: restarted nagios after a chmod of /etc/nagios/puppet_checks.d
* 02:22 logmsgbot: LocalisationUpdate completed (1.18) at Mon Oct 17 02:25:29 UTC 2011
== October 16 ==
* 23:28 mutante: knsq[17-28] - dist-upgraded/kernel,rebooted
* 22:58 logmsgbot: reedy synchronizing Wikimedia installation... : Push all Chads merges
* 22:54 mutante: restarted lsearchd (lucene) on search3 and 7
* 22:32 mark: Pooled cp3001 into the bits.esams pool
* 21:33 RobH: pybal upload config changed to remove sq47 per rt1597
* 20:59 notpeter: something seems srsly messed up with nagios. restarting it again...
* 20:55 notpeter: chmoded /etc/nagios/puppet_checks.d/ and restarted nagios
* 20:49 mark: Made varnishhtcpd not run as root on cp104[34]
* 20:47 notpeter: Restarted lsearchd on search7
* 20:38 mark: Accidently broke knsq6, will fix
* 19:41 RoanKattouw: Removing all broken symlinks in common/docroot
* 19:22 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Remove action=parse hack'
* 18:42 mutante: changed loglevel to ERROR in /usr/sbin/ganglia_parser
* 18:34 mutante: killed ganglia_parser.log to save spence from running out of disk space
* 18:29 mutante: knsq10,knsq11,knsq15,knsq16 - dist-upgrade/kernel,reboot
* 18:27 RobH: nikerabbit now has shell deploy access via rt 1475
* 17:29 RobH: reviewed exim panic log on lily, williams, and mchenry for non zero size notifications. all errors were quite old (week or more) so they have been cleared out to stop the log spamming emails
* 16:12 binasher: wally wally wally wally wally wally wally wally world
* 16:04 logmsgbot: tstarling synchronized php-1.18/extensions/ExtensionDistributor/svn-invoker.conf 
* 16:03 Tim: fixed ExtensionDistributor
* 13:47 mark: Restarted lsearchd on search7
* 13:46 mark: Restarted pdns on ns2
* 02:19 logmsgbot: LocalisationUpdate completed (1.18) at Sun Oct 16 02:22:10 UTC 2011
== October 15 ==
* 23:24 mutante: knsq1,knsq2,knsq4 - dist-upgrade/kernel,reboot
* 23:20 logmsgbot: reedy synchronized php-1.18/includes/specials/SpecialBrokenRedirects.php 
* 22:41 mutante: all amssq* servers upgraded now
* 21:58 LeslieCarr: reloading apache2 on spence
* 21:54 mutante: amssq50-amssq61 - dist-upgrade/kernel,reboot
* 21:50 mutante: amssq62 - power up,dist-upgrade/kernel,squid clean,reboot..
* 20:51 mark: Stopped puppet on cp1043 and 1044
* 20:48 RobH: updating dns again, forgot to change the mgmt entries
* 20:42 RobH: updating dns for cp3001/3002 per rt#1734
* 20:37 Tim: running refreshImageMetadata.php on all wikis, in screen on hume
* 19:35 logmsgbot: tstarling synchronizing Wikimedia installation... : 
* 19:00 mutante: amssq42-amssq50 - dist-upgrade/kernel,reboot (still one at a time)
* 18:33 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Reverted live hack as tim fixed symlinks'
* 18:33 logmsgbot: tstarling synchronizing Wikimedia installation... : 
* 18:15 logmsgbot: tstarling synchronizing Wikimedia installation... : 
* 18:08 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Fix CategoryTree'
* 17:55 logmsgbot: reedy synchronized php-1.18/extensions/CategoryTree/CategoryTree.css 'pushpush'
* 16:46 Ryan_Lane: adding wmflabs domain in DNS
* 16:28 mutante: amssq41,amssq42 - dist-upgrade/kernel,reboot
* 03:18 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Sitename for bdwikimedia'
* 02:54 binasher: replaced the puppet manifests and vcl templates for mobile. now with more shiny!
* 02:39 RobH: updated dns for magnesium.w.o
* 02:23 logmsgbot: reedy ran sync-common-all 'Rest of bdwikimedia'
* 02:23 logmsgbot: LocalisationUpdate completed (1.18) at Sat Oct 15 02:25:53 UTC 2011
* 01:54 logmsgbot: reedy ran sync-common-all 
* 01:46 RobH: updated dns for copper.w.o
* 01:40 mutante: amssq35-amssq40 - dist-upgrade/kernel, reboot
* 01:39 RobH: updating dns for some eqiad hosts
* 00:29 mutante: amssq31-amssq35 - dist-upgraded/kernel, reboot
* 00:27 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Finish account creation throttle exemption for Asaf and uncomment it'
* 00:15 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Put in a commented-out account creation throttle exemption for a Botswana event'
== October 14 ==
* 23:10 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'missing file for mobile frontend'
* 23:05 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/stylesheets/common.css 'missing file for mobile frontend'
* 22:19 preilly: pushing critical mobile frontend fix�es
* 22:18 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'emergency update to mobile frontend'
* 22:18 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/application.min.js 'emergency update to mobile frontend'
* 22:18 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.html.php 'emergency update to mobile frontend'
* 22:17 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/_search_webkit.html.php 'emergency update to mobile frontend'
* 21:18 Jeff_Green: payments1/payments2 back online with mw 1.17
* 21:14 awjr: mw 1.17 deployment to payments1 complete
* 20:46 awjr: mw 1.17 deployment to payments2 complete
* 18:50 preilly: finish pushing fix for mobile frontend 
* 18:50 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/_search_webkit.html.php 'emergency update to mobile frontend'
* 18:50 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/_search_webkit.html.php 'emergency update to mobile frontend'
* 18:41 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'emergency update to mobile frontend'
* 18:40 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'emergency update to mobile frontend'
* 18:40 preilly: pushing emergency fix to xml view of mobile frontend
* 18:11 Jeff_Green: rm'd /var/log/ganglia/ganglia_parser.log to recover 27GB on spence
* 18:11 Jeff_Green: rm'd /var/log/ganglia/ganglia_parser.log to recover 27GB
* 17:57 Jeff_Green: stopping various ganglia services too on spence b/c / is at 99% and the box nearly a doorstop
* 17:56 Jeff_Green: stopping nagios on spence b/c / is at 99% and the box nearly a doorstop
* 17:00 Jeff_Green: stopped mysql on db10 before it crashes from overrunning disk . . .
* 02:22 logmsgbot: LocalisationUpdate completed (1.18) at Fri Oct 14 02:24:42 UTC 2011
== October 13 ==
* 22:24 logmsgbot: aaron synchronized php-1.18/includes/api/ApiBlock.php 'r99739'
* 20:39 preilly: pushing weekly update
* 20:39 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'normal update to mobile frontend'
* 20:37 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'normal update to mobile frontend'
* 19:53 Tim: running svn cleanup on ExtensionDistributor trunk
* 18:56 logmsgbot: catrope synchronized php-1.18/includes/Title.php 'r99717'
* 18:56 logmsgbot: catrope synchronized php-1.18/includes/parser/ParserCache.php 'r99717'
* 18:56 logmsgbot: catrope synchronized php-1.17/includes/Title.php 'r99717'
* 18:56 logmsgbot: catrope synchronized php-1.17/includes/parser/ParserCache.php 'r99717'
* 16:56 RobH: raised alert limits on ps1-b5-sdtpa from 24 to 26 amps (it tops out at 28 amps)
* 16:42 Reedy: r99694 being merged, even
* 16:40 logmsgbot: reedy synchronized php-1.18/includes/api/ApiFeedWatchlist.php 'r99696'
* 16:40 Tim: rebooting srv171, not responding on ssh or serial but still responding on HTTP
* 16:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix logos per bug 31525 bug 31668'
* 02:17 logmsgbot: LocalisationUpdate completed (1.18) at Thu Oct 13 02:20:22 UTC 2011
== October 12 ==
* 23:07 Tim: ran puppetd -tv on srv171 which started apache
* 23:04 Tim: ran sync-common on srv171
* 22:58 RobH: rebooting srv171, its serial console is unresponsive
* 22:54 logmsgbot: reedy synchronized php-1.18/extensions/UploadWizard/resources/mw.fileApi.js 'r99654'
* 22:53 logmsgbot: reedy synchronized php-1.18/resources/mediawiki.special/mediawiki.special.upload.js 'r99654'
* 22:40 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 're-enabling ArticleFeedback on the other wikis where it was temporarily disabled'
* 21:44 Tim: running PrefSwitch-addusertext.sql on hiwiki, huwiki, ptwiki, ptwikibooks
* 21:36 Ryan_Lane: adding wmflabs.* and wikimedialabs.* domains to DNS
* 21:28 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 're-enabling article feedback for enwiki'
* 21:13 logmsgbot: asher synchronized php-1.18/includes/job/JobQueue.php 'temporarily removing select for update'
* 20:56 Jeff_Green: moved donate wiki VirtualHost from remnants.conf to main.conf so it can catch donate.wikipedia.org
* 19:55 logmsgbot: py synchronized wmf-config/mc.php 'swapping two mc instances for LAST TWO upgrades to lucid'
* 19:17 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'disabling ArticleFeedback because it is generating SQL fatal errors due to broken SimpleSurvey'
* 19:17 Reedy: Created WikiLove tables on all wikis that require it
* 19:10 logmsgbot: reedy synchronized php-1.18/includes/UserMailer.php 'r99646 Hopefully reduce number of enotif jobs by not inserting them in the first place if they're not going to send anything'
* 18:55 Reedy: Created wikilove tables on nowiki
* 18:38 Tim: running various tests against en.wikipedia.org to try to find rogue apaches and/or "can't connect to localhost" db errors
* 17:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31295 - Define transwiki import source for kmwikt (Khmer Wiktionary) from en, fr, th, and lo'
* 17:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31372 - Install Narayam in all Sinhala wiki projects'
* 17:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31525 - Change logo on sg.wikipedia'
* 17:28 logmsgbot: reedy synchronized php-1.18/resources/startup.js 'touch'
* 17:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31538 - Narayam Installation Request'
* 17:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31553 - Enable preference to send notification emails for watchlists on the WMUK wiki'
* 17:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'For bug 31539 Swap some urls to relative'
* 17:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable ClickTracking as prequisite for AF on mlwiki'
* 16:37 RobH: sq31 reinstalling, had dead drives replaced
* 16:30 logmsgbot: py synchronized wmf-config/mc.php 'swapping two mc instances for upgrade to lucid'
* 16:29 RobH: sq31 is being poked at
* 15:39 notpeter: also removed srv192 and srv249 from bits lb pool for upgrades to lucid
* 15:36 notpeter: restarted nagios
* 05:47 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 31634 - qqq extra language missing for Translate on Meta'
* 02:31 Jeff_Green: tweaked remnants.conf and redirects.conf, running sync-apache and apache-graceful-all
* 02:14 logmsgbot: LocalisationUpdate completed (1.18) at Wed Oct 12 02:17:03 UTC 2011
* 01:38 Aaron|away: running mwscriptwikiset extensions/FlaggedRevs/maintenance/fixBug28348.php flaggedrevs.dblist
* 00:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Translate on meta'
* 00:25 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Re-enable CUS on mw.org, Erik made me'
* 00:22 binasher: built udpprofile 1.1 and deployed on spence
* 00:20 awjr: archived and removed banner impression and landing page impression udp log files from hume -> storage3 (from 9/15/2011- 10/11/2011)
* 00:17 mutante: sq31 - not coming back after reboot - nothing on console
* 00:07 mutante: sq31 - dist-upgrade/kernel,reboot
== October 11 ==
* 23:42 logmsgbot: reedy synchronized php-1.18/extensions/Translate/_autoload.php 'r99598'
* 23:42 logmsgbot: reedy synchronized php-1.18/extensions/Translate/check-blacklist.php 'r99598'
* 23:42 logmsgbot: reedy synchronized php-1.18/extensions/Translate/tag/TPParse.php 'r99598'
* 23:41 logmsgbot: reedy synchronized php-1.18/extensions/Translate/MediaWikiMessageChecker.php 'r99598'
* 23:08 logmsgbot: reedy synchronized php-1.18/resources/startup.js 'touch'
* 23:05 logmsgbot: reedy synchronizing Wikimedia installation... : Just syncing everything
* 23:03 logmsgbot: reedy synchronized php-1.18/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'r99594'
* 22:42 logmsgbot: LocalisationUpdate completed (1.18) at Tue Oct 11 22:45:19 UTC 2011
* 22:15 logmsgbot: aaron synchronized php-1.18/extensions/FlaggedRevs/presentation/FlaggedPageView.php 'deployed r99584'
* 22:08 mutante: sq86 - dist-upgrade/kernel reboot (done with upload squids:)
* 22:05 logmsgbot: LocalisationUpdate failed
* 21:58 mutante: gallium - security updates,kernel upgrade,reboot
* 21:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Allow 'crats on wikis running translate to grant user group'
* 21:55 RoanKattouw: Rebuilding the LocalisationUpdate cache from scratch
* 21:49 logmsgbot: LocalisationUpdate completed (1.18) at Tue Oct 11 21:52:28 UTC 2011
* 21:48 notpeter: re rt 1422, srv217 is messed up, going to powerdown for now
* 21:44 awjr: restarted jenkins on grosley after mysterious death
* 21:43 mutante: sq85 - dist-upgrade/kernel,reboot
* 21:43 logmsgbot: reedy synchronized php-1.18/extensions/Translate/Translate.php 'Kill version_compare for jquery.form'
* 21:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Translate on TestWiki'
* 21:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add base Translate config from http://etherpad.wikimedia.org/translate'
* 21:32 Reedy: Created Translate tables on testwiki
* 21:21 mutante: sq84 - dist-upgrade/kernel, reboot
* 21:20 RobH: depooled payments1/2 per awjr, payments 3/4 are in service pool
* 21:19 RobH: payments3/4 repooled per arthur
* 21:00 mutante: sq83 - dist-upgrade/kernel,reboot
* 20:53 Reedy: Created Translate tables on metawiki
* 20:36 mutante: sq82 - dist-upgrade/kernel,reboot
* 20:26 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add requirce_once of translate if'
* 20:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add default off wmgUseTranslate'
* 20:11 logmsgbot: reedy synchronizing Wikimedia installation... : Initial push of translate extension
* 19:04 notpeter: putting srv250-srv257 back into api pool
* 18:50 mutante: sq81 - dist-upgrade/kernel,reboot
* 18:40 Tim: added libvips-tools to image scalers via puppet (second attempt)
* 18:24 mutante: sq80 - dist-upgrade/kernel,reboot
* 17:59 Reedy: For last push "Per RT #1623, comment out srv172 from external loads"
* 17:59 logmsgbot: reedy synchronized wmf-config/db.php 'Per RT'
* 17:51 mutante: sq79 - dist-upgrade/kernel, rebooting..
* 17:48 notpeter: putting srv187-srv189 and srv214-srv218 back into api pool
* 17:42 mutante: amssq49 - power up,dist-upgrade,squid clean,reboot
* 17:41 logmsgbot: catrope synchronized php-1.18/languages/Names.php 'r99504'
* 17:40 logmsgbot: catrope synchronized php-1.18/languages/messages/MessagesEn.php 'r99504'
* 17:40 logmsgbot: reedy synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'r99507'
* 17:40 logmsgbot: catrope synchronized php-1.18/extensions/Narayam/js/ext.narayam.rules.si-singlish.js 'r99504'
* 17:39 logmsgbot: reedy synchronized ../common/php-1.18/extensions/MobileFrontend/MobileFrontend.php 'r99507'
* 17:38 logmsgbot: reedy synchronized ../common/php-1.18/extensions/MobileFrontend/MobileFrontend.php 'r99507'
* 17:11 Tim: installed libvips-tools on srv220 experimentally
* 15:46 logmsgbot: catrope synchronized php-1.18/includes/api/ApiQueryAllUsers.php 'r99485'
* 15:23 logmsgbot: catrope synchronized php-1.18/includes/diff/DifferenceEngine.php 'Fix error'
* 15:06 logmsgbot: catrope synchronized php-1.18/includes/api/ApiQueryWatchlist.php 'r99480'
* 14:43 logmsgbot: py synchronized wmf-config/mc.php 'shuffling 1 more memecache instance for lucid upgrades'
* 14:37 notpeter: RoanKattouw also restarting mw68
* 14:36 RoanKattouw: And mw73 too
* 14:35 RoanKattouw: Gracefulling Apache on mw69, mw70, mw72
* 14:34 RoanKattouw: Gracefulling Apache on mw71, it's throwing weird notices
* 14:31 logmsgbot: py synchronized wmf-config/mc.php 'shuffling 2 more memecache instances for lucid upgrades'
* 14:20 logmsgbot: py synchronized wmf-config/mc.php 'shuffling 2 more memecache instances for lucid upgrades'
* 14:19 logmsgbot: catrope synchronized php-1.18/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'Fix syntax error (!!)'
* 14:15 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'null -> false'
* 14:14 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'null -> false'
* 14:11 notpeter: removing srv187-srv189, srv214-srv218, and srv250-srv257 from api pool for upgrades to lucid
* 14:11 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Fix undefined var in ArticleFeedback config'
* 14:10 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Fix undefined var in ArticleFeedback config'
* 13:53 mark: Backported lcms2 to lucid and installed it in apt repository lucid-wikimedia (updates RT #1696)
* 08:23 p858snake|l: Test wiki broken (500 error) since at least 1513AEST (bz is being silly for me, and so I don't forget)
* 02:03 logmsgbot: LocalisationUpdate failed
* 01:11 mutante: amssq51 - power up, dist-upgrade, reoot, squid clean
* 01:07 mutante: sq77,sq78 - dist-upgraded/kernel, rebooted
* 00:30 mutante: sq74,sq75,sq76 - dist-upgrade/kernel, reboot
* 00:28 apergos: running gzip on september banner log files in /a/static/uncompressed/udplogs on hume, form root screen session
== October 10 ==
* 23:24 mutante: amssq33 - power up,dist-upgrade,squid clean,reboot
* 22:40 mutante: sq70 - power up,dist-upgrade,reboot
* 22:28 mutante: sq71,sq72,sq73 - dist-upgraded/kernel,rebooted
* 21:11 logmsgbot: LocalisationUpdate completed (1.18) at Mon Oct 10 21:14:28 UTC 2011
* 21:09 logmsgbot: LocalisationUpdate failed
* 21:03 logmsgbot: LocalisationUpdate completed (1.18) at Mon Oct 10 21:05:53 UTC 2011
* 20:47 mutante: sq66,sq68 - upgraded kernel/reboot
* 20:34 logmsgbot: LocalisationUpdate completed (1.18) at Mon Oct 10 20:36:46 UTC 2011
* 20:29 logmsgbot: LocalisationUpdate completed (1.18) at Mon Oct 10 20:32:22 UTC 2011
* 20:17 logmsgbot: LocalisationUpdate completed (1.18) at Mon Oct 10 20:20:06 UTC 2011
* 20:04 notpeter: putting srv290-srv301 back into api pool
* 19:58 logmsgbot: reedy synchronized php-1.18/maintenance/wmf/clearMessageBlobs.php 
* 19:50 logmsgbot: LocalisationUpdate completed (1.18) at Mon Oct 10 19:53:32 UTC 2011
* 19:25 logmsgbot: LocalisationUpdate completed (1.18) at Mon Oct 10 19:28:09 UTC 2011
* 18:24 Jeff_Green: spence gets a cleansing reboot
* 17:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable ClickTracking on zhwiki as prequisite for bug 31578 Install AF'
* 17:33 notpeter: removing srv290-srv301 from api lb pool for upgrades to lucid
* 17:09 RobH: memcached started on srv256, should clear spence check
* 16:55 notpeter: putting mw62-mw74 into api pool
* 16:16 RobH: mw1102 voltage errors, poking around system, it may possibly reboot
* 14:15 notpeter: restarting nagios and ganglia on spence
* 02:22 logmsgbot: LocalisationUpdate completed (1.18) at Mon Oct 10 02:25:33 UTC 2011
* 00:47 Jeff_Green: srv290, srv292, srv294, srv298 power cycled, see RT #1685 for more info
== October 9 ==
* 17:58 Jeff_Green: power cycled srv188, it was unresponsive and com2 was stuck in use
* 17:52 Jeff_Green: power cycled srv187, it was unresponsive and com2 was stuck in use
* 02:15 logmsgbot: LocalisationUpdate completed (1.18) at Sun Oct 9 02:17:53 UTC 2011
== October 8 ==
* 02:20 logmsgbot: LocalisationUpdate completed (1.18) at Sat Oct 8 02:23:10 UTC 2011
* 01:27 mutante: sq63,sq64,sq65 - upgraded kernel/reboot (earlier today, with lots of time in between)
* 00:53 logmsgbot: robla synchronized php-1.18/tests/ 'syncing tests for r99292'
* 00:43 logmsgbot: robla synchronized php-1.18/languages/classes/LanguageTr.php 'deployed r99292 to fix bug 31490 (Turkish lcfirst/ucfirst prob)'
* 00:19 logmsgbot: aaron synchronized php-1.18/includes/parser 'deployed r99265'
* 00:07 mutante: /dev/mapper/static-uncompressed (local on hume) is 100% full!
* 00:04 logmsgbot: aaron synchronized php-1.18/extensions/GlobalUsage/GlobalUsageQuery.php 'deployed r99284'
== October 7 ==
* 22:50 mutante: installing security updates on fenari
* 22:27 logmsgbot: aaron synchronized php-1.18/resources/jquery/jquery.js 'deployed r99264'
* 22:12 logmsgbot: neilk synchronized php-1.18/extensions/UploadWizard/resources/mw.UploadWizardLicenseInput.js 'fix issue with UploadWizard next button on licensing step and jquery 1.6.4'
* 22:11 logmsgbot: aaron synchronized php-1.18/includes/media/Bitmap.php 'deployred r99274'
* 21:22 maplebed: ganglios is installed on spence; nagios can now query ganglia metrics.
* 20:30 RobH: srv284 os install in progress
* 19:26 logmsgbot: aaron synchronized php-1.18/maintenance/addwiki.php 
* 19:14 logmsgbot: aaron synchronized php-1.18/maintenance/addwiki.php 'r99238'
* 19:13 RobH: updated netboot.cfg for cp1001-1042 on brewster
* 19:12 RobH: installing os on cp1001-cp1042 per RT 1679
* 18:25 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.18
* 16:12 RobH: mw59-mw74 OS installed, awaiting puppetization
* 15:47 RobH: installing os on mw59-mw75
* 14:58 mark: Merged lots of puppet changes, gonna hide now
* 14:03 RobH: updated dns for bd.wikimedia.org, project still not live
* 13:52 notpeter: stopping apache and job queue runners on all external store boxes
* 13:15 notpeter: powercycling srv266, as this is one of our favorite pastimes.
* 12:21 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Replace expired account creation throttle exemption for an India event with one for a Brazil event'
* 05:46 logmsgbot: aaron synchronized php-1.18/extensions/GlobalBlocking/SpecialGlobalBlockStatus.php 'r99198'
* 05:43 logmsgbot: aaron synchronized php-1.18/languages/messages/MessagesEn.php 'r99179'
* 05:42 logmsgbot: aaron synchronized php-1.18/maintenance/language/messages.inc 'r99179'
* 02:26 logmsgbot: LocalisationUpdate completed (1.17) at Fri Oct 7 02:29:03 UTC 2011
* 02:21 logmsgbot: LocalisationUpdate completed (1.18) at Fri Oct 7 02:24:11 UTC 2011
* 00:37 logmsgbot: aaron synchronized php-1.18/includes/specials/SpecialUnblock.php 'deployed r99173'
== October 6 ==
* 23:20 mutante: sq60,sq61,sq62 - updated kernel, rebooted
* 22:57 logmsgbot: aaron synchronized php-1.18/includes/specials/SpecialBlock.php 'deployed r99163'
* 22:57 Tim: restarted nginx on ms5
* 22:40 logmsgbot: aaron synchronized php-1.18/extensions/FlaggedRevs/presentation/RevisionReviewFormUI.php 'deployed r99159'
* 22:28 ^demon: That was for r99156
* 22:26 logmsgbot: catrope synchronized php-1.18/languages/Language.php 'Chad made me'
* 21:29 RobH: srv266 placed back in apache service
* 21:27 logmsgbot: robh synchronized wmf-config/mc.php 'putting'
* 21:25 guillom: image scalers have been having issues for the last ~20 minutes
* 21:12 mutante: sq5[0-9] - upgraded kernel
* 21:11 logmsgbot: robh synchronized wmf-config/mc.php 'pulling'
* 20:59 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.17 for comparison
* 20:56 RobH: depooled srv266 so chris can troubleshoot hardware issues per RT 1432
* 20:55 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'normal update to mobile frontend'
* 20:54 preilly: pushing weekly mobile frontend changes 
* 20:54 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/javascripts/application.js 'normal update to mobile frontend'
* 20:53 logmsgbot: preilly synchronized php/extensions/MobileFrontend/javascripts/application.js 'normal update to mobile frontend'
* 20:52 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'normal update to mobile frontend'
* 20:41 logmsgbot: catrope synchronized php-1.18/includes/api/ApiFormatXml.php 'r99137'
* 20:28 notpeter: removing srv159,srv162,srv171,srv183,srv186 from the apaches pool for upgrade to lucid
* 20:15 notpeter: taking all external store apaches out of the apaches pool (we have a lot of extra apache capacity...)
* 19:35 notpeter: putting srv190, srv194-srv205, srv207-srv213, srv225 into the apaches pool
* 19:22 mutante: sq45,sq46,sq49 - dist-upgrade/kernel,reboot
* 19:11 logmsgbot: catrope synchronized docroot/mediawiki/xml/api/index.html 'bug 31437'
* 19:11 logmsgbot: dzahn synchronized wmf-config/InitialiseSettings.php 'updating logo for szlwiki'
* 18:55 mutante: sq48,sq43,sq44 - dist-upgrade/kernel,reboot
* 14:51 notpeter: and what the f-bomb is up with srv225? why was it never in the apaches pybal pool?
* 14:51 notpeter: rebootin' and reimagin' srv190, srv194-srv213, srv225
* 14:08 logmsgbot: catrope synchronized php-1.18/includes/objectcache/MemcachedClient.php 'r99107'
* 14:05 logmsgbot: py synchronized wmf-config/mc.php 'no substantive change, just record keeping.'
* 13:47 logmsgbot: catrope synchronized php-1.18/includes/specials/SpecialContributions.php 'r99105'
* 13:44 logmsgbot: catrope synchronized php-1.18/includes/specials/SpecialContributions.php 'r99103'
* 13:16 logmsgbot: catrope synchronizing Wikimedia installation... : r99094, r99098
* 13:06 logmsgbot: catrope synchronized php-1.18/extensions/WikiEditor/modules/jquery.wikiEditor.toolbar.config.js 'stray comma'
* 12:43 mark: Rebooting ms1002
* 12:21 logmsgbot: catrope synchronized php-1.18/skins/common/shared.css 'r99090'
* 10:34 mark: Reverted varnish backend timeout increase
* 10:10 mark: Fixed iptables rules on gallium for contint
* 04:35 logmsgbot: tstarling synchronized php-1.18/includes/specials/SpecialBlockList.php 
* 02:40 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: switching remaining tr wikis back to 1.18
* 02:36 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: switching trwiktionary to 1.18
* 02:34 logmsgbot: tstarling synchronized php-1.18/languages/classes/LanguageTr.php 
* 02:25 logmsgbot: LocalisationUpdate completed (1.17) at Thu Oct 6 02:27:40 UTC 2011
* 02:20 logmsgbot: LocalisationUpdate completed (1.18) at Thu Oct 6 02:22:58 UTC 2011
* 01:50 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files: rolling back on tr*
* 01:40 logmsgbot: reedy synchronized php-1.18/resources/startup.js 'For good measure'
* 01:40 logmsgbot: reedy synchronized php-1.18/extensions/skins/Donate/Donate.class.php 'r99073'
* 01:39 logmsgbot: reedy synchronized php-1.18/extensions/skins/Schulenburg/Schulenburg.class.php 'r99073'
* 01:39 logmsgbot: reedy synchronized php-1.18/extensions/skins/Tomas/Tomas.class.php 'r99073'
* 01:27 AaronSchulz: It was robla who pushed "enter" though
* 01:27 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: all wikis to 1.18
* 01:21 logmsgbot: tstarling synchronized php-1.18/includes/parser/StripState.php 'r99071'
* 01:17 logmsgbot: aaron synchronized php-1.18/includes/SpecialPage.php 'deployed r99069'
* 01:06 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: jawiki,ruwiki to 1.18
* 01:03 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: arwiki to 1.18
* 00:53 logmsgbot: tstarling synchronized php-1.18/includes/Wiki.php 'r99065'
* 00:53 logmsgbot: tstarling synchronized php-1.18/includes/cache/MessageCache.php 'r99065'
* 00:52 logmsgbot: tstarling synchronized php-1.18/includes/DefaultSettings.php 'r99065'
* 00:15 logmsgbot: reedy synchronized php-1.18/extensions/OpenSearchXml/ApiOpenSearchXml.php 'toString to getMessage'
* 00:10 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add bug 27452 debug group'
* 00:08 logmsgbot: reedy synchronized php-1.18/extensions/OpenSearchXml/ApiOpenSearchXml.php 'Brions patch https://gist.github.com/1266127'
== October 5 ==
* 23:39 logmsgbot: reedy synchronized php-1.18/includes/objectcache/MemcachedClient.php 'r99061'
* 23:38 logmsgbot: aaron synchronized php-1.18/extensions/AbuseFilter/AbuseFilter.parser.php 'deployed r99060'
* 23:27 logmsgbot: midom synchronized wmf-config/InitialiseSettings.php 
* 23:15 logmsgbot: midom synchronized wmf-config/InitialiseSettings.php 
* 23:07 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: dewiki to 1.18
* 23:02 awjr: MW 1.17 deployment to payments4 done
* 22:52 awjr: updating MW instance on payments4 to 1.17, deploying from http://svn.wikimedia.org/svnroot/mediawiki/branches/fundraising/deployment/payments_1.17
* 22:50 awjr: reenabling payments4 as mysqlhost in $wgDBservers on payments2
* 22:49 awjr: reenabling payments4 as mysqlhost in $wgDBservers on payments1
* 22:33 awjr: MW 1.17 deployment to payments3 done
* 22:31 awjr: updating MW instance on payments3 to 1.17, deploying from http://svn.wikimedia.org/svnroot/mediawiki/branches/fundraising/deployment/payments_1.17
* 21:59 RobH: updated jetpack plugin on blog.w.o
* 21:33 logmsgbot: py synchronized wmf-config/mc.php 'aaaaaand, one last mc instances switch'
* 21:31 logmsgbot: py synchronized wmf-config/mc.php 'pushing new mc.php to swap 2 more mc instances'
* 21:27 Ryan_Lane: restarting apache on payments2
* 21:25 Ryan_Lane: restarting apache on payments1
* 21:11 Ryan_Lane: technically, I'm blocking all bots on bits for itwiki
* 21:11 Ryan_Lane: blocking googlebot from itwiki on bits, per request from google
* 21:09 logmsgbot: py synchronized wmf-config/mc.php 'pushing new mc.php to swap 2 more mc instances'
* 21:02 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Adding CH to list of priority countries for LandingCheck'
* 20:53 mutante: srv155 - started apache
* 20:51 mutante: re-installed mysql-server on srv155 to fix dpkg state "pi"
* 20:49 logmsgbot: py synchronized wmf-config/mc.php 'pushing new mc.php to swap 2 more mc instances'
* 20:35 mutante: ms1002 - dpkg <defunct> at kernel install
* 20:34 mutante: mw23 - sits at partition disk dialog
* 20:26 logmsgbot: py synchronized wmf-config/mc.php 'pushing new mc.php to swap 2 more mc instances'
* 20:15 logmsgbot: catrope synchronized php-1.18/includes/cache/SquidUpdate.php 'r99043'
* 20:14 logmsgbot: catrope synchronized php-1.18/includes/GlobalFunctions.php 'r99043'
* 20:14 logmsgbot: catrope synchronized php-1.18/includes/Defines.php 'r99043'
* 20:11 logmsgbot: py synchronized wmf-config/mc.php 'pushing new mc.php to swap 2 more mc instances'
* 20:03 logmsgbot: catrope synchronized php-1.18/includes/SquidPurgeClient.php 'r99042'
* 19:58 logmsgbot: py synchronized wmf-config/mc.php 'pushing new mc.php to swap 2 more mc instances'
* 19:54 logmsgbot: demon synchronized php-1.18/includes/specials/SpecialAllmessages.php 'r99039'
* 19:35 logmsgbot: py synchronized wmf-config/mc.php 'pushing new mc.php to swap 2 more mc instances'
* 19:32 RobH: mw1123, mw1124, mw1132 fixed, os installing on them
* 19:29 logmsgbot: catrope synchronized php-1.18/resources/jquery/jquery.tablesorter.js 'r99032'
* 19:21 RobH: set payments3 and payments4 to false in pybal, depooling them per request of jeff via arthur
* 19:19 RobH: mw1111 & mw1122 fixed
* 19:15 logmsgbot: py synchronized wmf-config/mc.php 'pushing new mc.php to swap 2 more mc instances'
* 19:01 mutante: db12 - fix date/NTP
* 18:58 logmsgbot: py synchronized wmf-config/mc.php 'pushing new mc.php to swap 2 more mc instances'
* 18:56 RobH: mw1120 fixed
* 18:52 RobH: mw1097 & mw1112 fixed, os installing
* 18:49 logmsgbot: catrope synchronized php-1.18/extensions/LiquidThreads/lqt.js 'Actually sync the right file for r99024'
* 18:48 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/stylesheets/iphone.css 'update to mobile frontend emergency fix for search bar width'
* 18:48 preilly: pushing style fix for new search bar width 
* 18:48 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/stylesheets/iphone2.css 'update to mobile frontend emergency fix for search bar width'
* 18:47 logmsgbot: preilly synchronized php/extensions/MobileFrontend/stylesheets/iphone.css 'update to mobile frontend emergency fix for search bar width'
* 18:47 logmsgbot: preilly synchronized php/extensions/MobileFrontend/stylesheets/iphone2.css 'update to mobile frontend emergency fix for search bar width'
* 18:46 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/stylesheets/iphone2.css 'update to mobile frontend emergency fix for search bar width'
* 18:46 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/stylesheets/iphone.css 'update to mobile frontend emergency fix for search bar width'
* 18:44 notpeter: removing srv190, srv194-srv213, and srv225 from apaches pool for upgrades
* 18:38 logmsgbot: catrope synchronized php-1.18/extensions/LiquidThreads/classes/View.php 'r99024'
* 18:38 mutante: amssq59 - power up,dist-upgrade/kernel,reboot
* 18:31 logmsgbot: catrope synchronized php-1.18/extensions/LiquidThreads/classes/View.php 'r99017'
* 18:29 RobH: mw1113 fixed
* 18:26 RobH: fixed mw1090, mw1092, os installing
* 18:25 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/_search_webkit.html.php 'update to mobile frontend emergency fix for search bar width'
* 18:24 mutante: sq48 is running but offline - cable connected? eth0: <NO-CARRIER
* 18:24 preilly: push fix for search bar width 
* 18:24 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/_search_webkit.html.php 'update to mobile frontend emergency fix for search bar width'
* 18:21 notpeter: putting srv226-srv242 back into apaches pool
* 18:07 notpeter: putting srv243-srv247 back into apaches pool
* 17:26 RobH: cleared errors on mw1067 and mw1070, power was loose
* 17:15 RobH: mw1019 repaired, os install in progress
* 17:10 RobH: mw1002 & mw1016 repaired, os install in progress
* 16:24 Jeff_Green: swinging payments.wikimedia.org DNS from 208.80.152.7
* 16:12 logmsgbot: catrope synchronized php-1.18/includes/MessageBlobStore.php 
* 16:07 RobH: mw1010 mgmt fixed, os installing
* 16:07 logmsgbot: reedy synchronized php-1.18/extensions/CentralAuth/CentralAuthUser.php 
* 16:06 logmsgbot: reedy synchronized php-1.18/extensions/Quiz/Quiz.class.php 
* 16:03 logmsgbot: reedy synchronized php-1.18/includes/api/ApiUpload.php 'r99003 AGAIN'
* 15:57 logmsgbot: reedy synchronizing Wikimedia installation... : Seems we might have a few files out of sync
* 14:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31371 - Assamese Vs Latin numerals in templates using parser functions, Set $wgTranslateNumerals = false; for aswiki'
* 14:42 mark: Allocated LVS service IPs for mobile-lb.pmtpa, and payments-lb.(pmtpa|eqiad)
* 14:21 logmsgbot: reedy synchronized php-1.18/extensions/UserDailyContribs/api/ApiUserDailyContribs.php 'r99006'
* 14:20 logmsgbot: reedy synchronized php-1.18/extensions/UserDailyContribs/UserDailyContribs.php 'r99006'
* 13:58 logmsgbot: reedy synchronized php-1.18/includes/specials/SpecialEmailuser.php 'r99004'
* 13:53 logmsgbot: reedy synchronized php-1.18/includes/api/ApiUpload.php 'r99003'
* 13:39 mark: Started oprofile on ms1002
* 13:15 logmsgbot: catrope synchronized php-1.18/includes/api/ApiQueryCategoryMembers.php 'r98998'
* 13:14 robh: updated blog theme per RT 1534
* 13:11 robh: removed binlogs on db9 older than 20 days to free up space (was in warning in nagios)
* 12:19 mark: LDAP sucks, news at 11
* 12:16 mark: Attempted reconfiguring and stopping opendj on sanger, didn't work. Ended up with SIGKILL.
* 08:44 logmsgbot: catrope synchronized php-1.18/resources/jquery/jquery.js 'touch'
* 06:31 logmsgbot: tstarling synchronized php-1.18/extensions/Collection/js/collection.js 
* 04:54 AaronSchulz: ran fixPageFields.php to clean up pages with bad page_latest
* 03:58 logmsgbot: tstarling synchronized php-1.18/includes/WikiPage.php 
* 03:57 logmsgbot: aaron synchronized php-1.18/includes/WikiPage.php 'removed log hack'
* 03:30 logmsgbot: aaron synchronized php-1.18/includes/WikiPage.php 
* 03:25 logmsgbot: aaron synchronized php-1.18/includes/WikiPage.php 
* 03:13 logmsgbot: aaron synchronized php-1.18/includes/WikiPage.php 'r98979'
* 02:43 logmsgbot: aaron synchronized php-1.18/includes/specials/SpecialBlock.php 'deployed r98976'
* 02:43 logmsgbot: reedy synchronized php-1.18/includes/Exception.php 'r98978'
* 02:26 logmsgbot: LocalisationUpdate completed (1.18) at Wed Oct 5 02:29:23 UTC 2011
* 02:24 logmsgbot: LocalisationUpdate completed (1.17) at Wed Oct 5 02:27:08 UTC 2011
* 02:10 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Remove $wgMessageCache->addMessages() call from enwiki hack'
* 02:10 logmsgbot: reedy synchronized php-1.18/extensions/OpenSearchXml/ApiOpenSearchXml.php 'Kill my live hacks as they are doing nothing'
* 02:06 logmsgbot: demon synchronized php-1.18/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'r98974'
* 02:05 logmsgbot: tstarling rebuilt and synchronized wikiversions files: 
* 02:04 Tim: switching commonswiki to 1.18
* 01:38 logmsgbot: aaron synchronized php-1.18/skins/Standard.php 'Check if $browseLinks is an array'
* 01:35 logmsgbot: reedy synchronized php-1.18/extensions/OpenSearchXml/ApiOpenSearchXml.php 'Try to fix/suppress array-map callback errors'
* 01:28 logmsgbot: reedy synchronized php-1.18/extensions/OpenSearchXml/ApiOpenSearchXml.php 'Try to fix/suppress array-map callback errors'
* 01:19 logmsgbot: aaron synchronized php-1.18/skins/Standard.php 
* 01:14 logmsgbot: aaron synchronized php-1.18/skins/Standard.php 
* 01:13 logmsgbot: aaron synchronized php-1.18/skins/Standard.php 
* 00:44 logmsgbot: reedy synchronized php-1.18/includes/SkinLegacy.php 'r98964'
* 00:40 logmsgbot: reedy synchronized php-1.18/includes/api/ApiQueryWatchlist.php 'r98962'
* 00:32 logmsgbot: aaron synchronized php-1.18/extensions/CategoryTree/CategoryTree.php 'added default to $options, just as 1.17wmf did...'
* 00:25 logmsgbot: aaron rebuilt and synchronized wikiversions files: enwiki to 1.18
* 00:23 logmsgbot: reedy synchronized php-1.18/resources/startup.js 'touch'
* 00:23 logmsgbot: reedy synchronized php-1.18/resources/Resources.php 'r98956'
* 00:22 logmsgbot: reedy synchronized php-1.18/resources/mediawiki/mediawiki.Uri.js 'r98956'
* 00:22 logmsgbot: reedy synchronized php-1.18/resources/mediawiki/mediawiki.Title.js 'r98956'
* 00:15 logmsgbot: tstarling synchronized php-1.18/includes/specials/SpecialRecentchanges.php 'quick hack for RC warning'
* 00:10 logmsgbot: tstarling synchronized php-1.18/includes/specials/SpecialRecentchanges.php 'quick hack for RC warning'
== October 4 ==
* 23:55 logmsgbot: ben synchronized wmf-config/db.php 'putting db44 back in rotation; the schema changes are done.'
* 23:31 logmsgbot: aaron rebuilt and synchronized wikiversions files: plwiki to 1.18
* 23:25 logmsgbot: aaron rebuilt and synchronized wikiversions files: frwiki to 1.18
* 23:24 logmsgbot: aaron synchronized multiversion 'r98984'
* 23:20 logmsgbot: ben synchronized wmf-config/db.php 'pulling out db44 for a schema change - adding indexes on the flaggedrevs table for 1.18'
* 22:55 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 
* 22:54 logmsgbot: tstarling synchronized wmf-config/killscripts.php 
* 22:43 Reedy: Set ssl_redirect to on for bugzilla
* 22:41 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'itwiki hack'
* 22:37 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'itwiki hack'
* 22:32 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'itwiki hack'
* 22:02 mutante: upgraded Bugzilla to 4.0.2 (RT #1495)
* 21:59 logmsgbot: aaron synchronized multiversion 'Pushed r98933 verson of code'
* 21:52 logmsgbot: aaron synchronized php-1.18/includes/WikiPage.php 
* 21:33 logmsgbot: aaron synchronized php-1.18/includes/specials/SpecialUndelete.php 'deployed r98928'
* 21:21 robh: squid instances crashed out with disk swapping, restarted instances on sq31
* 21:19 robh: sq31 raid rebuilt
* 21:14 robh: manually started raid rebuild on sq32, its looking good and is in progress
* 20:53 notpeter: restarted gammu-smsd. it seems to be sending pages now. just in time for the 1.18 deploy! you're welcome ;)
* 20:46 notpeter: editing spence:/etc/gammurc and spence:/etc/gammu-smsdrc to point to /dev/ttyUSB1. because that's where it seems the sms sending stick decided to mount....
* 18:43 Jeff_Green: dist-upgrade and rebooting grosley
* 18:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove old yaseo references'
* 18:15 mutante: sq39,sq40,sq41,sq42 - upgrade kernel,reboot
* 17:44 preilly: need to push fix for double ? in view link
* 17:44 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'update to mobile frontend emergency fix to querystring'
* 17:43 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'update to mobile frontend emergency fix to querystring'
* 17:37 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'update to mobile frontend emergency fix to querystring'
* 17:36 preilly: fix issue with mobile frontend view on regular link
* 17:36 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'update to mobile frontend emergency fix to querystring'
* 17:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Update NS 105 for fawikisource'
* 17:01 notpeter: restarting and upgrading to lucid srv226-247
* 16:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30392 - zh-yue.wikipedia should use English namespace names as well as Wikipedia/Wikipedia_talk for NS_PROJECT[_TALK]'
* 16:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Update NS 105 for fawikisource'
* 16:43 notpeter: stopping all services (memcache, job-runner,apache, and puppet) on srv226-247
* 16:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Add index NS to fawikisource'
* 16:06 logmsgbot: py synchronized wmf-config/mc.php 'new mc.php to swap 1 more memecache instances for lucid upgrades'
* 15:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30721 - add namespace alias on latin wikisource (la.ws)'
* 15:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Set more mznwiki aliases'
* 15:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Set portal canonical ns for mznwiki'
* 15:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Revert that'
* 15:33 robh: installing os on mw1001-mw1160
* 15:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Set portal canonical ns for mznwiki'
* 15:31 logmsgbot: py synchronized wmf-config/mc.php 'new mc.php to swap 2 memecache instances for lucid upgrades'
* 15:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Set mznwiki metanamespaces'
* 15:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove wgGrammarForms as its in WikimediaMessages now'
* 15:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Swap space for _ for ruwiki incubator NS'
* 15:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30752 - Update namespaces for Mazandarani Wikipedia (mznwiki)'
* 15:06 robh: sq47 offline for system tests
* 15:00 logmsgbot: py synchronized wmf-config/mc.php 'new mc.php to swap 2 memecache instances for lucid upgrades'
* 14:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Make ruwiki incubator ns non indexed'
* 14:44 robh: updating dns with row B eqiad mw servers
* 14:34 Reedy: run namespaceDupes on azwikiquote
* 14:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31245 - Change project namespace on azwikiquote to Vikisitat'
* 14:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31245 - Change project namespace on azwikiquote to Vikisitat'
* 14:30 logmsgbot: py synchronized wmf-config/mc.php 'new mc.php to swap 2 memecache instances for lucid upgrades'
* 14:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'set wgMetanamespace for azwikiquote'
* 14:16 logmsgbot: py synchronized wmf-config/mc.php 'new mc.php to swap 2 memecache instances for lucid upgrades'
* 14:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Set azwikiquote sitename to Vikisitat'
* 14:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Set azwikiquote sitename to Vikisitat'
* 13:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31299 - Rename namespace and enable $wgExtraGenderNamespaces on pl.wiktionary'
* 13:43 logmsgbot: py synchronized wmf-config/mc.php 'new mc.php to swap 2 memecache instances for lucid upgrades'
* 13:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31142 - Narayam Install request'
* 12:43 notpeter: gracefulling apache on srv155 to load new php ini's
* 02:41 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 're-enabled Collection on eswiki'
* 02:39 Tim: killed mw-qserve on pdf1 and pdf3 and let supervise restart it. Apparently the queue size counters drifted and they thought they were busy when they weren't.
* 02:30 Tim: killed some stuck mw-zip processes on pdf1 and pdf3
* 02:29 logmsgbot: LocalisationUpdate completed (1.18) at Tue Oct 4 02:31:47 UTC 2011
* 02:26 logmsgbot: LocalisationUpdate completed (1.17) at Tue Oct 4 02:28:59 UTC 2011
* 02:24 Tim: killed some stuck imagemagick processes on pdf1, some were running since 2010!
* 02:20 Tim: restarted gmond on pdf2, it was probably screwed up because of the presence of the ganglia-monitor package
* 01:21 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'disabling collection extension on eswiki due to DoS'
* 01:14 logmsgbot: tstarling synchronized php-1.17/extensions/Collection/Collection.body.php 'patch to send XFF header'
* 01:13 logmsgbot: tstarling synchronized php-1.18/extensions/Collection/Collection.body.php 'patch to send XFF header'
* 01:12 mutante: sq36,sq37,sq38 - upgrade kernel to 2.6.32-34/dist-upgrade,reboot (RT #1612)
* 00:49 mutante: sq33,sq34,sq35 - upgrade kernel to 2.6.32-34/dist-upgrade,reboot (RT #1612)
* 00:39 maplebed: powercycled pdf2 because it wouldn't respond to ssh
== October 3 ==
* 23:59 maplebed: added ganglios to lucid apt repo
* 23:03 logmsgbot: aaron rebuilt and synchronized wikiversions files: Moving enwikibooks to 1.18
* 22:42 AaronSchulz: Added sync-dir script, modified sync-common-file a bit to handle it
* 22:40 logmsgbot: aaron synchronized php-1.18/extensions/FlaggedRevs/presentation/modules 
* 22:37 AaronSchulz: Testing sync-dir on fenari's /usr/ dir, looks OK
* 22:37 logmsgbot: aaron synchronized php-1.18/extensions/FlaggedRevs/presentation/modules 
* 22:36 logmsgbot: aaron synchronized php-1.18/extensions/FlaggedRevs/presentation/modules 
* 22:23 AaronSchulz: Made sync-file immediatly error out when given directories rather than flooding rsync errors
* 22:19 logmsgbot: aaron synchronized php-1.18/extensions/FlaggedRevs/presentation/modules 
* 22:16 logmsgbot: aaron synchronized php-1.18/extensions/FlaggedRevs/README.txt 
* 21:50 logmsgbot: aaron ran sync-common-all 'deployed r98807'
* 21:50 robh: dns updated for neon.wikimedia.org
* 21:31 logmsgbot: catrope synchronized wmf-config/secure.php 'Fake $_SERVER["HTTPS"] = "on" in secure.php'
* 21:28 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Fix logo URL for svwiktionary'
* 21:23 logmsgbot: aaron synchronized php-1.18/extensions/FlaggedRevs 
* 21:00 Jeff_Green: rolling through payments(1-3) with dist-upgrade, disable ipv6, reboot
* 20:57 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'monday update to mobile frontend'
* 20:57 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/views/layout/application.wml.php 'monday update to mobile frontend'
* 20:56 preilly: mobile frontend WML fixes 
* 20:56 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/application.wml.php 'monday update to mobile frontend'
* 20:55 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'monday update to mobile frontend'
* 20:38 logmsgbot: aaron synchronized php-1.18/extensions/FlaggedRevs/maintenance/fixBug28348.inc 'deployed r98803'
* 20:16 Jeff_Green: payments4 dist-upgrade, disable ipv6, reboot
* 20:02 logmsgbot: py synchronized wmf-config/mc.php 'swapping 2 memcache instances for lucid upgrades'
* 19:36 logmsgbot: py synchronized wmf-config/mc.php 'swapping 2 memcache instances for lucid upgrades'
* 19:16 logmsgbot: py synchronized wmf-config/mc.php 'swapping 2 memcache instances for lucid upgrades'
* 18:54 logmsgbot: py synchronized wmf-config/mc.php 'swapping 2 memcache instances for lucid upgrades'
* 18:45 notpeter: removing srv226-srv247 from apaches pool as step one of taking them down for upgrades to lucid
* 18:38 notpeter: putting srv258-272 back into the apaches pool
* 18:31 robh: updating dns for mw1001-mw1064
* 18:01 robh: updated bugzilla extension.pm for wikimedia extensions on kaulen
* 17:44 robh: srv278 shutdown for chris to troubleshoot
* 17:39 logmsgbot: catrope synchronizing Wikimedia installation... : i18n deployment (r98777)
* 17:10 notpeter: restarting srv172
* 16:50 notpeter: adding srv278 to apaches pool
* 16:26 logmsgbot: catrope synchronized php-1.18/includes/specials/SpecialUserlogin.php 'r98775'
* 16:26 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialUserlogin.php 'r98775'
* 16:19 logmsgbot: root synchronized wmf-config/CommonSettings.php 'Sync out just in case'
* 16:06 mutante: fixed nagios - chmod 644 mediawiki-lb.esams.wikimedia.org.cfg
* 16:02 logmsgbot: mark synchronized wmf-config/CommonSettings.php 'Add all SSL servers to the MediaWiki trusted XFF sources list'
* 15:24 mark: Updated bits.pmtpa service IP in varnish VCL in esams (live hack)
* 15:17 mark: Rebooting lvs2
* 15:13 mark: Rebooting lvs1
* 14:59 mark: Rebooting lvs1
* 14:58 Jeff_Green: depooling payments4 for more multicast fail testing
* 13:52 notpeter: upgrading srv258-srv272 and srv278 to lucid
* 13:35 mark: Rebooting lvs1
* 13:19 logmsgbot: catrope synchronizing Wikimedia installation... : r98757, r98758
* 13:14 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Set $wgVaryOnXFP, will be the new name of $wgVaryOnXFPForAPI'
* 13:02 logmsgbot: catrope synchronized search-redirect.php 'Support HTTPS in search-redirect.php'
* 12:58 logmsgbot: reedy synchronized php-1.18/extensions/WikiEditor/WikiEditor.php 'r98753'
* 12:57 logmsgbot: reedy synchronized php-1.18/extensions/WikiEditor/modules/ext.wikiEditor.toolbar.hideSig.js 'r98753'
* 12:56 logmsgbot: reedy synchronized php-1.18/extensions/ProofreadPage/proofread.js 'r98753'
* 12:51 logmsgbot: reedy synchronized php-1.18/includes/specials/SpecialBlockList.php 'r98752'
* 12:50 logmsgbot: reedy synchronized php-1.18/includes/SpecialPage.php 'r98752'
* 12:50 logmsgbot: reedy synchronized php-1.18/includes/Html.php 'r98752'
* 12:49 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'bug 31294 - Allow protocol-relative URLs in $wgCaptchaWhitelist'
* 12:42 logmsgbot: reedy synchronized php-1.18/extensions/UploadWizard/UploadWizard.i18n.php 'r98751'
* 10:55 logmsgbot: catrope synchronized php-1.18/extensions/CentralAuth/CentralAuthHooks.php 'r98749'
* 10:55 logmsgbot: catrope synchronized php-1.17/extensions/CentralAuth/CentralAuthHooks.php 'r98749'
* 10:49 logmsgbot: catrope synchronized php-1.18/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'r98746'
* 10:49 logmsgbot: catrope synchronized php-1.18/extensions/CentralAuth/CentralAuthHooks.php 'r98746'
* 10:49 logmsgbot: catrope synchronized php-1.18/includes/WikiMap.php 'r98746'
* 10:48 logmsgbot: catrope synchronized php-1.17/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'r98746'
* 10:48 logmsgbot: catrope synchronized php-1.17/extensions/CentralAuth/CentralAuthHooks.php 'r98746'
* 10:48 logmsgbot: catrope synchronized php-1.17/includes/WikiMap.php 'r98746'
* 02:24 logmsgbot: LocalisationUpdate completed (1.18) at Mon Oct 3 02:27:01 UTC 2011
* 02:22 logmsgbot: LocalisationUpdate completed (1.17) at Mon Oct 3 02:24:49 UTC 2011
== October 2 ==
* 22:07 logmsgbot: catrope synchronized php-1.17/extensions/MobileFrontend/MobileFrontend.php 'fix fatal'
* 22:03 logmsgbot: catrope synchronized php-1.17/includes/Message.php 'Fix fatal by adding isBlank()'
* 21:56 logmsgbot: catrope synchronizing Wikimedia installation... : Push out recent loginend/signupend fixes
* 20:10 logmsgbot: catrope synchronized php-1.18/includes/templates/Userlogin.php 'r98708'
* 20:09 logmsgbot: catrope synchronized php-1.17/includes/templates/Userlogin.php 'r98708'
* 20:09 logmsgbot: catrope synchronized php-1.18/includes/specials/SpecialUserlogin.php 'r98708'
* 20:09 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialUserlogin.php 'r98708'
* 19:38 logmsgbot: catrope synchronized php-1.18/extensions/LiquidThreads/classes/ParserFunctions.php 'r98704'
* 18:23 RoanKattouw: Edited extract2.php to remove duplicated listing of wikinews portal and template page
* 14:39 AaronSchulz: Fixed a page on nlwiki via "UPDATE page SET page_latest = '27371034' WHERE page_id = '2322234';"
* 10:06 rainman-sr: restarted search7 again, comes up with "too many open files" error, maybe it is worth to increase ulimit in /etc/init.d/lsearchd ?
* 04:48 logmsgbot: aaron synchronized php-1.18/includes/specials/SpecialBlock.php 'deployed r98655'
* 02:22 logmsgbot: LocalisationUpdate completed (1.18) at Sun Oct 2 02:24:44 UTC 2011
* 02:19 logmsgbot: LocalisationUpdate completed (1.17) at Sun Oct 2 02:22:16 UTC 2011
* 00:45 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'bumped $wgStyleVersion appendix per bug 31301'
== October 1 ==
* 20:37 Ryan_Lane: restarting etherpad again
* 20:34 Ryan_Lane: reconfiguring apache on etherpad box to fix etherpad https and to ensure https is always used with etherpad
* 20:33 Ryan_Lane: restarting etherpad
* 20:01 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Make Apple Touch icon URLs protocol-relative'
* 06:15 logmsgbot: aaron synchronized php-1.18/resources/mediawiki.special/mediawiki.special.block.js 'deployed r98613'
* 05:56 logmsgbot: aaron synchronized php-1.18/includes/specials/SpecialBlock.php 'deployed r98611'
* 05:38 logmsgbot: aaron synchronized php-1.18/includes/ProtectionForm.php 'deployed r98609'
* 05:25 logmsgbot: aaron synchronized php-1.18/extensions/GlobalBlocking/SpecialGlobalBlockList.php 'deployed r98608'
* 05:13 logmsgbot: aaron synchronized php-1.18/includes/IP.php 'deployed r98604, r98606'
* 02:19 logmsgbot: LocalisationUpdate completed (1.18) at Sat Oct 1 02:21:50 UTC 2011
* 02:17 logmsgbot: LocalisationUpdate completed (1.17) at Sat Oct 1 02:19:37 UTC 2011
* 00:05 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Use interwiki-pr.cdb for wikis with $wmgHTTPSExperiment (right now that means all wikis)'
* 00:02 logmsgbot: catrope synchronized php-1.18/cache/interwiki-pr.cdb 
* 00:02 logmsgbot: catrope synchronized php-1.18/cache/interwiki.cdb 
* 00:02 logmsgbot: catrope synchronized php-1.17/cache/interwiki-pr.cdb 
* 00:02 logmsgbot: catrope synchronized php-1.17/cache/interwiki.cdb 
== September 30 ==
* 23:42 RoanKattouw: Generated a list of all / and /wiki URLs that redirect to index.php in /home/catrope/purges and fed it to purgeList.php
* 23:25 logmsgbot: catrope synchronized php-1.17/includes/OutputPage.php 'Apply hack here too'
* 23:18 logmsgbot: catrope synchronized php-1.18/includes/OutputPage.php 'Different hack, attempt 2'
* 23:17 logmsgbot: catrope synchronized php-1.18/includes/OutputPage.php 'Revert fatl'
* 23:17 logmsgbot: catrope synchronized php-1.18/includes/OutputPage.php 'Different hack'
* 23:15 logmsgbot: catrope synchronized php-1.18/includes/OutputPage.php 'Comment out hack broken'
* 23:09 logmsgbot: catrope synchronized php-1.18/includes/OutputPage.php 'Experimental hack to hopefully fix https redirect issue'
* 23:05 logmsgbot: catrope synchronized wmf-config/checkers.php 'Remove HTTPS/XFP debug loggin'
* 23:03 logmsgbot: catrope synchronized wmf-config/checkers.php 
* 22:31 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Fix two settings for wgServer that erroneously were not protocol-relative'
* 22:24 AaronSchulz: deployed r98569 and r98577
* 22:23 logmsgbot: aaron synchronized php-1.18/extensions/FlaggedRevs/maintenance/updateStats.inc 
* 21:51 Ryan_Lane: switching mobile-lb entries to point to mobile-lb.eqiad
* 21:43 logmsgbot: aaron synchronized php-1.18/extensions/FlaggedRevs/maintenance/updateStats.php 
* 21:30 Ryan_Lane: Changing DNS for other registered top level names to point to correct top level domains
* 21:23 Ryan_Lane: changing mobile from text to mobile-lb
* 21:23 Ryan_Lane: moving wikipedia from text to wikipedia-lb
* 21:18 Ryan_Lane: moving wikimedia domains in DNS from text to wikimedia-lb
* 21:13 Ryan_Lane: moving wikiversity DNS from text to wikiversity-lb
* 21:08 Ryan_Lane: moving wikinews DNS from text to wikinews-lb
* 21:00 Ryan_Lane: moving wiktionary DNS from text to wiktionary-lb
* 20:57 Ryan_Lane: changing dns for mediawiki from text to wikimedia-lb
* 20:49 domas: added db39 back to service
* 20:49 logmsgbot: midom synchronized wmf-config/db.php 
* 20:28 RoanKattouw: Killing three really long-running FlaggedRevs queries on db25, its replag is insane
* 20:11 notpeter: taking down srv278 and srv258-srv272 for upgrades to lucid (not doing upgrades yet, am going to see if thigns are stable without them first)
* 19:57 logmsgbot: catrope synchronized php-1.18/includes/parser/ParserCache.php 'Bump pcache number for action=parse hack'
* 19:57 logmsgbot: catrope synchronized php-1.17/includes/parser/ParserCache.php 'Bump pcache number for action=parse hack'
* 19:52 logmsgbot: catrope synchronized wmf-config/secure.php 'Apply article path hack in GetLocalURL too, needed for API parse hack'
* 19:26 logmsgbot: py synchronized wmf-config/mc.php 'pushing out new mc.php to rotate out 1 more of a coming many boxes for upgrade to lucid'
* 19:05 Ryan_Lane: making DNS changes for wikiquote to point to wikiquote-lb instead of text; includes all langlist cnames
* 18:51 logmsgbot: py synchronized wmf-config/mc.php 'pushing out new mc.php to rotate out 3 more of a coming many boxes for upgrade to lucid'
* 18:23 Ryan_Lane: making DNS changes for wikibooks to point to wikibooks-lb instead of text; includes all langlist cnames
* 18:13 logmsgbot: py synchronized wmf-config/mc.php 'pushing out new mc.php to rotate out 2 more of a coming many boxes for upgrade to lucid'
* 17:59 Ryan_Lane: making DNS changes for wikisource to point to wikisource-lb instead of text; includes all langlist cnames
* 17:58 notpeter: putting mw28-49 and mw51-58 into the apaches pool
* 17:15 logmsgbot: py synchronized wmf-config/mc.php 'pushing out new mc.php to rotate out 2 more of a coming many boxes for upgrade to lucid'
* 16:28 logmsgbot: py synchronized wmf-config/mc.php 'pushing out new mc.php to rotate out 2 more of a coming many boxes for upgrade to lucid'
* 16:27 Ryan_Lane: making a *lot* of DNS changes. Please don't push DNS changes right now, if you know what's good for you.
* 16:19 Ryan_Lane: pooling ssl3004
* 16:14 Ryan_Lane: changed aggregators for ssl_esams from maerlant to ssl3001 and ssl3002 in gmetad
* 16:13 Ryan_Lane: pooling ssl3003
* 16:08 Ryan_Lane: pooling ssl3002
* 16:06 Ryan_Lane: installing new wikimedia-task-dns-auth package on dobson
* 16:03 Ryan_Lane: pooled ssl3001, depooling maerlant
* 16:02 Ryan_Lane: ipv6and4.labs moved to ssl3001
* 15:50 robh: ssl3002-ssl3004 os installed
* 15:33 Ryan_Lane: restarting pybal on amslvs1
* 15:29 RoanKattouw: Clearing /tmp directories on all mediawiki-installation hosts of files that haven't been accessed in more than 2 hours and are not in a subdirectory (this protects mw-cache files)
* 15:26 robh: ssl3002-ssl3004 installing
* 15:25 robh: cleaned up netboot.cfg (removing entries for servers that dont exist anymore), added in ssl esams range, updated dhcp
* 15:25 Ryan_Lane: restarting pybal on amslvs2
* 15:24 Ryan_Lane: restarting pybal on amslvs4
* 15:22 robh: ssl3001 installed with os, ready for puppet deployment
* 15:18 RoanKattouw: Using dsh to remove /tmp/mw-cache from all mediawiki-installation boxes. It's an obsolete caching dir with 300-400MB of dead data on each server
* 15:14 Ryan_Lane: restarting pybal on amslvs3
* 15:12 RoanKattouw: Removed all /tmp files that hadn't been accessed in >48 hours on srv255
* 15:05 Ryan_Lane: make that repository
* 15:05 Ryan_Lane: adding new wikimedia-task-dns-auth to repositoru
* 15:03 logmsgbot: py synchronized wmf-config/mc.php 'pushing out new mc.php to rotate out 2 more of a coming many boxes for upgrade to lucid'
* 14:51 Ryan_Lane: updating dns, adding mediawiki-lb using text's IP
* 14:50 robh: updating dns with ssl3001-ssl3004
* 14:32 logmsgbot: py synchronized wmf-config/mc.php 'pushing out new mc.php to rotate out 2 of a coming many boxes for upgrade to lucid'
* 13:55 mark: Configured csw1-esams switchports for ssl3001-3006
* 12:37 mark: shutdown server iris for decommissioning
* 05:22 mutante: sq69 -power up,dist-upgrade/kernel,reboot
* 05:15 mutante: sq67 -power up,dist-upgrade/kernel,reboot (varnish cache should be cleared on restart anyways)
* 02:25 logmsgbot: LocalisationUpdate completed (1.18) at Fri Sep 30 02:28:28 UTC 2011
* 02:23 logmsgbot: LocalisationUpdate completed (1.17) at Fri Sep 30 02:25:42 UTC 2011
== September 29 ==
* 22:22 maplebed: nagios is busted; rerunning puppet to fix nagios cfg permissions
* 21:12 preilly: pushed update to 1.18 mobile frontend
* 21:12 logmsgbot: preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'tuesday update mobile of frontend code part two'
* 21:10 preilly: push mobile frontend extension update
* 21:10 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'tuesday update mobile of frontend code part two'
* 21:02 preilly: pushing this weeks changes for mobile frontend
* 21:01 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'tuesday update mobile of frontend code'
* 20:38 logmsgbot: reedy synchronized php-1.18/resources/mediawiki.special/mediawiki.special.css 'r98444'
* 20:38 logmsgbot: reedy synchronized php-1.18/languages/messages/MessagesEn.php 'r98444'
* 20:36 logmsgbot: reedy synchronized php-1.18/skins/common/commonPrint.css 'r98444'
* 20:36 logmsgbot: reedy synchronized php-1.18/skins/common/shared.css 'r98444'
* 20:21 notpeter: adding mw28-58 to all dsh node groups that they belong in
* 16:30 robh: starting OS installation on mw28-mw58
* 14:02 Reedy: Why isn't enwikibooks on 1.18wmf1 per the deployment schedule?
* 04:09 mutante: sq47 is "dazed and confused" - most likely broken RAM -> RT#1597
* 03:09 mutante: knsq12 - power up,dist-upgrade/kernel,squid clean,reboot...
* 02:24 logmsgbot: LocalisationUpdate completed (1.18) at Thu Sep 29 02:26:48 UTC 2011
* 02:20 logmsgbot: LocalisationUpdate completed (1.17) at Thu Sep 29 02:23:25 UTC 2011
* 01:26 logmsgbot: tstarling synchronized php-1.17/includes/specials/SpecialIpblocklist.php 'privacy patch'
* 01:24 logmsgbot: tstarling synchronized php-1.17/includes/specials/SpecialIpblocklist.php 'privacy patch'
* 01:20 logmsgbot: tstarling synchronized php-1.17/includes/specials/SpecialIpblocklist.php 'privacy patch'
* 00:54 Tim: disabled api action=login logging on locke since it was apparently causing packet loss
== September 28 ==
* 23:17 logmsgbot: awjrichards synchronized php/extensions/ContactPageFundraiser/SpecialContact.php 'r98385'
* 23:09 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Updateing priority countries for LandingCheck'
* 22:19 AaronSchulz: running mwscriptwikiset extensions/FlaggedRevs/populateRevTimesamp.php flaggedrevs.dblist
* 22:16 AaronSchulz: copied populateRevTimesamp.php and populateRevTimesamp.inc into 1.17 dir
* 21:10 logmsgbot: reedy synchronizing Wikimedia installation... : Deploying 1.18 LQT
* 20:14 Reedy: Manually fix LQT schema on testwiki
* 20:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable LQT on test'
* 19:34 logmsgbot: reedy synchronized php-1.18/includes/api/ApiQueryUsers.php 'r98356'
* 19:34 logmsgbot: reedy synchronized php-1.18/includes/api/ApiQueryAllUsers.php 'r98356'
* 19:34 logmsgbot: reedy synchronized php-1.18/includes/api/ApiQueryUserInfo.php 'r98356'
* 19:23 logmsgbot: catrope synchronized php-1.18/extensions/WikiLove/modules/ext.wikiLove/ext.wikiLove.core.js 'r98353'
* 19:23 logmsgbot: catrope synchronized php-1.17/extensions/WikiLove/modules/ext.wikiLove/ext.wikiLove.core.js 'r98353'
* 18:58 logmsgbot: aaron synchronized php-1.18/extensions/Collection/js/collection.js 'deployed r98347'
* 18:15 logmsgbot: midom synchronized php-1.18/includes/parser/Parser.php 
* 18:08 domas: someone pushed some new code that took down profiling.. 
* 17:58 notpeter: pushing out reverted pdns templates...
* 17:53 maplebed: corrected typo bawikitionary -> bawiktionary in closed.dblist
* 17:34 notpeter: pushing out new dns templates for rt 1344
* 15:00 mark: Restarted apache on fenari
* 14:56 RoanKattouw: Stopped Apache on fenari, server was going crazy
* 13:48 logmsgbot: catrope synchronized php-1.17/includes/parser/ParserCache.php 'r98311'
* 13:47 logmsgbot: catrope synchronized php-1.17/includes/parser/ParserCache.php 'r98310'
* 13:46 logmsgbot: catrope synchronized php-1.17/includes/Title.php '..but they weren't. Reinstating'
* 13:44 logmsgbot: catrope synchronized php-1.17/includes/Title.php 'I think the Title.php changes may have been unnecessary, reverting'
* 13:39 logmsgbot: catrope synchronized php-1.18/includes/parser/ParserCache.php 'r98309'
* 13:39 logmsgbot: catrope synchronized php-1.18/includes/Title.php 'r98309'
* 13:36 logmsgbot: catrope synchronized php-1.17/includes/parser/ParserCache.php 'Change renderkey for action=parse from 2 to 3 to force invalidation'
* 13:32 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add hacks to make various config vars non-protocol-relative for action=parse'
* 13:24 logmsgbot: catrope synchronized php-1.17/includes/Title.php 'oops'
* 13:20 logmsgbot: catrope synchronized php-1.18/includes/parser/ParserCache.php 'r98306'
* 13:20 logmsgbot: catrope synchronized php-1.18/includes/Title.php 'r98306'
* 13:14 logmsgbot: catrope synchronized php-1.17/includes/parser/ParserCache.php 'r98305'
* 13:14 logmsgbot: catrope synchronized php-1.17/includes/Title.php 'r98305'
* 12:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Part of bug 30752, add NS_PROJECT alias'
* 12:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31057 - Incubator namespace in ruwiki'
* 12:04 Reedy: Ran namespace dupes on azwikibooks
* 12:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31058 - Add Recipe namespace to Azerbaijani Wikibooks'
* 12:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix tlwikiboos typo'
* 11:59 logmsgbot: tstarling synchronized php-1.18/includes/libs/jsminplus.php 
* 11:58 Reedy: Ran namespaceDupes on azwikisource
* 11:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31112 - Add Author namespace to Azerbaijani Wikisource'
* 11:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'WikiLove doesn't work on incubator'
* 11:43 logmsgbot: reedy synchronized php-1.18/resources/startup.js 'touch'
* 11:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31185 - To enable Special:Import right on orwiki'
* 11:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31209 - Enable the WikiLove extension for incubator'
* 11:16 logmsgbot: reedy synchronized php-1.18/extensions/CheckUser/CheckUser_body.php 'r98302'
* 10:35 Tim: on formey: apt-get upgrade
* 02:23 logmsgbot: LocalisationUpdate completed (1.18) at Wed Sep 28 02:25:34 UTC 2011
* 02:20 logmsgbot: LocalisationUpdate completed (1.17) at Wed Sep 28 02:22:57 UTC 2011
* 00:53 logmsgbot: tstarling synchronized php-1.18/tests/qunit/index.html 
* 00:52 logmsgbot: tstarling synchronized php-1.18/includes/resourceloader/ResourceLoaderFileModule.php 'r98295'
== September 27 ==
* 23:02 Tim: cleaning up /tmp on srv256
* 22:54 RoanKattouw: Exposed wikiversions.dat at http://noc.wikimedia.org/conf , and removed 1.17.dblist (obsolete)
* 22:51 logmsgbot: tstarling synchronized php-1.18/extensions/ParserFunctions/ParserFunctions_body.php 'r 98282'
* 22:49 logmsgbot: aaron synchronized multiversion/getMWVersion 
* 22:10 Reedy: that was fixing a typo in config for bug 31200
* 22:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 
* 22:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31168 - Enable email notification for Watchlist for hi Wiki'
* 22:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31200 - Add configuration for Babel extension on la.wikipedia to InitialiseSettings.php'
* 22:03 binasher: new S1 master position - db32-bin.000056 3001
* 22:02 logmsgbot: asher synchronized wmf-config/db.php 'new s1 master, setting to writeable'
* 22:01 logmsgbot: asher synchronized wmf-config/db.php 'new s1 master, setting to read-only'
* 22:00 binasher: switching the S1 (enwiki) master
* 21:44 binasher_: running 1.18 schema migrations on former s4 master (db31)
* 21:42 binasher_: added srv171 back to dsh apache + mwinstall grps
* 21:38 binasher_: new S4 master position - db22-bin.000040 516445005
* 21:37 logmsgbot: asher synchronized wmf-config/db.php 'writes on s4'
* 21:34 logmsgbot: asher synchronized wmf-config/db.php 'new s4 master, setting read-only'
* 21:31 binasher_: preparing to �swap masters on s4 (commonswiki) - will be read-only briefly 
* 21:17 logmsgbot: aaron synchronized multiversion/MWScript.php 
* 20:59 cmjohnson1: project2 rebooted per rt#1589
* 20:14 cmjohnson1: ms1 rebooted per rt#1577
* 20:13 robh: ms1 back online
* 20:05 logmsgbot: catrope synchronized php-1.17/cache/interwiki.cdb 'bug 22407 - Make mw: et al local interwiki prefixes'
* 19:57 logmsgbot: catrope synchronized php-1.18/cache/interwiki.cdb 'Attempting to make mw: et al local interwiki prefixes'
* 19:48 logmsgbot: catrope synchronized php-1.18/cache/interwiki.cdb 'Attempting to make mw: et al local interwiki prefixes'
* 19:40 Ryan_Lane: rebooting gurvin
* 19:34 Ryan_Lane: purged all ssl proxy configuration (and certs/keys) from yvon and gurvin
* 19:29 logmsgbot: catrope synchronized php-1.18/includes/specials/SpecialUserlogin.php 'r98238'
* 19:26 notpeter: restarting nagios on spence
* 19:21 Jeff_Green: revised apache config puppetized and installed on grosley
* 19:17 Ryan_Lane: repooling gurvin
* 19:14 Ryan_Lane: depooling gurvin and yvon, and pooling ssl2 and ssl3
* 19:05 Ryan_Lane: depooling ssl1-4 and repooling gurvin and yvon
* 19:00 Ryan_Lane: depooling gurvin for testing
* 18:55 Ryan_Lane: depooling yvon, and adding ssl1-4 to https lvs group
* 18:52 Ryan_Lane: pushed version of nginx with udplogging module to repo
* 18:25 notpeter: restarting srv159 and srv189. swapdeath.
* 17:58 RoanKattouw: Fixing corrupted article_feedback_revisions rows by hand in the enwiki DB
* 17:56 apergos: restarted lighthttpd on dataset2, it had stopped
* 17:37 Ryan_Lane: installing ssl1-4
* 17:35 logmsgbot: py synchronized wmf-config/mc.php 'putting mw24-mw27 into mc.php as memcache spares as part of RT 1550'
* 17:27 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/api/ApiArticleFeedback.php 'r98232'
* 17:05 Jeff_Green: stopping mysql on db1008 and cloning to db1048
* 16:22 logmsgbot: catrope synchronized php-1.18/resources/startup.js 'touch'
* 16:20 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Temporarily set $wgResourceLoaderValidateJS to false for bug 31187'
* 16:20 notpeter: stopping jobrunners on all external store apaches and removing from puppet/dsh group
* 15:56 Jeff_Green: throwing payments4 back in the pool to test new firewall rules
* 15:22 robh: updated dns for ssl1-ssl4
* 15:05 logmsgbot: py synchronized wmf-config/db.php 'putting srv180 and srv163 back into db.php as they are back up'
* 14:34 notpeter: restarting srv180
* 14:33 notpeter: restarting srv163 (this time the right one...)
* 14:18 logmsgbot: ariel synchronized php-1.18/includes/HistoryBlob.php 'mhash workaround from 117wmf1'
* 14:11 notpeter: restarting srv161
* 13:56 notpeter: powercycling srv188
* 13:54 RoanKattouw: Fixed ownership of /apache/common/php-1.17/cache/l10n from nagios:wikidev to mwdeploy:mwdeploy
* 13:38 notpeter: restarting srv171
* 13:37 logmsgbot: catrope synchronized wmf-config/db.php 'Comment out srv163 and srv180 from ES list, they are dead'
* 13:12 notpeter: powercycled srv179 to bring it back up
* 12:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31172 - Deploy WikiLove on orwiki'
* 12:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31178 - Enable wikilove in Portuguese Wikipedia'
* 11:36 RoanKattouw: Edited /home/wikipedia/bin/sync-common-file and added -o -oSetupTimeout=10 to the dsh command line. This makes sync-file not hang forever when an Apache is down. Thanks to maplebed for the tip
* 11:35 logmsgbot: catrope synchronized php-1.18/extensions/WikiLove/modules/ext.wikiLove/ext.wikiLove.core.js 'r98218'
* 11:35 logmsgbot: catrope synchronized php-1.17/extensions/WikiLove/modules/ext.wikiLove/ext.wikiLove.core.js 'r98218'
* 11:20 RoanKattouw: Killed job runners on srv155
* 10:45 mark: Stopped job runner on srv161
* 10:44 mark: Stopped memcached on srv161
* 09:56 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Disable interwiki-pr.cdb, again. I swear I disabled this before but I guess it came back'
* 09:52 logmsgbot: catrope synchronized php-1.17/includes/parser/Parser.php 'Actually deploy r98197, grr, forgot to ran svn up before scapping last night'
* 09:51 logmsgbot: catrope synchronized php-1.18/includes/parser/Parser.php 'Actually deploy r98195, grr'
* 05:45 mutante: amssq34 - powercycle,dist-upgrade,puppet,squid clean,reboot
* 05:27 mutante: amssq32 - powercycle,dist-upgrade,puppet,squid clean,reboot
* 04:47 mutante: deleted files from /tmp that had not been read for 30 days+ on several srv's to free disk space (RT 1571)
* 02:29 logmsgbot: LocalisationUpdate completed (1.18) at Tue Sep 27 02:31:52 UTC 2011
* 02:26 logmsgbot: LocalisationUpdate completed (1.17) at Tue Sep 27 02:28:40 UTC 2011
* 02:21 logmsgbot: aaron rebuilt and synchronized wikiversions files... : enwikiquote, betawikiversity, eowiki to 1.18
* 02:09 logmsgbot: aaron rebuilt and synchronized wikiversions files... : metawiki,incubatorwiki to 1.18
* 02:04 logmsgbot: tstarling synchronized php-1.18/extensions/OggHandler/OggHandler_body.php 
* 01:38 logmsgbot: aaron synchronized php-1.18/includes/Skin.php 'fixed newFromName() related fatal'
* 01:03 logmsgbot: aaron synchronized wikiversions.cdb 
* 00:58 logmsgbot: aaron rebuilt and synchronized wikiversions files... : switched nlwiki to 1.18
* 00:44 logmsgbot: aaron synchronized php-1.18/includes/cache/MessageCache.php 'deployed r98201'
* 00:35 Tim: cleared the message cache on mediawikiwiki
* 00:18 logmsgbot: tstarling synchronized php-1.18/languages/messages/MessagesEn.php 
* 00:12 logmsgbot: ben synchronized wmf-config/db.php 'retabbed the db.php; no content changes.'
* 00:00 logmsgbot: ben synchronized wmf-config/db.php 'putting two apache hosts into rotation for cluster8 since they're up, happy, and have correct content.'
== September 26 ==
* 23:17 logmsgbot: catrope synchronizing Wikimedia installation... : Push bug 31176 fix and 1.17 messages update
* 23:06 Tim: updated /etc/dsh/group/mysql
* 23:02 RoanKattouw: Commented out srv171 in the mediawiki-installation dsh group as well
* 22:59 binasher: new s5 master pos db45-bin.000011 658053810
* 22:59 logmsgbot: catrope synchronizing Wikimedia installation... : Deploy r98187, second attempt
* 22:58 binasher: pulled srv171 from dsh apache group
* 22:55 logmsgbot: asher synchronized wmf-config/db.php 'new s5 master'
* 22:55 maplebed: cleared out tmp files on ms3, disk space back down to 90% (from 94)
* 22:52 logmsgbot: catrope synchronizing Wikimedia installation... : Deploy r98187
* 22:35 logmsgbot: asher synchronized wmf-config/db.php 'setting s5 to ro for master swap'
* 22:32 binasher: setting s5 to ro, swapping master
* 22:20 logmsgbot: ben synchronized wmf-config/db.php 'removing ms1 from rotation again until its mysql configs get fixed. too much system cpu and connection timeouts'
* 22:09 logmsgbot: catrope synchronized php-1.18/languages/messages/MessagesEn.php 'r98184'
* 21:50 binasher: running 1.18 migrations on former s6 master (db43)
* 21:49 binasher: new s6 master position - db47-bin.000022 678399893
* 21:48 logmsgbot: asher synchronized wmf-config/db.php 'new s6 mater'
* 21:47 logmsgbot: asher synchronized wmf-config/db.php 'setting s6 to ro for master switch'
* 21:46 logmsgbot: catrope synchronizing Wikimedia installation... : Reinstate MobileFrontend changes that I reverted while debugging the broken Nagios check for mobile
* 21:45 binasher: setting db cluster S6 to read only for master rotation
* 21:37 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Revert r2352: make $wgRightsUrl and $wgRightsIcon protocol-relative again. Should be fine with the WikimediaMessages fix'
* 21:34 logmsgbot: catrope synchronized php-1.18/extensions/WikimediaMessages/WikimediaMessages.php 'r98177'
* 21:34 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMessages/WikimediaMessages.php 'r98177'
* 21:21 notpeter: pushing new dns templates after a typo correction
* 21:08 RoanKattouw: So to set the record straight, I removed $urlprotocol from $wgRights* and that may or may not have been what fixed the copyright issue
* 21:05 RoanKattouw: I meant remove $urlprotocol
* 21:05 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Remove from * because I don't know what else to try'
* 20:14 logmsgbot: catrope synchronizing Wikimedia installation... : Revert recent MobileFrontend changes, got pushed accidentally
* 20:14 Jeff_Green: rebooting payments4 for apt dist-upgrade fun
* 20:07 Ryan_Lane: restarting gerrit. its error log stopped logging
* 19:50 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Here goes... Set $wgServer protocol-relative on all wikis, and enable $wmgHTTPSExperiment on all wikis'
* 19:48 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Fix stupid typo'
* 19:46 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Use $urlprotocol in two more places'
* 19:44 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'When $wmgHTTPSExperiment is used, do NOT use protocol-relative URLs for en.wikipedia.org; that domain does not support https right now'
* 19:38 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Make most of InitialiseSettings protocol-relative. Left out domains that do not yet have HTTPS, and wgServer'
* 19:14 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Kill prstdlogo and make stdlogo protocol-relative'
* 19:14 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Make all logo URLs protocol-relative'
* 18:36 logmsgbot: asher synchronized wmf-config/db.php 'commenting out db39'
* 18:34 binasher: new s7 master position db16-bin.000221 515053560
* 18:31 logmsgbot: asher synchronized wmf-config/db.php 'new s7 master'
* 18:22 logmsgbot: asher synchronized wmf-config/db.php 'put S7 in readonly section, preparing to rotate master'
* 18:22 logmsgbot: reedy synchronized php-1.17/includes/api/ApiUpload.php 'r98153'
* 18:18 binasher: preparing to rotate S7 master 
* 17:57 logmsgbot: ben synchronized wmf-config/db.php 'putting ms1 back in rotation to ease load on the rest of the rc1 cluster.'
* 17:45 logmsgbot: catrope synchronizing Wikimedia installation... : Deploy r98144
* 17:32 notpeter: putting mw15-mw22, mw24-27 into apaches pool
* 17:24 logmsgbot: catrope synchronized php-1.17/resources/jquery/jquery.js 'r98145'
* 17:13 notpeter: putting mw4-mw14 into the apaches pool
* 16:59 logmsgbot: catrope synchronizing Wikimedia installation... : Push out r98137, r98138
* 16:17 notpeter: adding mw1, mw2, and mw3 to apaches pool for testing.
* 16:11 logmsgbot: py synchronizing Wikimedia installation... : scappin' to make sure that mw1-27 boxxies are up to date before I put them into apaches pool
* 15:38 notpeter: moving dsh group job-runners2 to root's home dir on fenari, as it seems out of date. if this breaks anything, that's where it can be found.
* 15:38 mark: Configured switchports for ssl1-4
* 13:29 RoanKattouw: Fixed ownership&permissions for /h/w/common/*.dblist , they were a mess and caused sync-dblist to fail for non-roots. Ran chgrp wikidev *.dblist; chmod g+w *.dblist;
* 10:01 RoanKattouw: Updating prototype.wikimedia.org/rc-en to 1.18wmf1
* 03:02 Tim: on srv182: restarted memcached to reduce memory usage, and started apache and puppet
* 03:01 logmsgbot: tstarling synchronized wmf-config/mc.php 'removed srv182 from memcached since it only has 4GB, added srv237 instead'
* 02:23 logmsgbot: LocalisationUpdate completed (1.18) at Mon Sep 26 02:26:09 UTC 2011
* 02:21 logmsgbot: LocalisationUpdate completed (1.17) at Mon Sep 26 02:23:39 UTC 2011
== September 25 ==
* 20:32 logmsgbot: aaron synchronized php-1.18/resources/mediawiki/mediawiki.util.js 'deployed r97882'
* 20:30 logmsgbot: aaron synchronized php-1.18/includes/Title.php 'deployed r97859'
* 13:29 notpeter: restarting gammu-smsd on spence
* 07:30 apergos: yet another search7 lsearchd restart. I did not see one cpu pegged, in theory the box would not have been depooled, right? alas I was not awake enough to remember to check that
* 02:24 logmsgbot: LocalisationUpdate completed (1.18) at Sun Sep 25 02:26:52 UTC 2011
* 02:21 logmsgbot: LocalisationUpdate completed (1.17) at Sun Sep 25 02:24:17 UTC 2011
== September 24 ==
* 19:09 Ryan_Lane: adding the python-rtkit package to the lucid wikimedia repo
* 18:38 logmsgbot: aaron rebuilt and synchronized wikiversions files... : testing message; no changes
* 17:12 logmsgbot: demon synchronized php-1.18/extensions/ExtensionDistributor/ExtensionDistributor.i18n.php 'r98009'
* 17:12 logmsgbot: demon synchronized php-1.18/extensions/ExtensionDistributor/ExtensionDistributor_body.php 'r98009'
* 13:17 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Set $wgBabelMainCategory for frwikisource'
* 13:05 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'bug 31119 - Pluralize Babel categories for frwikisource'
* 02:25 logmsgbot: LocalisationUpdate completed (1.18) at Sat Sep 24 02:28:23 UTC 2011
* 02:23 logmsgbot: LocalisationUpdate completed (1.17) at Sat Sep 24 02:25:44 UTC 2011
== September 23 ==
* 23:26 binasher: fixed report.py xml parsing
* 20:46 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'fixed multiversion error message'
* 20:35 logmsgbot: hashar: ci.tesla: cleaned /tmp/
* 20:25 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'removed b/c Het Deploy hack for maintenance scripts'
* 19:43 notpeter: purged binlogs on db9 back to 2011-09-01
* 18:55 logmsgbot: reedy synchronized php-1.18/includes/HTMLForm.php 'r97862'
* 17:53 notpeter: putting mw10 into the apaches pool for test purposes.
* 17:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 31119 - Configuration of Extension:Babel on fr.wikisource'
* 17:50 logmsgbot: catrope synchronized php-1.18/extensions/Babel/BabelAutoCreate.class.php 'r97936'
* 17:50 logmsgbot: catrope synchronized php-1.17/extensions/Babel/BabelAutoCreate.class.php 'r97936'
* 17:08 mark: Enabled LLDP on all Foundry access switches (asw-a4-sdtpa, asw-a5-sdtpa, asw-d1-sdtpa, asw-d2-sdtpa, asw-d3-sdtpa)
* 16:41 robh: updated blog to add fundraising banner
* 15:37 robh: mw6 os installed, ready for puppet/service
* 15:25 robh: did some tinkering on blog themes, any odd issues in the past 5 minutes with blog appearance are due to that. all changes reverted and blog is back to normal
* 15:17 robh: mw5 os installed, ready for puppet/service
* 15:04 mark: Setup firewall filter on cr1-sdtpa:ae0.105 and cr2-pmtpa:irb.105; reactivated rpf-check
* 15:03 robh: mw2 installed, ready for puppet/service
* 15:01 Jeff_Green: depooling payments4 for testing fun
* 14:59 robh: shutting down dataset1 for mainboard swap, confirmed with ariel the other day that its not serving live traffic
* 14:51 robh: mw13 installed, ready for puppetization and service
* 08:22 logmsgbot: ariel synchronized php-1.17/maintenance/dumpTextPass.php 'mwscript wrapper for script call'
* 08:20 logmsgbot: ariel synchronized php-1.18/maintenance/dumpTextPass.php 'mwscript wrapper for script call'
* 02:25 logmsgbot: LocalisationUpdate completed (1.18) at Fri Sep 23 02:27:39 UTC 2011
* 02:22 logmsgbot: LocalisationUpdate completed (1.17) at Fri Sep 23 02:25:11 UTC 2011
* 01:07 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'update mobile frontend code'
* 00:29 preilly: pushing TestCanonicalRedirect hook 
* 00:29 logmsgbot: preilly synchronized php/includes/Wiki.php 'update mobile frontend code'
* 00:23 logmsgbot: preilly synchronized php/extensions/MobileFrontend/javascripts/application.js 'update mobile frontend code'
* 00:19 logmsgbot: preilly ran sync-common-all 'update mobile frontend code'
* 00:19 preilly: pushing today's mobile frontend push
== September 22 ==
* 23:07 binasher: running schema migrations against eqiad and analytics slaves
* 22:19 Ryan_Lane: tweeking performance settings for gerrit, restarting
* 21:50 Ryan_Lane: rebooting virt3, and virt2
* 21:50 Ryan_Lane: rebooting virt4
* 20:59 logmsgbot: aaron synchronized php-1.18/skins/common/edit.js 
* 20:49 notpeter: putting srv274 and srv273 back in the apaches pool
* 20:20 binasher: additive liquidthread 1.18 migrations completed on all slaves, still need to do masters. some slaves could use an index drop
* 19:58 mark: Set static routes of 10.4.0.0/24 and 208.80.153.192/32 (in addition to 208.80.153.192/28) to 10.4.16.3 on cr1-sdtpa and cr2-pmtpa
* 19:55 logmsgbot: ben synchronized wmf-config/db.php 'db13 is now caught up with replication. Putting it back in rotation.'
* 19:49 mark: Deconfigured cr1-sdtpa and cr2-pmtpa for vlan 103; they are no longer having an interface in that vlan
* 19:40 binasher: preparing to run liquidthreads 1.18 schema migrations against slaves
* 19:21 Jeff_Green: rebooting silicon to test firewall rule setup
* 18:16 robh: installing mw1-mw27
* 18:14 robh: updated netboot apaches.cfg to include a tmp partition of 2gb
* 17:59 logmsgbot: ben synchronized wmf-config/db.php 'pulling db13 out of s2 because it looks like the check slave status cache does not work - the host is being pummelled by show slave status checks.'
* 17:55 logmsgbot: ben synchronized wmf-config/db.php 'db13 back in rotation now that slaving is fixed, though it\'s still lagged,which is ok because MW will not use lagged slaves'
* 17:31 AaronSchulz: Hacked in common/edit.js symlink into 1.18 for LQT
* 17:22 robh: updating dns for new mw1-mw74
* 17:18 maplebed: pulled db13 out of rotation (cluster s2) because slaving is broken
* 17:17 logmsgbot: ben synchronized wmf-config/db.php 'slaving is broken on db13. removing it from rotation in cluster s2.'
* 16:38 notpeter: returning srv273 and srv274 to apaches pool
* 16:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Gendered namespaces for test2wiki'
* 16:00 notpeter: putting srv275 back into apaches pool and removing srv273 for upgrade to lucid
* 15:42 notpeter: returning srv276 to apaches and removing srv274 for upgrade to lucid
* 15:30 notpeter: removing srv275 from apaches pool for upgrade to lucid
* 15:06 logmsgbot: reedy synchronized php-1.18/includes/Linker.php 'Error handling for null given rather than a Title'
* 15:05 notpeter: removing srv276 from apaches pool for upgrade to lucid
* 15:04 Reedy: Who left the live hack in php-1.18 Linker.php type coercing public static function makeMediaLinkObj( Title $title, $text = , $time = false ) {
* 15:01 logmsgbot: reedy synchronized php-1.18/includes/Linker.php 'Sync please?'
* 15:00 logmsgbot: reedy synchronized php-1.18/includes/Linker.php 'Testing for PHP fatal error in /home/wikipedia/common/php-1.18/includes/Linker.php line 831'
* 14:58 logmsgbot: reedy synchronized php-1.18/includes/Linker.php 'Testing for PHP fatal error in /home/wikipedia/common/php-1.18/includes/Linker.php line 831'
* 14:57 logmsgbot: reedy synchronized php-1.18/includes/Linker.php 'Testing for PHP fatal error in /home/wikipedia/common/php-1.18/includes/Linker.php line 831'
* 12:01 mutante: lvs1003's disk died - see RT #1549
* 11:40 mutante: power cycling lvs1003
* 07:41 awjr: restarting jenkins on grosley
* 07:27 logmsgbot: aaron synchronized php-1.18/extensions/SpamBlacklist/SpamBlacklist_body.php 'disabled wfDebugLog() call; UDP packets were way too large'
* 06:31 logmsgbot: tstarling synchronizing Wikimedia installation... : 
* 06:20 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'remove debug log group since the bug is isolated now'
* 06:18 logmsgbot: tstarling synchronized php-1.18/includes/User.php 'revert debugging hack'
* 05:11 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'added extension namespace to $wgFlaggedRevsNamespaces for mediawikiwiki'
* 04:58 logmsgbot: tstarling synchronized php-1.18/includes/User.php 'debugging for getGroupPermissions warning'
* 04:26 logmsgbot: tstarling synchronized php-1.18/includes/User.php 'debugging patch for getGroupPermissions warning'
* 04:26 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'debugging for getGroupPermissions'
* 04:14 logmsgbot: tstarling synchronized php-1.18/includes/User.php 'debugging patch for getGroupPermissions warning'
* 04:13 logmsgbot: aaron synchronized php-1.18/includes/Linker.php 
* 04:06 logmsgbot: aaron synchronized php-1.18/includes/Linker.php 
* 04:02 logmsgbot: aaron synchronized php-1.18/includes/User.php 
* 03:59 logmsgbot: aaron synchronized php-1.18/includes/api/ApiQuerySiteinfo.php 
* 03:58 logmsgbot: aaron synchronized php-1.18/includes/api/ApiQuerySiteinfo.php 'deployed r97789'
* 03:53 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'fixed thumbDir in $wgForeignFileRepos, should not be used for anything but it may as well be set correctly'
* 03:45 logmsgbot: tstarling synchronized php-1.18/includes/media/Bitmap.php 'r97787'
* 03:39 Tim: stopped apache and puppet on srv182 again and sabotaged it in configuration to stop it from coming up again
* 03:27 logmsgbot: aaron synchronized php-1.18/includes/Xml.php 'deployed r97784'
* 02:39 logmsgbot: aaron synchronized wikiversions.cdb 
* 02:38 logmsgbot: aaron synchronized wikiversions.dat 'strategywiki/mediawiki back to 1.18'
* 02:30 logmsgbot: aaron ran sync-common-all 'pushing 1.18wmf LQT code out to apaches'
* 02:27 logmsgbot: LocalisationUpdate completed (1.18) at Thu Sep 22 02:30:05 UTC 2011
* 02:24 logmsgbot: LocalisationUpdate completed (1.17) at Thu Sep 22 02:26:40 UTC 2011
* 02:20 logmsgbot: aaron synchronized wikiversions.cdb 
* 02:20 logmsgbot: aaron synchronized wikiversions.dat 'hewikisource to 1.18'
* 02:13 logmsgbot: aaron synchronized wikiversions.cdb 
* 02:13 Tim: also stopped job runner on srv182
* 02:13 logmsgbot: aaron synchronized wikiversions.dat 
* 02:12 Tim: stopping apache on srv182, some kind of disk I/O overload
* 02:12 logmsgbot: aaron synchronized wikiversions.cdb 'strategywiki and mediawikiwiki back to 1.18 due to LQT schema fail'
* 01:46 logmsgbot: aaron synchronized wikiversions.cdb 'simplewiki & simplewiktionary to 1.18'
* 01:46 logmsgbot: aaron synchronized wikiversions.dat 'simplewiki & simplewiktionary to 1.18'
* 01:42 logmsgbot: aaron synchronized wikiversions.cdb 'usabilitywiki & strategywiki to 1.18'
* 01:41 logmsgbot: aaron synchronized wikiversions.dat 'usabilitywiki & strategywiki to 1.18'
* 01:35 logmsgbot: aaron synchronized wikiversions.cdb 'mediawikiwiki to 1.18'
* 01:35 logmsgbot: aaron synchronized wikiversions.dat 'mediawikiwiki to 1.18'
* 01:11 logmsgbot: aaron synchronized php-1.18/includes/resourceloader/ResourceLoaderFileModule.php 'deployed r97771, r97775'
* 01:06 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'setting $wgResourceBasePath'
* 01:03 logmsgbot: aaron ran sync-common-all 
* 00:47 logmsgbot: aaron synchronized php-1.18/includes/resourceloader/ResourceLoaderFileModule.php 'deployed r97771, r97775'
* 00:45 logmsgbot: aaron synchronized php-1.18/includes/DefaultSettings.php 'deployed r97771, r97775'
== September 21 ==
* 23:17 Tim: starting job runners
* 22:00 Tim: stopped all job runners for testing
* 21:20 notpeter: putting srv277 back into apaches pool
* 20:41 logmsgbot: aaron synchronized php-1.18/extensions/FlaggedRevs/dataclasses/FlaggedRevs.class.php 'deployed r97764'
* 20:36 notpeter: removing srv277 from apaches pool for upgrade to lucid
* 20:28 logmsgbot: aaron synchronized php-1.18/includes/objectcache/MemcachedClient.php 'deployed r97762'
* 19:52 logmsgbot: reedy synchronized php-1.18/includes/SkinTemplate.php 'r97759'
* 19:36 notpeter: putting srv279 back in the apaches pool
* 19:03 Reedy: test.wikipedia.org is now on 1.18wmf1
* 19:02 logmsgbot: reedy synchronized wikiversions.dat 'test to 1.18wmf1'
* 19:00 logmsgbot: reedy synchronized wikiversions.cdb 'test to 1.18wmf1'
* 18:58 AaronSchulz_: created flaggedrevs_statistics tables on flaggedrevs wikis
* 18:46 notpeter: returning srv280 to apaches pool
* 18:30 notpeter: returning srv282 to apaches pool
* 18:30 notpeter: removing srv279 from apaches pool for upgrades
* 18:18 notpeter: removing srv280 from apaches pool for upgrade to lucid
* 18:03 notpeter: removing srv282 from apaches pool for upgrade to lucid
* 18:00 notpeter: putting srv283, srv285, and srv286 back in the apaches pool
* 17:47 maplebed: stopping mysql on ms1 to create a replica in eqiad
* 16:17 Jeff_Green: locked out ssh from fenari, etc. to payments[1-4], leaving only the new bastion host
* 15:39 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add Babel settings for Italian wikis'
* 15:37 RobH: updating dns for datacenter cameras
* 15:32 logmsgbot: catrope synchronized php-1.18/extensions/Babel/Babel.i18n.php 'r97741'
* 15:31 logmsgbot: catrope synchronized php-1.17/extensions/Babel/Babel.i18n.php 'r97741'
* 14:29 robh: es1003 hdd replaced per rt#1453
* 14:05 logmsgbot: catrope synchronized php-1.18/extensions/WikiEditor/WikiEditor.php 'r97734'
* 13:45 notpeter: removing srv283 from apaches pool for upgrade to lucid
* 13:43 logmsgbot: catrope synchronized php-1.18/extensions/WikimediaMessages/WikimediaMessages.php 'r97730'
* 13:32 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable Babel extension on all wikis'
* 13:23 logmsgbot: catrope synchronized php-1.17/includes/Message.php 'r97729'
* 13:12 notpeter: removing srv285 from apaches pool for upgrade to lucid
* 13:06 notpeter: putting srv286 back in apaches pool
* 13:06 logmsgbot: catrope synchronized php-1.18/extensions/Babel/Babel.i18n.php 'r97728'
* 13:06 logmsgbot: catrope synchronized php-1.18/extensions/Babel/Babel.class.php 'r97728'
* 13:06 logmsgbot: catrope synchronized php-1.17/extensions/Babel/Babel.i18n.php 'r97728'
* 13:06 logmsgbot: catrope synchronized php-1.17/extensions/Babel/Babel.class.php 'r97728'
* 13:03 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add config for Babel, pulled from wikipage on meta by SPQRobin'
* 12:53 logmsgbot: catrope synchronized php-1.17/includes/OutputPage.php 'r97723'
* 12:53 logmsgbot: catrope synchronized php-1.17/includes/parser/ParserOutput.php 'r97723'
* 12:43 logmsgbot: catrope synchronized php-1.18/includes/OutputPage.php 'r97722'
* 12:41 logmsgbot: catrope synchronized php-1.18/includes/parser/ParserOutput.php 'r97722'
* 12:22 logmsgbot: catrope synchronizing Wikimedia installation... : Add Babel to extension-list and enable it on testwiki and test2wiki
* 12:18 notpeter: removing srv286 from apaches pool for upgrade to lucid
* 12:16 logmsgbot: catrope synchronizing Wikimedia installation... : Pushing out code for Babel, not enabled yet
* 10:27 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'revert test'
* 10:25 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add group so I can briefly view Special:MoodBar'
* 03:12 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'removed unused ReaderFeedback settings from some wikis'
* 03:09 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'removed flaggedrevs_labswikimedia config duplication'
* 02:23 logmsgbot: LocalisationUpdate completed (1.18) at Wed Sep 21 02:26:17 UTC 2011
* 02:20 logmsgbot: LocalisationUpdate completed (1.17) at Wed Sep 21 02:23:08 UTC 2011
* 02:17 Tim: rebooting srv182: ssh hangs after connecting, nothing on serial console
* 02:14 logmsgbot: tstarling synchronized php-1.18/includes/profiler/ProfilerSimple.php 'r97690'
* 02:12 Tim: cleaning up /tmp on srv215, which is almost out of disk space in its root partition
* 01:06 logmsgbot: asher synchronized php-1.17/includes/ExternalStoreDB.php 'bug 31052 : live hack to support reading from old non-slave external store servers'
* 00:07 maplebed: shutting down mysql on ms1 to replicate to eqiad. It's already out of rotation.
== September 20 ==
* 23:31 logmsgbot: reedy synchronizing Wikimedia installation... : Pushing out 1.18wmf1 merges, mainly image rotation fixes
* 23:30 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/information/disable.html.php 'emergency fix for disable link endless loop'
* 23:29 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/information/disable.html.php 'emergency fix for disable link endless loop'
* 23:26 preilly: emergency fix for disable link endless loop on mobile frontend
* 23:21 maplebed: increased ganglia's tmpfs from 700M to 1G and restarted gmetad. it's all happy again, but we've lost ~8hrs ganglia data.
* 23:08 maplebed: ganglia's tmpfs is out of space.
* 22:55 logmsgbot: asher synchronized wmf-config/db.php 'experimenting with an externalstore cluster in readOnlyBySection'
* 22:50 notpeter: putting srv287 back in the apaches poolwq
* 22:13 notpeter: removing srv287 from apaches pool to upgrade to lucid
* 22:05 logmsgbot: asher synchronized php-1.17/includes/resourceloader/ResourceLoaderFileModule.php 'live hack - prevent resourceloader from serving errors when a db master is set to read-only'
* 21:05 binasher: invalidated varnish mobile cache due to cached pages linking to broken javascript
* 20:52 logmsgbot: aaron synchronized wmf-config/flaggedrevs.php 'removed duplication with delabs config; tabify some w/s'
* 20:18 Jeff_Green: rebuilding db1048 per ticket #1477
* 20:02 notpeter: putting srv288 back into apaches pool
* 19:58 binasher: fully done pmtpa s2/s3 1.18 schema upgrades
* 19:55 logmsgbot: aaron synchronized wikiversions.cdb 'moving delabs to 1.18'
* 19:55 logmsgbot: aaron synchronized wikiversions.dat 'moving delabs to 1.18'
* 19:31 preilly: fix caching issue for javascript 
* 19:30 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/application.html.php 'fix javascript issue with wrong version in cache'
* 19:20 binasher: s3/DEFAULT new master / log pos = db34-bin.000374 113472055
* 19:20 binasher: s2 new master / log pos = db30-bin.000031 414029948
* 19:12 binasher: running 1.18 schema migrations against all the wikis on the former s2/s3 master hosts
* 19:05 logmsgbot: asher synchronized wmf-config/db.php 'setting S2 and DEFAULT clusters to writable on new masters'
* 18:59 binasher: switching s2 master to db30
* 18:55 binasher: switching default cluster master to db34
* 18:54 logmsgbot: asher synchronized wmf-config/db.php 'setting S2 and DEFAULT clusters to read-only'
* 18:51 binasher: preparing to set db clusters S2 and S3(default) to read only, in prep for swapping masters
* 18:01 logmsgbot: reedy synchronized php-1.18/extensions/FlaggedRevs/FlaggedRevs.php 'r97648'
* 17:40 robh: dns updated for rt#1279
* 17:30 robh: updated dns for rt$1225
* 16:47 mark: installed OS on arsenic and niobium (eqiad bits servers)
* 16:42 logmsgbot: reedy synchronized wmf-config/xrumer/XRumerCookieBug.php 'Fix function definition'
* 16:40 logmsgbot: reedy synchronized wmf-config/xrumer/XRumerCookieBug.php 'Fix function definition'
* 16:37 logmsgbot: reedy synchronized wmf-config/xrumer/XRumerCookieBug.php 'Fix function definition'
* 16:35 logmsgbot: reedy synchronized wmf-config/xrumer/XRumerCookieBug.php 'Fix function definition'
* 16:32 logmsgbot: reedy synchronized wmf-config/xrumer/XRumerCookieBug.php 'Fix function definition'
* 16:14 logmsgbot: catrope synchronized php-1.18/extensions/PrefSwitch/PrefSwitch.php 'r97640'
* 16:14 logmsgbot: catrope synchronized php-1.18/extensions/PrefSwitch/PrefSwitch.hooks.php 'r97640'
* 15:59 logmsgbot: catrope synchronizing Wikimedia installation... : Push out 1.18wmf1 fixes
* 14:50 robh: pushed puppet update for erik z public key. stat1 old authorized keys wiped and new file in place
* 14:29 ^demon: if you run a 1.17 maintenance script on a 1.18 wiki, you get a fatal about not being able to find Math extension. Odd, since the check in CommonSettings.php should be sufficient (and $wmfVersionNumber is correct in saying 1.18)
* 13:42 Reedy: Finished creating user_former_groups table on all wikis
* 13:38 Reedy: Creating user_former_groups tables on all wikis
* 13:29 Reedy: Created user_former_groups tables on test2wiki
* 13:24 logmsgbot: reedy synchronized php-1.18/includes/OutputPage.php 'r97624'
* 10:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30938 - Enable $wgExtraGenderNamespaces on plwiki *BEFORE* MW1.18 deployment'
* 10:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30982 - Subpages activation for some namespaces on WMRU wik'
* 10:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30988 - Activate subages for Portail namespace on fr.wikisource'
* 09:29 RoanKattouw-BSc: Added skins-1.18 symlink to /h/w/common/docroot/secure
* 09:24 logmsgbot: catrope synchronized wmf-config/secure.php 'Fix hardcoded -1.17 in secure.php'
* 09:06 logmsgbot: catrope synchronized php-1.18/resources/startup.js 'touch'
* 09:05 logmsgbot: catrope synchronized php-1.18/includes/resourceloader/ResourceLoaderContext.php 'Experimental fix for another ResourceLoader back compat issue'
* 09:00 logmsgbot: catrope synchronized php-1.18/resources/mediawiki/mediawiki.js 'Experimental fix for ResourceLoader back compat issue'
* 08:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable LQT on test2wiki'
* 08:53 Reedy: Created LiquidThreads tables on test2wiki
* 08:44 logmsgbot: catrope synchronized php-1.18/resources/jquery/jquery.textSelection.js 'r97604'
* 08:34 logmsgbot: catrope synchronized php/resources/jquery/jquery.textSelection.js 'r97602'
* 03:55 Tim: in LDAP, added myself to the wmf and ops groups
* 03:48 AaronSchulz: Removed $oldLabsWikis cruft from CommonSettings.php
* 03:47 logmsgbot: aaron ran sync-common-all 'enabled FlaggedRevs on test2wiki'
* 03:18 Tim: created a password for myself in LDAP
* 02:40 binasher: completed 1.18 schema migrations against all slaves. masters still need to be rotated and upgraded
* 02:29 logmsgbot: LocalisationUpdate completed (1.18) at Tue Sep 20 02:32:24 UTC 2011
* 02:24 logmsgbot: LocalisationUpdate completed (1.17) at Tue Sep 20 02:26:48 UTC 2011
* 02:14 binasher: finished 1.18 migrations on s6 slaves, started s7
* 01:48 Tim: stopped puppet on fenari since I need to change a file and puppet on wikitech doesn't have any documentation about the new git system
* 01:43 binasher: finished 1.18 migrations on s3/DEFAULT slaves, started s6
* 01:38 preilly: sync file to fix uncaught type error with reading null property 
* 01:38 logmsgbot: preilly synchronized php/extensions/MobileFrontend/javascripts/application.js 'fix javascript error with null object'
* 01:30 logmsgbot: tstarling synchronized php-1.18/includes/profiler/ProfilerSimpleUDP.php 'r97591'
* 01:18 logmsgbot: aaron synchronized wikiversions.cdb 'set test2wiki to 1.18 again (for real this time)'
* 01:17 binasher: finished 1.18 migrations on s5 slaves, started s3/DEFAULT
* 01:16 logmsgbot: aaron synchronized wikiversions.cdb 'set test2wiki to 1.18 again'
* 01:16 logmsgbot: aaron synchronized wikiversions.dat 'set test2wiki to 1.18 again'
* 01:12 logmsgbot: aaron synchronized wikiversions.cdb 'set test2wiki back to 1.17'
* 01:12 logmsgbot: aaron synchronized wikiversions.dat 'set test2wiki back to 1.17'
* 00:33 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'fixed $wgTexvc dir for 1.18 (override extension default)'
* 00:28 logmsgbot: reedy synchronized php-1.18/includes/OutputPage.php 'r97588'
* 00:21 logmsgbot: reedy synchronized php-1.18/extensions/TitleBlacklist/TitleBlacklist.list.php 'r97583'
* 00:02 binasher: finished 1.18 migrations on s4 slaves, started s5
== September 19 ==
* 23:58 logmsgbot: reedy synchronized php-1.18/StartProfiler.php 'r97581'
* 23:52 binasher: finished 1.18 migrations on s2 slaves, started s4
* 23:48 maplebed: took ms1 out of rotation in the external data store hosts to create a new slave.
* 23:47 logmsgbot: ben synchronized wmf-config/db.php 'taking ms1 out of rotation to create a new slave in eqiad RT-1530'
* 23:29 logmsgbot: aaron synchronized php-1.18/includes/profiler/Profiler.php 'deployed r97577'
* 23:12 logmsgbot: aaron synchronizing Wikimedia installation... : switching testwiki back to regular 1.17
* 22:55 logmsgbot: aaron synchronizing Wikimedia installation... : switching test2wiki to 1.18
* 22:54 binasher: test2wiki on s3 has been migrated by hand 
* 22:52 binasher: 1.18 schema migrations completed on all s1 slaves, starting s2
* 22:04 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Pre-emptively disable file movers for "user" group (for 1.18, r93871)'
* 19:48 binasher: 1.18 schema migrations completed against enwiki on the first slave (db36)
* 19:04 logmsgbot: neilk synchronizing Wikimedia installation... : regular UploadWizard update, a few changes to make JS message parser throw less fatals, other misc small fixes
* 19:02 binasher: begin run of /home/w/common/php-1.18/maintenance/upgrade-1.18wmf1-1.php
* 18:39 logmsgbot: LocalisationUpdate failed
* 18:38 preilly: removing show/hide from blackberry for now
* 18:38 logmsgbot: preilly synchronized php/extensions/MobileFrontend/DeviceDetection.php 'remove show/hide for now from blackberry'
* 18:05 logmsgbot: catrope synchronizing Wikimedia installation... : Deploy Narayam changes
* 17:50 RoanKattouw-BSc: Moved /home/wikipedia/common/php-1.17-test/includes/.swp to /home/midom/php-1.17-test-includes-.swp , was causing sync errors due to restrictive perms
* 17:47 logmsgbot: aaron ran sync-common-all 'pushing 1.18 files out to apaches'
* 17:40 logmsgbot: aaron ran sync-common-all 'pushing 1.18 files out to apaches'
* 17:37 AaronSchulz: created php-1.18/cache/trusted-xff.cdb
* 17:28 AaronSchulz: Ran checkoutMediaWiki script just add php-1.18 non-svn files/symlinks (reedy checked out already)
* 16:54 logmsgbot: reedy synchronizing Wikimedia installation... : Deployment of 1.18 files to apaches
* 16:51 Reedy: Copy of php-1.18 to /home/wikipedia/common/php-1.18 finished, taking just under 46 minutes
* 16:03 Reedy: copying /tmp/php-1.18 to /home/wikipedia/common/php-1.18 in a screen session on fenari as reedy
* 16:02 Reedy: svn co done
* 16:01 Reedy: Running svn co svn+ssh://svn.wikimedia.org/svnroot/mediawiki/branches/wmf/1.18wmf1 on fenari to /tmp/php-1.18 in a screen session as reedy
* 14:24 mark: Upgraded asw-a4-dtpa and asw-a5-sdtpa to 2.7.02d to resolve memory corruption issues (RT #1113)
* 13:55 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30984 - Filemover group in ruwiki, allow sysop to remove filemover'
* 12:27 mark: Fixed Puppet on amslvs* [I4a710a8d] (RT #1448)
* 12:01 mark: Corrected wrong LVS service ip of search_pool3 [Icd27ac04]
* 11:49 mark: Changed depool thresholds for search servers LVS services [Ib58b6178]
* 11:07 mutante: owa2 - power up,run puppet,dist-upgrade,reboot - RT #1510
* 09:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30984 - Filemover group in ruwiki'
* 02:28 logmsgbot: LocalisationUpdate completed (1.17-test) at Mon Sep 19 02:31:18 UTC 2011
* 02:23 logmsgbot: LocalisationUpdate completed (1.17) at Mon Sep 19 02:25:34 UTC 2011
== September 18 ==
* 23:52 Tim: added comments to lsearch-global-2.1.conf indicating that the "x" hostnames are to disable services
* 23:13 Tim: restarted lsearchd on search7, that seems to have done something
* 23:09 Tim: search7 has been broken since ~09:00 UTC, trying to fix
* 07:18 apergos: restarted lsearchd on search7 about 5 min ago 
* 04:06 AaronSchulz: created l10n in php-1.17-test/cache
* 02:19 logmsgbot: LocalisationUpdate failed (1.17-test)
* 02:19 logmsgbot: LocalisationUpdate completed (1.17) at Sun Sep 18 02:21:34 UTC 2011
== September 17 ==
* 17:43 AaronSchulz: fixed 10update script, same bug with $mwVerDbSets
* 13:00 domas: manually placed /etc/php5/conf.d/apc-aux.conf on lucid apaches, apparently working APC cache helps a lot with request processing
* 11:26 mark: PyBal on lvs1-3 was in a BGP colliding loop with cr2-pmtpa, restarted PyBal on all three hosts
* 10:44 domas: made profile hitrate collection less crashy, restarted it (from ~tstarling/src/udpprofile/, not sure where else it should've been)
* 10:44 mark: Upgrade of cr2-eqiad complete
* 10:34 mark: Upgrading re1.cr2-eqiad to junos 10.4R7.5
* 10:28 mark: Performing failover from cr2-eqiad re1 to re0
* 10:18 mark: Upgrade of cr1-eqiad complete
* 10:08 mark: Upgrading re0.cr1-eqiad to junos 10.4R7.5
* 09:55 mark: Upgrading re0.cr2-eqiad to junos 10.4R7.5
* 09:46 mark: Performing failover from cr1-eqiad re0 to re1
* 09:33 mark_: Upgrading re1.cr1-eqiad to junos 10.4R7.5
* 02:22 logmsgbot: LocalisationUpdate completed (1.17=aawiki 1.17-test) at Sat Sep 17 02:25:20 UTC 2011
* 00:00 notpeter: removing srv288 from the apaches pool for upgrade to lucid
== September 16 ==
* 23:48 maplebed: started copying data from all srv hosts with external storage databases to es1001. I expect it to run all weekend.
* 23:39 notpeter: putting srv289 back in the apache pool
* 23:12 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'added require for Math extension, checks $wmgUseMath and version compare with $wmfVersionNumber so it does nothing yet'
* 22:50 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'added wmgUseMath setting (on by default); not yet used'
* 21:27 AaronSchulz: fixed usage of $mwVersionNums in set-group-write2 (same double quotes bug the sync scripts had before)
* 20:51 logmsgbot: aaron synchronized php-1.17/extensions/FlaggedRevs/specialpages/ReviewedVersions_body.php 'deployed r97331'
* 20:51 Jeff_Green: putting http redirect back in front of donate.wikimedia.org b/c fundraisers aren't ready for it
* 20:51 logmsgbot: aaron synchronized php-1.17/extensions/FlaggedRevs/api/FlaggedRevsApi.hooks.php 'deployed r97331'
* 20:29 AaronSchulz: fixed geshi on test2 & secure testwiki
* 20:25 logmsgbot: aaron ran sync-common-all 
* 20:22 AaronSchulz: changed mwscript & mwscriptwikiset to use the /home version of scripts (people usually did this before and some things like TrustedXFF expect it)
* 20:07 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Set wgAutopromoteOnceLogInRC for plwiki,plwikisource ahead of time - no effect yet - bug 29655'
* 20:02 logmsgbot: aaron synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevision.php 'deployed r97323'
* 19:55 Jeff_Green: repooled payments4 after its long strange testing hiatus
* 19:55 logmsgbot: aaron synchronized php-1.17-test/extensions/FlaggedRevs/FlaggedRevision.php 'deployed r97323'
* 19:03 preilly: fix for undefined method on wml card 
* 19:02 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix method call to undefined method'
* 18:45 AaronSchulz: added php-1.17-test/libs/geshi symlink to common/libs geshi, working on testwiki now
* 18:43 AaronSchulz: created common/lib directory and placed a copy of GeSHi-1.0.8.10 inside
* 18:19 preilly: add version to mobile application js to bust cache
* 18:18 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/application.html.php 'remove jquery add version id to bust cache'
* 17:55 preilly: size correction for mobile icons
* 17:55 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'config project icons for mobile size corrections'
* 15:19 logmsgbot: reedy synchronized php/extensions/OggHandler/OggHandler_body.php 'r97289'
* 15:18 logmsgbot: reedy synchronized php/extensions/OggHandler/OggPlayer.js 'r97289'
* 15:17 logmsgbot: reedy synchronized php/languages/messages/MessagesKk_cyrl.php 'r97287'
* 15:17 logmsgbot: reedy synchronized php/includes/HttpFunctions.php 'r97287'
* 14:53 domas: killed whoever was running full table scans with subselects on enwiki and hitting http://bugs.mysql.com/bug.php?id=46947 - thanks, you locked up enwiki for many users :)
* 12:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30928 - Make Portal namespace for shwikipedia'
* 11:47 Reedy: Run namespaceDupes on hywiki
* 02:22 logmsgbot: LocalisationUpdate completed (1.17=aawiki 1.17-test) at Fri Sep 16 02:24:43 UTC 2011
* 01:49 logmsgbot: aaron synchronizing Wikimedia installation... : 
* 01:41 AaronSchulz: moved test2wiki to php-1.17-test
* 01:36 AaronSchulz: testing with testwiki on php-1.17-test
* 01:04 logmsgbot: aaron synchronizing Wikimedia installation... : 
* 00:47 logmsgbot: aaron synchronized wikiversions.dat 'set testwiki to 1.17-test'
* 00:47 logmsgbot: aaron synchronized wikiversions.cdb 'set testwiki to 1.17-test'
== September 15 ==
* 23:32 Tim: running apt-get upgrade on hume
* 22:31 binasher: upgraded varnish on mobile cache servers to pull in udplog fix
* 22:15 preilly: today's push for mobile frontend
* 22:15 logmsgbot: preilly synchronized php/extensions/MobileFrontend/javascripts/application.js 'remove jquery'
* 22:14 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'wml paging'
* 22:14 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/application.html.php 'remove jquery'
* 21:08 Jeff_Green: purged hume:/a/static/uncompressed and switched it to xfs. see RT #1474 for more info
* 20:59 robh: updated puppet for new pdus in row d-pmtpa as well as chris shell access
* 20:50 notpeter: removing srv289 from apaches pool for upgrade to lucid. also stopping memecache and apache.
* 20:40 Ryan_Lane: restating gerrit to change subject lines in change emails
* 19:30 notpeter: putting srv219 back into rending pool
* 18:40 mark_: Deactivate static routes on cr2-pmtpa
* 17:50 mark_: Upgraded cr2-pmtpa to JUNOS 10.4R7.5
* 17:50 notpeter: rebooting spence
* 17:44 notpeter: restarting torrus
* 17:40 mark_: Rebooting cr2-pmtpa
* 16:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Revert bug 30594 (namespaces for plwiki)'
* 15:15 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'bug 30808: wgRestrictDisplayTitle for enwikibooks, should've been false'
* 02:27 logmsgbot: LocalisationUpdate completed (1.17) at Thu Sep 15 02:29:44 UTC 2011
== September 14 ==
* 23:32 logmsgbot: aaron synchronizing Wikimedia installation... : 
* 23:25 logmsgbot: aaron synchronizing Wikimedia installation... : 
* 23:08 logmsgbot: aaron synchronized multiversion/MWScript.php 'r96507'
* 23:07 logmsgbot: aaron synchronized multiversion/MWVersion.php 'r97110'
* 22:59 Ryan_Lane: finished switchover to git for puppet repos
* 22:14 Ryan_Lane: switching the svn puppet repo to the git one. going to lock the svn repo
* 22:11 binasher: re-imaged db14 to lucid
* 21:28 binasher: disabled nagios notifs for db14, preparing to re-image
* 20:50 logmsgbot: reedy synchronized php/resources/startup.js 'touch'
* 20:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 28081 - Install ArticleFeedback on Portuguese Wikipedia (ptwiki)'
* 20:40 logmsgbot: reedy synchronized php/resources/startup.js 'touch'
* 20:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 28081 - Install ArticleFeedback on Portuguese Wikipedia (ptwiki)'
* 19:46 Reedy: That was bug 30726
* 19:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Setup of confirmed group for ptwiki'
* 19:42 logmsgbot: py synchronized wmf-config/CommonSettings.php 'syncing out correct location of oggThumb'
* 19:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30830 - Please add a Portal namespace to hywiki'
* 19:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Swap NS_PORTAL for 100'
* 19:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30432 - set meta namespaces for ay.wikipedia'
* 19:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30808 - Set to false on English Wikibooks'
* 18:20 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Add cp1043 and cp1044 to wgSquidServersNoPurge'
* 18:07 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30798 - Logo for Sami Wikipedia'
* 17:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30899 - Timezone for lmo.wikipedia'
* 15:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'fawiki -'
* 15:02 Jeff_Green: correction: removed donate.wikiPedia.org from redirects.conf, sync-apache, apache-graceful-all
* 15:01 Jeff_Green: removed donate.wikimedia.org from redirects.conf, sync-apache, apache-graceful-all
* 14:57 robh: ps1-b5-eqiad
* 14:57 robh: lowered the humidity alarm on ps1-b25-eqiad, also fixed some of its configuration
* 14:24 logmsgbot: catrope synchronized php/extensions/LiquidThreads/classes/Hooks.php 'Fix stupid typo'
* 14:17 logmsgbot: catrope synchronized php/extensions/LiquidThreads/classes/Hooks.php 'More live hacks for fatals that Ariel has been seeing in the dumps'
* 14:10 logmsgbot: catrope synchronized php/extensions/LiquidThreads/classes/Hooks.php 'Live hack to work around fatal'
* 14:08 logmsgbot: catrope synchronized php/extensions/LiquidThreads/classes/Hooks.php 'Live hack to work around fatal'
* 14:07 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30832 - Set $wgOverrideSiteFeed to use Special:NewsFeed from GNSM extension, on Persian Wikinews (fawikinews)'
* 14:01 Reedy: Created ArticleFeedback tables on ptwiki
* 13:59 logmsgbot: reedy synchronized php/resources/startup.js 'Touch'
* 13:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable ClickTracking on ptwiki as prequisite for AF'
* 12:24 mutante: srv281 - powercycle (one more time but created a hardware check ticket now)
* 11:56 mark: Upgrading asw-d-pmtpa to junos 10.4R3.4
* 11:45 mark: Added asw-d-pmtpa to rancid, torrus, observium
* 11:36 mark: Configured asw-d-pmtpa
* 11:12 RoanKattouw: Started Apache again on srv219 now that it's depooled
* 11:10 Tim: removed srv219 from /home/wikipedia/conf/pybal/pmtpa/rendering on reports that it is giving inappropriate PHP errors
* 10:59 RoanKattouw: Stopped Apache on srv219, was spewing lots of PHP errors. Needs an ops person to depool the box
* 10:51 RoanKattouw: logmsgbot: !log catrope synchronized wmf-config/CommonSettings.php 'Raise account creation throttle for India event per IRC request'
* 10:51 RoanKattouw: Restarted morebots
* 02:30 logmsgbot: LocalisationUpdate completed (1.17) at Wed Sep 14 02:32:57 UTC 2011
* 02:16 logmsgbot: root synchronized wmf-config/InitialiseSettings.php 'Enabling protocol-relative urls for donatewiki'
* 02:04 logmsgbot: root ran sync-common-all 
* 00:34 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/_footmenu_default.html.php 'remove feedback'
* 00:15 preilly: fix for wml issues
* 00:15 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix for wml issues'
== September 13 ==
* 23:26 notpeter: removing srv219 from rendering pool for upgrade to lucid
* 23:23 notpeter: putting srv220 back into the rendering pool
* 22:26 notpeter: removing srv220 from the rendering pool for upgrade to lucid
* 22:14 notpeter: putting srv221 back into rendering pool
* 21:35 binasher: srv207 was serving traffic with an old deploy and incorrect db.php. synced, and added back to mediawiki-installation grp
* 21:23 binasher: mysql 4.0.40 is dead! 
* 21:22 binasher: S6 master changed to db43, master-pos is db43-bin.000227 144900190
* 21:21 logmsgbot: asher synchronized wmf-config/db.php 'setting S6 to writeable on new master'
* 21:18 logmsgbot: asher synchronized wmf-config/db.php 'setting S6 ({ru,fr,ja}wiki) to read only, replacing mysql master'
* 19:46 notpeter: removing srv221 from rendering pool for upgrade to lucid
* 19:39 maplebed: wikitech ran out of space on /. removed old log files. installed logrotate.
* 18:44 logmsgbot: asher synchronized wmf-config/db.php 'setting S5 (dewiki) to read only, replacing mysql master'
* 18:37 binasher: preparing to replace S5 master - will set to readonly shortly
* 18:32 notpeter: removing srv222 from rendering pool for upgrade to lucid
* 18:20 preilly: fix mismatched div
* 18:20 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix for wml card mismatched div'
* 18:05 mark: Rebooted asw-d-pmtpa to recycle VC member IDs
* 18:04 preilly: wml fixes
* 18:04 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/application.wml.php 'remove element title'
* 18:03 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix for wml card id attribute'
* 14:30 logmsgbot: reedy synchronized php/languages/messages/MessagesEn.php 'r96969'
* 09:01 mutante: knsq10 - powercycle, dist-upgrade/kernel,squid clean,reboot
* 08:55 mutante: srv281 - powercycle,start apache
* 02:24 logmsgbot: LocalisationUpdate completed (1.17) at Tue Sep 13 02:27:04 UTC 2011
* 01:16 neilk_: just deployed a config change (wmf-config r2266) that should fix bug #30797
* 01:13 logmsgbot: neilk synchronized wmf-config/CommonSettings.php 'the URL for the UploadStash image scaler should not be protocol-relative -- this is invoked with curl, behind the scenes to fetch a scaled image.'
* 00:48 notpeter: putting srv223 back into rendering pool
== September 12 ==
* 23:37 Ryan_Lane: add CVE-2011-3192 workaround to noc.wm.o
* 23:35 logmsgbot: midom synchronized wmf-config/db.php 'srv157 down'
* 23:23 notpeter: removing srv223 from rendering pool for upgrade
* 22:35 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'config project icons for mobile wikiquote correction'
* 22:32 preilly: pushing mobile project icon change
* 22:31 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'config project icons for mobile'
* 22:06 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix section title with link'
* 21:50 logmsgbot: reedy synchronized php/extensions/UploadWizard/resources/jquery/jquery.mwCoolCats.js 'Bug 30879 - UploadWizard adds categories without Category: prefix'
* 21:47 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Scrap that'
* 21:43 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Fix UW category addition bug'
* 21:38 robh: mc1001-mc1008 have sas controllers, ready for installation
* 21:30 notpeter: readding srv224 to rending pool
* 20:34 notpeter: removing srv224 from rendering pool for upgrade to lucid
* 19:23 logmsgbot: neilk synchronizing Wikimedia installation... : deploying for UploadWizard (up to r96823) - minor fixes to CSS/button position, WLM, categories
* 19:04 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/application.html.php 'fix view'
* 18:22 preilly: fix wml view for mobile
* 18:22 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/application.wml.php 'fix wml view'
* 18:04 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable Narayam on tawikibooks and tawikisource per bug 29798'
* 17:40 logmsgbot: catrope synchronizing Wikimedia installation... : Deploy Narayam bugfixes
* 17:39 logmsgbot: catrope synchronized php/extensions/CategoryTree/CategoryTreeFunctions.php 'r96696'
* 17:36 logmsgbot: catrope synchronized php/resources/mediawiki/mediawiki.js 'r96699'
* 16:48 logmsgbot: reedy synchronized php-1.17/includes/Linker.php 'r96864'
* 16:47 logmsgbot: reedy synchronized php-1.17/extensions/CodeReview/ui/CodeRevisionAuthorView.php 'r96864'
* 16:40 logmsgbot: reedy synchronized php-1.17/includes/Linker.php 'Revert'
* 16:38 logmsgbot: reedy synchronized php-1.17/includes/Linker.php 'r96862, r96858'
* 16:34 logmsgbot: reedy synchronized php-1.17/includes/Linker.php 'Revert push'
* 16:32 logmsgbot: reedy synchronized php-1.17/extensions/EmailCapture/SpecialEmailCapture.php 'r96860'
* 16:31 logmsgbot: reedy synchronized php-1.17/extensions/EmailCapture/EmailCapture.i18n.php 'r96860'
* 16:31 logmsgbot: reedy synchronized php-1.17/extensions/CodeReview/ui/CodeRevisionAuthorView.php 'r96860'
* 16:30 logmsgbot: reedy synchronized php-1.17/includes/Linker.php 'r96858'
* 16:26 robh: updating dns for new misc servers mgmt
* 15:41 Reedy: That was scapping r96856
* 15:40 logmsgbot: reedy synchronizing Wikimedia installation... : 
* 15:01 logmsgbot: catrope synchronized php/extensions/UploadWizard/resources/mw.UploadWizardDeed.js 'r96848'
* 14:25 logmsgbot: reedy synchronized php-1.17/skins/common/shared.css 'r96829'
* 11:42 mutante: sq78 - powercycle, dist-upgrade/kernel, squid clean, reboot
* 11:06 mutante: sq66 - powercycle, dist-upgrade/kernel, squid clean, reboot
* 10:44 mark: Setup IPv6 transit to AS2828 on cr1-sdtpa
* 10:42 mutante: srv187 - power up, dist-upgrade/kernel, reboot
* 10:29 mutante: amssq56 - power cycle, dist-upgrade/kernel, squid clean,reboot
* 07:09 apergos: restarted searchd on search7
* 02:23 logmsgbot: LocalisationUpdate completed (1.17) at Mon Sep 12 02:25:37 UTC 2011
== September 11 ==
* 02:21 logmsgbot: LocalisationUpdate completed (1.17) at Sun Sep 11 02:23:26 UTC 2011
== September 10 ==
* 22:02 Ryan_Lane: purged binlogs on db9 to free up some space
* 02:24 logmsgbot: LocalisationUpdate completed (1.17) at Sat Sep 10 02:26:45 UTC 2011
== September 9 ==
* 19:06 logmsgbot: reedy synchronized php-1.17/resources/startup.js 'touch'
* 19:05 logmsgbot: reedy synchronized php-1.17/resources/mediawiki/mediawiki.js 'r96682'
* 18:54 logmsgbot: reedy synchronized php-1.17/resources/startup.js 'touch'
* 18:51 logmsgbot: reedy synchronized php-1.17/resources/mediawiki/mediawiki.js 'Revert push of r96679, stuff seems broken'
* 18:40 logmsgbot: catrope synchronized php/resources/mediawiki/mediawiki.js 'r96679'
* 18:04 Ryan_Lane: gave Danny B. editor rights on wikitech
* 16:36 mutante: amssq37 - power up, dist-upgrade/kernel,squid clean, reboot
* 15:44 mutante: ran sync-apache and apache-graceful-all to add 2 redirects (RT 1445 + 1449)
* 11:33 logmsgbot: catrope synchronized php/resources/jquery/jquery.highlightText.js 'r96625'
* 10:41 mutante: sq60 - power cycle,dist-upgrade/kernel ,reboot
* 10:28 mutante: sq65 - power up, dist-upgrade/kernel,squid clean,reboot
* 09:07 mark: Removed configuration for XO on csw5-pmtpa
* 09:00 mark: Set local-pref of most XO routes to 90, to favour traffic to AS 3257
* 02:22 logmsgbot: LocalisationUpdate completed (1.17) at Fri Sep 9 02:24:42 UTC 2011
* 01:24 preilly: fix random link
* 01:24 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix random link'
* 01:11 Tim: restarted parser cache hit rate stats collector on fenari
* 00:38 preilly: only show feedback link on en wiki
* 00:38 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/_footmenu_default.html.php 'only show feedback on en wiki'
* 00:23 logmsgbot: aaron synchronizing Wikimedia installation... : 
== September 8 ==
* 23:32 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/_footmenu_default.html.php 'fix footer links'
* 23:32 preilly: pushing fix for footer links
* 23:31 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix footer links'
* 22:02 binasher: changed m.wikipedia.org dns cname to point to mobile-lb.eqiad.wikimedia.org
* 21:56 binasher: upgraded wikimedia-task-appserver to 1.49 (het deploy) in the hardy repo
* 21:49 binasher: moving m.wikipedia.org nagios monitor to point to mobile-lb.eqiad.wikimedia.org
* 16:21 mark: Shutdown csw5-pmtpa:e5/3
* 16:21 mark: Migrated XO transit connection from csw5-pmtpa:e5/3 to cr1-sdtpa:xe-0/0/2
* 14:55 mark: Shutdown BGP sessions to AS 2828 on csw5-pmtpa
* 12:50 logmsgbot: catrope synchronizing Wikimedia installation... : 
* 12:50 RoanKattouw: Seems I forgot to deploy my 1.17wmf1 merges from yesterday, scapping them out now
* 12:35 mark: Activated XO BGP session on cr1-sdtpa
* 02:28 logmsgbot: LocalisationUpdate completed (1.17) at Thu Sep 8 02:30:26 UTC 2011
* 00:42 preilly: fix for home page with styles that float images
* 00:42 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix homepage'
* 00:11 preilly: fix wildcard selections for mobile main page
* 00:10 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix homepage'
* 00:01 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.i18n.php 'fix homepage'
* 00:00 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix homepage'
* 00:00 preilly: pushing fix for homepage
* 00:00 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/notices/notice_1.html.php 'fix homepage'
* 00:00 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/application.html.php 'fix homepage'
== September 7 ==
* 23:59 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/_footmenu_default.html.php 'fix homepage'
* 23:59 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/information/leave_feedback.html.php 'fix homepage'
* 19:46 logmsgbot: reedy synchronized php-1.17/includes/OutputPage.php 'r96472'
* 18:49 Reedy: That sync was Bug 30621 - Set $wgSiteFeed['rss'] to use the new Special:NewsFeed from GNSM on English Wikinews
* 18:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 
* 17:50 logmsgbot: catrope synchronized php/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'r96455'
* 17:50 logmsgbot: catrope synchronized php/extensions/UploadWizard/resources/mw.UploadWizard.js 'r96455'
* 17:03 Reedy: Run namespaceDupes on sourceswiki again, this time with --fix
* 16:07 mutante: srv281 power cycled
* 14:22 logmsgbot: catrope synchronized php/includes/SquidUpdate.php 'r96439'
* 14:22 logmsgbot: catrope synchronized php/includes/GlobalFunctions.php 'r96439'
* 14:21 logmsgbot: catrope synchronized php/includes/Defines.php 'r96439'
* 13:50 logmsgbot: catrope synchronized php/includes/SquidUpdate.php 'Reinstate Tim's live hack'
* 12:35 Reedy: Run namespaceDupes.php on sourceswiki
* 12:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30724 - Author namespace at multilingual wikisource'
* 12:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30724 - Author namespace at multilingual wikisource'
* 12:29 mark: Setup port and BGP configuration for XO on cr1-sdtpa
* 12:29 mark: Setup generic BGP_transit_in import policy statement on cr1-eqiad, cr2-eqiad, cr1-sdtpa, cr2-pmtpa, and migrated TiNet neighbor to use it
* 12:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30724 - Author namespace at multilingual wikisource'
* 12:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30724 - Author namespace at multilingual wikisource'
* 12:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30756 - Enable WikiLove on Swedish Wikipedia'
* 11:05 mutante: db15 - dist-upgrade/kernel, reboot
* 10:53 mutante: db15 - reset/power up via Sun LOM
* 10:20 logmsgbot: catrope synchronized php/includes/SquidUpdate.php 'Revert emergency patch for bug 30792'
* 10:19 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'bug 30792: set $wgInternalServer'
* 09:19 mutante: sq72 - powercycle,dist-upgrade/kernel,squid clean,reboot
* 02:53 logmsgbot: tstarling synchronized php-1.17/extensions/OggHandler/OggHandler_body.php 'live patch to fix total breakage of OggHandler due to protocol-relative upload URLs'
* 02:45 logmsgbot: tstarling synchronized php-1.17/includes/SquidUpdate.php 'emergency patch for bug 30792'
* 02:21 logmsgbot: LocalisationUpdate completed (1.17) at Wed Sep 7 02:24:09 UTC 2011
* 01:40 logmsgbot: asher synchronized 503.html 'running sync-file to test new wikimedia-task-appserver'
== September 6 ==
* 22:38 logmsgbot: catrope synchronized php/languages/messages/MessagesEn.php 'r96194'
* 22:20 logmsgbot: ben synchronized wmf-config/mc.php 'moved mc servers that passed mctest.php from the down list to the spares list. Whitespace cleanup. Linked to memcache wikitech page. -ben'
* 22:13 binasher: wiped enwiki slave on db1048, re-enabled binlogging
* 20:06 Jeff_Green: rebooting db1008 for kernel update
* 19:58 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Make wgUploadPath protocol-relative for commons, meta and test'
* 19:22 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable HTTPS on meta'
* 19:19 logmsgbot: catrope synchronized php/includes/Title.php 'r96360'
* 19:18 logmsgbot: catrope synchronized php/extensions/CentralNotice/CentralNotice.php 'r96360'
* 19:18 logmsgbot: catrope synchronized php/extensions/Collection/Collection.body.php 'r96360'
* 17:42 robh: ns1 pdns restarted
* 17:42 robh: ns1/linne pdns died. initial post sync check with dig showed new info populated, odd.
* 17:38 robh: all dns servers still responsive after update
* 17:37 robh: updating dns for sodium server per RT#1423
* 07:56 apergos: rsyncd set up and running on dataset2 for mirroring recent xml dumps
* 07:30 apergos: restarted lighttpd on dataset2
* 02:24 logmsgbot: LocalisationUpdate completed (1.17) at Tue Sep 6 02:26:41 UTC 2011
== September 5 ==
* 18:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30756 Enable wikilove on sewiki'
* 16:21 RoanKattouw: Restarting indexer on searchidx2
* 16:20 RoanKattouw: Installing new build of LuceneSearch.jar on searchidx2. Contains fix for protocol-relative URLs that only applies to the indexer
* 15:48 logmsgbot: catrope synchronized php-fatal-error.html 'Update for r96283'
* 15:42 logmsgbot: catrope synchronized live-1.5/404.php 'Per bug 30733, fix HTTPS detection and make URLs protocol-relative. Grr, why is this file not versioned?'
* 13:30 mutante: sq31 shutdown -> RT #1431 (degraded RAID, just on one disk and squid would not run anyways)
* 13:18 mark: Extended /var LVM LV on lily by 5 GB
* 13:00 mutante: sq33 - dist-upgrade/kernel, reboot | sq31 - hardware failure? ->RT #1431
* 12:22 mutante: sq31 - installed smartmontools to check disk health, squid backend does not start due to missing /dev/sdd | srv266 run puppet manually
* 11:42 mutante: srv217,srv266 - fix NTP/ntpdate | sq31 - dist-upgrade/kernel, start squid
* 11:34 mutante: srv266 - power cycle (too often since July)
* 11:25 mutante: sq76 - power up,dist upgrade/kernel,squid clean,reboot
* 11:12 mutante: also cleared paniclogs on mchenry (spamd socket fail on Aug31) and lily (temp. failure to write log due to disk space, see Tim's log entries Sep 2nd)
* 11:05 mutante: reviewd paniclog on srv207 due to non-zero size. ran out of mem once on Aug 11, cleared logs to stop email alerts
* 10:43 mutante: sq55 - power up,dist upgrade/kernel,squid clean,reboot
* 02:27 logmsgbot: LocalisationUpdate completed (1.17) at Mon Sep 5 02:29:45 UTC 2011
== September 4 ==
* 16:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'From bug 29940, kill MW/MWT namespace aliases from hiwiki. MW is an interwiki, and mwt is a languagecode'
* 02:27 logmsgbot: LocalisationUpdate completed (1.17) at Sun Sep 4 02:30:04 UTC 2011
== September 3 ==
* 02:19 logmsgbot: LocalisationUpdate completed (1.17) at Sat Sep 3 02:22:07 UTC 2011
== September 2 ==
* 22:08 logmsgbot: catrope synchronized php/extensions/UploadWizard/resources/mw.UploadWizardDetails.js 'r96154'
* 22:07 logmsgbot: catrope synchronized php/extensions/UploadWizard/includes/UploadWizardCampaign.php 'r96154'
* 22:07 logmsgbot: catrope synchronized php/extensions/UploadWizard/UploadWizard.config.php 'r96154'
* 22:07 logmsgbot: catrope synchronized php/extensions/UploadWizard/UploadWizard.i18n.php 'r96154'
* 19:15 logmsgbot: catrope synchronizing Wikimedia installation... : 
* 19:14 RoanKattouw: Running scap to repair any damage that might have been caused by the above
* 19:13 RoanKattouw: Removed empty line in /etc/dsh/group/mediawiki-installation that seemed to be breaking dsh and therefore break sync
* 18:59 logmsgbot: catrope synchronized php/resources/startup.js 'touch'
* 18:52 logmsgbot: LocalisationUpdate completed (1.17) at Fri Sep 2 18:54:34 UTC 2011
* 18:17 RoanKattouw: Removing the entire LocalisationUpdate cache and regenerating it from scratch
* 18:17 logmsgbot: catrope synchronized php/extensions/UploadWizard/UploadWizard.i18n.php 'r96136'
* 17:52 RoanKattouw: Put l10nupdate cron job on fenari back at 2:00 UTC
* 17:31 logmsgbot: catrope synchronized php/includes/parser/Preprocessor_DOM.php 'Pass XML_PARSE_HUGE flag'
* 16:21 Tim: restarting mailman on lily
* 15:45 notpeter: starting lighttpd on dataset2
* 15:11 logmsgbot: LocalisationUpdate completed (1.17) at Fri Sep 2 15:13:32 UTC 2011
* 14:53 mutante: srv217 - apache2 kept becoming a defunct zombie, there were apache2 package updates, installed them
* 14:50 RoanKattouw: Testing new l10nupdate cron job by hacking the time to 14:53 (one minute from now)
* 14:45 RoanKattouw: Moved l10nupdate cron job from hume to fenari
* 14:30 mutante: srv217 - there were zombies again "[apache2] <defunct>", kill/restart, hints on debugging?
* 14:12 Jeff_Green: depooling payments4 for fundraising testing/staging
* 14:04 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Make $wgLogo protocol-relative on foundationwiki'
* 13:24 mutante: fenari - loaded apache php module by creating symlinks from mods-available to mods-enabled (to fix http://noc.wikimedia.org/conf/)
* 12:47 mutante: changed puppet to make fenari include class mediawiki::packages which installs wikimedia-task-appserver (RT#1420)
* 11:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 
* 11:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30680 - Enable Collection for KM WP'
* 10:56 mutante: sq59 - power up,dist-upgrade/kernel,squid clean,reboot
* 10:03 Tim: on lily: reduced /var reserved space to 1% to get an extra 2GB free
* 09:56 Tim: on lily: did "apt-get clean" to get an extra 1 GB of space and restarted exim4
* 09:51 Tim: mail on lily is broken due to full /var partition
* 02:20 logmsgbot: LocalisationUpdate completed (1.17) at Fri Sep 2 02:22:27 UTC 2011
* 00:33 logmsgbot: aaron synchronizing Wikimedia installation... : 
* 00:26 logmsgbot: aaron synchronizing Wikimedia installation... : 
* 00:21 AaronSchulz: updated set-group-write2 for het deploy
== September 1 ==
* 23:03 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Setting CentralNotice fundraising banners to use https on foundationwiki'
* 20:49 logmsgbot: awjrichards synchronized php/extensions/ContributionTracking/ContributionTracking_body.php 'r96038'
* 20:49 logmsgbot: awjrichards synchronized php/extensions/ContributionTracking/ContributionTracking.processor.php 'r96037'
* 20:35 maplebed: removed all vim turds (foo~) from within ~/puppet on sockpuppet
* 19:11 preilly: fix issue with enable/disable links for images on Mobile Frontend
* 19:11 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix caching issue on enable/disable image links try two'
* 18:17 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix caching issue on enable/disable image links'
* 18:17 preilly: fix a caching issue with enable/disable images links
* 18:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Synchronise wmgWikiLoveDefault with wmgUseWikiLove, seems most places want it on by default, not off by default'
* 15:30 logmsgbot: catrope synchronized php/includes/parser/Preprocessor_DOM.php 'Revert debugging hack'
* 15:22 logmsgbot: catrope synchronized php/includes/parser/Preprocessor_DOM.php 'udp:// not udp:/ , d'oh'
* 15:19 logmsgbot: catrope synchronized php/includes/parser/Preprocessor_DOM.php 'Debug DOMDocument exceptions'
* 14:54 logmsgbot: catrope synchronized php/resources/startup.js 'touch'
* 14:03 mutante: srv266 - power cycled
* 13:53 notpeter: removing srv155 from apache pool for bug fix
* 13:51 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Don't use the protocol-relative interwiki table on secure'
* 12:26 mutante: sq74 - power up, dist-upgrade/kernel, squid clean,reboot
* 07:35 Ryan_Lane: dropped the nova database on virt1; restarting virt2, virt3, and virt4
* 02:16 logmsgbot: LocalisationUpdate completed (1.17) at Thu Sep 1 02:18:53 UTC 2011
* 00:20 maplebed: started gathering stats about storage blobs in a screen session on fenari. feel free to kill if it's causing problems.
== August 31 ==
* 23:12 preilly: fix the broken links for disable/enable images
* 23:12 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fix broken links'
* 23:02 preilly: add view that was missed in merge
* 23:02 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/layout/_footmenu_default.html.php 'add view missed in merge'
* 22:35 maplebed: increased freshness timer from 6 hours to 10 hours to reduce flapping. rt-1408
* 21:21 awjr: accidentally ran 'rsync -var grosley:/ /' rather than 'rsync -nvar grosley:/ /' on aluminium >_<
* 21:08 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'Enable HTTPSExperiment on commonswiki and set protocol-relative settings'
* 21:00 logmsgbot: asher synchronized wmf-config/db.php 'putting db44,db45 (s5) and db46,db47 (s6) into production'
* 20:27 RoanKattouw: Fix ownership of l10n cache directory (from nagios to mwdeploy) on all Apaches using dsh as root
* 20:26 Ryan_Lane: (I gave Roan permission to run that as root)
* 20:25 RoanKattouw: rsyncing /h/w/c/php/cache/l10n to the Apaches manually as root to fix permissions/ownership issues (hopefully)
* 20:12 logmsgbot: LocalisationUpdate completed (1.17) at Wed Aug 31 20:15:00 UTC 2011
* 20:10 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Fix Apple Touch Icon URL on foundationwiki'
* 20:05 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable HTTPSExperiment on foundationwiki and make wgServer protocol-relative'
* 19:56 RoanKattouw: Running LocalisationUpdate again
* 19:55 logmsgbot: LocalisationUpdate failed
* 19:48 notpeter: restarting apache on srv155.
* 19:45 Ryan_Lane: changing all cnames in wikimediafoundation.org to point to foundation-lb and changing the A record as well
* 19:38 Ryan_Lane: changing cname of wikimediafoundation.org to point to foundation-lb address
* 19:26 logmsgbot: catrope synchronizing Wikimedia installation... : Deploy HTTPS fixes
* 19:09 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Define wgCanonicalServer for all wikis, needed for upcoming deployment'
* 18:33 logmsgbot: LocalisationUpdate failed
* 18:33 RoanKattouw: Running LocalisationUpdate to hopefully fix Arabic message issue in UploadWizard
* 18:28 logmsgbot: catrope synchronizing Wikimedia installation... : Deploy MoodBar update
* 18:25 Ryan_Lane: adding new log collector on locke, running on port 8421, for aft clicktracking
* 18:24 logmsgbot: catrope synchronized php/extensions/UploadWizard/UploadWizard.i18n.php 'touch'
* 17:58 logmsgbot: catrope synchronized php/extensions/UploadWizard/UploadWizard.i18n.php 'r95893'
* 17:48 logmsgbot: catrope synchronizing Wikimedia installation... : Deploy WikimediaIncubator fixes
* 17:46 binasher: started replicating db46 + db47 from db29 (s6 master)
* 17:31 logmsgbot: catrope synchronizing Wikimedia installation... : Deploying UploadWizard bug fixes
* 17:18 binasher: started replication on db44 + db45 from db23 (s5 master)
* 15:03 mutante: db1025 apt-get --reinstall libmysql-common libmysqlclient16 - fixed the broken dpkg status
* 14:18 mutante: db1025 - dist-upgrade/kernel, kill mysql,reboot..
* 13:50 mutante: sq35 - dist-upgrade/kernel,squid clean,showed I/O errors,but squid-backend runs now,RAID recovering
* 12:38 mutante: sq64 - powercycle,dist-upgrade/kernel,squid clean,reboot
* 12:20 mutante: sq35 - start squid backend but exits again soon after (Exiting due to repeated, frequent failures-> RT 1404
* 12:10 mutante: srv217 - restart apache ([apache2] <defunct>)
* 12:07 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30491'
* 12:04 mutante: srv266 - power cycle
* 02:19 logmsgbot: LocalisationUpdate completed (1.17) at Wed Aug 31 02:21:24 UTC 2011
== August 30 ==
* 22:26 Ryan_Lane: scratch that. instances in labs will need LDAP access as well
* 22:25 Ryan_Lane: remove ldap from virt1, push new schema to svn ldap. point labs at svn ldap.
* 22:20 binasher: imaged db44-db47
* 22:12 Ryan_Lane: imported svn ldap to labs ldap
* 21:09 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 
* 21:06 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'bug 30638, set upload wizard policy page'
* 19:56 logmsgbot: hashar: previous sync was for r2238: removing old commented out parts
* 19:55 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 
* 19:18 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '30523 - Portal namespace for id.wikisource'
* 19:10 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '30594 - Change namespaces configuration - pl.wikipedia'
* 19:09 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '30594 - Change namespaces configuration - pl.wikipedia'
* 18:38 Ryan_Lane: wiping the LDAP database on virt1, re-creating using same domain as svn LDAP.
* 18:26 logmsgbot: aaron synchronizing Wikimedia installation... : 
* 17:37 logmsgbot: aaron synchronizing Wikimedia installation... : 
* 17:34 logmsgbot: aaron synchronizing Wikimedia installation... : 
* 17:30 logmsgbot: aaron synchronizing Wikimedia installation... : 
* 17:30 aaron: updated scap script file
* 17:21 Ryan_Lane: removed duplicate ip in badbadip list in squid
* 16:31 mutante: srv266 - power cycle
* 16:22 mutante: amssq38 - power up,dist-upgrade/kernel,squid clean,reboot
* 12:31 Tim: on db9 and db10, raised max_connect_errors from 10 to 1 billion so that it will stop blocking kaulen. These servers are not publically accessible, nobody can SYN flood them
* 12:25 Tim: ran "flush hosts" on db9 to temporarily fix bugzilla connection error
* 09:22 RoanKattouw: Reverted yesterday's changes on srv162 (disabled core dumps and suid_dumpable)
* 02:22 logmsgbot: LocalisationUpdate completed (1.17) at Tue Aug 30 02:25:02 UTC 2011
* 01:16 binasher: upgrading varnish on mobile caching servers to 3.0.0-1wmf5
* 00:19 binasher: added new udplog-1.7 pkg to lucid-wikimedia repo, will be upgraded everywhere via puppet on next run
== August 29 ==
* 22:01 awjr: updating payflowpro_gateway.i18n on payments1-4 - r95708
* 19:41 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 95690: 
* 19:41 RoanKattouw: Running scap to deploy Narayam changes
* 19:03 logmsgbot: neilk synchronized php/includes/Exception.php 'adding these subclasses in an attempt to stop logged errors in prod - r95683'
* 18:54 RoanKattouw: Re-enabling Apache core dumps on srv162, this time with suid_dumpable enabled
* 18:18 Jeff_Green: taking payments3 out of production to test a mediawiki config change
* 18:03 logmsgbot: neilk synchronizing Wikimedia installation... Revision: 95681: 
* 15:36 mutante: srv207 -stop NTP/ntpdate dobson.wm/start NTP (fixes Nagios CRIT), start apache | sq35 -start squid
* 15:15 mutante: srv278, srv281 - started apache
* 15:09 mutante: srv207 - was unusable due to overload/freeze - powercycle, dist-upgrade/kernel, puppet run, reboot (log entry from July 31st (RAID issues) not confirmed)
* 14:57 mutante: srv266 started apache
* 14:51 mutante: srv281 - power up, dist-upgrade/kernel, puppet run, reboot (note: see 'srv281' in comments of RT#22 and Server_admin_log)
* 14:39 mutante: srv278 - power up, dist-upgrade/kernel, puppet run, reboot
* 14:28 mutante: srv266 - power up, dist-upgrade/kernel, puppet run, reboot
* 14:08 mutante: srv217 - power up, dist-upgrade/kernel, puppet run, reboot
* 14:06 mutante: nagios-wm - ok, just needed restart to talk again
* 13:54 mutante: srv188 - power up, dist-upgrade/kernel, puppet run, reboot
* 13:45 mutante: nagios-wm is on channel but does not speak!? (not ignoring it)
* 13:45 mutante: srv174 - confirmed hardware failure, new RT#1379, acked in Nagios
* 13:29 mutante: srv156 - power up, dist-upgrade/kernel, puppet run, reboot
* 12:57 RoanKattouw: Reverted all of my changes to srv162 and started puppet again. Need to do more to get a core dump, will do that later
* 09:26 RoanKattouw: ... on srv162
* 09:26 RoanKattouw: Changed the core dump directory to /a/tmp/apachecore because the root partition doesn't have much free space but /a does
* 09:23 RoanKattouw: Set up Apache core dumping on srv162 *correctly* by uncommenting CoreDumpDirectory /tmp/apache-core locally in /etc/apache2/wmf/main.conf
* 09:03 RoanKattouw: Changed ownership of /mnt/upload6/math/8/0/0/800618943025315f869e4e1f09471012.png from root:root to apache:apache, permissions errors were causing PHP warnings
* 07:39 RoanKattouw: Reverted my changes on srv163 and started puppet
* 07:38 RoanKattouw: Stopped puppet on srv162, set Apache's cwd to /a/tmp/apachecore in /etc/apache2/envvars , and set ulimit -c 1000000 in /etc/default/apache2
* 07:34 RoanKattouw: Moving my core dump for segfault debugging test to srv162 instead of srv163, for disk space reasons
* 07:32 RoanKattouw: Stopped puppet on srv163 to prevent it from reverting my hacks
* 07:26 RoanKattouw: Restarting Apache on srv163 so these changes take effect
* 07:26 RoanKattouw: Enabled core dumps for Apache on srv163 by editing /etc/default/apache2
* 07:19 RoanKattouw: Changing Apache's cwd on srv163 by editing /etc/apache2/envvars
* 02:18 logmsgbot: LocalisationUpdate completed (1.17) at Mon Aug 29 02:20:17 UTC 2011
== August 28 ==
* 17:55 logmsgbot: ariel synchronized php-1.17/includes/upload/UploadFromStash.php 'fix fatal Call to a member function getId() on a non-object'
* 02:23 logmsgbot: LocalisationUpdate completed (1.17) at Sun Aug 28 02:26:07 UTC 2011
== August 27 ==
* 02:21 logmsgbot: LocalisationUpdate completed (1.17) at Sat Aug 27 02:23:37 UTC 2011
== August 26 ==
* 21:06 mutante: amssq48 - power back up, clean squid, dist-upgrade
* 16:37 robh: updating text-settings to move sq36 into the squid api cluster. puppet updated already for the same, and pybal updated to remove sq36 frontend from normal text service
* 02:27 logmsgbot: LocalisationUpdate completed (1.17) at Fri Aug 26 02:29:15 UTC 2011
* 00:11 robh: change reverted, nothing bad, but undesired result. hooper back to normal
* 00:09 robh: hooper apache config change for https redirection on etherpad
* 00:09 robh: i meant to paste the rt link
* 00:09 robh: testing something in hooper apache config, should result in nothing noticeable to users, unless i did it wrong.
* 00:08 maplebed: changed puppet client run interval from the default (30m) to 2hrs to reduce load on the master.
== August 25 ==
* 23:29 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 95505: 
* 23:20 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 95505: 
* 23:10 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 
* 23:09 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 
* 23:08 logmsgbot: awjrichards synchronized php/extensions/LandingCheck/LandingCheck.i18n.php 'r95542'
* 23:08 logmsgbot: awjrichards synchronized php/extensions/LandingCheck/SpecialLandingCheck.php 'r95542'
* 22:48 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 95505: 
* 22:47 mark: Rebooting lvs1002, lvs1003, lvs1005, lvs1006
* 22:43 robh: gallium deployed for continuous integration testing per RT#1204. Requires further development input for final system configuration.
* 22:29 mark: Deployed cr2-pmtpa as backup bootp forwarder and PIM router
* 22:24 mark: Deployed cr2-pmtpa as backup VRRP router on all production subnets
* 22:05 mark: Enabled cr2-pmtpa:irb.105 (subnet virt-hosts) family inet; activated cr2-pmtpa as backup VRRP router for that subnet
* 21:54 logmsgbot: awjrichards synchronized php/extensions/CentralNotice/centralnotice.css 'r95518'
* 21:48 mark: Setup iBGP on cr2-pmtpa to all other AS14907 routers
* 21:46 logmsgbot: awjrichards synchronized php/extensions/CentralNotice/centralnotice.js 
* 21:41 Jeff_Green: taking payments4 out of the pool for testing
* 21:13 robh: installing gallium for CI testing.
* 19:52 logmsgbot: catrope synchronized php/includes/api/ApiQuerySiteinfo.php 'Fix PHP warning'
* 19:49 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 95505: 
* 19:49 RoanKattouw: Run scap to deploy HTTPS / prot rel fixes
* 19:22 awjr: deployed contrib log auditing fixes on Grosley
* 19:20 logmsgbot: catrope synchronized wmf-config/secure.php 'Factor out URL fixing into fixupUrl, add hook functions for GetCanonicalURL and IRCLineURL'
* 19:13 logmsgbot: catrope synchronized wmf-config/secure.php 'rm debugging'
* 19:12 logmsgbot: catrope synchronized wmf-config/secure.php 'Fix setting of $wgInternalServer for protocol-relative URL wikis (was broken since introduction of prot rel) and set $wgCanonicalServer'
* 19:05 logmsgbot: catrope synchronized wmf-config/secure.php 'Debugging'
* 18:54 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Unbreak wmgArticleFeedbackBlacklistCategories, thanks Krinkle'
* 18:54 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Add wmgArticleFeedbackBlacklistCategories line for enwiki'
* 17:12 Ryan_Lane: pushed out workaround for CVE-2011-3192
* 14:43 logmsgbot: reedy synchronized php-1.17/resources/startup.js 'touch'
* 14:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30418 - Enable ArticleFeedback on Hungarian Wikipedia'
* 14:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30523 - Portal namespace for id.wikisource'
* 13:11 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Disable WikimediaIncubator on testwiki, is causing issues'
* 10:47 logmsgbot: catrope synchronized php/languages/messages/MessagesPi.php 'r95472'
* 02:20 logmsgbot: LocalisationUpdate completed (1.17) at Thu Aug 25 02:22:40 UTC 2011
* 02:02 mutante: sq77,sq81,sq85 - power back up, squid clean, dist-upgrade
* 01:46 mutante: sq52,sq54 - power back up, squid clean, dist-upgrade
* 00:10 robh: manually pushed ntpdate on dataset1 cuz its ugly red in nagios.
== August 24 ==
* 19:32 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 
* 19:24 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Re-enable WikimediaIncubator on incubatorwiki'
* 19:07 robh: sq34 unresponsivle to serial console, rebooted
* 19:04 robh: sq34 offline, no rt ticket or admin log entry for a reason to be offline, investigating
* 19:03 logmsgbot: catrope synchronized php/extensions/WikimediaIncubator/IncubatorTest.php 'r95416'
* 18:37 logmsgbot: catrope synchronized php/extensions/WikimediaMessages/WikimediaMessages.php 'r95411, right path this time'
* 18:37 apergos: doing testing of some new xml dump code on snapshot4 on root, it's writing locally so don't worry about jumps in local disk usage
* 18:36 logmsgbot: catrope synchronized php/includes/EditPage.php 'r95411'
* 18:28 Ryan_Lane: powercycling virt3
* 17:47 Ryan_Lane: disabling access_log on srv193
* 17:41 Ryan_Lane: temporarily enabled the access log on srv193
* 17:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Tidying up wmgUseAbuseFilter'
* 17:07 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Setting wmgUseAbuseFilter to true for all wikis'
* 17:06 Reedy: About to enable AbuseFilter accross all wikis
* 16:38 maplebed: running a puppet experiment - turning off the puppet client on spence from now until 2pm - nagios operation will remain unaffected
* 16:24 maplebed: added roan's key to root's authorized keys
* 15:58 robh: rebooting emery, appears to be hung in boot, but cannot see due to serial console not retaining a backlog when not connected
* 15:37 mark: Reenabled cr1-sdtpa:xe-0/0/0
* 14:53 mark: Disabled cr1-sdtpa:xe-0/0/0 due to port errors
* 14:31 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30538 - Change Cantonese Wikipedia (zh-yue.wikipedia) autoconfirmed threshold'
* 09:38 logmsgbot: catrope synchronized php/extensions/MobileFrontend/MobileFrontend.php 'r95388'
* 02:19 logmsgbot: LocalisationUpdate completed (1.17) at Wed Aug 24 02:21:20 UTC 2011
== August 23 ==
* 23:49 apergos: limited downloads on dataset2 to 2 simultaneous connections per ip, except our hosts
* 23:12 apergos: forgot to log earlier, limited bandwidth in lighttpd on dataset2 per download except for our hosts
* 21:48 Ryan_Lane1: moved catrope from mortals to roots
* 21:12 notpeter: upgrading squid and squid-frontend on sq86
* 20:10 notpeter: upgrading squid and squid-frontend on sq85
* 19:56 logmsgbot: demon synchronized php/includes/specials/SpecialEmailuser.php 'r95328'
* 18:22 notpeter: upgrading squid and squid-frontend on sq83
* 18:03 robh: wiped old binlogs off db9 older than 20 days
* 17:16 maplebed: installed a new /etc/sudoers on db36 to placate nagios
* 16:36 apergos: added webzilla.com hosts to urls denied on dataset2, bulk garbage requests and using a pile of bandwidth
* 16:27 robh: reviewed paniclog on lily and mchenry due to non-zero size. cleared logs to stop email alerts
* 10:58 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable WikimediaIncubator on testwiki for debugging'
* 10:51 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Disable WikimediaIncubator again'
* 10:45 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Comment out $wmincClosedWikis'
* 10:38 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Re-enable WikimediaIncubator extension so I can do some profiling'
* 10:37 logmsgbot: catrope synchronized wmf-config/StartProfiler.php 'Add profiling group for debugging incubatorwiki slowness, triggered by a request parameter'
* 10:30 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Temporarily disable WikimediaIncubator extension due to slowness on incubatorwiki'
* 09:15 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 95304: 
* 09:15 RoanKattouw: Running scap to undo the accidental deployment of yesterday's failed UploadWizard deployment
* 02:19 logmsgbot: LocalisationUpdate completed (1.17) at Tue Aug 23 02:21:27 UTC 2011
* 00:28 ^demon: ran refresh-dblist and sync-dblist
* 00:21 ^demon: fixed refresh-dblist since wmf-config was moved
== August 22 ==
* 22:39 RoanKattouw: Added UploadWizard tables to testwiki, for real this time
* 22:32 preilly: pushing fix for disable link
* 22:32 logmsgbot: preilly synchronized php/extensions/MobileFrontend/views/information/disable.html.php 'issue with en always being the target of the form'
* 22:21 preilly: remove content type header for now from XHTML view on Mobile Frontend
* 22:21 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'remove content type header for now'
* 22:07 notpeter: upgrading squid and squid-frontend on sq55
* 21:45 notpeter: upgrading squid and squid-frontend on sq52
* 20:59 logmsgbot: catrope synchronized php/extensions/TitleBlacklist/api/ApiQueryTitleBlacklist.php 'r95249'
* 20:30 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 95237: 
* 20:30 RoanKattouw: Deploy new TitleBlacklist code (trunk version) site-wide
* 20:22 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php 'Bug 30507 - Please lock flaggedrevs.labs.wikimedia.org'
* 20:19 logmsgbot: jeluf synchronized closed.dblist 
* 20:17 notpeter: restarting torrus. again....
* 20:13 RoanKattouw: ..or not. The SQL is not in 1.17wmf1 yet, will have to wait a few minutes
* 20:09 RoanKattouw: Creating tables for UploadWizard on testwiki and commonswiki
* 20:09 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php 'Bug 30509 - Closure of Asturianu Wikibooks'
* 20:08 logmsgbot: jeluf synchronized closed.dblist 
* 20:08 logmsgbot: jeluf synchronized closed.dblist 
* 20:00 notpeter: restarting torrus
* 19:33 logmsgbot: catrope synchronized php/extensions/WikimediaIncubator/IncubatorTest.php 'r95234'
* 19:08 notpeter: upgrading squid and squid-frontend on sq50
* 18:47 RoanKattouw: Fixed /h/w/bin/scap to update ExtensionMessages-1.17.php instead of ExtensionMessages.php 
* 18:44 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.17.php 'Sync the right file'
* 18:37 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Set $mincClosedWikis for Incubator extension per SPQRobin'
* 18:35 logmsgbot: catrope synchronized wmf-config/ExtensionMessages.php 'Fix i18n issue with WikimediaIncubator, hopefully'
* 18:21 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 95216: 
* 18:21 RoanKattouw: Running scap to deploy WikimediaIncubator bugfixes
* 18:06 Jeff_Green: added fundraising-analytics.wikimedia.org cname for grosley for RT #854
* 17:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable HTML5 on mediawikiwiki. Feel free to revert if it breaks too much'
* 16:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable wikilove on commons per http://commons.wikimedia.org/wiki/Commons:Village_pump#WikiLove'
* 15:23 logmsgbot: catrope synchronized php/extensions/WikimediaMessages/WikimediaGrammarForms.php 'r95209'
* 13:03 Tim: set up ufw on ersch and alsted
* 11:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Turn wikilove on by default on mlwiki'
* 11:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'mlwikipedia to mlwiki'
* 11:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30473 - Please create the rollback group at es.wikibooks'
* 11:35 logmsgbot: reedy synchronized php-1.17/resources/startup.js 'touch startup for good measure'
* 11:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30500 - Enable WikiLove Extension on Malayalam Wikipedia'
* 11:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30491 - Enable WikiLove on Swedish Chapter Wiki'
* 07:12 Tim: on ersch and alsted: set up a temporary firewall to block external access to miscellaneous swift services such as the object servers
* 06:41 Tim: on alsted and ersch: adjusted rsyncd.conf to deny access from outside the cluster and restarted rsync
* 06:22 Tim: on ersch: reconfigured MW to send its debug log to a place which is not on the public web, and did a chmod 000 in case the config is reverted
* 06:18 Tim: on ersch apache2.conf: deny access to vim editor backup files
* 02:21 logmsgbot: LocalisationUpdate completed (1.17) at Mon Aug 22 02:23:31 UTC 2011
== August 21 ==
* 18:41 logmsgbot: reedy synchronized php-1.17/extensions/OAI/OAIRepo_body.php 'r95170 for bug 26304'
* 18:36 rainman_: restarted lucene on search7 which somehow got hung
* 02:17 logmsgbot: LocalisationUpdate completed (1.17) at Sun Aug 21 02:20:08 UTC 2011
== August 20 ==
* 11:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29834 - Reconfigured AF for hiwiki'
* 02:21 logmsgbot: LocalisationUpdate completed (1.17) at Sat Aug 20 02:23:29 UTC 2011
== August 19 ==
* 23:43 logmsgbot: asher synchronized wmf-config/db.php 'adding srv155 back to ext store'
* 20:00 logmsgbot: laner synchronized wmf-config/db.php 'Repooling db12'
* 19:58 Ryan_Lane: pushing squid config
* 19:56 Ryan_Lane: banning a bad ip in squid list
* 19:49 RoanKattouw: Ryan uncommented srv155 and srv156 from the ES list at 19:11 today
* 19:28 logmsgbot: laner synchronized wmf-config/db.php 'Removing db12 temporarily to let it catch up on replication'
* 19:16 logmsgbot: laner synchronized wmf-config/db.php 'srv156 is also down, removing it from db.php'
* 19:09 logmsgbot: laner synchronized wmf-config/db.php 'removing srv155 from external store list'
* 18:47 logmsgbot: reedy synchronized wmf-deployment/languages/Names.php 'r94830 r95027'
* 18:46 logmsgbot: reedy synchronized wmf-deployment/languages/messages/MessagesQug.php 'r94830'
* 17:39 mutante: sq53,sq58 - power back up, squid clean, dist-upgrade
* 17:35 notpeter: adding srv155 back into pool for testing
* 17:10 notpeter: upgrading squid and squid-frontend on sq49
* 16:08 logmsgbot: reedy synchronized php-1.17/resources/startup.js 'Touch startup.js'
* 16:07 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable ClickTracking on huwiki as dependancy of AF'
* 14:08 RobH: confirmed serial on scs-c1-pmtpa working for asw-d2-pmtpa, ps1-d2-pmtpa, & cr2-pmtpa
* 13:16 logmsgbot: reedy synchronized php-1.17/resources/startup.js 'Touch'
* 13:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Fix'
* 13:11 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Revert'
* 13:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Revert'
* 13:10 Reedy: Fail summary. Bug 29834 - Enable Extension:ArticleFeedback on Hindi Wikipedia. Also, add wmg global for $wgArticleFeedbackBlacklistCategories
* 13:10 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 29834 - Enable Extension:ArticleFeedback on Hindi Wikipedia. Also, add wmg Global for Bug 29834 - Enable Extension:ArticleFeedback on Hindi Wikipedia'
* 13:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 29834 - Enable Extension:ArticleFeedback on Hindi Wikipedia. Also, add wmg Global for Bug 29834 - Enable Extension:ArticleFeedback on Hindi Wikipedia'
* 11:07 Andrew: (in a screen on hume)
* 11:07 Andrew: Sending 753994 emails to qualified voters in the referendum.
* 10:57 mark: Converted routing subinterfaces on cr2-pmtpa:ae0 to IRB and configured bridge-domains
* 02:19 logmsgbot: LocalisationUpdate completed (1.17) at Fri Aug 19 02:21:42 UTC 2011
* 01:17 Jeff_Green: flipped $wgContributionReportingDBserver back to db10
* 00:39 logmsgbot: reedy synchronized php-1.17/resources/startup.js 'Touching startup.js'
* 00:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29834 Enable AF on hiwiki - Enabling ClickTracking first'
== August 18 ==
* 23:24 notpeter: adding srv155 back to the pool after upgrade to 10.04 and testing
* 22:43 notpeter: upgrading squid and squid-frontend on sq48
* 22:01 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 94793: 
* 22:00 RoanKattouw: Running scap to deploy WikiLove changes
* 21:43 logmsgbot: catrope synchronized php/resources/jquery.ui/themes/vector/jquery.ui.button.css 'r94947, JUI CSS fixes for WikiLove'
* 21:03 Jeff_Green: db10 is re-cloned with innodb-file-per-table enabled and a significantly smaller db footprint, /a is down from 94% to 88% use
* 20:34 RobH: serial connection to cr2-pmtpa tested working. port 13 on scs-c1-pmtpa
* 16:21 mark: Allocated IPs for pmtpa row D PDUs and access switches
* 11:15 Jeff_Green: stopping mysql on db10 for re-clone
* 11:04 Jeff_Green: swinging $wgContributionReportingDBserver from db10.pmtpa.wmnet to db1008.eqiad.wmnet
* 02:22 logmsgbot: LocalisationUpdate completed (1.17) at Thu Aug 18 02:24:15 UTC 2011
== August 17 ==
* 23:59 mutante: deleted srv173 manually from db9 hosts and resources so Nagios would start again
* 22:53 logmsgbot: reedy synchronized php-1.17/resources/startup.js 'Touch startup'
* 22:31 preilly: pushing updates to Mobile Frontend for opt-in
* 22:31 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 94793: 
* 22:01 binasher: deploy of frontend.conf to all squids
* 21:43 Reedy: That was Setting $wgCategoryTreeDisableCache to false. Stupid escaping
* 21:42 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Setting to false'
* 21:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable poolcounter for commmonswiki'
* 21:31 maplebed: extended logic in mysql.pp to always set variables used in my.cnf.erb
* 21:22 Reedy: Added PoolCounterSettings.php to noc
* 21:04 Reedy: Allowed StartProfiler.php to be shown on noc
* 21:02 mutante: sq57 - power back up, clean squid, dist-upgrade
* 20:53 notpeter: upgrading squid and squid-frontend on sq47
* 20:50 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 94793: 
* 20:49 RoanKattouw: Running scap to deploy Neil's UploadWizard/UploadStash changes
* 20:43 RoanKattouw: Neil added uploadstash table to all wikis
* 20:42 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 29983 - Install ArticleFeedback on Portuguese Wikibooks (ptwikibooks). Enabling ClickTracking'
* 20:37 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 29983 - Install ArticleFeedback on Portuguese Wikibooks (ptwikibooks). Enabling ClickTracking'
* 20:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable PoolCounter for Commons'
* 20:24 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 29983 - Install ArticleFeedback on Portuguese Wikibooks (ptwikibooks)'
* 20:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 29983 - Install ArticleFeedback on Portuguese Wikibooks (ptwikibooks)'
* 20:14 Reedy: Disabled ReaderFeedback on ptwikibooks, part of bug 29983
* 19:53 Reedy: Creating ArticleFeedback tables on huwiki
* 19:53 Reedy: Creating ArticleFeedback tables on ptwikibooks
* 19:52 Reedy: Creating ArticleFeedback tables on hiwiki
* 19:16 logmsgbot: catrope synchronized php/extensions/ArticleFeedback/SpecialArticleFeedback.php 'r94792'
* 19:11 binasher: repooled srv264
* 19:07 logmsgbot: reedy synchronized php-1.17/extensions/CodeReview/backend/CodeRevision.php 'r94793'
* 19:07 logmsgbot: reedy synchronized php-1.17/extensions/CodeReview/CodeReview.i18n.php 'r94793'
* 18:52 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'bug 30417 setting use wikilove default to true for huwiki'
* 18:43 logmsgbot: reedy synchronized php-1.17/includes/parser/ParserCache.php 'Revert debugging here too'
* 18:41 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Disabling parsercache debug logging. Nothing of interest'
* 18:32 binasher: pulling srv264 from lvs for testing
* 18:27 logmsgbot: reedy synchronized php-1.17/includes/parser/ParserCache.php 'Add in a not saved in parser cache message if appropriate'
* 18:20 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Log parsercache log groups to file'
* 18:18 logmsgbot: reedy synchronized php-1.17/includes/parser/ParserCache.php 'Hack in some logging for when parser output is uncacheable'
* 14:26 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Keep $wgInternalServer on http when visiting wikis that do not have a protocol-relative $wgServer over https. For bug 29925'
* 14:22 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Remove debugging code'
* 14:15 logmsgbot: catrope synchronized php/includes/RecentChange.php 'r94755, fix prot rel links in IRC feeds'
* 13:27 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'More debugging'
* 13:24 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Fix stupid mistake in debugging hack'
* 13:23 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Add temp debugging hack for $wgInternalServer'
* 13:14 logmsgbot: catrope synchronized wmf-config/secure.php 'Remove debugging hack'
* 13:12 logmsgbot: catrope synchronized wmf-config/secure.php 'tweak hack'
* 13:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30417 Enable WikiLove on huwiki'
* 13:10 logmsgbot: catrope synchronized wmf-config/secure.php 'Debugging code so I can figure out what $wgInternalServer is on commons via secure'
* 12:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30362 Enable newusermessage and wikilove on zhwiki'
* 12:22 mark: restarted etherpad on hooper
* 02:27 logmsgbot: LocalisationUpdate completed (1.17) at Wed Aug 17 02:29:32 UTC 2011
== August 16 ==
* 22:31 notpeter: pulling srv155 from pybal so I can take it down and upgrade to 10.04
* 22:10 notpeter: upgrading squid and squid-frontend on sq45 and sq46
* 21:55 mutante: resolved RT 1274 - we now get mail incl. the diff and comment after every svn commit on sockpuppet
* 21:34 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'For bug 26304 set oaiChunkSize to 40'
* 21:32 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesEn.php 'Push r94698'
* 18:48 mutante: sq61,sq73 - power back up, clean squid, dist-upgrade, rebootb
* 18:12 mutante: sq40,sq56 - power back up, clean squid, dist-upgrade, reboot
* 18:10 preilly: pushing updates to Mobile Frontend extension to allow for search page to work
* 18:10 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 94562: 
* 18:08 binasher: deploying frontend.conf to all squids - change to mobile acl to support opt-in trial
* 17:53 mutante: amssq42,amssq61 - power back up, clean squid, dist-upgrade, reboot
* 17:38 mutante: added new class svn::server:notify which installed libsvn-notify-perl on sockpuppet (to get svn commit mails in the future)
* 17:29 logmsgbot: demon synchronized php/extensions/CodeReview/backend/CodeRevision.php 'r94664'
* 16:23 mark: Increased prefix limit on BGP session between br1-knams and csw2-esams from 10 to 50 prefixes
* 16:00 RobH: lvs1004, lvs1005, lvs1006 both network patches attached
* 15:57 mark: Added Nagios purge script on 'reload' action in its init script
* 15:44 mark: Setup switchports for cross rack row connections from lvs1001-1006 on asw-a-eqiad and asw-b-eqiad
* 15:40 Jeff_Green: package upgrade on payments4, no reboot
* 14:21 mark: Noticed the mobile LVS service had a depool-threshold set at '1.5', preventing it from depooling any hosts, ever. Corrected it by setting it to .6
* 14:07 mark: Setup 'new-mobile' LVS service on lvs1001 and lvs1004
* 14:07 mark: Setup eqiad LVS cluster (classes 'high-traffic' and 'testing') consisting of lvs1001 (A4) and lvs1004 (B4); still awaiting cross-row connections
* 13:54 mark: Increased bgp group PyBal multihop ttl from 1 to 2 on all core routers
* 13:54 mark: Configured PyBal bgp group and policy statements on cr1/cr2-eqiad
* 13:14 RobH: reviewed exim paniclog on lily and williams, both from a refusal in spam acl, over 24 hours ago, and no issues since. cleared each paniclog back to 0
* 11:16 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'bug 30392: Add Wikipedia_talk and MediaWiki_talk aliases on zh-yuewiki'
* 10:42 mark: Corrected DNS for lvs1004-1006
* 10:03 mark: Setup switchports for lvs1001-1006
* 02:17 logmsgbot: LocalisationUpdate completed (1.17) at Tue Aug 16 02:19:43 UTC 2011
* 00:37 mutante: gmond.conf was an empty file on search12. copied it in place and restarted gmond
== August 15 ==
* 23:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30320 Enable AbuseFilter for cswiki'
* 23:34 Reedy: Testwiki job queue is now empty
* 22:57 preilly: pushing Mobile Frontend updates for mobile friendly main page
* 22:57 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 94562: 
* 21:58 logmsgbot: reedy synchronized wmf-config/codereview.php 'Auto defer semantic tags'
* 21:35 notpeter: upgrading squid and squid-frontend on sq43 and sq44
* 21:15 notpeter: upgrading squid and squid-frontend on sq41 and sq42
* 21:14 RoanKattouw: Fixed scap so mergeExtensionMessages didn't fail anymore. Was broken due to het deploy changes
* 21:14 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 94562: 
* 21:12 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 94562: 
* 21:12 RoanKattouw: Ended up moving extensions/SecurePoll/auth-api.php.save to /home/catrope . I could do that because I have write access to the parent directory, even though I don't have read access to the file
* 21:07 RoanKattouw: Permissions for extensions/SecurePoll/auth-api.php.save are messed up, breaking scap. Running set-group-write in an attempt to fix it
* 21:01 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 94562: 
* 21:01 RoanKattouw: Running scap to push my latest 1.17wmf1 commits plus Patrick's MobileFrontend changes site-wide
* 20:31 RoanKattouw: Running svn up on fenari to deploy my recent 1.17wmf1 revisions to testwiki
* 19:19 Ryan_Lane: assigning svc ips for bits, upload, and text in 10.2.2.x range for eqiad
* 18:25 notpeter: upgrading squid and squid-frontend on knsq20 and knsq21 and knsq22
* 18:10 notpeter: upgrading squid and squid-frontend on knsq18 and knsq19
* 18:04 preilly: pushing mobile fronted changes for image enable disable
* 18:04 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 94253: 
* 17:53 mutante: added libnet-stomp-perl to puppet to be installed on spence
* 17:49 notpeter: upgrading squid and squid-frontend on knsq16 and knsq17
* 17:04 RobH: updating dns with lvs1001-1006 mgmt info
* 16:58 Ryan_Lane: pushing varnish changes to allow XFF from HTTPS cluster
* 15:39 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/auth-api.php 
* 15:38 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/auth-api.php 
* 15:36 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/auth-api.php 
* 15:34 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/auth-api.php 
* 15:25 RobH: updating dns for lvs1001-lvs1006
* 15:19 RobH: restarted dns on ns2
* 15:04 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/includes/ballots/Ballot.php 'Deploy r94343'
* 15:04 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/SecurePoll.php 'Deploy r94343'
* 15:03 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/includes/ballots/RadioRangeCommentBallot.php 'Deploy r94343'
* 14:37 mark: Enabled NAT on the pmtpa management LAN
* 12:10 mark: Setup ongoing rsync of ms7 images to ms1002
* 12:03 mark: Setup rsyncd on ms7
* 02:17 logmsgbot: LocalisationUpdate completed (1.17) at Mon Aug 15 02:19:19 UTC 2011
== August 14 ==
* 17:49 JeLuF: rebooted srv183
* 17:30 JeLuF: stopped apache and jobrunner on srv183, logfiles indicate memory problems
* 17:18 logmsgbot: jeluf synchronized wmf-config/mc.php 'replace srv183 by srv229'
* 13:22 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Make $wgServer protocol-relative on wikimania2005wiki, for consistency with the other HTTPS experiment wikis'
* 10:12 JeLuF: rebooting sq40 using echo b > /proc/sysrq-trigger, syslog shows lots of SCSI errors
* 09:10 apergos: stopped and restarted lsearchd on search7. had to shoot the old one, it wouldn't die
* 02:19 logmsgbot: LocalisationUpdate completed (1.17) at Sun Aug 14 02:21:23 UTC 2011
== August 13 ==
* 21:23 RoanKattouw: sq40 keeps flapping in Nagios all the time
* 10:39 RoanKattouw: ...on enwiki
* 10:39 RoanKattouw: Running a hacked-up version of extensions/ArticleFeedback/populateAFRevisions.php to fix bug 30227 (data corruption in ArticleFeedback tables), using a list of affected revisions produced on the toolserver
* 02:16 logmsgbot: LocalisationUpdate completed (1.17) at Sat Aug 13 02:18:42 UTC 2011
* 00:22 notpeter: upgrading squid and squid-frontend on knsq12 and knsq15
* 00:19 notpeter: upgrading squid and squid-frontend on knsq10 and knsq11
== August 12 ==
* 23:15 preilly: pushing fix for mobile WML view and disable caching on cookie set page
* 23:14 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 94253: 
* 22:40 notpeter: upgrading squid and squid-frontend on knsq8 and knsq9
* 21:54 notpeter: upgrading squid and squid-frontend on amssq59 and amssq62
* 21:46 preilly: updating the mobile frontend extension on production
* 21:46 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 94253: 
* 21:31 maplebed: localization update permissions changes complete and tested.
* 21:29 logmsgbot: LocalisationUpdate completed (1.17) at Fri Aug 12 21:31:45 UTC 2011
* 21:26 AaronSchulz: Running LU to confirm permission fix
* 21:25 notpeter: upgrading squid and squid-frontend on amssq56 and amssq58
* 21:17 logmsgbot: LocalisationUpdate completed (1.17) at Fri Aug 12 21:19:39 UTC 2011
* 21:10 maplebed: changing ownership of /usr/local/apache/common-local/php-1.17/cache/l10n from nagios to mwdeploy on all affected srv hosts
* 21:03 binasher: deploying new squid frontend.conf - bypass mobile redirector in case of trial opt-in/out pages
* 20:53 notpeter: upgrading squid and squid-frontend on amssq51 and amssq52
* 20:52 logmsgbot: LocalisationUpdate completed (1.17) at Fri Aug 12 20:54:15 UTC 2011
* 18:44 maplebed: made ariley an account on rt to submit tickets for email list maintenance
* 18:24 JeLuF: DNS: added bugs.mediawiki.org as alias to text.wikimedia.org 
* 17:25 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '30268 - Point eowp logo to Wiki.png'
* 16:25 logmsgbot: mark synchronized wmf-config/CommonSettings.php 'Raise multicast ttl from 2 to 8'
* 16:10 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Set $wgVaryXFPForAPI = true for HTTPS experiment wikis. This splits the API Squid cache between HTTP and HTTPS, fixing cache pollution issues'
* 15:08 mark: Turning off ethernet hw offloading GRO on all lvs servers with Puppet
* 14:30 mark: Turned off all forms of hardware segmentation on lvs4, fixing the slow upload problem
* 14:29 mark: Turned tcp segment offloading back on on sq51..86
* 13:57 mark: Manually turned off TCP segmentation offloading on sq51..86
* 12:59 logmsgbot: catrope synchronized wmf-config/StartProfiler.php 'Remove upload profiling, hasn't produced any useful data'
* 12:52 logmsgbot: catrope synchronized wmf-config/StartProfiler.php 'Profile uploads on officewiki separately, I wanna try something'
* 11:39 apergos: reran ppuppet by hand on spence, sq32 entries in nagios conf files were not recreated, restarted nagios, seems to be running
* 11:11 apergos: er... because sq32 is in the decommissioned list but the script to purge resources from nagios is broken right now, which means nagios fails to start
* 11:10 apergos: purged sq32 resources and host references from puppet db manually on db9, and from nagios conf files on spence. will run puppet manually on spence shortly
* 10:52 Andrew: (screen on hume)
* 10:52 Andrew: Running populatePifEditCount.php on all wikis
* 10:45 Andrew: Adding pif_edits table to all wikis for personal image filter voter list
* 10:39 mark: Enabled cr1-sdtpa:xe-0/0/2; a cross connect has been ordered, expect Nagios to complain
* 10:24 apergos: uncommented the monitor_group line in varnish.pp which defines the cache_mobile_eqiad group in puppet (thanks ma rk), will run puppet shortly on spence
* 08:44 apergos: revert change to site.pp, try applying to spence
* 08:29 apergos: doing repeated manual runs of puppet on spence til we catch up to current config (it is quite out of date)
* 07:21 apergos: nagios was failing to start because of unknown host group cache_mobile_eqiad in /etc/nagios/puppet_hosts.cfg; commented out line $nagios_group = "cache_mobile_${site}" in site.pp, waiting for puppet run to complete on spence
* 02:15 logmsgbot: LocalisationUpdate completed (1.17) at Fri Aug 12 02:17:38 UTC 2011
== August 11 ==
* 21:24 JeLuF: added 'ttf-ubuntu-font-family' to the list of required packages for image scalers in puppet (bug 30288)
* 21:07 JeLuF: virt2 root filesystem has switched to read-only due to a disk failure
* 21:03 logmsgbot: LocalisationUpdate completed (1.17) at Thu Aug 11 21:06:09 UTC 2011
* 20:24 binasher: re-pooling mobile2
* 20:16 logmsgbot: LocalisationUpdate failed
* 18:57 binasher: depooling mobile2 from lvs for mobile extension opt in proxy conf testing
* 18:19 preilly: pushing new mobile frontend changes to production 
* 18:19 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 94253: 
* 18:12 mark: Temporarily serving li.wikipedia.org from srv153 (bypassing LVS) as backend on the text squids
* 18:02 mark: Test complete, change reverted
* 17:57 mark: Temporarily moved squids->apaches LVS traffic from lvs4 to lvs3 for testing
* 16:48 Ryan_Lane: upping the nginx upload size too 100m for the https and ipv6 cluster
* 16:06 Reedy: SVN and related services should be ok now. High HTTP load causing OOM
* 15:51 Reedy: SVN may be unavailable due to issues with Formey
* 15:32 logmsgbot: catrope synchronized php/includes/filerepo/LocalFile.php 'r94252 by Chad - Try wrapping ss_images update in a transaction'
* 15:17 RoanKattouw: All file uploads were returning HTTP 500 errors between 15:07 and 15:16 UTC, my apologies. It's fixed now
* 15:16 logmsgbot: catrope synchronized wmf-config/StartProfiler.php 'Fix it for real this time'
* 15:15 logmsgbot: catrope synchronized wmf-config/StartProfiler.php 'Unbreak uploads, oops'
* 15:07 logmsgbot: catrope synchronized wmf-config/StartProfiler.php 'Make upload profiling 1:1 instead of 1:50'
* 15:01 mark: Set maximum TCP window size on nas1-a
* 15:00 RoanKattouw: ...and that worked. Lesson of the day: it seems you can't use dashes in your profile IDs
* 14:59 logmsgbot: catrope synchronized wmf-config/StartProfiler.php 'Remove dashes from profile IDs just because I'm paranoid'
* 14:56 logmsgbot: catrope synchronized wmf-config/StartProfiler.php 'Add profiling groups upload-commons and upload-other, based on count($_FILES)'
* 14:10 mark: Setup semi-sync snapmirror replication from nas1001-a:images to nas1-a:images, started initial transfer
* 13:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 8886 - Install DynamicPageList extension for Vietnamese Wiktionary'
* 13:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 18390 Set [bureaucrat][] = sysop on enwiki'
* 13:34 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30307 Allow bureaucrats to remove sysop flag on Russian Wikipedia'
* 12:45 mark: Migrated nas1-a and nas1-b to new 64 bit root volumes
* 09:45 logmsgbot: catrope synchronized wmf-config/checkers.php 'Remove broken profiling in wfACLBlocks. Not needed because everything in $wgExtensionFunctions is automatically profiled anyway'
* 09:17 mutante: sq75 - power cycle, squid clean
* 03:00 logmsgbot: LocalisationUpdate completed (1.17) at Thu Aug 11 03:02:09 UTC 2011
* 02:32 logmsgbot: LocalisationUpdate completed (1.17) at Thu Aug 11 02:34:20 UTC 2011
* 02:31 Reedy: LocalisationUpdate is giving Warning: array_diff_assoc(): Argument #1 is not an array in /home/wikipedia/common/php-1.17/extensions/LocalisationUpdate/LocalisationUpdate.class.php on line 373
* 01:58 logmsgbot: LocalisationUpdate completed (1.17) at Thu Aug 11 02:00:53 UTC 2011
* 01:56 logmsgbot: LocalisationUpdate completed (1.17) at Thu Aug 11 01:58:21 UTC 2011
* 01:46 logmsgbot: LocalisationUpdate completed at Thu Aug 11 01:48:34 UTC 2011
* 01:46 logmsgbot: LocalisationUpdate failed
* 01:30 logmsgbot: LocalisationUpdate failed
* 01:27 logmsgbot: LocalisationUpdate failed
* 01:12 logmsgbot: LocalisationUpdate failed
* 00:42 logmsgbot: demon synchronized wmf-config/StartProfiler.php 'Enabled profiling group for commonswiki'
== August 10 ==
* 23:02 logmsgbot: LocalisationUpdate failed
* 22:59 logmsgbot: LocalisationUpdate failed
* 22:50 aaron: testing/debugging Het Deploy updates to l10 script
* 21:25 preilly: pushing device change back into production 
* 21:25 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 93960: 
* 20:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Revert my change'
* 20:49 preilly: fixing mobile view
* 20:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Temporarily disabling MObileFrontend'
* 20:49 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 93960: 
* 20:48 RoanKattouw: MobileFrontend broke, serving mobile pages to everyone. Fixing
* 20:43 preilly: pushing fix for Mobile Frontend device detection
* 20:43 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 93960: 
* 19:02 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 93960: 
* 19:02 preilly: updating mobile frontend extension on production
* 18:41 notpeter: upgrading squid and squid-frontend on amssq49
* 17:01 mark: Deployed new HTCP loss ganglia monitoring plugin on ganglia aggregator squids in pmtpa
* 16:32 mutante: sq51 & sq62 - dist-upgrade (incl. kernel) / reboot
* 16:01 mutante: sq51,sq62: power cycle, squid clean
* 02:18 logmsgbot: LocalisationUpdate completed at Wed Aug 10 02:21:00 UTC 2011
* 00:12 maplebed: trying out a change to puppet's nagios config generation - moving service checks to one file per host.
== August 9 ==
* 19:01 logmsgbot: catrope synchronized php/extensions/ArticleFeedback/SpecialArticleFeedback.php 'Add missing !'
* 18:33 notpeter: upgrading squid and squid frontend on amssq47 and amssq48
* 17:41 maplebed: added microsecond query duration logging to sockpuppet's apache log format
* 02:25 logmsgbot: LocalisationUpdate completed at Tue Aug 9 02:27:12 UTC 2011
== August 8 ==
* 20:41 binasher: added self to auth portion of nagios cgi.cfg
* 20:35 binasher: powered off knsq30, disabling in nagios
* 19:23 logmsgbot: reedy synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js 'blanking file'
* 18:51 logmsgbot: tfinc synchronized wmf-config/CommonSettings.php 'Disabling WikimediaMobile client side redirect'
* 18:49 logmsgbot: tfinc synchronized wmf-config/InitialiseSettings.php 'Disabling WikimediaMobile client side redirect'
* 18:06 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bug 25957 - Update'
* 17:57 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php 'bug 29911'
* 17:46 logmsgbot: reedy synchronized php-1.17/languages/Names.php 'r94069'
* 17:27 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'fix language code for bug 30045'
* 17:15 Ryan_Lane: RobH is removing the sep11.wikipedia.org subdomain redirect. It didn't post to the SAL, and he can't get onto IRC
* 13:53 mark: Fixed IPv6 router advertisements on cr1-sdtpa
* 11:57 mark: Killed squid on knsq30
* 11:52 mutante: restarted squid on knsq30
* 11:29 mark: Restored routing of pmtpa - esams path
* 11:19 mark: Restarted lsearchd on search7
* 02:18 logmsgbot: LocalisationUpdate completed at Mon Aug 8 02:21:05 UTC 2011
== August 7 ==
* 07:56 mark: Rerouted pmtpa <--> esams traffic in both directions
* 02:22 logmsgbot: LocalisationUpdate completed at Sun Aug 7 02:24:47 UTC 2011
== August 6 ==
* 15:52 mutante: amssq41 - and while we're on it: dist-upgrade incl. kernel and reboot. did not include squid package, that was downgraded on Apr 20
* 15:25 mutante: amssq41 - power back up after it went down, clean squid cache
* 13:07 RobH: puppet run on spence is via screen on root user
* 13:05 RobH: spence is running puppet to update with new nagios info that won't contain the facilities fasle reports. if it fails to work, ~/rob_puppet_services.cfg is on spence
* 13:03 RobH: nagios is being regenerated, i hope this works!
* 12:23 RobH: srv169 depooled, needs work and to be restored to service
* 12:10 logmsgbot: root synchronized wmf-config/mc.php 'replacing srv169 in active mc role due to its constant flapping'
* 06:46 RobH: emptied the /var/log/exim4/paniclog on project2, formey, lily, mchenry, sockpuppet, williams, srv196. All errors were quite old due to various network blips over time.
* 02:19 logmsgbot: LocalisationUpdate completed at Sat Aug 6 02:21:34 UTC 2011
* 01:49 Reedy: srv169 seems to be flapping
* 01:48 Reedy: Ganglia seems brokened "There was an error collecting ganglia data (127.0.0.1:8654): fsockopen error: Connection refused"
== August 5 ==
* 23:50 tfinc: removed knsq30 from LVS esams rotation
* 23:36 Reedy: Even though Asher pulled knsq30 from rotation, it seems to be still serving users
* 21:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'more whitespace'
* 21:33 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'whitespace'
* 21:32 ^demon: fixed permissions on new hook scripts so they're executable
* 21:28 ^demon: deployed new svn pre-commit hook framework
* 21:01 binasher: pulled knsq30 from all frontend squids
* 20:44 logmsgbot: root synchronized wmf-config/db.php 'returning db18 to s7'
* 19:02 apergos: power cycled dataset2, it was unresponsive to pings and form mgmt console
* 18:24 binasher: squid frontend deploy - no mobile redirect for android 3 (tablets)
* 18:15 Reedy: running namespaceDupes on plwikibooks
* 18:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30164 Change namespaces configuration - pl.wikibooks'
* 18:00 logmsgbot: reedy synchronized wmf-config/abusefilter.php 'Tidy up some redundant userrights definitions'
* 17:56 Reedy: That sync was also bug 30207 Enable AbuseFilter on Russian Wikinews
* 17:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 
* 17:55 logmsgbot: reedy synchronized wmf-config/abusefilter.php 'bug 30207 Enable AbuseFilter on Russian Wikinews'
* 17:51 logmsgbot: reedy synchronized wmf-config/abusefilter.php 'Remove old abusefilter-private rights entries'
* 17:48 binasher: deploying squid frontend.conf (commented out nokia mobile acl until a jquery netfront browser compatibility issue is fixed)
* 17:02 maplebed: dns templates in svn and changes to authdns-update tested successfully
* 15:34 mark: Uncommented srv301 from mediawiki_installation node list; killed apache; reran puppet
* 15:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30221 Enable NewUserMessage on ladwiki'
* 15:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'whitespace'
* 15:18 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'whitespace'
* 15:10 logmsgbot: reedy synchronized wmf-config/wgConf.php 'Remove initialise noop call'
* 15:04 logmsgbot: reedy synchronized php-1.17/includes/SquidUpdate.php 'r93971 r93972'
* 14:24 Reedy: Running svn up to get r93971/r93972 on test
* 13:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30247 - Enable Botadmin user group on Hindi Wikipedia'
* 13:29 Ryan_Lane: added timo alias in exim for krinkle
* 13:28 Ryan_Lane: added timo alias in exim for krinkle
* 12:21 preilly: updated the MobileFrontend extension on production 
* 12:13 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 93960: 
* 11:39 logmsgbot: laner synchronized wmf-config/CommonSettings.php 'Setting HTTPS mode when using XFP'
* 10:06 mark: Fixed memcached check on spence
* 09:47 mark: srv301 didn't have the right (API) service IPs bound; manually ran dpkg-reconfigure wikimedia-lvs-realserver to fix this
* 04:10 maplebed: wikimedia-task-dns-auth package upgraded on ns0 to put zone files in revision control.
* 02:38 logmsgbot: aaron synchronized php/maintenance/addwiki.php 'deployed r93593'
* 02:31 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'renamed versionNumber variables to have wmf prefix'
* 02:25 logmsgbot: LocalisationUpdate completed at Fri Aug 5 02:27:52 UTC 2011
* 00:59 notpeter: knsq30 is seems to have some kind of hardware failure. stopping squid and squid frontend proc
* 00:48 AaronSchulz: /home/wikipedia/bin version with a working repo again
== August 4 ==
* 23:57 binasher: deploying mobile squid acls with nokia and netfront enabled for a brief test to see if todays mobile gateway changes will allow it to support the load
* 23:10 binasher: squid mobile redirect deployment completed
* 22:38 binasher: mobile redirect deploy complete to all pmtpa squids. mobile cluster looks good, starting amsterdam
* 22:17 binasher: starting an extra slow rollout of mobile redirect acls to frontend squids, minus nokia and netfront
* 21:43 binasher: varnish 3.0 is now on all RoR mobile gw servers - only caching static non-article content
* 21:33 binasher: mobile2 is back / varnish for static content / apache relax
* 19:52 binasher: pulling mobile2 from lvs for testing
* 17:17 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php 'r93902'
* 17:17 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php 'r93901'
* 17:12 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php 'r93901'
* 17:11 AaronSchulz: Fixed thumbnails on testwiki (bug 30210)
* 17:10 logmsgbot: aaron synchronized live-1.5/MWVersion.php 
* 15:40 mark: Mounted nas1001-a:/vol/images on ms1002:/vol/images for testing, started rsync copy from ms1002:/export/upload to /vol/images
* 15:20 mark: Created 12-disk aggregate test0 on nas1001-a
* 15:01 RoanKattouw: Staging some HTTPS support / prot rel URL code on testwiki, see today's 1.17wmf1 merges. DO NOT DEPLOY SITEWIDE until tomorrow
* 13:19 RobH: updated wikibugs on mchenry about 10 minutes ago and forgot to admin log
* 13:17 mark: Created new 64 bit aggregate and root volume on nas1001-a, copied over root data, destroyed old root volume, moving new aggregate to first disks
* 13:00 RobH: disabled line numbers in vimrc.local on puppet because i hate line numbers (it already says the line number yer on in the corner)
* 12:21 mark: Started disk replace operation of 0a.10.0 by 0a.10.3 and 0a.10.1 by 0a.10.4 on nas1-b
* 12:12 mark: Started disk replace operation of 0b.00.0 by 0a.01.1, and 0b.00.1 by 0a.01.2 on nas1-a
* 12:08 mutante: ragweed - package updates, restart required
* 11:05 mutante: amssq46 - power cycle, clean squid, dist-upgrade/kernel, reboot
* 02:18 logmsgbot: LocalisationUpdate completed at Thu Aug 4 02:20:48 UTC 2011
== August 3 ==
* 21:22 binasher: previous log line should have read $ppsize + 50
* 21:21 binasher: dropping apache maxclients on mobile servers to ($ppsize - 50)
* 18:24 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'deployed r93845'
* 18:23 logmsgbot: aaron synchronized multiversion/MWScript.php 'deployed r93845'
* 16:19 logmsgbot: tfinc synchronized wmf-config/InitialiseSettings.php 'testing collectinos on hewiki'
* 16:13 mutante: installed Net::Stomp Perl module on spence via cpan shell which fixed RT #703
* 14:32 RobH: updating dns for magnesium and zinc eqiad misc servers to public IPs RT#1245
* 13:24 logmsgbot: tstarling synchronized php-1.17/extensions/CodeReview/ui/WordCloud.php 
* 13:24 logmsgbot: tstarling synchronized php-1.17/extensions/CodeReview/CodeReview.php 
* 12:21 mark: Rebooting both nas1001 controllers with corrected network config
* 10:02 logmsgbot: root synchronized wmf-config/mc.php 'swapped out all down servers returning 0 with ones returning 100, cleaned up spares and down list to be accurate'
* 09:44 RobH: swapping out down memcached servers iwth tested up ones
* 09:44 RobH: incorrect, turns out there are some down, they reply but do not return full weight
* 09:44 logmsgbot: robh synchronized wmf-config/mc.php 'replacing down servers'
* 09:40 RobH: nagios shows spence memcached error, yet manual testing shows all memcached are pooled fine.
* 09:01 Ryan_Lane: installing ssl1001-4
* 08:09 apergos: live hack to /usr/share/etherpad/etherpad/bin/run-local.sh on hooper to force maxthreads to 512 for etherpad. restarted etherpad, this may or may not help connection issues (maybe we'll just oom instead?)
* 07:40 domas: stupid bot and its quotes, http://p.defau.lt/?UUberPb2hRtPxOzG7Q6OwQ is the url
* 07:39 logmsgbot: midom synchronized php/includes/db/DatabaseMysql.php 'livehacking out sql_mode and set charset code - http://p.defau.lt/?UUberPb2hRtPxOzG7Q6OwQ'
* 02:20 logmsgbot: LocalisationUpdate completed at Wed Aug 3 02:22:48 UTC 2011
== August 2 ==
* 23:25 binasher: done with that epic test
* 23:04 binasher: redeploying revised mobile acls with the redirector only matching /wiki/ urls for brief (15min) test
* 22:19 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'added $wgExtendedVersionNumber (Het Deploy), should have no effect now'
* 19:19 binasher: reverted mobile redirect acls
* 18:58 binasher: deploying mobile redirect acls/helper to all frontend text squids (--slow)
* 18:16 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '30186 - Install WikiLove on the Hebrew Wikipedia'
* 18:03 notpeter: restarting torrus again. cuz it's my favorite thing to do...
* 17:16 Reedy: As part of fatal coming from OAIRepository decreased by 10% from 50 to 45. No fatals in logs for nearly a week though
* 17:15 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'bug 29304 Set to 45'
* 17:04 logmsgbot: reedy synchronized php-1.17/extensions/OAI/OAIRepo_body.php 'r93770'
* 17:03 logmsgbot: reedy synchronized php-1.17/extensions/OAI/OAIRepo.php 'r93770'
* 17:02 logmsgbot: reedy synchronized php-1.17/includes/parser/Preprocessor_DOM.php 'r93763'
* 15:28 logmsgbot: catrope synchronized php/includes/media/SVG.php 'r93750'
* 14:23 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'Changing main namespace for ladwiki'
* 12:49 logmsgbot: midom synchronized wmf-config/InitialiseSettings.php 'reenabling disk cache'
* 12:42 domas: synced http://p.defau.lt/?Jze2xAwLV4Fvh8JC2B_KqQ to fix memcached side of pcache
* 12:40 logmsgbot: midom synchronized php/includes/objectcache/MemcachedClient.php 
* 12:07 logmsgbot: midom synchronized wmf-config/InitialiseSettings.php 'disabling disk cache'
* 12:00 Ryan_Lane: changing management dns entry for phosphorus to ssl1004
* 11:57 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'removed l10n debug log group'
* 11:56 logmsgbot: tstarling synchronized php-1.17/includes/LocalisationCache.php 'reverted live hack'
* 11:52 RobH: updated wikibugs on mchenry per rt#1179, restarted the service
* 11:44 logmsgbot: tstarling synchronized multiversion/MWVersion.php 'r93727 fix l10n cache'
* 11:38 RobH: restarted pdns on linne
* 11:31 RobH: updated dns for ssl and silicon servers
* 11:31 logmsgbot: tstarling synchronized php-1.17/includes/LocalisationCache.php 'log l10n file list changes'
* 11:30 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'Removed unmatched wfProfileOut()'
* 11:26 RobH: phosphorus changed to sl1004 in dns and racktables
* 11:20 logmsgbot: tstarling synchronized php-1.17/includes/LocalisationCache.php 'enough data'
* 11:16 logmsgbot: tstarling synchronized php-1.17/includes/LocalisationCache.php 'more logging'
* 11:15 logmsgbot: tstarling synchronized php-1.17/includes/LocalisationCache.php 'more logging'
* 11:11 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'logging hack for recache tracking'
* 11:09 logmsgbot: tstarling synchronized php-1.17/includes/LocalisationCache.php 'logging hack for recache tracking'
* 11:04 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'Removed unmatched wfProfileOut()'
* 11:04 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'Adding sitename for ladwiki, per bug 30181'
* 10:16 mark: Restarting all PyBal instances in a controlled manner
* 10:13 mark: Split LVS class high-traffic into high-traffic1 (text, bits) and high-traffic2 (upload), and made a separate class for https with BGP disabled
* 09:29 Ryan_Lane: powercycling srv281
* 08:14 mark: Increased TTL of bits.pmtpa.wikimedia.org from 60 to 1H
* 08:09 mark: Changed IP address of upload.pmtpa.wikimedia.org from .3 to 208.80.152.211
* 08:03 mark: Increased etherpad memory from 1G to 2G in /usr/local/etherpad/bin/run.sh
* 07:39 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'in parser cache multiwrite configuration, putting memcached first instead of mysql'
* 05:51 mark: Restarted nginx on yvon and gurvin; apparently nginx needs to be restarted after ip addresses changes, as it binds each individual address instead of IN_ADDR_ANY
* 04:23 notpeter: stopping puppet on spence temporarily
* 02:21 logmsgbot: LocalisationUpdate completed at Tue Aug 2 02:23:56 UTC 2011
* 00:26 AaronSchulz: deployed r93695 to fix WURFL infinite loop
* 00:24 logmsgbot: aaron synchronized php/extensions/MobileFrontend/library/WURFL/Storage/Mwmemcache.php 
* 00:23 logmsgbot: aaron synchronized php/extensions/MobileFrontend/library/WURFL/CustomDeviceRepository.php
== August 1 ==
* 23:41 ^demon: deleted another one of Roan's silly files from /h/w/c/p/ that contain less --help
* 23:38 notpeter: upgrading squid and squid-frontend on sq31. sq32 is down and still needs an upgrade.
* 23:25 logmsgbot: aaron synchronized php/extensions/MobileFrontend/library/WURFL/CustomDeviceRepository.php 'pre-OOM logging'
* 23:19 notpeter: upgrading squid and squid-frontend on sq33 and sq34
* 23:15 notpeter: restarting broken torrus again again.
* 22:58 logmsgbot: aaron synchronized live-1.5/robots.php 'commented out redundant code'
* 22:57 notpeter: upgrading squid and squid-frontend on sq35 and sq36
* 22:49 logmsgbot: aaron synchronized multiversion/MWVersion.php 'fix for robots.php hits on bogus wikis'
* 22:47 logmsgbot: aaron synchronized multiversion/MWVersion.php 
* 22:26 notpeter: upgrading squid and squid-frontend on sq37 and sq38
* 22:09 notpeter: upgrading squid and squid-frontend on sq39 and sq40
* 21:50 notpeter: restarting broken torrus again
* 21:18 Reedy: srv187 although up and alive, isn't reporting any metrics to ganglia, and uptime is completely wrong
* 21:15 notpeter: upgrading squid and squid-frontend on sq59 and sq60
* 21:12 logmsgbot: aaron synchronized multiversion/MWVersion.php 
* 21:10 logmsgbot: aaron synchronized wmf-config 'multiversion/MWVersion.php missing.php fix'
* 20:46 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'moved two lines of code down a bit to allow usage of missing.php'
* 20:33 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'removed transition hack (cdb rebuilt)'
* 20:29 AaronSchulz: wikiversions.cdb rebuilt
* 20:21 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132: 
* 20:17 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'hardcode to 1.17 for safe transition'
* 20:09 logmsgbot: aaron synchronized live-1.5/MWVersion.php 
* 19:59 notpeter: upgrading squid and squid-frontend on sq61 and sq62
* 19:37 notpeter: upgrading squid and squid-frontend on sq64
* 16:15 mark: Deployed puppet generated pybal.conf on amslvs1,2,3,4
* 15:47 mutante: amssq55: power cycle, fsck forces add. reboot, clean squid cache, dist-upgrade,..
* 15:43 mark: Deployed PyBal generated pybal.conf on amslvs3
* 15:25 mutante: amssq53: cleaned squid cache, now dist-upgrade and another reboot
* 15:07 mark: Temporarily filtering traffic from 10.0.0.0/16 to puppet on sockpuppet
* 14:56 mutante: power cycle amssq53
* 12:36 mutante: restarted ntpd on ms3 to fix a Nagios CRIT
* 12:14 mark: Changed IP address of bits.pmtpa.wikimedia.org from .118 to 208.80.152.210
* 11:02 mark: Moved LVS services configuration out of site.pp into lvs.pp, in puppet
* 10:03 mark: Started db20
== July 31 ==
* 12:20 Reedy: srv207 seems to be dead. Nagios suggests RAID issues
* 12:19 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30150 Fix timezone at xmfwiki'
* 02:21 logmsgbot: LocalisationUpdate completed at Sun Jul 31 02:23:17 UTC 2011
== July 30 ==
* 23:54 logmsgbot: reedy synchronized php-1.17/extensions/MobileFrontend/MobileFrontend.php 'htmlspecialchars url'
* 23:48 logmsgbot: reedy synchronized php-1.17/extensions/MobileFrontend/MobileFrontend.php 'htmlspecialchars url'
* 23:39 logmsgbot: reedy synchronized php-1.17/extensions/MobileFrontend/MobileFrontend.php 'Urlencode url'
* 23:32 logmsgbot: reedy synchronized php-1.17/extensions/MobileFrontend/MobileFrontend.php 'Urlencode url'
* 23:02 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Bump wgWikimediaMobileVersion to .3'
* 22:59 Reedy: WikimediaMobile javascript giving 404 error whilst log out. Going to bump $wgWikimediaMobileVersion
* 22:15 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '30122 - Add "WP" namespace alias to Test Wikipedia'
* 22:15 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '30122'
* 22:08 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '30138 - Please enable AbuseFilter for ee.wikipedia'
* 22:06 logmsgbot: jeluf synchronized wmf-config/abusefilter.php '30138 - Please enable AbuseFilter for ee.wikipedia'
* 20:35 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '30121 - Test Wikipedia request for file mover user group'
* 20:27 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '25181 - Enable transwiki import from zhwiki to other zh projects'
* 20:20 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '30136 - Enable recent changes patrolling on Portuguese Wikibooks'
* 19:09 binasher: db18 mysqld is back up and catching up on replication. may have a bad dimm (lots of ecc errors), leaving out of rotation for now
* 18:48 binasher: pulling db18 from rotation, appears to be having hw troubles
* 18:48 logmsgbot: root synchronized wmf-config/db.php 'pulling db18'
* 02:38 logmsgbot: LocalisationUpdate completed at Sat Jul 30 02:40:45 UTC 2011
* 00:09 notpeter: upgrading squid and squid-frontend on sq65 and sq66
== July 29 ==
* 23:34 notpeter: upgrading squid and squid-frontend on sq71 and sq72
* 23:08 logmsgbot: preilly synchronized php/extensions/MobileFrontend/library/WURFL/CustomDeviceRepository.php 'remove logging of memory exhausted error'
* 23:02 logmsgbot: preilly synchronized php/extensions/MobileFrontend/library/WURFL/CustomDeviceRepository.php 'possibly logging memory exhausted error bump limit to 15000'
* 23:00 logmsgbot: preilly synchronized php/extensions/MobileFrontend/library/WURFL/CustomDeviceRepository.php 'remove logging of memory exhausted error'
* 22:57 logmsgbot: preilly synchronized php/extensions/MobileFrontend/library/WURFL/CustomDeviceRepository.php 'possibly logging memory exhausted error bump limit to 10000'
* 22:56 logmsgbot: preilly synchronized php/extensions/MobileFrontend/library/WURFL/CustomDeviceRepository.php 'possibly logging memory exhausted error bump limit to 1000'
* 22:55 logmsgbot: preilly synchronized php/extensions/MobileFrontend/library/WURFL/CustomDeviceRepository.php 'possibly logging memory exhausted error'
* 22:53 notpeter: upgrading squid and squid-frontend on sq73 and sq74
* 22:17 notpeter: upgrading squid and squid-frontend on sq75 and sq76
* 21:57 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'move call for props'
* 21:11 notpeter: upgrading squid and squid-frontend on sq78 and sq77
* 21:10 RobH: fixed symlink for wikimedia.org/w/ in docroot
* 19:28 notpeter: restarting torrus. again. poor torrus, it just wants to die.
* 19:26 RobH: updated dsh files and restarted job runners
* 19:15 notpeter: upgrading squid and squid-frontend on knsq24
* 18:43 notpeter: upgrading squid and squid-frontend on knsq25 and knsq26
* 18:01 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'add mb_convert_encoding call'
* 17:01 mutante: amssq40 - dist-upgrade->new kernel->reboot
* 16:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Revert last commit'
* 16:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29911'
* 16:22 mutante: power cycle amssq40
* 15:45 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php 'Bug 29911 - customise FlaggedRevs with whole reviewer system on hi.wikipedia.org'
* 15:28 mark: Setup cross link between cr1-sdtpa:ae0.400 and cr2-pmtpa:ae0.400, using VLAN 400 across csw1-sdtpa/csw5-pmtpa
* 14:48 Jeff_Green: purging master logs on db9 up to the time of the last full db dump
* 14:30 mark: Removed old cross link between csw5-pmtpa:ve1 and csw1-sdtpa:ve1, freeing up VLAN 400
* 14:30 mark: Setup temporary cross link between cr1-sdtpa:ae0.404 and csw1-sdtpa:ve2
* 14:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30098 - Install extension WikiLove on Arabic Wikipedia'
* 13:53 mark: Deployed full (modified) replica of configuration of cr1-sdtpa on cr2-pmtpa
* 13:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 29956 Customise hiwiki add/remove groups'
* 13:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 30003 - Please set wgUploadNavigationUrl + wgUploadMissingFileUrl for nl.wikisource.org nl.wikiquote.org nl.wiktionary.org'
* 13:34 mark: Upgraded cr2-pmtpa to JUNOS 10.4R6.5
* 13:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30045 define transwiki import source for kmwiki from be, en and simple'
* 13:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30065 replace the Hebrew Wikipedia logo'
* 13:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 30085 Name update for cy.wikibooks.org'
* 02:16 logmsgbot: LocalisationUpdate completed at Fri Jul 29 02:18:38 UTC 2011
* 00:10 AaronSchulz: Updated wikimedia-periodic-update (for FR) to use MW script wrapper
* 00:05 AaronSchulz: Updated mw-central-notice, mw-tor-list to use MW script wrapper
== July 28 ==
* 23:36 AaronSchulz: Updated update-special-pages, update-special-pages-small, jobs-loop.sh to use MWScript.php
* 23:06 binasher: power cycled srv 173
* 22:56 logmsgbot: reedy synchronized wmf-config/db.php 'srv173 is dead to the world commenting out'
* 22:52 Reedy: srv173 seems deaded
* 22:40 logmsgbot: hashar: made proofreadpage.php configuration publicly available through http://noc.wikimedia.org/conf/ . Usual notice is at the top of the file. Requested by phe (frwiki).
* 22:39 logmsgbot: preilly synchronized php/extensions/MobileFrontend/MobileFrontend.php 'fixing caching'
* 22:37 logmsgbot: aaron synchronized wmf-config/ExtensionMessages-1.17.php 'removed ArticleAssessmentPilot'
* 22:31 logmsgbot: aaron synchronized wmf-config/extension-list 'removed ArticleAssessmentPilot'
* 22:28 Reedy: srv178 flapping due to full OS disk
* 22:14 logmsgbot: preilly synchronized wmf-config/InitialiseSettings.php 'enable Mobile Frontend'
* 22:07 logmsgbot: preilly synchronizing Wikimedia installation... Revision: 93132: 
* 21:28 Reedy: Full run through of LocalisationUpdate, to take into account r93425, Successful!
* 21:22 logmsgbot: LocalisationUpdate completed at Thu Jul 28 21:24:14 UTC 2011
* 20:45 binasher: enabling mobile redirect acls on sq78 (5% of all text requests)
* 20:27 logmsgbot: LocalisationUpdate failed
* 19:56 Reedy: New localisation files added for Ks-arab, Jam, Brh, Khw, Ks-deva, Anp, Dtp. Bh renamed Bho. Ks was split
* 19:55 logmsgbot: reedy synchronized php-1.17/languages/Names.php 'r93423'
* 19:54 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesBh.php 'r93420'
* 19:54 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesBho.php 'r93420'
* 19:51 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesKs.php 'r93420'
* 19:51 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesDtp.php 'r93420'
* 19:51 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesAnp.php 'r93420'
* 19:50 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesKs_deva.php 'r93420'
* 19:50 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesKhw.php 'r93420'
* 19:50 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesBrh.php 'r93420'
* 19:49 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesJam.php 'r93420'
* 19:49 logmsgbot: reedy synchronized php-1.17/languages/messages/MessagesKs_arab.php 'r93420'
* 19:41 logmsgbot: hashar: ci.tesla: deleted cc/projects/mw/extensions/FCKeditor directory
* 19:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Kill lqt alpha'
* 19:23 logmsgbot: reedy synchronized wmf-config/liquidthreads.php 'Kill lqt alpha'
* 18:50 logmsgbot: LocalisationUpdate completed at Thu Jul 28 18:52:41 UTC 2011
* 18:46 notpeter: pushing out new dns zone file (only changing cname survey to point to argon.w.o)
* 18:40 Reedy: LocalisationUpdate seems to be working correctly now
* 18:35 Reedy: Fixed localisationupdate for FR, added symlink in /home/wikipedia/l10n/trunk/extensions/FlaggedRevs for the language folder to presentation/language
* 18:32 notpeter: removing survey.w.o from sites enabled on singer for limesurvey migration, and doing a graceful on singer
* 18:22 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Remove old ArticleAssessment stuff'
* 18:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Remove old ArticleAssessment stuff'
* 18:14 logmsgbot: reedy synchronized php-1.17/languages/Names.php 'r93407'
* 18:14 logmsgbot: reedy synchronized php-1.17/includes/DefaultSettings.php 'r93407'
* 17:44 logmsgbot: LocalisationUpdate failed
* 17:23 Reedy: Manual run of /home/wikipedia/bin/sync-l10nupdate after updated permissions
* 16:56 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'touched file'
* 16:52 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 're-pushed Het Deploy changes with $dbSuffix fix'
* 16:51 RobH: nagios wasnt running, didnt investigate, just restarted service
* 16:44 AaronSchulz: Updated set-group-write2 to handle wmf-config being move up a dir
* 12:45 Reedy: Also, LocalisationUpdate doesn't seem to be logging to /var/log/l10nupdate as it is supposed to
* 12:43 Reedy: Seems LocalisationUpdate may be broken due to permission issues on l10n files on apaches
* 12:39 mark: Replaced SSDs in amssq31 and dist-upgrade
* 12:27 mark: Brought up BGP session to AS 38930
* 12:21 Reedy: Running l10nupdate manually on fenari
* 12:18 Reedy: Reports of LocalisationUpdate may have stopped working again
* 12:12 mark: Shutdown session to AS 16265
* 01:18 andrewS: dongle found. reconnected serial to pdu d3-sdtpa.
== July 27 ==
* 23:47 preilly: pushing bad redirect logging to mobile cluster
* 23:37 maplebed: started reslave from db42 to db1047; poor performance expected for the duration of the sync.
* 23:33 notpeter: restarting torrus again
* 23:19 preilly: pushing badredirect log to mobile cluster
* 22:46 notpeter: upgrading squid and squid-frontend on knsq27 and knsq28
* 22:13 notpeter: torrus stuck. assuming in deadlock. following instructions for restart
* 22:01 AaronSchulz: Re-installed moodbar & edittracking. Cleaned up wmf-config so that we are on HEAD (but with the reverted stuff still gone)
* 22:00 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132: 
* 20:19 notpeter: upgrading squid and squid-frontend on knsq29 and knsq30
* 19:54 binasher: pulled sq78 frontend from lvs
* 19:40 Reedy: Reports of users using non mobile browsers being redirected to mobile sites (enwiki and dewiki)
* 19:04 mark: Temporarily disabled facilities.pp file in puppet, until there is time to debug
* 18:34 binasher: pulled sq78
* 18:32 binasher: sq78 frontend is in service with mobile redirection
* 17:08 binasher: pulling sq78 from lvs prior to mobile testing
* 16:56 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'deployed r93255'
* 13:40 Reedy: Do not push any configuration change from wmf-config without speaking to Reedy/Ariel first
* 13:33 Reedy: Seems some configuration files between r2087 and r2088 weren't pushed to site, pushing them broke most projects. Aaron to investigate when he's in the office
* 13:20 logmsgbot: reedy synchronized wmf-config/extension-list 'r2087'
* 13:19 logmsgbot: reedy synchronized wmf-config/liquidthreads.php 'r2087'
* 13:19 logmsgbot: reedy synchronized wmf-config/ExtensionMessages.php 'r2087'
* 13:18 Reedy: Site should be somewhat back to normal, reverted to r2087 of config from private SVN
* 13:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'r2087'
* 13:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'r2087'
* 13:15 logmsgbot: reedy synchronized wmf-config/CommonSettings-old.php 
* 13:14 logmsgbot: reedy synchronized wmf-config/CommonSettings-new.php 
* 13:12 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'r2088'
* 13:12 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'r2088'
* 12:36 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Reverting last push'
* 12:36 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Reverting last push'
* 12:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Reverting last push'
* 12:31 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Reverting last push'
* 12:27 andrewS: miguel gave OK to fill up d1 in sdtpa as the heat flow issues have been temporarily resolved (also said construction on new AC units was to begin Aug 1st but has not heard anything yet).
* 12:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 
* 12:20 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 
* 11:55 mutante: sq63 - restored software RAID & dist-upgrade & reboot
* 11:15 Jeff_Green: adjusted firewall rules on payments servers per RT #1223
* 10:28 apergos: chowned /home/w/logs/*.log to udp (was root, /usr/local/bin/demux.py on nfs1 was failing to write to these files since July 21). This is probably the wrong solution but I don't know what the right setup is; can someone who does, make it right?
* 08:55 andrewS: 74th r410 dell server has arrived in tampa. uploaded packing slip to rt#611.
* 06:13 Jeff_Green: repooling payments3 & payments4, they're dist-updated and tested
* 05:05 logmsgbot: aaron synchronized php/maintenance/jobs-loop.sh 'added --wiki to another spot'
* 02:49 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'fixed texvc path'
* 01:31 notpeter: restarting jorb runners on srv151-180. also wishing there was a script for htis
* 01:05 logmsgbot: aaron synchronized php/maintenance/jobs-loop.sh 'added wiki=aawiki param to nextJobDB.php'
* 00:54 ^demon: job runners stuck in futex deadlock again :(
== July 26 ==
* 23:39 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132: 
* 23:34 Jeff_Green: dist-upgrade and reboot payments3, payments4
* 23:26 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132: 
* 23:23 notpeter: upgrading squid and squid-frontend on amssq31 and 32
* 23:23 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132: 
* 23:15 AaronSchulz: CommonSettings.php is no longer just a wrapper (it now has the CommonSettings-new.php contents)
* 23:07 Jeff_Green: dist-upgrade and reboot payments2
* 23:00 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132: 
* 22:47 notpeter: upgrading squid and squid-frontend on amssq33 and 34
* 22:42 notpeter: (retroactive) upgraded squid and squid-frontend on amssq38 and 39
* 22:18 Jeff_Green: dist-upgrading and rebooting payments1
* 21:51 logmsgbot: catrope synchronized wmf-config/CommonSettings-old.php 'Correct MoodBar policy URL per Howie'
* 21:50 logmsgbot: catrope synchronized wmf-config/CommonSettings-new.php 'Correct MoodBar policy URL per Howie'
* 21:38 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'moved setting of $secure back up'
* 21:21 notpeter: upgrading amssq40 and 41 squid and squid-frontend
* 20:45 Jeff_Green: purging master logs on db9
* 20:20 logmsgbot: awjrichards synchronized php/extensions/CentralNotice/special/SpecialCentralNotice.php 'r93242'
* 20:19 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'deployed r93240'
* 19:42 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'deployed r93236'
* 19:26 notpeter: upgrading squid and squid frontend on amssq35 and 36
* 19:22 logmsgbot: aaron synchronized wmf-config/CommonSettings-new.php 'removed warning in b/c script code'
* 19:16 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132: 
* 19:11 logmsgbot: aaron synchronized wmf-config/CommonSettings-new.php 'fix b/c code for scripts ran without wrapper'
* 18:48 RobH: cp1043 and cp1044 installed per rt#1157
* 18:46 logmsgbot: aaron synchronized wmf-config/CommonSettings-new.php 
* 18:44 logmsgbot: aaron synchronized wmf-config/CommonSettings-new.php 
* 18:40 AaronSchulz: CommonSettings.php is now at CommonSettings-new.php for now, the former is just a wrapper (making it easy to go back to the old one)
* 18:38 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132: 
* 18:34 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'switched all wikis to new commonsettings'
* 18:33 logmsgbot: aaron synchronized wmf-config/CommonSettings-new.php 'make sure this is on all wikis'
* 18:33 logmsgbot: aaron synchronized wmf-config/CommonSettings-old.php 'make sure this is on all wikis'
* 18:28 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'r93229'
* 16:06 mutante: power cycle owa1
* 15:55 mutante: power cycle srv281
* 15:53 mutante: power cycle srv266
* 14:55 mark: Migrated LVS services 'text' and 'upload' to lvs5/lvs6
* 14:51 mark: Migrated all HTTPS LVS services to lvs5/6
* 14:45 mark: Migrated bits-lb and bitssvc to lvs5/6
* 14:16 mutante: power cycle sq84 | clean squid cache & dist-upgrade on sq80,82,84
* 14:10 mutante: power cycle sq82
* 14:07 mutante: power cycle sq80
* 13:50 mark: Deploying lvs5 and lvs6 as LVS balancers (class high-traffic)
* 13:44 mutante: OS installed on lvs5 and lvs6
* 13:26 mutante: reinstall lvs6
* 13:23 mutante: power cycle lvs6
* 13:23 mark: Fixed switchport settings for lvs5
* 12:10 mutante: power cycle and dist-upgrade sq79
* 11:58 mutante: power cycle and dist-upgrade sq63
* 11:52 mutante: power cycle and dist-upgrade knsq5
* 11:31 mutante: power cycle and dist-upgrade knsq25
* 11:18 mutante: power cycle and dist-upgrade knsq23
* 11:13 mutante: dist-upgrade and reboot knsq20
* 10:56 mutante: power cycle knsq20
* 10:50 mutante: power cycle knsq13
* 10:26 mutante: rebooting amssq54,amssq57,amssq60
* 10:14 mutante: dist-upgrade on amssq60
* 10:14 mutante: dist-upgrade on amssq57
* 10:13 mutante: dist-upgrade on amssq54
* 09:45 mutante: running dist-upgrade on amssq50
* 09:40 mutante: power cycled amssq60
* 09:37 mutante: power cycled amssq57
* 09:23 mutante: power cycled amssq54
* 09:22 mutante: power cycled amssq50
* 00:44 logmsgbot: catrope synchronized php/resources/startup.js 'touch'
* 00:43 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Set privacyUrl properly for MoodBar'
* 00:39 AaronSchulz: Note - addwiki.php won't let you run it until Het deploy stage 2C is done (planned for tomorrow morning)
* 00:34 logmsgbot: aaron synchronized php/maintenance/addwiki.php 'deployed r93152'
* 00:11 notpeter: upgrading squid and squid-frontend on amssq42
== July 25 ==
* 23:52 AaronSchulz: changed scap & CommonSettings.php to use ExtensionMessages-1.17.php (instead of ExtensionMessages.php)
* 23:50 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132: 
* 23:45 notpeter: (re)deploying squid.confs to amssq46
* 23:43 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132: 
* 23:42 logmsgbot: aaron synchronized live-1.5/MWVersion.php 'switched all wikis to use new Het wrapper'
* 23:41 logmsgbot: aaron synchronized extract2.php 
* 23:23 notpeter: upgrading squid and squid-frontend on amssq44
* 23:09 Andrew: MoodBar deployment to enwiki complete.
* 23:03 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 93132: 
* 23:02 logmsgbot: aaron synchronized live-1.5/MWVersion.php 'avoid /home for secure testwiki'
* 23:00 Andrew: Deploying MoodBar to enwiki
* 22:43 logmsgbot: aaron synchronized extract2.php 'done with debug code'
* 22:40 Andrew: Adding MoodBar tables to all databases.
* 22:34 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 93132: 
* 22:33 Andrew: Deploying EditPageTracking to enwiki
* 22:33 logmsgbot: aaron synchronized extract2.php 
* 22:26 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 93132: 
* 22:25 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 93132: 
* 22:24 Andrew: Running scap to push message updates
* 22:21 Andrew: EditPageTracking tables are all added.
* 22:12 Andrew: Creating EditPageTracking tables on all wikis
* 22:08 Andrew: Deploying EditPageTracking to testwiki
* 22:08 Andrew: EditPageTracking and MoodBar user registration cutoff is 20110725221004.
* 22:04 Andrew: added EditPageTracking and MoodBar to extension-list
* 22:02 Andrew: EditPageTracking rollout to enwiki is going ahead.
* 22:01 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 93132: 
* 22:00 Andrew: updated prototype MoodBar / EditPageTracking to trunk state
* 21:57 logmsgbot: aaron synchronized extract2.php 
* 21:33 AaronSchulz: www.wikipedia.org portal site didn't like the change
* 21:32 logmsgbot: aaron synchronized live-1.5/MWVersion.php 'revert'
* 21:29 RobH: cp1001-cp1044 have had sas controllers installed and cabled. ready for os installation
* 21:21 logmsgbot: aaron synchronized live-1.5/MWVersion.php 'switch all wikis to new Het wrapper'
* 20:57 logmsgbot: catrope synchronized php/resources/startup.js 'touch'
* 20:56 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 100%'
* 20:55 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.27%'
* 20:40 logmsgbot: catrope synchronized php/resources/startup.js 'touch'
* 20:39 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 95%'
* 20:38 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.28%'
* 20:37 logmsgbot: aaron synchronized wmf-config/liquidthreads.php 'fixed reference to undefined var'
* 20:32 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'fixed testwiki notice'
* 20:29 logmsgbot: catrope synchronized php/resources/startup.js 'touch'
* 20:26 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 90%'
* 20:26 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.3%'
* 20:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29940 hiwiki namespaces'
* 20:02 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'fixed use of "tru" and bad NS_PORTAL constants'
* 20:00 logmsgbot: aaron synchronized php-1.17/maintenance/Maintenance.php 'deployed r93099'
* 19:45 notpeter: stopping, purging, and upgrading squid and squid frontened on amssq46 (still has faulty version on it)
* 19:32 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29956 userrights/groups for hiwiki'
* 18:44 binasher: installing mobile redirect helper to all caching proxies but leaving out of the squid config
* 18:43 binasher: added percona xtrabackup to wikimedia-lucid and installing on all lucid based db's via puppet
* 18:17 notpeter: updating squid and squid frontend on amssq37
* 17:48 Reedy: Trunk version of CodeReview has been deployed to mediawiki.org. Yaay
* 17:46 logmsgbot: demon synchronizing Wikimedia installation... Revision: 92888: Deploying updates to CodeReview
* 17:17 RobH: cp1033-cp1044 sas cards installed
* 17:12 mutante: Disabled account rcole
* 16:52 mark: Added user account and full root access to dzahn
* 16:08 mark: Removed old puppet resources for lvs3.pmtpa.wmnet (replaced by lvs3.wikimedia.org) on db9
* 01:17 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92888: 
== July 24 ==
* 18:12 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '29641 - Create Appendix'
== July 23 ==
* 09:50 logmsgbot: aaron synchronized php-1.17/includes/specials/SpecialUserlogin.php 'deployed r92938'
* 09:36 logmsgbot: aaron synchronized php-1.17/extensions/CategoryTree/CategoryTree.php 'live hack to efCategoryTreeAjaxWrapper to shut up $option notices'
* 09:28 logmsgbot: aaron synchronized php-1.17/includes/specials/SpecialResetpass.php 'deployed r92935'
* 09:27 logmsgbot: aaron synchronized php-1.17/includes/specials/SpecialUserlogin.php 'deployed r92935'
* 02:57 logmsgbot: catrope synchronized php/extensions/CategoryTree/CategoryTree.php 'remove debugging code; no point as long as the log collector is broken'
* 02:56 logmsgbot: catrope synchronized php/extensions/CategoryTree/CategoryTree.php 'debugging a PHP warning'
* 02:54 logmsgbot: catrope synchronized php/extensions/CategoryTree/CategoryTree.php 'debugging a PHP warning'
== July 22 ==
* 21:10 logmsgbot: catrope synchronized php/resources/jquery/jquery.localize.js 'r92890'
* 19:24 binasher: enabled local file debug logging on amssq45's backend squid
* 19:13 Ryan_Lane: adding ci2.tesla to DNS
* 19:03 Jeff_Green: new db10 dump restored to db1008 and replication is started
* 18:08 notpeter: upgrading squid-frontend/squid on amssq45
* 18:06 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Turning account creation project off for mw.org. Wasnt fully disabled'
* 18:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29940 hiwiki ns aliases'
* 16:59 Reedy: Running namespaceDupes.php on ukwiki
* 16:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29210'
* 16:56 RobH: rebooting srv187, unresponsive to serial console
* 16:55 Reedy: srv187 seems somewhat dead, SSH takes over 6 minutes to die with "srv187: ssh_exchange_identification: read: Connection reset by peer"
* 16:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29210'
* 16:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'bug 29210'
* 16:40 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'replaced undefined CT_MODE_PAGES with 10, the extension constant value'
* 14:57 Reedy: sync-common-file seems broken. Not returning after executing a command...
* 10:21 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92561: 
* 10:09 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'setupFundraiserPortal must declare $urlprotocol as global'
* 10:07 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'error suppression for REQUEST_METHOD check'
* 04:49 Jeff_Green: restarted replication on db10
* 04:47 Jeff_Green: repooled pyayments4
* 02:36 AaronSchulz: updated scap to new wmf-config dir
* 02:34 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92561: 
* 02:25 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92561: 
* 00:33 awjr: updating DonationInterface on payments cluster to r92830
* 00:30 logmsgbot: aaron synchronized wikiversions.cdb 
* 00:26 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 
== July 21 ==
* 23:54 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 
* 23:28 Jeff_Green: stopping slave on db10 for a mysqldump
* 23:21 Jeff_Green: depooling payments4 for awjr staging code changes
* 23:04 aaron: new MWVersion/MWMultiVersion code running on testwiki
* 23:04 logmsgbot: aaron synchronized live-1.5/MWVersion.php 
* 23:03 logmsgbot: aaron synchronized live-1.5/MWVersion.php 
* 23:00 logmsgbot: aaron synchronized wikiversions.cdb 
* 22:53 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92561: 
* 22:46 RobH: psw1-eqiad racked with serial connection to scs-a8-eqiad. RT#1207
* 22:25 awjr: updating CiviCRM install on Grosley to r368
* 22:04 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92561: 
* 21:31 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 92561: 
* 21:30 awjr: running scap to deploy CentralNotice updates to r92794
* 21:18 RobH: psw1-eqiad added into racktables, racked, powered, and outlets configured. need to still run serial and get network port assignemtns
* 20:55 logmsgbot: catrope synchronized php/resources/startup.js 'touch'
* 20:50 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 85%'
* 20:49 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.318%'
* 20:46 Ryan_Lane: make that 5, since that's the lowest allowed
* 20:45 Ryan_Lane: setting autosupport retry to 0 on eqiad netapp
* 20:40 Ryan_Lane: disabled autosupport on eqiad netapp
* 20:28 logmsgbot: catrope synchronized php/resources/startup.js 'touch'
* 20:28 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 77%'
* 20:27 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.35%. Also includes lots of whitespace changes and an unused global removal by Aaron'
* 19:52 RobH: ms1004 rt1164 passes full dell hardware testing suite
* 19:08 notpeter: upgrading squid and squid frontend on amssq46
* 18:56 logmsgbot: jeluf synchronized wmf-config/InitialiseSettings.php '29971 - Set wgUploadMissingFileUrl for nlwiki'
* 17:26 Ryan_Lane: powering off srv173, since it is dead, and is messing up the stats
* 16:33 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'unused var in wfNoDeleteMainPage'
* 16:18 mark: Deployed lvs3 as backup LVS balancer for 'low-traffic' class LVS services, backing up lvs4
* 15:22 RobH: lvs3 reinstalled, puppet not run
* 15:07 RobH: updated dns again for more changes pertaining to lvs
* 15:02 RobH: updated dns to add lvs3.wikimedia.org to forward dns, was already in reverse file.
* 14:56 RobH: lvs6 reinstalled, puppet not synced and not signed as it may need further work
* 14:50 mark: Added new service IPs for bits and upload to varnish and squid classes in puppet
* 14:43 mark: Allocated out of subnet IPs for bits and upload.pmtpa
* 14:35 RobH: lvs3, lvs5, & lvs6 being (re)installed and setup, please disregard any warnings.
* 14:34 mark: Redeployed RunCommand monitor on lvs4, now fixed
* 14:33 RobH: lvs3 being reinstalled
== July 20 ==
* 22:51 Ryan_Lane: started memcached on srv260
* 21:53 Ryan_Lane: added monitoring for esams and pmtpa -lb addresses
* 21:10 Jeff_Green: db10 mysqldump is done, replication restarted
* 21:05 Ryan_Lane: adding lvs IPs for all esams -lb addresses to amslvs servers
* 20:54 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
* 20:52 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 70%'
* 20:52 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.386%'
* 20:47 Ryan_Lane: added all new -lb services to esams pybal
* 20:45 Ryan_Lane: made esams pybal files consistent. restarting pybal on each amslvs system, in order
* 20:39 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
* 20:37 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 63%'
* 20:37 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking percentage to 0.428571%'
* 20:25 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
* 20:24 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 57%'
* 20:22 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking rate to 0.47%'
* 19:02 logmsgbot: awjrichards synchronized php/extensions/ContactPageFundraiser/SpecialContact.php 'r92653'
* 19:02 logmsgbot: awjrichards synchronized php/extensions/ContactPageFundraiser/README 'r92653'
* 19:02 logmsgbot: awjrichards synchronized php/extensions/ContactPageFundraiser/install.settings 'r92653'
* 19:01 logmsgbot: awjrichards synchronized php/extensions/ContactPageFundraiser/ContactPage.i18n.php 'r92653'
* 19:01 logmsgbot: awjrichards synchronized php/extensions/ContactPageFundraiser/ContactPage.alias.php 'r92653'
* 18:59 logmsgbot: awjrichards synchronized wmf-config/CommonSettings.php 'Variable-izing enabling of ContactPageFundraiser extension, enabling on testwiki and foundation wiki'
* 18:56 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Variable-izing enabling of ContactPageFundraiser extension'
* 18:54 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 
* 18:42 AaronSchulz: Changed sync-file to be relative to common/ instead of php-1.17/
* 18:41 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92561: 
* 18:41 RobH: rebooting the offline sq63 to poll its SAS controller info
* 18:25 Ryan_Lane: restarting pybal on lvs1
* 18:12 Jeff_Green: stopping replication on db10 and redoing dump with mysqldump instead of mydumper...
* 18:03 logmsgbot: awjrichards synchronized php-1.17/extensions/LandingCheck/SpecialLandingCheck.php 'r92647'
* 18:03 logmsgbot: awjrichards synchronized php-1.17/extensions/LandingCheck/LandingCheck.i18n.php 'r92647'
* 17:40 RobH: lvs3 racked, but powered off. mgmt confirmed working, network port confirmed up
* 16:09 mark: Disabling RunCommand monitor on lvs4; not working right
* 15:50 mark: Restarted PyBal on lvs4
* 15:27 RobH: powercycling srv283
* 15:27 RobH: srv283 is hung up again, and now wont take my commands in serial console...
* 15:22 RobH: srv283 loads are insane, its very slugging, and only serial console works. kicking the apache process over to free up memory and resume normal function
* 14:39 RobH: srv239 unresponsive, rebooting
* 14:02 mark: Setup nas1 with clustering failover
* 08:44 mark: Fixed misc::mediawiki-logger manifest
* 00:04 Ryan_Lane: test bug 1
== July 19 ==
* 23:59 binasher: also set long_query_time to 0.8 on the enwiki read slaves
* 23:09 Ryan_Lane: added new ganglia clusters for ssl cluster in pmtpa and esams
* 22:50 binasher: reduced long_query_time on db36 to 800ms
* 22:46 binasher: setting up mk-query-digest slow query processing on db36
* 22:43 binasher: installing maatkit on core dbs via puppet
* 22:13 Jeff_Green: installing mysql 5.0.51a on db1025 to for mydumper/myloader mysqld version upgrade testing
* 21:40 Ryan_Lane: make that changing the CNAME to wikipedia-lb
* 21:40 Ryan_Lane: changing DNS entry for test.wikipedia.org to wikipedia-lb.wikimedia.org
* 21:04 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
* 21:03 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Kill underscores in blacklist category name'
* 20:58 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
* 20:58 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Set up Category:Article Feedback Blacklist as an AFT blacklist category on enwiki'
* 20:45 mark: Flipped all backup static routes for LVS service ips from lvs3 to lvs4
* 20:42 mark: lvs3 is now unused.
* 20:40 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
* 20:40 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 50%'
* 20:40 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking percentage to 0.54%'
* 20:38 mark: Migrated apaches LVS service from lvs3 to lvs4
* 20:22 mark: Migrated LVS service 'api' from lvs3 to lvs4
* 20:19 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
* 20:19 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 43%'
* 20:19 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking percentage to 0.63%'
* 20:07 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
* 20:07 mark: Migrated LVS service 'rendering' from lvs3 to lvs4
* 20:05 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 37%'
* 20:04 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 0.73%'
* 19:56 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.css 'r92561'
* 19:56 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r92561'
* 19:55 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'r92561'
* 19:55 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.i18n.php 'r92561'
* 18:58 Ryan_Lane: installing nginx with udplogging module on ssl proxies
* 18:51 logmsgbot: aaron synchronized php-1.17/maintenance/Maintenance.php 'deploy r92550'
* 18:27 Ryan_Lane: pushing change to squids to strip X-Forwarded-Proto, if not set by the ssl proxies
* 18:08 Ryan_Lane: repooling sq69 with geoiplookup xff changes, for load testing
* 18:02 mark: Fixed LVS on lvs1/lvs2
* 17:44 mark: Disabled RPF check on ae2.1002 on cr1/cr2-eqiad
* 17:07 Jeff_Green: rebooting db1025 after dist-upgrade
* 17:02 Ryan_Lane: depooling sq69 for geoiplookup tests
* 16:01 mark: Enabled BGP on lvs4 and migrated LVS service search_pool3 to lvs4
* 15:35 RobH: rebooting sq63 to check its bios, its arleady down due to other issues and depooled in pybal
* 15:31 mark: Added BGP configuration for PyBal to cr1-sdtpa
* 14:14 Jeff_Green: commented out puppet-related cron jobs on db1008 as well :-P
* 13:58 Jeff_Green: stopping puppet daemon on db1008 to facilitate local my.cnf testing
* 08:55 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92324: 
* 07:14 AaronSchulz: Fixed require in PrivateSettings.php
* 07:13 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 
* 07:10 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 
* 01:14 Ryan_Lane1: repooling sq69
* 00:34 logmsgbot: reedy synchronized wmf-config/abusefilter.php 'bug 29922 Remove abusefilter-private right so it cannot be accidentally granted'
* 00:34 Jeff_Green: db10 slaving restarted
* 00:30 Ryan_Lane: depooling sq69 to test varnish changes
== July 18 ==
* 23:57 AaronSchulz: changed 1.17/StartProfiler.php to symlink to the one in wmf-config
* 23:57 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92324: 
* 23:41 logmsgbot: catrope synchronized docroot/bits/test.txt 
* 23:40 AaronSchulz: copied StartProfiler.php to wmf-config
* 23:40 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92324: 
* 22:32 rainman-sr: build search indexes for the recently created wikis
* 21:34 AaronSchulz: moved PrivateSettings.php to wmf-config and symlinked 1.17 copy of file
* 21:31 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92324: 
* 21:23 JeLuF: removed some privacy related mails from wikide-l archive
* 20:58 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
* 20:56 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 30%'
* 20:55 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking rate to 0.9%'
* 20:44 notpeter: running apache-graceful-all
* 20:38 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'toch'
* 20:38 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 27%'
* 20:37 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking percentage to 1%'
* 20:37 notpeter: syncing apaches again
* 20:26 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
* 20:25 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Raise AFT percentage to 23%'
* 20:24 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Lower AFT clicktracking to 1.17% and bump cache epoch on HTTPS experiment wikis'
* 20:20 logmsgbot: laner synchronized wmf-config/InitialiseSettings.php 'setting wgServer for officewiki to be protocol relative'
* 20:11 Ryan_Lane: renaming chlorine, germanium, and sulfer to ssl1001, ssl1002, and ssl1003, respectively
* 20:00 Jeff_Green: stopping replication on db10 and dumping db to db1008
* 19:40 Jeff_Green: recloned payments3 from payments4, restored db pool config to the way it was before
* 19:12 Jeff_Green: pointing payments[1-4] db reads to payments2 specifically (rather than localhost) to repair replication on payments3
* 18:57 notpeter: syncing apaches again
* 18:41 notpeter: syncing apaches
* 18:02 notpeter: pushing new dns confs (just a cname, ryan looked over it)
* 17:55 binasher: varnish3-3.0 packages with udp multicast logging support built and added to the wmf repo
* 17:31 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92324: 
* 17:29 logmsgbot: aaron synchronized php-1.17/maintenance/doMaintenance.php 'deployed r92452'
* 17:18 logmsgbot: aaron synchronized php-1.17/LocalSettings.php 'use new wmf-conf dir'
* 17:12 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 92324: 
* 17:12 aaron copied wmf-config up one directory (to common/)
* 16:08 RobH: ms1004 started hardware testing. system is using the build in diag tests for dell, but do NOT work via serial redirection. Thus ms1004 appears offline, but is running tests. DO NOT REBOOT IT
* 14:51 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29942 Enable UploadNavigationUrl for Hindi Wikipedia'
* 14:49 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29942 Enable UploadNavigationUrl for Hindi Wikipedia'
* 14:29 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'revert bug 29942'
* 14:21 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29942 Enable UploadNavigationUrl for Hindi Wikipedia'
* 14:08 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 14:03 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29941 Enable Extension Quiz on hiwiki'
* 13:55 RobH: ms1004 coming back online, the UEFI doesn't support console redirection
* 13:52 RobH: ms1004 coming down for testing
== July 17 ==
* 22:00 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29689'
* 20:59 logmsgbot: hashar: investigating hacked sysop account on hewiki for matanya
* 20:28 Reedy: Running namespaceDupes on brwiktionary
* 20:27 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29924 Add appendix namespace for brwiktionary even'
* 20:24 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29924 Add appendix namespace for brwiki'
* 19:50 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29210 Add Portal alias for ukwiki'
* 19:28 Reedy: running cleanupImages.php against commons in dry-run mode
* 19:20 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 25181 allow import from zhwiki to other zh projects'
* 18:34 Reedy: Ran populateSha1 on commons for images with the hash "phoiac9h4m842xq45sp7s6u21eteeq1"
* 18:05 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29534 set to true multilingual Wikisource'
* 17:57 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29486 Rename user namespace on ka.wiki'
* 17:44 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 25752 Add some rights to enwiki crats'
* 17:35 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29830, set wmgWikiLoveDefault for hiwiki'
== July 16 ==
* 23:36 Ryan_Lane: depooling sq63
* 21:25 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/SpecialArticleFeedback.php 'Fix fatal'
* 12:49 logmsgbot: ariel synchronized php-1.17/wmf-config/abusefilter.php 'disable abusefilter-private for es wikt, it would reveal ips, see bug 29910'
* 07:34 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29648 - Set $wgCollectionHierarchyDelimiter = "/" for Wikibooks projects'
* 07:33 logmsgbot: jeluf synchronized php-1.17/wmf-config/CommonSettings.php '29648 - Set $wgCollectionHierarchyDelimiter = "/" for Wikibooks projects'
* 07:18 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28221 - Romanian Wiktionary $wgSitename change and change of namespace name for extra namespace'
* 07:01 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29913 - ScoWp Import sources'
* 00:06 Ryan_Lane: restarting job runners
== July 15 ==
* 23:19 RobH: ms1004 refusing to boot of cd burned, updating ticket and will work on it later, booted back online
* 23:09 RobH: upgraded blog.wikimedia.org to newest wp version 3.2.1
* 23:04 RobH: rebooting ms1004 into hardware tests per RT#1164
* 22:11 logmsgbot: reedy synchronized php-1.17/extensions/ProofreadPage/proofread.js 'r92321'
* 22:10 logmsgbot: reedy synchronized php-1.17/extensions/ProofreadPage/ProofreadPage_body.php 'r92321'
* 22:10 logmsgbot: reedy synchronized php-1.17/extensions/ProofreadPage/ProofreadPage.php 'r92321'
* 22:10 logmsgbot: reedy synchronized php-1.17/extensions/ProofreadPage/ProofreadPage.i18n.php 'r92321'
* 21:54 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'bumping epoch for https experiment'
* 21:50 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'global wmfConfigDir needed, breaks otherwise'
* 21:48 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Setting wgServer to be protocol relative for testwiki'
* 21:45 logmsgbot: catrope synchronized php-1.17/includes/MessageBlobStore.php 'Fix fatals'
* 21:37 logmsgbot: reedy synchronized php-1.17/includes/ImageGallery.php 'r92310'
* 21:25 RobH: db1037 controller replaced, ready for installation
* 21:25 logmsgbot: reedy synchronized php-1.17/includes/ImageGallery.php 'r92294'
* 21:24 Reedy: "log reedy synchronized php-1.17/skins/common/shared.css 'r92278'" also merged r92294
* 21:23 logmsgbot: reedy synchronized php-1.17/includes/upload/UploadFromUrl.php 'r92289'
* 21:23 logmsgbot: reedy synchronized php-1.17/includes/MessageBlobStore.php 'r92289'
* 21:22 logmsgbot: reedy synchronized php-1.17/includes/HttpFunctions.php 'r92289'
* 21:21 logmsgbot: reedy synchronized php-1.17/languages/Names.php 'r92287'
* 21:21 logmsgbot: reedy synchronized php-1.17/includes/media/SVGMetadataExtractor.php 'r92884 r98226'
* 21:20 logmsgbot: reedy synchronized php-1.17/includes/api/ApiQueryCategoryMembers.php 'r92283'
* 21:19 logmsgbot: reedy synchronized php-1.17/includes/api/ApiQueryCategories.php 'r92283'
* 21:19 logmsgbot: reedy synchronized php-1.17/resources/Resources.php 'r92278'
* 21:18 logmsgbot: reedy synchronized php-1.17/includes/libs/CSSMin.php 'r92278'
* 21:18 logmsgbot: reedy synchronized php-1.17/skins/common/commonPrint.css 'r92278'
* 21:17 logmsgbot: reedy synchronized php-1.17/skins/common/shared.css 'r92278'
* 21:16 logmsgbot: reedy synchronized php-1.17/extensions/CentralAuth/CentralAuthGroupMembershipProxy.php 'r92270'
* 21:15 logmsgbot: reedy synchronized php-1.17/extensions/TitleBlacklist/TitleBlacklist.hooks.php 'r92265'
* 21:03 RobH: mobile1001-mobile1005 & stat1001 racked and mgmt online in eqiad. available for allocation and installation.
* 19:35 binasher: restarted pdns on ns2
* 19:29 RobH: updating dns for mgmt on mobile1001-1005, stat1001, and all of the now named misc servers in eqiad.
* 18:51 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29910 - Activation of AbuseFilter on Spanish Wiktionary'
* 18:51 binasher: finished setting up replication for s1-7 to eqiad dbs
* 18:50 logmsgbot: jeluf synchronized php-1.17/wmf-config/abusefilter.php '29910 - Activation of AbuseFilter on Spanish Wiktionary'
* 18:14 logmsgbot: aaron synchronized php-1.17/wmf-config/CommonSettings.php 'Use $wmfConfigDir temp in the other places'
* 18:11 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29830 - Enable Extension:WikiLove on Hindi Wikipedia'
* 18:08 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29825 - Enable Extension:NewUserMessage on Hindi Wikipedia'
* 18:07 awjr: disabled OWA tracking on all payments boxes ~1 hour ago (forgot to log)
* 18:07 Jeff_Green: testing mydumper by pulling db1040's database to db1008
* 17:38 logmsgbot: aaron synchronized php-1.17/wmf-config/CommonSettings.php 
* 17:30 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable Narayam on tawiki per bug 29798'
* 11:36 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'remove some old commented code'
* 04:56 Jeff_Green: rebooting db1008 after dist-upgrade
* 01:12 Ryan_Lane: restarting pybal on lvs2
* 00:42 Ryan_Lane: adding nginx configuration for all domains to the https clusters
* 00:41 Ryan_Lane: adding all -lb addresses to the https cluster for LVS. also adding the text.esams.wikimedia.org address to the esams https cluster, since that address will switch to a -lb address in the future.
* 00:33 Jeff_Green: installed packages on db1008 to support compiling mydumper: python-sphinx libglib2.0-dev libmysqlclient15-dev zlib1g-dev libpcre3-dev gcc make cmake g++
* 00:21 Ryan_Lane: adding wiktionary-lb, wikiquote-lb, wikibooks-lb, wikisource-lb, wikinews-lb, wikiversity-lb, mediawiki-lb, and foundation-lb addresses to geodns, and forward and reverse for esams and pmtpa
== July 14 ==
* 23:38 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'moved $wgCacheDirectory out'
* 23:38 logmsgbot: aaron synchronized php-1.17/wmf-config/CommonSettings.php 'moved $wgCacheDirectory here'
* 23:23 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'removed commented out wgCacheDirectory code'
* 20:49 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
* 20:49 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT percentage from 17% to 20%'
* 20:48 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Take down AFT clicktracking from 1.6% to 1.35%'
* 20:27 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
* 20:27 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT percentage from 13% to 17%'
* 20:26 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Take down AFT clicktracking percentage from 2% to 1.6%'
* 20:14 Ryan_Lane: reenabling portmap and nfs-common services on fenari
* 20:13 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'touch'
* 20:10 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT percentage from 10% to 13%'
* 20:08 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Take down AFT clicktracking percentage from 2.7% to 2%'
* 18:47 Ryan_Lane: disabling portmap and nfs-common services on fenari
* 18:46 logmsgbot: catrope synchronized php-1.17/includes/parser/Parser.php 'r92178'
* 18:44 JeLuF: importing 6 huge images to the commons (bug 29872)
* 18:31 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump cache epoch for HTTPS experiment wikis'
* 18:25 logmsgbot: catrope synchronized php-1.17/cache/interwiki-pr.cdb 'touching'
* 18:13 logmsgbot: laner synchronized wmf-deployment/cache/interwiki-pr.cdb 'Re-pushing the protocol relative interwiki cdb, since it had bad write permissions'
* 18:12 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Adding testwiki back into the https experiment'
* 18:07 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Taking testwiki out of https experiment'
* 18:04 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Changing stdlogopr to prstdlogo'
* 18:03 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Changing stdlogopr to prstdlogo'
* 17:59 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Adding stdlogopr, for protocol relative url stdlogo for https experiment'
* 17:57 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Putting testwiki into the https experiment'
* 17:33 RobH: replaced disk in db1006, fixed
* 17:07 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29738 - add auto-patrolled group to he.wikiquote'
* 16:13 mark: Disabled STP on asw-d3-sdtpa
* 16:02 mark: Halted nas1001-a and nas1001-b for recabling
* 15:54 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29798 - Install Narayam on Tamil wiki projects'
* 15:40 mark: Reran initial setup on nas1001-b, rebooting it now
* 15:30 mark: Reran initial setup on nas1001-a, rebooting it now
* 15:23 mark: Setup switch ports and LACP LAGs for nas1001
* 14:16 awjr: disabling puppet on grosley (including its cronjobs)
* 11:54 mark: Increased MTU on both pmtpa-eqiad links to 9192
* 11:32 mark: Renamed core-database puppet class into db::core
* 01:47 RoanKattouw: nfs1 overloaded because I tried to scp too much data to fenari at once. nfs1 and fenari have high wait CPU, /home kinda useless right now. This should sort itself out over time, right?
== July 13 ==
* 23:50 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Fixing the x-forwarded-proto check'
* 23:47 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Enabling wgSecureCookie for wikis in the https experiment'
* 23:13 RobH: more dns updates for same servers, forgot to set the mgmt to the new nice names instead of just the asset tag names
* 23:00 RobH: updating dns for new servers argon, chlorine, germanium, sulfur
* 22:09 logmsgbot: catrope synchronized php-1.17/extensions/Narayam/ext.narayam.rules.ta.js 'r92105'
* 22:09 logmsgbot: catrope synchronized php-1.17/extensions/Narayam/ext.narayam.rules.ta99.js 'r92105'
* 22:00 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Making the apple touch icon for officewiki protocol relative'
* 21:58 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling protocol relative URLs for officewiki'
* 21:53 Ryan_Lane: changing cname for office wiki to wikimedia-lb
* 21:42 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28192 - Patrolled edits on ro.wikipedia'
* 21:09 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'Touch startup.js, workaround for RL bug'
* 21:06 Jeff_Green: repooling payments3 & payments4
* 21:02 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Increase AFT percentage from 7% to 10%'
* 21:01 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Lower AFT clicktracking rate from 3.85% to 2.7%'
* 21:00 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29727 - Enable the NewUserMessage extension on Wikimedia Incubator'
* 20:58 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29727 - Enable the NewUserMessage extension on Wikimedia Incubator'
* 20:57 Jeff_Green: rebooting payments3 & payments4
* 20:50 notpeter: taking down grosley for reimaging/upgrading
* 20:43 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29825 - Enable Extension'
* 20:42 Jeff_Green: depooling payments3 and payments4 for package updates and reboots
* 20:38 Jeff_Green: repooling payments2
* 20:38 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise ArticleFeedback percentage from 4% to 7%'
* 20:38 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Lower AFT clicktracking from 6.75% to 3.85%'
* 20:27 Jeff_Green: rebooting payments2
* 20:22 Jeff_Green: depooling payments2 for patches and reboot
* 20:20 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Increase AFT percentage from 2.7% to 4%'
* 20:19 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Lower AFT clicktracking rate from 10% to 6.75%'
* 20:18 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29796 - Create Wikiversity Arabic'
* 15:17 RobH: pushed dns change to fix dns error for db12 mgmt
* 13:26 mark: Rebooting asw-b-sdtpa
* 13:09 mark: Temporarily disabled BGP aggregates announcement on cr1/cr2-eqiad
* 00:32 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Setting the powered-by link to a protocol relative url for wikis in https experiment'
* 00:08 Ryan_Lane: updated dumpInterwiki.php to add a protocolrelative option, which will output the interwiki cdb with all wikimedia interwiki urls set as protocol relative
== July 12 ==
* 23:55 logmsgbot: laner synchronized wmf-deployment/cache/interwiki-pr.cdb 'Pushing out updated protocol relative interwiki cdb again'
* 23:51 logmsgbot: laner synchronized wmf-deployment/cache/interwiki-pr.cdb 'Pushing out updated protocol relative interwiki cdb'
* 23:33 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Use interwiki-pr.cdb in a more sane way'
* 23:21 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Setting interwiki cdb to interwiki-pr.cdb for wikis in https experiment'
* 23:20 logmsgbot: laner synchronized wmf-deployment/cache/interwiki-pr.cdb 'Pushing protocol relative interwiki cdb for https experiment'
* 22:55 Jeff_Green: payments1 is back online and repooled
* 22:40 Jeff_Green: rebooting payments1 for kernel upgrade
* 22:39 Jeff_Green: depooling payments1 for kernel upgrade
* 22:20 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Setting more URLs to be protocol relative for HTTPS experiment'
* 22:20 binasher: running xtrabackup of db33 for clone to db1020
* 22:20 Jeff_Green: running db hot backup on db43 for clone to db1006
* 20:57 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'Disabling interwiki cache for wikis in the https experiment'
* 20:56 Ryan_Lane: disabling interwiki cache for wikimedia2005wiki
* 19:39 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Setting url relative URL for apple touch icon for wikimania2005wiki'
* 19:34 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Set $wgCacheEpoch to today for wikis with $wmgHTTPSExperiment'
* 19:26 logmsgbot: laner synchronized php-1.17/wmf-config/CommonSettings.php 'enabling protocol relative urls for wikimania2005wiki'
* 19:26 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling protocol relative urls for wikimania2005wiki'
* 19:11 logmsgbot: demon synchronized php-1.17/extensions/LiquidThreads/classes/Hooks.php 'r91986'
* 19:03 Ryan_Lane: switching meta and commons from using text.wikimedia.org to use wikimedia-lb.wikimedia.org
* 18:56 Ryan_Lane: updated CommonSettings.php and InitialiseSettings.php, if you push changes right now, you're crazy ;)
* 15:42 logmsgbot: ariel synchronized php-1.17/includes/Import.php 'er, now with the svn updated file :-P'
* 15:41 logmsgbot: ariel synchronized php-1.17/includes/Import.php 'add LIBXML_PARSEHUGE to override 10b text node limit in recent versions of libxml'
* 15:41 logmsgbot: ariel synchronized php-1.17/maintenance/backupPrefetch.inc 'add LIBXML_PARSEHUGE to override 10b text node limit in recent versions of libxml'
* 09:42 JeLuF: created four new wikis, bugs 29456, 29715, 29758, 29796
* 09:19 logmsgbot: jeluf ran sync-common-all 
* 07:44 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27557 - Create Sakha Wikisource'
* 05:50 Tim: restarting job runners
* 00:27 Tim: running chown/chmod to fix up the permissions on the php-1.17/cache directory, broken by sync-l10nupdate
* 00:20 Tim: fixed sync-l10nupdate to run as mwdeploy
* 00:01 MegaTroll: installing vmwaretools on internproxy
== July 11 ==
* 23:59 MegaTroll: updated internproxy. rebooting
* 23:37 logmsgbot: reedy synchronized docroot/mediawiki/xml/export-0.5.xsd 
* 23:32 logmsgbot: reedy ran sync-common-all 
* 23:28 Tim: fixed sync-docroot and sync-common-all, they were both trying to push files out without switching user to mwdeploy
* 23:28 logmsgbot: tstarling ran sync-common-all 
* 23:10 logmsgbot: reedy synchronized docroot/mediawiki/xml/export-0.5.xsd 
* 23:09 Reedy: Downloaded export-0.5.xsd on fenari
* 22:41 Jeff_Green: payments1 up and repooled
* 22:23 Jeff_Green: rebooting payments1
* 22:14 Jeff_Green: depooled payments1 for package updates and reboot
* 21:16 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r91909'
* 21:14 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Doh! $wmgClickTrackingLog needs udp:// prefix'
* 21:11 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Try using locke's IP address instead of FQDN'
* 20:56 Ryan_Lane: powercycling payments1.wikimedia.org
* 20:14 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Using FQDN for lock, for udp clicktracking logging instead of the short name'
* 19:40 Jeff_Green: purged master logs before 20110611 on db9
* 19:18 notpeter: pushing out new zone file (only one minor change to cname)
* 19:00 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'CommonSettings code for wmfClickTrackingLog'
* 19:00 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add wmgClickTrackingLog setting, logging enwiki to hume:8421'
* 18:57 logmsgbot: catrope synchronized php-1.17/extensions/ClickTracking/ApiClickTracking.php 'r91899'
* 18:57 logmsgbot: catrope synchronized php-1.17/extensions/ClickTracking/ClickTracking.hooks.php 'r91899'
* 18:56 logmsgbot: catrope synchronized php-1.17/extensions/ClickTracking/ClickTracking.php 'r91899'
* 18:15 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29702 - Enable desysoping by bureaucrats on Hindi Wiktionary'
* 18:12 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29707 - New logo for Khmer Wiktionary km.wiktionary.org'
* 12:48 apergos: rebooting snapshot1, after upgrade to new kernel
* 11:31 mark: Enabled LLDP on csw1-sdtpa
* 06:06 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling mysql parser cache on all wikis'
* 06:02 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabling mysql parser cache on enwiki'
* 05:59 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabling mysql parser cache on enwikinews'
* 05:51 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'increased parser cache expiry to 365 days'
* 05:47 Tim: dropping existing pc* tables and recreating them with different table options
* 05:40 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling mysql parser cache for server configuration change'
* 05:38 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 05:28 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling mysql parser cache on enwikinews only'
* 05:25 Tim: restarted profile-stats-logger.py
* 05:22 Tim: fixed umask in reedy's .bashrc
* 05:19 Tim: dropping old parser cache data from db40, was there since previous pilot
== July 10 ==
* 10:47 Ryan_Lane: restarted the lsearchd daemon on search7. it was dead.
== July 9 ==
* 11:23 mark: IGMP snooping is broken on csw1-sdtpa
* 10:33 mark: Disabled PIM, multicast routing on csw5-pmtpa, and enabled IGMP snooping on all VLANs
* 10:25 mark: Disabled PIM, multicast routing on csw1-sdtpa, and enabled IGMP snooping on all VLANs
* 10:12 mark: Removed static routes on csw1-sdtpa, set default route towards csw5-pmtpa
* 10:12 mark: Shutdown BGP sessions on csw1-sdtpa
* 02:45 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 18511 remove hiderevision from oversight group'
* 02:19 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 27706'
* 02:18 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27706'
* 02:18 logmsgbot: reedy synchronized php-1.17/extensions/RSS/RSSHooks.php 'r91775'
* 02:17 logmsgbot: reedy synchronized php-1.17/extensions/RSS/RSS.php 'r91775'
* 02:17 logmsgbot: reedy synchronized php-1.17/extensions/RSS/RSS.i18n.php 'r91775'
* 02:06 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 28140'
* 01:51 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 27078'
* 01:50 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27078'
* 01:44 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'Testing showhiddencats for commons'
* 01:43 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Testing showhiddencats for commons'
* 01:42 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Testing showhiddencats for commons'
* 01:40 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Testing showhiddencats for commons'
* 00:55 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'Lowering ptwikibooks account creation throttle again: event in bug 27172 is since over'
* 00:43 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Revert that'
* 00:43 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Testing showhiddencats for commons'
* 00:42 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Testing showhiddencats for commons'
* 00:40 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Revert that'
* 00:39 Reedy: bug 22689 Commons __HIDDENCAT__ (change the default setting for logged-in users)
* 00:38 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Testing showhiddencats for commons'
* 00:28 Reedy: running namespacedupes on eowikisource finished
* 00:28 Reedy: running namespacedupes on eowikisource
* 00:28 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28290 Add eowikisource NSs'
* 00:23 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28290 Add eowikisource NSs'
* 00:20 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29641 Appendix namespace for zhwikt'
== July 8 ==
* 23:40 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Revert bug 28686 per request'
* 23:32 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'bug 29483 add some extra abusefilter lines'
* 23:23 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Actually sync my brwikimedia changes, turns out I pushed Reedy's changes for bug 29627 (dewiki). Yay conflict resolution'
* 23:18 Reedy: bug 29267 Namespace aliases for dewiki pushed in Chads last push
* 23:15 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28686: new namespaces for brwikimedia'
* 23:03 Reedy: run namespacedupes on mkwikisource
* 23:02 Reedy: run namespacedupes on rowikisource
* 22:54 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29190 NSs for rowikisource'
* 22:40 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'revert logo, incorrect size'
* 22:39 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29557, fix sah wikisource logo'
* 22:32 Reedy: make that ukwiki
* 22:32 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29740 Enable rollbacker on viwiki'
* 22:29 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29717 Enable rollbacker on viwiki'
* 22:07 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29166 Author NS for mkws'
* 21:59 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28860 botadmin for frwikt'
* 21:42 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28221 update rowikt namespaces'
* 21:32 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27874'
* 21:30 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27874'
* 21:27 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27874 enable patrol'
* 21:24 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27874, allow mlwiki sysop add/remove patroller, autopatroller, and rollbacker'
* 21:22 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27874, allow mlwiki sysop add/remove patroller, autopatroller, and rollbacker'
* 21:16 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28192 patrol stuff for rowiki'
* 20:48 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29702'
* 19:45 logmsgbot: reedy synchronized php-1.17/extensions/Collection/Collection.php 'r91753'
* 19:32 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'kill purgethumbnail logging'
* 19:31 logmsgbot: reedy synchronized php-1.17/includes/filerepo/LocalFile.php 'r91751'
* 19:11 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable wikilove on mediawikiwiki'
* 18:57 logmsgbot: reedy synchronized php-1.17/includes/filerepo/LocalFile.php 'add success logging'
* 18:51 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php '[purgeThumbnails] = udp://10.0.5.8:8420/purgethumbs'
* 18:44 logmsgbot: reedy synchronized php-1.17/includes/filerepo/LocalFile.php 'r91746 debugging for testing thumbnail purging'
* 17:47 mark: Migrated routing to management gateway from csw1-sdtpa to cr1-sdtpa:ae0.402
* 17:34 mark: Moved transport link between csw1-sdtpa and cr1-eqiad to cr1-sdtpa:xe-0/0/1
* 17:06 RobH: All ops should be watching eqiad temps and humidity closely over the next few hours/days as the airflow is adjusted to lower relative humidity to acceptable ranges
* 16:44 mark: Migration completed
* 16:40 mark: Migrated routing of vlan 100 (squid+lvs) from csw1-sdtpa to cr1-sdtpa
* 16:28 mark: Migrated routing of vlan 2 (private) from csw1-sdtpa to cr1-sdtpa
* 16:16 mark: Migrated routing of vlan 101 (public services) from csw1-sdtpa to cr1-sdtpa
* 16:13 RobH: srv278 shutdown
* 16:10 mark: Migrated routing of vlan 102 (sandbox) from csw1-sdtpa to cr1-sdtpa
* 15:51 mark: Migrated routing of vlan 103 (virt) and vlan 104 (pub-svc2) from csw1-sdtpa to cr1-sdtpa
* 15:30 mark: Migrated routing of vlan 105 (virt-hosts) from csw1-sdtpa to cr1-sdtpa
* 15:29 mark: Moved transport link between csw1-sdtpa to cr2-eqiad, to cr1-sdtpa port xe-1/1/0
* 15:01 RobH: db1023 drac fixed, had typo in ip allocation, its ready for os install, rt#1105
* 14:42 RobH: confirmed that high humidity readings in eqiad are valid, reopened ticket with eq support
* 14:35 mark: Powercycled asw-a5-sdtpa
* 07:25 Tim: removed the part of scap that copied skins to /mnt/upload6. This is no longer required, I added a redirect to bits instead.
* 07:17 Tim: adding redirect from http://upload.wikimedia.org/skins/ to http://bits.wikimedia.org/skins/ using ms7 obj.conf
* 07:14 Tim: adding http://bits.wikimedia.org/skins/ alias to the latest version of skins, for backwards-compatibility with hot linkers
* 06:26 apergos: updated snapshot hosts and search hosts to use sudoer file that allows non root syncs 
* 06:25 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 91659: non-root scap test
* 06:21 logmsgbot: tstarling synchronized php-1.17/README 'non-root sync-file test'
* 05:58 logmsgbot: tstarling synchronized php-1.17/README 'non-root sync-file test'
* 05:57 logmsgbot: tstarling synchronized php-1.17/README 'non-root sync-file test'
* 03:56 Tim: in puppet, removed the subscribe from the rsyncd service for /home, since the init script is buggy and causes the service to go down (bind() failed: Address already in use). Rsyncd doesn't need it anyway, it re-reads the configuration file at each request.
* 03:34 Tim: on nfs1: changed rsyncd.conf to allow client IPs from a subnet that includes mobile1, since it looks like patrick tried to set up mediawiki on it. Ran sync-common on mobile1.
* 03:00 Tim: deleting and recreating /usr/local/apache/common-local on all servers, to get mwdeploy ownership
* 02:39 Tim: updated sudoers.appserver and /h/w/b/sync-common-file to support the sync script changes
* 02:33 Tim: updated wikimedia-task-appserver manually on searchidx1, searchidx2, fenari, hume. Manually removed the php.ini-dist diversion on searchidx1, see CR r76095
* 02:00 Tim: built wikimedia-task-appserver 2.2 for lucid, uploading to brewster, since we have some lucid boxes in production
* 01:35 Tim: uploading wikimedia-task-appserver 1.48 to brewster. Waiting for puppet to immediately wreak havoc in response.
* 01:25 Tim: re-enabled apache on srv255. Note that sync-file pushes to it will fail until I finish updating everything
* 01:17 Tim: did a test install of wikimedia-task-appserver version 1.48 on srv255
* 01:06 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28221 rowikt sitename'
* 00:56 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29217 Add Portal namespace to Azerbaijani Wikisource'
* 00:40 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29381 Change title of Kannada wiktionary'
* 00:34 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'revert'
* 00:34 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29381 Change title of Kannada wiktionary'
* 00:30 logmsgbot: reedy synchronized closed.dblist 
* 00:22 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'fixing sitemap for fawikinews'
* 00:21 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'fixing sitemap for fawikinews'
== July 7 ==
* 23:26 Tim: temporarily stopping apache on srv255 for sync script testing
* 22:59 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 26732'
* 22:29 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'bug 29483'
* 21:45 Ryan_Lane: changing the spammer blocks from blocking post to blocking all requests
* 21:22 Ryan_Lane: dropping the weight of ms1004 to 5
* 21:15 Ryan_Lane: upping weight of sq79 and sq82
* 21:14 Ryan_Lane: repooling sq79 sq82
* 20:59 Ryan_Lane: depooling sq79 and sq82 in lvs
* 20:47 Ryan_Lane: lowering carp weight of sq79 and 82
* 20:45 Ryan_Lane: repooling sq79 and sq82 in lvs
* 20:44 Ryan_Lane: temporarily depooling sq79 and sq82
* 20:00 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29360'
* 18:47 Ryan_Lane: adding another IP to that search block
* 18:38 Ryan_Lane: deploying new squid conf with a blocked ip, to block a search spammer
* 18:04 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'More GNSM: had to set comment namespace and fallback category'
* 17:59 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'GNSM for fawikinews: bug 29563'
* 17:52 Ryan_Lane: that's powering
* 17:52 Ryan_Lane: powing off srv154
* 17:51 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap.alias.php 
* 16:39 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27840, moving wiki'
* 15:45 mark: Established full iBGP mesh between cr1-sdtpa, csw5-pmtpa, cr1-eqiad, cr2-eqiad (NOT csw1-sdtpa)
* 00:25 awjr: disabling fundraising-related squid log filtering for udp2log
== July 6 ==
* 23:45 Ryan_Lane: powercycling srv281
* 23:44 Ryan_Lane: powercycling srv266
* 23:44 Ryan_Lane: powercycling srv217
* 23:43 Ryan_Lane: powercycling srv206
* 23:42 Ryan_Lane: powercycling srv154
* 23:40 Ryan_Lane: powercycling srv276, it's dead
* 22:56 mark: Setup cr1-sdtpa with initial config; connected to csw5-pmtpa (via L2 csw1-sdtpa); OSPF up
* 22:36 RobH: wmf_ops: you can disregard the humidity alarms for eqiad that are spamming alerts to email. eq confirms no humidity issue on site and I will investigate the actual sensors this friday
* 22:21 Ryan_Lane: repooling srv169, it was missing the wikimedia-lvs-realserver package. fixed in puppet
* 22:04 Ryan_Lane: rebooting srv169
* 21:59 Ryan_Lane: depooling srv169
* 21:59 Ryan_Lane: sync'd srv169, repooling to test
* 21:54 Ryan_Lane: depooling srv169
* 21:51 Ryan_Lane: restarted apache on singer
* 21:46 logmsgbot: laner synchronized php-1.17/wmf-config/db.php 'depooling srv154, repooling srv178'
* 21:30 logmsgbot: pdhanda ran sync-common-all 'Synced to r91606 for ArticleFeedback'
* 21:27 Ryan_Lane: setting proxy setting for secure back to original setting. removed ~ files from sites-enabled
* 21:17 Ryan_Lane: putting retry=3 back, and adding a timeout of 15 seconds to secure
* 21:16 Ryan_Lane: removed retry=3 from ProxyPass directive for secure. 3 seconds really isn't enough for this service...
* 21:06 RobH: running puppet on spence, this is going to take forever.
* 21:05 Ryan_Lane: restarting apache on singer
* 19:37 mark: Added DNS entries for cr1-sdtpa and cr2-pmtpa
* 19:25 logmsgbot: hashar: hexmode raised an user issue with blocking. It is a lock wait timeout happening from time to time on enwiki. 30 occurences in dberror.log for Block::purgeExpired. Could not reproduce it so I am assuming it was temporary issue.
* 19:15 logmsgbot: hashar: srv154 seems unreachable. dberror.log is spammed with "Error connecting to <srv154 IP>"
* 19:13 RobH: added webmaster@ to other top level domain mail routing to forward to the wikimedia.org webmaster for google securebrowsing stuff per RT#1122
* 18:08 pdhanda: running maintenance/cleanupTitles.php on commonswiki
* 17:51 pdhanda: Running maintenances/namespaceDupesWT.php on commonswiki
* 17:12 RobH: srv169 successfully back in service, tests fine and has all updated files, lvs3 updated to include it in pool
* 17:11 RobH: returning srv169 into service
* 15:37 mark: Removed ms5:/etc/cron.d/mdadm
* 15:37 mark: Stopped MD raid resync on ms5
* 15:28 RobH: search18 booted back up successfully
* 15:25 RobH: api lag issues known due to search server failure, being worked presently
* 15:24 RobH: search18 sas configuration bios confirms both disks are still in a non-degraded (according to it) mirror
* 15:23 RobH: search18 randomly rebooted after checking disks before the login prompt
* 15:19 RobH: rebooting search18
* 15:14 RobH: search18 appears to be completely offline, investigating lom logs before rebooting.
* 15:12 RobH: search18 offline, logging into mgmt to check it out
* 15:01 RobH: eqiad humidity levels ticket dispatched for fufillment
* 14:37 mark: Paused rsyncs on ms5
* 14:04 mark: Powercycled sq36
* 13:18 ^demon|away: fixed permissions on svn c/o on ci.tesla, ran svn cleanup. cruise control still not pleased and yelling about locks
* 13:16 mark: Upgrading firmware of scs-a1-sdtpa
* 12:51 mark: csw5-pmtpa crashed and reloaded
* 11:53 mark: Upgrading firmware of scs-c1-pmtpa
== July 5 ==
* 23:53 ^demon: scratch that....ssh just seems to have been rather slow in getting its act together. ci.tesla is just fine now
* 23:50 ^demon: well now I've locked myself out of ci.tesla. Seems it doesn't start ssh on reboot...what a silly thing to do
* 23:47 ^demon: rebooting ci.tesla since it was horribly hung up on the latest build--was it really stuck for the past 24hrs?
* 23:35 logmsgbot: reedy synchronized php-1.17/resources/mediawiki/mediawiki.js 'r91505'
* 21:28 logmsgbot: pdhanda ran sync-common-all 'Synced to r91494 for WikiLove'
* 21:10 logmsgbot: pdhanda synchronized php-1.17/resources/jquery.ui/themes/vector/jquery.ui.button.css 'Synced to r 91493.'
* 21:00 logmsgbot: pdhanda synchronized php-1.17/resources/jquery.ui/themes/vector/jquery.ui.button.css 'Synced to r 91490.'
* 19:53 logmsgbot: reedy synchronized php-1.17/includes/api/ApiQuery.php 'r91479'
* 19:07 logmsgbot: catrope synchronized php-1.17/extensions/Vector/modules/ext.vector.collapsibleTabs.js 'r91476'
* 19:07 logmsgbot: catrope synchronized php-1.17/extensions/Vector/modules/ext.vector.simpleSearch.js 'r91476'
* 16:49 RoanKattouw: Short CPU spike on the Apaches, approx 16:45-16:50 UTC. Things seem to be recovering now
* 16:04 mark: Put OSPF/OSPFv3 on csw1-sdtpa:e14/1 in active mode again; appears stable so far
* 15:45 mark: Reenabled csw1-sdtpa:e14/1 with OSPF/OSPFv3 in passive mode and high metric
* 15:31 mark: Shutdown csw1-sdtpa:e14/1 (l3 wave)
* 15:24 mark: Disabled IP load-sharing on csw1-esams (out of next-hops), disabled e14/1 (10G wave L3), reenabled it
* 13:38 mark: Rebooting streber; dpkg hung
* 08:34 mark: Rerouted traffic AS14907 -> AS43821 via 2828
* 07:11 Tim: creating 256 tables on db40 for DBOC
* 07:08 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'config change for the next DBOC trial, unused at present'
* 07:04 logmsgbot: tstarling synchronized php-1.17/includes/objectcache/SqlBagOStuff.php 'r91442: MySQL parser cache fix, ready for a new trial'
* 06:54 logmsgbot: root synchronizing Wikimedia installation... Revision: 91441: 
* 06:53 logmsgbot: tstarling synchronized php-1.17/includes/AutoLoader.php 
* 06:53 logmsgbot: tstarling synchronized php-1.17/includes/libs/IEUrlExtension.php 
* 06:48 Tim: doing svn up/scap to get r91440
== July 4 ==
* 19:29 mark: Restarted lsearchd on search7
* 14:02 apergos: restarted morebots
* 13:51 mark: Powercycled ms1004
* 12:52 mark: Started rsync of upload images on ms7 to ms1002 (eqiad) in a screen on ms1002
* 12:35 mark: OS-installed ms1002
* 08:43 morebots died because of netsplit
* 04:58 Tim: setting up lighttpd mod_auth on lily
== July 1 ==
* 20:18 logmsgbot: hashar: ci.tesla: installed exiv2 (required for some Api tests)
* 20:13 andrewS: mw1-mw22 racked and drac setup
* 19:49 RobH: typo in my number sequence update for the new 74 mw servers, updating dns to fix
* 19:09 Ryan_Lane: Changing text backend for https in esams to textsvc IP instead of text.esams.wikimedia.org IP
* 18:49 Ryan_Lane: restarting pybal on amslvs1 and amslvs3
* 18:34 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'removing mysql from pcache, need to repartition, current scheme too costly on memory'
* 18:25 Ryan_Lane: adding textsvc.esams IP to amssq and knsq, and amslvs servers
* 18:23 logmsgbot: hashar: ci.tesla install wbritish-small package to get a word list in /usr/share/dict
* 18:08 Ryan_Lane: restarting pybal on lvs2 to add textsvc service
* 17:40 andrewS: Rob Halsell is a Wizard. Period.
* 17:39 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 17:29 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'adding enwiki to the mix'
* 17:27 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'sending a bit lower cache load to db40, though I guess that was initial shock before'
* 17:25 RobH: Aware that #wikipedia is down and fixed, in less than a minute, woot
* 17:25 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'oops, should have started with lower load, opentable cache..'
* 17:23 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'sending cache load again to db40'
* 17:16 Ryan_Lane: adding text.svc address to pmtpa text squids and lvs2
* 17:08 Ryan_Lane: adding text.svc private address to esams and pmtpa DNS
* 14:56 RobH: pdns crash after rehup on ns1.wikimedia.org, restarting pdns
* 14:49 RobH: dns updated to remove the mgmt ip addresses for srv302+ and replace them with mw1-mw74
* 14:33 RobH: updating dns, removing entries for non-existent srv302+ and replacing them with mw1-mw74
* 11:18 andrewS: motion sensors are being installed today in sdtpa that are going to trip the lights when there is movement. not sure how this will affect cameras. will update once their work is done later today.
* 07:33 andrewS: the 3 new racks in pmtpa have power plugs running to them now.
* 05:51 apergos: cron job set up on snapshot1 to dump central auth tables (in subdir by date just like all the private wikis) on weekly basis, runs as backup user
* 00:01 RobH: disabled the notify requesters on resolve in the hw-decommission queue in RT