Server Admin Log/Archive 18

:This page archive server admin logs from January 1st 2011 till June 30th 2011
== June 30 ==
* 20:43 logmsgbot: awjrichards synchronized php-1.17/extensions/CentralNotice/special/SpecialBannerLoader.php 'r91218'
* 20:42 logmsgbot: awjrichards synchronized php-1.17/extensions/CentralNotice/special/SpecialBannerController.php 'r91218'
* 18:39 RoanKattouw: Running extensions/Vector/switchExperimentPrefs.php on enwiki to disable WikiLove for those with the "Exclude from experiments" preference
* 18:38 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikiLove by default on enwiki'
* 18:28 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikiLove on enwiki, non-default initially'
* 18:18 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 91190: 
* 18:17 RoanKattouw: Deploying updates for ArticleFeedback and WikiLove
* 18:09 RobH: srv169 still offline, do not repool (set to false in pybal on lvs3)
* 18:00 RobH: doing reboot testing on srv169 and puppet testing. server is depooled, disregard notices.
* 17:56 RobH: depooled srv169, its not updating its /var/local/apache via puppet for some reason.
* 17:53 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Add code enabling WikiLove by default if $wmgWikiLoveDefault is true'
* 17:52 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add wmgWikiLoveDefault variable, set to false'
* 17:48 RobH: blog.wikimedia.org updated to wp3.1.4
* 16:48 RobH: srv169 reinstalled and returned to service. (lvs3 is not repooling, citing a key error that has since been removed, perhaps pybal nees to be refreshed?)
* 16:25 RobH: reinstalling srv169
* 10:57 mark: Changed preferred path for AS14907 (pmtpa/eqiad) - AS43821 (esams) from 14907 2828 6908 43821 to 14907 3257 1299 43821 (bidirectionally)
== June 29 ==
* 23:39 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 91090: 
* 23:38 awjr: running scap to deploy CentralNotice changes (r91118)
* 23:25 logmsgbot: awjrichards synchronized php-1.17/extensions/CentralNotice/centralnotice.css 'r91118'
* 23:24 logmsgbot: awjrichards synchronized php-1.17/extensions/CentralNotice/centralnotice.js 'r91118'
* 23:03 RobH: db1001-db1048 (minus a few hosts mentioned in rt#1105) are installed, puppetized, and ready to be placed into replication service.
* 22:53 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Making CentralNotice config changes for test wiki (page path and CentralDBname)'
* 21:03 RobH: OS install running on db1033-db1048
* 20:53 RobH: db1033-db1048 raid setup, ready for OS installs
* 20:44 mark: Started rsync of ms5 commons thumb dir '1' to ms1004
* 20:23 RobH: working on db1033-db1048 (setting up hardware raid)
* 20:10 RobH: db1017-db1031 os installed, puppet updated, ready for replication setup (db1023 & db1032 need further troubleshooting)
* 19:54 RobH: os install on db1017-db1031 done, running puppet updates
* 19:45 mark: Started rsync of ms5 commons thumb dir '0' to ms1004
* 19:36 mark: Deployed ms1004 (eqiad) as caching thumbs server in a similar configuration as ms6.esams; requesting thumbs from ms5 and caching (and purging) them locally
* 19:35 RobH: db1023 and db1032 drac unresponsive, rt# 1111 abd 1112 to troubleshoot next onsite
* 19:23 RobH: rob is working installs on db1017-db1032
* 19:08 RobH: updated netboot.cfg for private1-b-eqiad broadcast
* 18:46 RobH: raid10 setup on db1017-db1027
* 18:30 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 91090: 
* 18:27 RoanKattouw: Deploying r91089 (jQuery in the head and position=top support) with scap
* 18:11 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikiLove on testwiki'
* 17:58 mark: OS-installed ms1004
* 17:33 RobH: os install running on db1017-db1021
* 17:22 mark: Deployed DNS entries for ms1001-1004
* 17:20 RobH: db1015 and db1016 os installed, puppet updated, ready to go into replication service
* 17:20 RobH: corrected dns, pushed db1017 and upwards into eqiad-b-private ip range
* 17:18 mark: Caught ms5 doing RAID10 resync check, stopped it using echo idle > /sys/block/md1/md/sync_action
* 16:43 RobH: updating dns for the all eqiad dbs
* 16:36 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'In the spirit of bug 29277: also set scriptDirUrl for $wgLocalFileRepo'
* 15:49 RobH: starting install on db1015-db1018
* 15:45 RobH: db1011, db1012, db1013, db1014 os installed, puppet updated, ready to go into replication service
* 14:49 RobH: installing db1011, db1012, db1013, & db1014
* 14:40 RobH: db1007, db1008, db1009, & db1010 os installed, puppet updated, ready to go into replication service
* 14:32 RobH: stat1.wikimedia.org server online for Erik Z
* 13:46 RobH: db1007, db1008, db1009, db1010 OS installed, needs puppet run
* 13:42 RobH: starting os install on stat1, db1010 and db1009. mislogged before, was finishing install on db1007 and db1008
* 13:39 RobH: updated dns with stat1 server info
* 13:31 RobH: db1008 & db1009 finishing OS install
* 01:50 RobH: db1005 & db1006 os installed, puppet updated, ready to go into replication service
* 00:42 logmsgbot: demon synchronizing Wikimedia installation... Revision: 91036: Pushing r91036 for Filepage.css fixes
== June 28 ==
* 23:29 RobH: db1003 and db1004 os installed, puppet updated, ready to be put into replication service
* 22:46 RobH: db1001 installed and updated with puppet, ready to be put into replication service
* 22:45 RobH: db1002 installed and updated with puppet, ready to be put into replication service
* 22:44 RobH: db1001 is installed with OS and up (has been for awhile), db1002 and db1003 finishing installs
* 22:24 RobH: carbon host online as install tftp host in eqiad.
* 22:10 RobH: syncing docroot and gracefully restarting apaches to take the update
* 22:01 RobH: carbon rebooting
* 21:43 RobH: testing tftp function of carbon from cp1001 as its on the same public vlan network
* 21:13 RobH: updated puppet to point tftp hosts in eqiad to carbon
* 21:13 RobH: rsyncing over the tftpboot directory from brewster to carbon
* 21:00 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28781. Change requested by jamesofur on irc'
* 20:51 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28781. This time without breaking the world'
* 20:33 RobH: reinstalling carbon
* 20:27 RobH: updated misc::install-server and brewster/carbon entires in puppet, install server roles are now called on a service by service bais. brewster also needs reboot for kernel update, so doing that now as well.
* 17:47 RobH: carbon host powered down, the misc::install-server class in puppet needs to be broken out into more specialized roles, carbon reinstalled and updated to just host the tftp roles of the install server
* 17:36 RobH: new install host carbon os installed, running puppet updates
* 17:11 RobH: copied db41 sudoers file to db40 since it was overwritten by old raid utils package. now maybe the log spam will stop
* 15:20 RoanKattouw: NTP has synced clock in prototype, it's got the correct time now
* 15:18 RoanKattouw: Installed ntp on prototype, its clock was 10 minutes behind. It seems to be slowly catching up now
* 14:58 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling while my eyes are on other stuff'
* 14:52 RobH: stat1 racked and remote accessible. has public vlan setup, needs IP allocated and os installed. (also needs to be added to racktables)
* 13:59 mark: Partly implemented European routing policy on AS14907 as well, and tuned a bit for routing southern routes via Tampa
* 13:27 logmsgbot midom cleared profiling data
* 13:27 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php
* 13:25 logmsgbot midom cleared profiling data
* 13:24 logmsgbot midom cleared profiling data 
* 13:20 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'one more db40 testing round'
* 13:19 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'one more db40 testing round'
* 13:13 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'one more db40 testing round ended'
* 13:12 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'one more db40 testing round'
* 11:35 Nemo_bis: !Wikipedia and other !Wikimedia sites problems at 6:15 and 9:30 UTC due to database overload fixed in 30 min, config now tweaked.
* 10:41 mark: Set 'send-community' on the iBGP peer groups on csw1-sdtpa and csw5-pmtpa
* 10:41 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 10:39 domas: reenabled full mysql pcache, watching with half of my right eye
* 10:39 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 10:38 domas: reduced max_connections on db40 to 1000, enabled "connection recycling" (disconnects longest idling connections when max_connections gets reached), set wait_timeout=2
* 10:30 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'adding enwiki to the mix'
* 10:23 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling db-pcache on eswiki for transaction analysis'
* 09:52 logmsgbot: mark synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable MySQL parser cache'
* 09:22 mark: Activated BGP sessions to TiNet / AS3257
* 06:49 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled SubPageList3 on wikiversity'
* 06:45 logmsgbot: tstarling synchronized php-1.17/extensions/SubPageList3/SubPageList3.php 'r90659, fix fatal error'
* 06:31 logmsgbot: tstarling synchronized php-1.17/includes/objectcache/SqlBagOStuff.php 'r90942, proper fix'
* 06:21 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabling MySQL parser cache'
* 06:20 logmsgbot: tstarling synchronized php-1.17/includes/objectcache/SqlBagOStuff.php 
* 06:16 Tim: temporarily disabled db40 parser cache due to connection errors
* 06:15 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 06:14 logmsgbot: root restarted all apaches
* 06:14 logmsgbot: root is doing a hard restart of all apaches (apaches overload)
* 00:31 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling MySQL parser cache on all wikis'
* 00:09 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling MySQL parser cache on enwiki'
== June 27 ==
* 23:22 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'no automatic purging'
* 23:15 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling MySQL parser cache on en.wikiquote'
* 21:54 Ryan_Lane: repooling sq69
* 21:09 RobH: grosley back online
* 21:00 RobH: grosley reboot in process, disk checking.
* 20:57 RobH: powercycling grosley
* 20:56 RobH: grossley entirely unresponsive from serial console
* 20:51 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 29603 - enabled abusefilter for iawiki. Bug 29279 - turned off Devanagari digits for hiwiki'
* 20:36 logmsgbot: pdhanda ran sync-common-all 'Enabled abusefilter for iawiki. Bug 29603'
* 20:18 RobH: allocated 11/41 on csw1-sdtpa to vlan101 for use by stat1
* 19:32 RobH: raised power warnign threshhold on phases on ps1-d1-sdtpa from 12 to 16
* 18:13 RobH: rebooting carbon for installation in eqiad.
* 18:06 RobH: updating dns to have carbon mgmt dns info
* 17:42 andrewS: 47 r410 servers now await placement @ sdtpa
* 17:13 andrewS: 36 new dell servers sit outside sdtpa cage and await placement
* 16:40 Ryan_Lane: depooling sq69 for continued work on XFF for geoiplookup
* 15:30 JeLuF: increased puppet's timeout for "apt-get update" from 2 to 4 minutes. Regular apt-get update on storage1 takes more than 2 minutes
* 11:50 mark: Changed ganglia / gmond configuration in puppet to use a different, associative hash structure
* 03:06 Tim: changed mailman site password
== June 26 ==
* 13:13 rainman-sr: built search index for nycwikimedia
== June 25 ==
* 03:26 andrewS: 8 new servers in sdtpa d1 awaiting labels and power/data hookups
* 03:25 andrewS: sdtpa feels hotter than normal. will keep an eye on cage temp next few days.
* 00:30 Ryan_Lane: repooling sq69
== June 24 ==
* 22:26 notpeter: initialized and started repl between nfs1 and 2 for opendj
* 22:08 Ryan_Lane: depooling sq69 to test geoiplookup changes for https
* 21:36 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28529 - Change move-rootuserpages right in Portuguese Wikipedia'
* 21:32 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28398 - Custom namespaces for ku.wiktionary and ku.wikipedia'
* 21:24 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29527 - Enable AbuseFilter on Catalan Wikipedia'
* 21:23 logmsgbot: jeluf synchronized php-1.17/wmf-config/abusefilter.php '29527 - Enable AbuseFilter on Catalan Wikipedia'
* 20:00 andrewS: pmtpa: we should have power and the overhead railings to run data cables by late monday, as per Matt.
* 19:20 Ryan_Lane: depooled srv301
* 12:37 Tim-away: finished fixing ExtensionDistributor, re-enabled cron job and xinetd
* 07:37 Tim: XO appears to be back up, after ~3 hours of downtime
* 07:30 Tim: fixing ExtensionDistributor snapshot permissions, was owned by mark instead of extdist
* 06:21 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29168 - Define namespace aliases for Author, Page, Index on all Wikisources'
* 06:10 Tim: configured csw5-pmtpa to send its syslogs to 10.0.5.8 instead of 10.0.0.1
* 00:19 logmsgbot: tstarling synchronized php-1.17/includes/db/DatabaseMysql.php 'reapplied r90644, it was fixing an actual problem'
== June 23 ==
* 22:36 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 14407. Added Login icon for incubatorwiki'
* 22:33 logmsgbot: pdhanda synchronized images/sul/incubatorwiki.png 'Bug 14407. Login icon for incubatorwiki'
* 22:18 logmsgbot: pdhanda synchronized php-1.17/wmf-config/CommonSettings.php 'Bug 14407. Added incubatorwiki to CentralAuthAutoLoginWikis'
* 21:30 logmsgbot: catrope synchronized php-1.17/maintenance/update.php 'Sync out update.php too, just because Ariel is being paranoid about it :D'
* 21:27 ^demon: svn up'd /h/w/c/p/m/update.php for nice warning about not running update.php on the cluster
* 21:06 logmsgbot: pdhanda ran sync-common-all 'Updating to r90676 of extensions/UploadWizard'
* 21:03 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Enabling wgNoticeInfrastructure for testwiki per request by Kaldari'
* 20:36 pdhanda: updated to r90667 of extensions/Narayam
* 20:33 logmsgbot: pdhanda ran sync-common-all 
* 18:38 JeLuF: #21237 - removed brion's email address from apache config
* 18:10 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28909 - Change path to Low Saxon Wiktionary logo'
* 18:04 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29391 - Enable e-mail notification for watchlists on br.wikimedia'
* 18:03 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29348 - Email notifications for stewardwiki'
* 18:00 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29360 - Define $wgUploadMissingFileUrl for ptwikibooks'
* 17:02 binasher: sync-file r90664 includes/db/LoadMonitor.php
* 17:01 logmsgbot: root synchronized php-1.17/includes/db/LoadMonitor.php 'max threads logging'
* 16:47 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 'attempting to set max threads on testwiki only'
* 16:32 RoanKattouw: Site came back up instantly after Tim disabled max_threads
* 16:28 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 'disabled max threads'
* 16:28 logmsgbot: catrope synchronized php-1.17/includes/db/LoadBalancer.php 'Make the "LB failure with no last connection" DB error log message more verbose'
* 16:24 binasher: upping max threads in db.php
* 16:24 logmsgbot: root synchronized php-1.17/wmf-config/db.php 'upping max threads'
* 16:17 logmsgbot: laner synchronized php-1.17/includes/db/DatabaseMysql.php 'Roll back Tim's getLag() change from 03:18 today (r90644)'
* 15:58 Ryan_Lane: restarting pybal on lvs3
* 15:52 logmsgbot: catrope synchronized php-1.17/includes/db/Database.php 'Roll back Tim's getLag() change from 03:18 today (r90644)'
* 15:50 RoanKattouw: Going to roll this back: 03:18 logmsgbot: tstarling synchronized php-1.17/includes/db/DatabaseMysql.php 'Database::getLag() fix from r90644' 
* 15:46 RoanKattouw: Lots of "LB failure with no last connection" errors in dberror.log, suddenly started at 10:53 UTC
* 15:43 RoanKattouw: Torrus broken, reporting lots of NaNs
* 15:40 RoanKattouw: Strike that, site not completely down. Intermittent service reported in US/Canada
* 15:39 RoanKattouw: AMS squids all down in Nagios. Load is approximately halved on app servers. TPA squids look like they should be fine, but the site is reported to be down worldwide
* 15:37 RoanKattouw: Mark not near a computer, texting Ryan instead
* 15:36 RoanKattouw: All wikis seem to be down worldwide. Have texted mark 
* 09:18 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29215 - Add import source for dearbcomwiki'
* 09:14 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29380 - Change logo of Kannada wiktionary (kn.wiktionary.org)'
* 08:48 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29515 - install Narayam in all Sanskrit wiki projects'
* 08:42 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29474 - Search Index and Author namespaces by default on all Wikisources which don't have a different configuration'
* 08:27 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '29172 - Add Page, Author and Index namespaces to $wgContentNamespaces on all Wikisources'
* 07:53 JeLuF: fixed broken ganglia installation on project2 (reported by puppetmon)
* 07:51 JeLuF: added new class mobile::disabled in puppet that has apache disabled, assigned mobile1 to it.
* 07:19 logmsgbot: tstarling synchronized php-1.17/extensions/SubPageList3/SubPageList3.php 'r90654'
* 07:12 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'enabling SubPageList3 on testwiki'
* 07:11 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 07:07 logmsgbot: root synchronizing Wikimedia installation... Revision: 90493: 
* 07:05 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 90493: 
* 07:04 Tim: running scap to push out SubPageList3 source directory (not enabled)
* 06:36 Tim: killed puppet-monitor on fenari, I'm pretty sure that's the thing that was spamming #wikimedia-tech every 3 minutes
* 05:34 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled mysql parser cache on testwiki'
* 05:33 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'mysql parser cache pre-deploy changes'
* 05:32 Tim: created "parsercache" database on db40 and added suitable wikiuser/wikiadmin grants
* 04:46 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'put db33 back'
* 04:46 Tim: restarted the parser cache hit rate monitor
* 03:20 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 'restored max lag to 30s, now that r90644 is deployed it shouldn't break anything'
* 03:18 logmsgbot: tstarling synchronized php-1.17/includes/db/DatabaseMysql.php 'Database::getLag() fix from r90644'
* 03:03 Tim: started ntpd on db36, not sure why it was stopped, it has an init script. Clock skew was 1.7 seconds.
* 02:55 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 'increasing max lag to 3000'
* 00:43 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'removed db33 for TS dump'
== June 22 ==
* 23:08 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabled extension NewUserMessage on rowikisource. Bug 28307.'
* 22:47 Ryan_Lane: repooling sq69
* 22:07 Ryan_Lane: depooling sq69 to test vcl changes for https geoiplookup
* 21:23 notpeter: resized db10:/a to 220G
* 19:10 Reedy: Wikimedia repo post commit hook fixed for CR
* 18:43 Reedy: Wikimedia repo set up as a CodeReview repo on mw.org
* 18:04 JeLuF: fixed puppetmon, log format has changed
* 13:29 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'put db21 back'
* 10:41 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'put db34 back'
* 02:14 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'remove db34 to dump for TS'
* 02:08 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'remove db21 to dump s6 for TS'
* 01:59 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'put db18 back'
== June 21 ==
* 23:33 logmsgbot: pdhanda ran sync-common-all 'Creating nyc.wikimedia.org Bug 29273'
* 23:29 pdhanda: creating nyc.wikimedia.org
* 22:41 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/SpecialArticleFeedback.php 'Attempt to fix fatal'
* 22:14 binasher: pulling srv301 from api lvs pool to prep mobile testing
* 21:40 binasher: returned mobile4 to lvs
* 20:32 logmsgbot: preilly synchronized php-1.17/extensions/WikimediaMobile/redirector_test.js 'r90541'
* 20:32 logmsgbot: preilly synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 'r90541'
* 20:31 logmsgbot: preilly synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.i18n.php 'r90541'
* 20:31 logmsgbot: preilly synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js 'r90541'
* 19:25 logmsgbot: kate synchronized php-1.17/wmf-config/db.php 'taking db18 out of rotation to dump s7 for TS'
* 19:26 river: taking db18 out of rotation to dump s7 for TS
* 18:56 Ryan_Lane: moving preilly from restricted to mortals, for deploy access
* 15:01 apergos: deleted the old ssh key for snap2.wm.o from the puppet db also, as fenari was whining about it similarly, hope that gets is. spence puppet run *finally* done, sheesh
* 13:17 apergos: after changing snapshot2 from external to internal subnet (oops forgot to log that huh), puppet on spence was complaining about the external resource Nagios_host[snapshot2] not being able to override the local resource, so in puppet db on db9 I deleted the old snapshot2 nagios resources, leaving only the new ones, puppet run is in process now on spence and is bleeping sloooow.
* 09:14 apergos: taking down snapshot2 in prep for os upgrade (reinstall)
* 01:52 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Fix checkuserwiki permissions back to how they were'
* 00:49 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Permission changes for checkuserwiki. Bug 28781.'
== June 20 ==
* 23:57 binasher: reconfiguring mobile1 for testing the new php mobile mediawiki extension
* 23:03 notpeter: added checks for google safe browsing for lots of MWF sites to nagios
* 22:53 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Missed Comment 1 in bug 28785.'
* 22:42 logmsgbot: reedy synchronized php-1.17/includes/api/ApiEditPage.php 'bug 29278 r90943'
* 21:57 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Changed language, sitename and namespace for checkuserwiki. Bug 28781.'
* 21:33 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Changed icons on checkuserwiki. Bug 28785.'
* 19:20 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r90480'
* 18:51 binasher: drop db29 (s6 master) weight to 0
* 18:51 logmsgbot: root synchronized php-1.17/wmf-config/db.php 'drop db29 weight to 0'
* 18:21 logmsgbot: root synchronized php-1.17/wmf-config/db.php 'adding db43 to s6 with a weight of 20'
* 18:21 binasher: increased db43 weight to 1500
* 18:07 binasher: just put db43 into rotation in the s6 cluster
* 18:04 logmsgbot: root synchronized php-1.17/wmf-config/db.php 'adding db43 to s6 with a weight of 20'
* 16:53 pdhanda: Upgrading wikitech.wikimedia.org to 1.17
* 15:43 apergos: started en pedia dumps run on snapshot 4, from root screen session as backup user. This will run as 32 jobs in parallel; we should keep a half an eye on the db in about 16-17 hours (previous runs had only 15 jobs going at once). 
* 14:18 mark: Configured TiNet transit on cr2-eqiad (BGP sessions deactivated)
* 14:18 mark: Configured border ACL filtering on the eqiad border
* 00:46 logmsgbot: tstarling synchronized php-1.17/includes/db/LoadBalancer.php 'r90423'
== June 17 ==
* 20:39 RobH: email is working again, but address in question is not filtered
* 20:38 RobH: pushed a small exim change to try to sort out a sender address, but it didnt work, rolled back changes
* 16:44 RobH: upgraded blog.wikimedia.org to wordpress 3.1.3
* 16:44 RobH: restarted morebots.
* 00:33 Ryan_Lane: restarting pybal on amslvs4
* 00:32 Ryan_Lane: restarting pybal on amslvs2
* 00:31 Ryan_Lane: restarting pybal on amslvs3
* 00:30 Ryan_Lane: restarting pybal on amslvs1
== June 16 ==
* 23:51 Ryan_Lane: restarting pybal on amslvs3
* 23:51 Ryan_Lane: restarting pybal on amslvs1
* 23:50 Ryan_Lane: adding uploadsvc and uploadsecure services to amslvs1 and amslvs3
* 22:25 Ryan_Lane: restarting pybal on lvs1
* 21:47 RobH: puppet now works on eqiad hosts.
* 21:38 Ryan_Lane: enabled https for bits. still needs varnish changes.
* 21:38 RobH: updating dns with puppet cname in eqiad to sockpuppet
* 21:37 RobH: dispatched gilman to peter for puppetize testing of grosley
* 21:21 RobH: dns update for puppetmaster cname in eqiad
* 21:13 RobH: updated dns with carbon install server and cp server ranges
* 21:10 mark: Added all eqiad subnets to dhcpd.conf
* 21:10 mark: Prepared puppet's base.pp for eqiad
* 21:05 Ryan_Lane: restarting pybal on amslvs3
* 21:03 Ryan_Lane: killing and starting pybal on amslvs1
* 21:03 Ryan_Lane: restarting pybal on amslvs1
* 21:01 Ryan_Lane: adding bitssvc address to amslvs systems
* 20:52 notpeter: restarting drbd replication on nfs2
* 20:52 RobH: db1001 is our first wmf proper installed host in eqiad, huzzah
* 20:44 mark: Added static route for 10.2.1.23/32 to lvs2 on csw1-sdtpa
* 20:09 mark: Added 10.64.0.0/12 to squid http_access on brewster
* 18:51 Ryan_Lane: bound bitssvc address to lvs2, restarting pybal
* 18:33 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 18:24 Ryan_Lane: restarting pybal on lvs2
* 18:24 Ryan_Lane: adding bitssvc and bitssecure services to pmtpa lvs
* 18:01 notpeter: reimaging nfs2
* 16:36 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 14:54 andrewS: srv169 plugged in and powered on
* 14:26 RobH: updating again, there are db1001-1016 in a single rack, doh
* 14:24 RobH: updating dns with new dbs in eqiad
* 12:00 RoanKattouw: Set $wgInstantCommons back to true on prototype
* 11:59 RoanKattouw: Enabled WikiLove on prototype
* 11:15 andrewS: will be working on srv278 later today
* 11:12 logmsgbot: root synchronized php-1.17/wmf-config/mc.php 
* 09:50 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'raise account creation throttle for el pedia workshop'
* 01:38 Tim: success, my SIP connection now works again
* 01:38 Tim: attempting to remove the SIP connection rate limit from sfoasterisk
== June 15 ==
* 21:32 awjr: killing epicly long running query in civicrm db on db9
* 21:28 logmsgbot: reedy synchronized php-1.17/wmf-config/codereview.php 'Couple more auto defers'
* 19:07 logmsgbot: catrope synchronized php-1.17/extensions/WikiLove/WikiLove.api.php 'r90138'
* 18:47 logmsgbot: catrope synchronized php-1.17/extensions/WikiLove/modules/ext.wikiLove/ext.wikiLove.core.js 'r90135'
* 18:45 logmsgbot: catrope synchronized php-1.17/extensions/WikiLove/WikiLove.hooks.php 'r90135'
* 18:18 logmsgbot: catrope synchronized php-1.17/resources/mediawiki.util/mediawiki.util.js 'r90128'
* 18:03 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikiLove on officewiki'
* 18:02 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Add switch for WikiLove'
* 18:01 mark: Upgraded msw1-eqiad to JUNOS 10.4R3.4
* 17:59 RoanKattouw: Adding WikiLove table on officewiki (extensions/WikiLove/patches/WikiLoveLog.sql)
* 17:58 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add $wmgUseWikiLove, false by default'
* 17:53 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 90124: 
* 17:52 RoanKattouw: Syncing WikiLove code. Code sync only, not enabled anywhere yet
* 16:19 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Revert raised account creation throttle for Lennart on testwiki'
* 12:37 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29351 Set timezone to America New_York for pcdwiki'
* 05:17 apergos: phase 2 of integrity checks on dump files (bz2s) running from screen on sn1 as root
== June 14 ==
* 19:41 RobH: chad is crazy it seems.
* 19:41 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29215: import sources for arbcom_dewiki'
* 19:40 logmsgbot: catrope synchronized php-1.17/README 'Test sync to confirm whether Chad is crazy'
* 18:29 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'bug 29331 - Set e-mail capture sender to improve@wikimedia.org'
* 17:50 notpeter: restarting udp2log on locke after upgrade
* 15:10 mark: Upgraded asw-a-eqiad to JUNOS 10.4R3.4
* 14:42 mark: Upgraded asw-b-eqiad to JUNOS 10.4R3.4 (jtac recommended version)
* 14:26 apergos: running a bunch of rudimentary integrity checks on xml dumps, out of screen as root on snapshot1
* 06:09 logmsgbot: tstarling synchronized php-1.17/includes/HistoryBlob.php 'r90036'
== June 13 ==
* 23:40 notpeter: taking down nfs-home on nfs2
* 23:03 Ryan_Lane: ran sync-docroot - how does this not get logged?
* 19:16 RobH: snapshot4 properly boots to serial console display on bootloader and os
* 18:31 RobH: snapshot4 rebooting to troubleshoot console redirection
* 13:27 apergos: snapshot4 installed and *almost* correct (stray apache started up but the rest looks ok)
== June 12 ==
* 19:47 JeLuF: Software RAID on fenari was broken, mirror sdb1 was failed. Re-added sdb1. If the error is persistent, sdb needs to be replaced.
* 19:19 JeLuF: cron.{hourly,daily,...} on hume was not running due to a broken record in /etc/crontab, so logrotate was not running
* 12:58 logmsgbot: ariel synchronized php-1.17/includes/HistoryBlob.php 'check for existence of mhash function, not extension (need for lucid php build) (take 2, after svn up :-P)'
* 12:51 logmsgbot: ariel synchronized php-1.17/includes/HistoryBlob.php 'check for existence of mhash function, not extension (need for lucid php build)'
== June 11 ==
* 07:37 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Move section disabling minordefault out of the if($wmgUseUsabilityInitiative) conditional and document it'
* 07:12 RoanKattouw: Running script to fix preferences of users that can't turn off minordefault on enwiki. See bug 24313 comment 55. Script is in maintenance/fixMinorPrefs.php
== June 10 ==
* 21:26 binasher: mobile2 goes natty
* 18:08 Ryan_Lane: added ip and dns entry for internproxy.wikimedia.org in the sandbox vlan
* 17:28 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'See whether this fixes bug 29314'
* 15:21 RobH: copying old data from ms4 for solaris specific info. do not disrupt ms4.
* 02:24 RobH: ms4 repaired and back online, ready for reinstall per ticket 885
* 01:15 Ryan_Lane: starting services on virt1, since iptables rules are in
== June 9 ==
* 23:54 binasher: upgrading mobile2 to maverick (pre natty)
* 23:04 Ryan_Lane: added preilly account as a restricted user. Added him to mobile1-5
* 22:30 binasher: upgrading mobile3 to natty ice
* 21:38 binasher: upgrading mobile3 to maverick (then natty)
* 20:02 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 89789: 
* 20:02 RoanKattouw: Running scap to deploy ArticleFeedback changes
* 18:59 binasher: mobile5 natty
* 18:42 binasher: mobile5 upgrade to maverick on the way to natty
* 15:28 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Raise account creation throttle for Lennart's IP on testwiki'
* 15:10 RobH: dns updated, all nameservers dig correctly.
* 15:10 RobH: pushing dns updates to add mgmt of all eqiad devices
* 01:22 Ryan_Lane: shutting off services on virt1 (including puppet) until I get a chance to firewall them properly
== June 8 ==
* 22:35 Ryan_Lane: rebooting virt1
* 21:58 hcatlin: deploying to mobile cluster
* 21:34 Ryan_Lane: changing controller.labs.wikimedia.org to a CNAME for virt1.wikimedia.org
* 21:27 mark: Started ircd on ekrem
* 21:20 mark: Moved routing of vlan 101 from csw1-sdtpa to csw5-pmtpa
* 21:06 binasher: changing pybal healthcheck url for mobile
* 20:47 Ryan_Lane: rebooting controller.labs.wikimedia.org
* 20:44 Ryan_Lane: restarting pdns on ns0, it's locked up
* 20:33 Ryan_Lane: make that deleting virt1.pmtpa.wmnet, and changing virt1.mgmt.pmtpa.wmnet to controller.labs.mgmt.pmtpa.wmnet
* 20:32 Ryan_Lane: changing virt1.pmtpa.wmnet to controller.labs.pmtpa.wmnet
* 20:27 Ryan_Lane: changing ip of controller.labs.wikimedia.org
* 20:17 mark: Rebooting ekrem
* 20:16 mark: Changed ekrem's ip
* 20:10 mark: Added 2nd ip to ekrem (208.80.152.178)
* 19:56 binasher: upgrade mobile4 to natty
* 19:53 mark: Shutdown ekrem's switch port and null routed the ip for a while
* 19:53 Ryan_Lane: deleting vmdev1 and vmdev2, and moving the controller.labs.wikimedia.org entry into the public services subnet
* 19:45 binasher: upgrading mobile4 to maverick on its way to natty
* 19:31 Ryan_Lane: reclaiming sandbox ips for owa.tesla, owa1.tesla, owa2.tesla, lvs1.tesla, and grid.tesla
* 18:10 Ryan_Lane: depooling mobile5
* 18:05 Ryan_Lane: changing weight for mobile5
* 17:42 Ryan_Lane: upping the weight of mobile5 to 24
* 17:27 Ryan_Lane: upping weight of mobile5 to 16
* 17:18 Ryan_Lane: depooling mobile4
* 17:16 Ryan_Lane: reducing weight of mobile4/5 to 8
* 17:13 Ryan_Lane: repooling mobile4/5 with a weight of 16
* 16:29 ^demon|away: irc feeds are down, somebody who has access should investigate.
* 15:39 mark: Rebooting sq71 to remove old temp eqiad tunnel config
* 14:49 mark: Setup PIM-SM on eqiad core routers and joined the pmtpa domain
* 14:49 mark: Changed pmtpa PIM-DM to PIM-SM
* 11:31 mark: Added A records for cr1-eqiad and cr2-eqiad loopback interfaces
* 10:47 mark: Set local-pref value for private peers on cr1-eqiad and cr2-eqiad
* 10:43 mark: Removed cr1-eqiad:ae1.666 configuration to remove the temporary tunneled link
* 10:39 mark: Configured OSPF and OSPFv3 on csw1-sdtpa:e16/1 and cr2-eqiad:xe-5/2/1
* 10:39 mark: Assigned IP addresses to csw1-sdtpa:e16/1 and cr2-eqiad:xe-5/2/1 to bring up the second 10G wave
* 00:37 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling CustomUserSignup on testwiki'
== June 7 ==
* 23:51 Ryan_Lane: rebooting virt1/2
* 23:21 Ryan_Lane: rebooting virt1
* 23:21 Ryan_Lane: moving nova-network to virt2, as it was causing issues accessing the public IP on virt1
* 23:12 Ryan_Lane: rebooting virt1
* 22:33 Ryan_Lane: removed nova-network from virt2-4, since multiple network nodes isn't supported. Added to virt1. Rebooting virt2-4.
* 22:11 hcatlin: deploying to mobile cluster
* 21:55 Ryan_Lane: restarting opendj on nfs2
* 19:56 logmsgbot: tfinc synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling collections extension on wikinews'
* 19:39 andrewS: ms4 ticket 729 updated, system is in testing
* 17:46 Ryan_Lane: rebooting virt1 to enable virtualization in the bios
* 17:42 Ryan_Lane: rebooting virt2 to enable virtualization in the bios
* 17:39 Ryan_Lane: rebooting virt3 to enable virtualization in the bios
* 17:35 Ryan_Lane: rebooting virt4 to enable virtualization in the bios
* 17:01 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Disabling Tidy on collabwiki per bug 29295'
* 16:50 mark: Adjusted BGP filters on cr1-eqiad to allow advertisement of pmtpa aggregates to BGP peers
* 16:29 mark: Removed static routes from cr1-eqiad and cr2-eqiad
* 16:18 mark: Setup full mesh of iBGP sessions between csw5-pmtpa, csw1-sdtpa, cr1-eqiad, cr2-eqiad (separate v4/v6 sessions)
* 15:17 mark: Removed IPv6-only iBGP session between cr1-eqiad and cr2-eqiad and converted the v4 iBGP session into multi-address-family
* 14:22 mark: Setup OSPFv3 between csw1-sdtpa and csw5-pmtpa
* 14:08 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28292'
* 14:04 logmsgbot: reedy synchronized php-1.17/includes/api/ApiPageSet.php 'r89646'
* 13:58 mark: Setup OSPF and OSPFv3 on the first 10G wave circuit between csw1-sdtpa and cr1-eqiad
* 13:39 RobH: updaed puppet giving andrew shields basic restricted shell access to fenari
* 11:08 mark: Setup quick log rotation of ipv6and4 on maerlant
* 04:51 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'Let normal users test/signoff on revisions, just like we let them post comments. Restricting it to "coders" kind of defeats the purpose'
== June 6 ==
* 20:22 logmsgbot: hashar: ci.tesla: cleaned up backlog and /tmp (I really need to fix this)
* 17:00 logmsgbot: hashar: ran namespaceDupes.php frwikiversity --fix (bug 29015)
* 15:45 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaIncubator/IncubatorTest.php 'r89571'
* 15:45 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaIncubator/SpecialViewUserLang.php 'r89571'
* 14:54 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaIncubator/SpecialRandomByTest.php 'r89565'
* 12:23 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaIncubator/IncubatorTest.php 'r89561'
* 11:41 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaIncubator/SpecialViewUserLang.php 'Attempted fix for fatal error'
* 11:36 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable WikimediaIncubator extension on incubatorwiki'
* 11:28 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Add incubator loading code'
* 11:24 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add $wmgUseIncubator, set to false'
* 11:14 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 89556: 
* 11:13 RoanKattouw: Running scap to deploy WikimediaIncubator extension (code push only, not yet enabled anywhere)
* 09:46 logmsgbot: catrope synchronized php-1.17/includes/json/Services_JSON.php 'r89555'
* 06:18 Ryan_Lane: rebooted virt2/3, so that nova-network would reconfigure itself properly
* 05:26 Ryan_Lane: depooled mobile4/5
== June 5 ==
* 23:22 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29269 Enable wgRawHtml on collabwiki - it is locked down wiki'
* 18:06 logmsgbot: reedy synchronized php-1.17/includes/api/ApiPageSet.php 'r89514 for bug 25734'
* 15:32 logmsgbot: reedy synchronized php-1.17/wmf-config/codereview.php 'Reduce semantic autodefer regex'
* 09:18 mark: killed old stray lsearchd process, started new one on search7 (search pool 3)
== June 4 ==
* 11:17 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29270'
== June 3 ==
* 22:44 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 22:43 logmsgbot: pdhanda synchronized closed.dblist 
* 22:10 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 29007. Permission changed per comment 2'
* 21:50 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 29007. Added autopatrol to fawiki'
* 21:34 Ryan_Lane: upping weights of mobile4/5 to 24
* 21:28 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.i18n.php 'r89444, r89445'
* 21:28 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'r89444, r89445'
* 21:27 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.css 'r89444, r89445'
* 21:27 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js 'r89444, r89445'
* 21:27 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r89444, r89445'
* 20:43 Ryan_Lane: added temporary dns entry controller.labs.wikimedia.org until we figure out a proper DNS scheme for labs
* 19:43 Ryan_Lane: repooling mobile4/5 with a weight of 8
* 19:09 notpeter: pushing mible 4 and 5 into pool
* 19:00 Ryan_Lane: reinstalling opendj on virt1 with a new domain; using labs instead of tesla
* 17:59 awjr: killing eternally running query on civicrm db on db9
* 17:28 pdhanda: running maintenance/namespaceDupes.php on frwikiversity
== June 2 ==
* 20:00 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'Live hack to disable join CTA'
* 19:03 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump AFT tracking version to 8'
* 19:02 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r89355'
* 19:02 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.startup.js 'r89355'
* 19:01 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.css 
== June 1 ==
* 21:46 logmsgbot: reedy synchronized php-1.17/includes/User.php 'bug 27514 Make unblockself available as a global permission'
* 21:24 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 26319 Enable collection on fawiki. Testing on rtl'
* 21:13 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 27323 WMF Logo/Button not served via Bits'
* 19:07 Ryan_Lane: switching test.wikipedia.org back to text cname, since it is using a fake cert. (testing was successful)
* 18:47 Ryan_Lane: changing cname for test.wikipedia.org to wikipedia-lb, to test https
* 18:41 Ryan_Lane: changing scheduler for https services from wrr to sh on pmtpa lvs servers. restarting pybal on lvs1
* 18:38 Ryan_Lane: make that ns1
* 18:38 Ryan_Lane: restarting pdns on ns2, it locked up after authdns-update
* 18:36 Ryan_Lane: adding wikipedia-lb geodns entry
* 18:30 Ryan_Lane: restarting pybal on amslvs1
* 18:29 Ryan_Lane: adding wikipedia-lb service to amslvs1
* 16:11 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29228'
* 15:43 logmsgbot: reedy synchronized php-1.17/extensions/CentralAuth/SpecialCentralAuth.php 'bug 28767'
* 15:40 logmsgbot: reedy synchronized php-1.17/extensions/CentralAuth/SpecialCentralAuth.php 'bug 28767'
* 14:39 Reedy: Poking at CentralAuth (bug 28767) on testwiki. If things get more broken, it's probably me
* 09:32 apergos: there's a hung tcpdump on amssq37 tht won't die. it doesn't seem to be doing much but it won't die either (that was me looking at purge issues)
* 08:18 thedj: Cache invalidations issues continue, but squid source unknown atm.
== May 31 ==
* 23:28 logmsgbot: asher synchronized php-1.17/wmf-config/db.php 'removing db7 from rotation'
* 23:19 Ryan_Lane: restarting pybal on lvs1
* 23:07 Ryan_Lane: adding forward entry for wikipedia-lb, for https testing
* 22:44 Ryan_Lane: restarting pdns on ns0, it's locked up
* 22:43 Ryan_Lane: changing cname for wikimania2005 to wikimedia-lb, for testing https
* 22:37 Ryan_Lane: adding geodns entry for wikimedia-lb
* 22:26 Ryan_Lane: adding wikimedia-lb forward entry for esams
* 21:56 Ryan_Lane: switching wikimania2005 back to text
* 21:45 Ryan_Lane: changing cname of wikimania2005 from text to wikimedia-lb to test https
* 20:51 hashar: wikitech 'ci.tesla: setup a cron for user ci to delete /tmp/mwParser* directories'
* 20:26 mark: Extended LVS prefix lists on csw1-esams and csw2-esams with 91.198.174.224/28
* 20:25 Ryan_Lane: restarting pybal on lvs1
* 19:52 Ryan_Lane: updating yvon and gurvin in dns, moving them to the squid-lvs address range
* 19:40 Ryan_Lane: adding wikimedia https service to pybal, and restarting pybal
* 18:47 mark: Disabled (redundant) PIM on csw5-pmtpa:ve3, so only csw1-sdtpa is running PIM now
* 18:20 Ryan_Lane: and restarting apache on it
* 18:20 Ryan_Lane: running sync common on srv186
* 16:07 logmsgbot: hashar: ci.tesla: was actually out of inodes on /. /tmp filled with mwParser-...-images directories
* 16:02 logmsgbot: hashar: ci.tesla: restarted cruisecontrol (no space left on device) error. It still has space though!
== May 29 ==
* 21:19 logmsgbot: demon synchronized php-1.17/extensions/LiquidThreads/classes/View.php 'r89136'
* 19:17 logmsgbot: hashar: ci.tesla installed imagemagick php-gd php5-gd
* 18:42 logmsgbot: hashar: resumed continuous integration on ci.tesla
* 18:37 logmsgbot: hashar: ci.tesla out of disk, cleaning up /dev/sda1 wich is / (a good idea would be to use separate partitions
* 17:12 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29119 Allow nlwiki crats to remove rollbacker group'
* 05:57 Ryan_Lane: restarted pdns on dobson, it wasn't responding to requests
* 03:07 Ryan_Lane: killed some long running civi queries on db9 that have been running for over 200,000 seconds.
* 02:54 mark: Killed oprofile daemon and unloaded oprofile module on db32
* 02:38 mark: Restarted squid-frontend on sq76
== May 28 ==
* 19:41 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'Set wgCodeReviewMaxDiffPaths = 30'
* 17:52 Andrew: Loaded board vote configuration from extensions/SecurePoll/cli/wm-setup/bv2011-*-final3-jump.xml
* 17:48 logmsgbot: andrew synchronized php-1.17/wmf-config/CommonSettings.php 'Hooks for SecurePoll'
* 15:01 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29119'
* 08:15 logmsgbot: andrew synchronized php-1.17/extensions/SecurePoll/includes/user/Auth.php 'deploy r89026'
* 07:49 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 89008: 
* 07:48 Andrew: Running scap to deploy SecurePoll updates
== May 27 ==
* 22:17 logmsgbot: demon synchronized closed.dblist 'bug 29064: cleanup wikimania wiki closures (2005-2010)'
* 22:17 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29064: cleanup wikimania wiki closures (2005-2010)'
* 22:00 logmsgbot: catrope synchronized php-1.17/includes/GlobalFunctions.php 'Remove wfMsgExt profiling again'
* 21:53 logmsgbot: catrope synchronized php-1.17/includes/MessageBlobStore.php 'Prospective fix for the UW bug'
* 21:38 logmsgbot: catrope synchronized php-1.17/includes/MessageBlobStore.php 'More profiling'
* 21:35 logmsgbot: catrope synchronized php-1.17/includes/GlobalFunctions.php 'Add profiling points in wfMsgExt()'
* 21:29 logmsgbot: catrope synchronized php-1.17/includes/MessageBlobStore.php 'Add more profiling'
* 21:15 logmsgbot: catrope synchronized php-1.17/StartProfiler.php 'Live hack for debugging UploadWizard RL weirdness'
* 21:13 logmsgbot: catrope synchronized php-1.17/load.php 'Revert live hack'
* 21:04 logmsgbot: catrope synchronized php-1.17/load.php 'Add live hack for profiling'
* 20:48 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 29129. Added the new namespace to be search by default for metawiki.'
* 20:30 logmsgbot: pdhanda synchronized php/wmf-config/InitialiseSettings.php 
* 12:10 Andrew: Installing SecurePoll DB tables on wikis in ~/centralauth_wikis_missing_securepoll
* 07:57 apergos: the images Rl2-quality-full-003.jpg and Picture_602.11.85.5.jpg have been removed from our image server permanently, office action (legal request).
* 06:37 apergos: thanks tim
* 06:37 apergos: restarted irc bot on ekrem
* 06:09 apergos: er... on ekrem :-P
* 06:09 apergos: restarted irc daemon, hopefully properly 
* 06:02 apergos: attempting to reboot ekrem (was not pingable)
== May 26 ==
* 23:20 notpeter: powercycling knsq5
* 21:34 Ryan_Lane: adding en.prototype, de.prototype and test.prototype cnames for prototype.wikimedia.org
* 18:58 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Raise AFT tracking percentage from 2% to 10% (restores pre-ramp-up volume) and put all users in the show-expertise bucket. Also bump tracking version to 7'
* 18:54 Ryan_Lane: adding wikimedialbsecure service to amslvs1, for testing https
* 18:52 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.js 'r88909'
* 18:47 Ryan_Lane: adding wikimedia-lb.esams address to ams and kn text squids, also adding lvs configuration for this address
* 18:31 Ryan_Lane: removing mediawiki-lb.esams and foundation-lb.esams reverse addresses, they were assigned to the same ips as text.esams and bits.esams respectively
* 17:48 Ryan_Lane: adding forward entry for wikimedia-lb.pmtpa.wikimedia.org
* 15:34 mark: Moved AS13030 connection configuration from port 2/11 to port 2/11 on br1-knams
== May 25 ==
* 22:39 awjr: svn up'd on civicrm.wikimedia.org to r189 - deploying further optimizations to contribution searching
* 21:22 Ryan_Lane: adding wikimedia-lb svc address to squids
* 21:22 Ryan_Lane: adding wikimedialbsecure service to lvs2, for testing https access to wikimedia.org sites.
* 21:21 Ryan_Lane: modified pybal configuration to add wikimedia-lb ip to text backend
* 19:52 logmsgbot: demon synchronized php-1.17/extensions/CentralAuth/SpecialWikiSets.php 'r88825'
* 18:32 Ryan_Lane: turning off the selenium grid virtual machines on tesla
* 17:04 RobH: updated bugzilla per rt 764 and 768. whine feature should work now
* 16:44 awjr: deployed patch to civicrm.wikimedia.org to optimize contribution search query (r185, wikimedia repo)
* 14:57 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Don't allow users to create accounts on closed wikis either'
* 14:54 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Cleanup wikimania200[58] entries: bug 29064'
* 14:53 logmsgbot: demon synchronized closed.dblist 'Add wikimania200[58] to closed.dblist: bug 29064'
* 14:45 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29064: close wikimania2009wiki'
* 14:43 logmsgbot: demon synchronized closed.dblist 'bug 29064: close wikimania2009wiki'
* 12:26 Tim: marked the deletion jobs in OTRS as "invalid" to avoid confusing error messages
* 12:25 Tim: re-enabled anti-deletion security patch in OTRS, disabled "temporarily" by Fred on June 4, 2009
* 11:58 domas: rebooting db9, looks unhealthy
* 11:06 apergos: so that was me on srv225: stopped apache, waited for puppet restart: it now syncs, then does the restart. oh btw srv225 isn't in the pybal apache conf file, weirdly... anyone know why?
* 09:15 apergos: restarted front and backend squids on sq75
* 08:47 apergos: restarted front and back end squids on sq81, front end was 20gb and back end was out to lunch
* 08:35 apergos: ekrem's root partition was full; cleared out some older logs from /var/log/apache2, compressed the most recently rotated (it had failed from disk full), reloaded puppet
* 07:20 Tim: set read_only=1 on db10, it's a slave so it should be read only. Nagios says it was broken for 6 months.
* 07:07 logmsgbot: midom synchronized php-1.17/wmf-config/db.php 
* 05:09 Tim: restarted nagios IRC bot
* 01:05 Tim: created an account for myself in watchmouse
* 00:48 Andrew: Running populateBv2011EditCount.php in a screen on hume
* 00:32 Ryan_Lane: pruning binlogs on db9
== May 24 ==
* 23:12 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 29129. Added a namespace alias for metawiki.'
* 20:36 notpeter_: resyncing drbd nfs1 to nfs2
* 20:28 RobH: mobile4 awaiting final configurations for service, mobile5 installation done, not yet run puppet updates
* 20:20 RobH: mobile4 is running puppet updates, mobile5 is mid-install
* 19:46 RobH: mobile4 & mobile5 racked and accessible for mgmt. installing OS on mobile4 now
* 18:00 Ryan_Lane: added new reverse zone for svc ips in esams; added entries for bits.svc.esams.wmnet and upload.svc.esams.wmnet
* 17:31 Ryan_Lane: removing bits-lb and upload-lb public ip enties, and adding bits.svc.pmtpa.wmnet and upload.svc.pmtpa.wmnet entries for https testing
* 16:46 mark: Upgraded cr2-eqiad to JUNOS 10.4R4.5
* 16:44 mark: PIM configuration was missing on some virtual interfaces on csw1-sdtpa, which broke purging to esams. Put back in place
* 16:09 mark: Reloading re1.cr2-eqiad
* 16:01 mark: Fixed tunnel hack between pmtpa and eqiad
* 15:01 RobH: db20 networking restarted, back online
* 14:58 RobH: Miguel is rebooting db18
* 13:55 mark: reloading csw5
* 13:52 domas: ns2 has been down for 3 hours, will try restarting pdns
* 13:47 mark: reloading asw-a4
* 13:45 mark: reloading asw-d3
* 13:39 domas: db18/20/28 are still down, may need power cycle. Took them out of rotation.
* 13:37 RobH: restarted networking on db33-39.
* 13:30 mark: restarted networking on db30-32.
* 13:25 domas: DB server issue appears to be a linux kernel issue, forcedeth-related, can be fixed with /etc/init.d/networking restart. Doing so on db10-30, asking mark to do the rest.
* 13:23 mark: restarted slot 3 since that's where the problematic database servers are
* 13:12 mark: csw1-sdtpa back up, but a whole lot of database servers are unreachable
* 13:07 mark: upgraded and reloaded csw1-sdtpa
* 12:05 mark: Upgraded bootrom and software on asw-d1-sdtpa and asw-a5-sdtpa
* 10:53 mark: Copied existing 2.7.0.2 image to secondary flash, and installed 2.7.0.3a image on primary flash on csw5-pmtpa
* 10:42 mark: Copied existing 2.6.0 image to secondary flash, and installed 2.7.0.3a image on primary flash on csw1-sdtpa
* 04:20 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php 'r88707'
* 04:03 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 04:03 Tim: enabling GoogleNewsSitemap on enwikinews
== May 23 ==
* 21:18 Ryan_Lane: added upload-lb.pmtpa.wm.o and bits-lb.pmtpa.wm.o addresses to reverse dns for https preparation
* 19:32 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29003'
* 18:45 apergos: reloading puppet on all hosts, with a minute sleep in between, running in screen on fenari as root. needed for enabling pluginsync on clients
* 18:20 Ryan_Lane: moved ipv6 proxy from iris to maerlant
* 17:35 RobH: db20 back online, needs mysql setup for replication on another cluster. created rt#836
* 17:29 RobH: db20 was offline from earlier hardware issues that appear to have been resolved. it is in no current deployment group. booting it online to see if it requires reinstallation
* 14:31 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/SitemapFeed.php 
* 14:31 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/FeedSMItem.php 'r88640'
* 14:30 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php 'r88640'
* 14:23 hcatlin: Deploying code update to Mobile wikipedia
* 05:08 Tim: in watchmouse: updated subversion monitoring, was broken since May 7
== May 22 ==
* 17:47 logmsgbot: reedy synchronized php-1.17/extensions/CodeReview/ui/CodeStatusChangeListView.php 'r88585'
* 17:46 logmsgbot: reedy synchronized php-1.17/extensions/CodeReview/ui/CodeCommentsListView.php 'r88585'
* 00:09 Reedy: Commons is back up and editable
== May 21 ==
* 23:38 Reedy: Intermittant MySQL errors occuring on hits to 10.0.6.41, it is being looked into
* 23:35 Ryan_Lane: powercycling db31, it's dead
* 10:27 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '25963 - Enable of Recent changes patrol for Serbian Wikisource (sr.wikisource.org)'
== May 20 ==
* 22:30 Reedy: StewardWiki security issues fixed
* 22:22 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'Stewardwiki'
* 21:23 awjr: running contribution auditing scripts on civicrm.wikimedia.org
* 21:21 awjr: enabling log audit module on civicrm.wikimedia.org
* 21:18 awjr: svn up'ing to r183 of wikimedia branch on production instance of civicrm on grosley
* 20:57 logmsgbot: pdhanda ran sync-common-all 'Bug 28717. Group changes and flagged revs enabled for bnwiki'
* 20:50 Ryan_Lane: maerlant too
* 20:50 Ryan_Lane: rebooting maerlent
* 20:13 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28773. more rights and namespaces for stewardwiki'
* 17:16 RobH: updated dns for new mobile servers mgmt
* 17:12 RoanKattouw: Applying index changes on click_tracking_user_properties from r88462 to the cluster. Using /home/catrope/patch-click_tracking_user_properties-index.sql
* 15:50 logmsgbot: pdhanda synchronized php-1.17/wmf-config/abusefilter.php 'Bug 28461. Custom abuse filter settings for be_x_oldwiki'
* 15:50 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28461. Custom abuse filter settings for be_x_oldwiki'
* 14:41 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28614. Added suppressredirect to autopatrolled on svwikisource'
* 14:33 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28576. Changed upload settings for huwiki'
* 03:12 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Touch config'
* 03:12 logmsgbot: demon synchronized closed.dblist 'bug 29044: closure of tkwikibooks'
* 02:48 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29058: import sources for arwikinews'
* 02:04 logmsgbot: LocalisationUpdate failed
* 00:06 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 29057'
== May 19 ==
* 22:26 ^demon: add new 'tools' group to public svn authz
* 21:02 logmsgbot: hashar: cruisecontrol: tests should now run in roughly 1min30! Still need attention though
* 20:16 logmsgbot: hashar: cruisecontrol: archived build logs for 2010 and 2011 from 01 till 04
* 18:48 logmsgbot: hashar: relaunched CruiseControl service on ci
* 02:01 logmsgbot: LocalisationUpdate failed
== May 18 ==
* 19:44 Ryan_Lane: restarting squid processes on sq71
* 18:11 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Turning GNSM back off for now -- apache spike'
* 17:50 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Deploying GNSM for enwikinews'
* 17:29 logmsgbot: demon synchronizing Wikimedia installation... Revision: 88303: 
* 16:06 ^demon: didn't work, had errors with ArticleAssessmentPilot, FlaggedRevs, LQT_alpha, and MessagesTp. Plus lots of rsync permission errors. Saved output to /home/demon/logs/l10nupdatefail.log
* 15:53 ^demon: doing manual run of l10nupdate
* 15:24 logmsgbot: demon synchronized php-1.17/wmf-config/extension-list 
* 02:05 logmsgbot: LocalisationUpdate failed
== May 17 ==
* 18:33 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'fix capitalisation fail'
* 18:30 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'bug 29017 again doesn't seem to have pushed out'
* 13:24 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Re-enable CLDR'
* 13:23 logmsgbot: catrope synchronized php-1.17/languages/Language.php 'r88303'
* 13:09 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable CLDR because of fatal'
* 13:03 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable CLDR extension on all wikis'
* 12:52 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 88299: 
* 12:51 RoanKattouw: Running scap to deploy CLDR extension. Only enabled on test for now, will go site-wide in a minute
* 11:57 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesEn.php 'r88295'
* 11:34 logmsgbot: catrope synchronized php-1.17/extensions/UploadWizard/resources/jquery/jquery.validate.wmCommonsBlacklist.js 'r88293'
* 09:52 Tim: creating an account on wikitech for priyanka
* 09:51 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Re-enable UploadWizard on Commons, adding note about how to switch it back off if it explodes'
* 09:39 logmsgbot: catrope synchronized php-1.17/extensions/UploadWizard/SpecialUploadWizard.php 'Fix fatal in live hack'
* 09:37 logmsgbot: catrope synchronized php-1.17/extensions/UploadWizard/SpecialUploadWizard.php 'Live hack to debug UploadWizard issue'
* 09:17 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Re-enabling UploadWizard on test to hopefully reproduce ResourceLoader issue'
* 05:21 mark: Reduced passenger pool size to 200 on all mobile servers
* 04:46 mark: Depooled mobile2 and rebalanced LVS weights on m.wikipedia
== May 16 ==
* 21:56 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28504 - Need to remove all tokipona settings from InitialiseSettings.php'
* 21:52 logmsgbot: hashar: cleaning up settings files for tokipona stuff (bug 28504)
* 19:03 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'bug 29017'
* 09:37 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'raise account throttle for el wiki for workshop (maybe I can talk the community into upping the threshhold generally??)'
== May 15 ==
* 21:13 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Closing 4 wikis: thwikinews (bug 28341), huwikinews (bug 28342), cowikibooks (bug 28644), astwikiquote (bug 28964)'
* 21:13 logmsgbot: demon synchronized closed.dblist 'Closing 4 wikis: thwikinews (bug 28341), huwikinews (bug 28342), cowikibooks (bug 28644), astwikiquote (bug 28964)'
* 21:02 logmsgbot: demon synchronized php-1.17/includes/DefaultSettings.php 'r88203'
* 21:01 logmsgbot: demon synchronized php-1.17/includes/User.php 'r88203'
* 12:27 logmsgbot: pdhanda ran sync-common-all 
* 12:12 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 28141'
* 11:48 logmsgbot: robh synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28987'
* 11:32 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28727 - please create Portal namespace on am.wikipedia'
* 11:05 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '25172 - Enable DynamicPageList extension on it.wikisource'
* 10:35 logmsgbot: hashar: had to manually delete a row from the ptwiki.langlink table (ll_from: 162503, ll_lang: zh-classic, ll_title: Template:Link A (bug 28805)
* 10:27 logmsgbot: hashar: running update ignore query on ptwiki for bug28805
* 10:23 logmsgbot: hashar: batch for bug 28805 finished.
* 10:17 logmsgbot: hashar: running batch queries on all wikis for bug 28805
* 08:44 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28609 - Disable uploads for eu.wikipedia.org'
* 01:14 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28914 - Adding 'ipblock-exempt' userright to the Bots usergroup in enwiki'
* 00:52 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28959 - si.wikipedia $wgSitename'
== May 14 ==
* 22:46 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27434'
* 22:08 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'wgMetanamespace for amwiki (bug 28727)'
* 22:05 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28727 - please create Portal namespace on am.wikipedia'
* 17:11 mark: Fixed IPv6 routing on csw5-pmtpa
* 16:34 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'bug 28153'
* 16:30 logmsgbot: reedy synchronized php-1.17/wmf-config/abusefilter.php 'bug 28153'
* 16:16 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 28141'
* 16:16 logmsgbot: reedy synchronized php-1.17/wmf-config/CommonSettings.php 'bug 28141'
* 16:16 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28141'
* 16:14 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28141'
* 15:57 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27078'
* 15:52 RobH: added dns entry for wikimania2012.w.o
* 15:23 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28773'
* 15:22 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 14:50 awjr: cleaned out hudson job files for 'Donations queue consume' on grosley and archived them to storage3
* 14:22 mark: Allocated service IPs for all projects for HTTPS in DNS
* 14:06 logmsgbot: tstarling synchronized php-1.17/lib/geshi 
* 13:53 mark: Upgraded kernel on streber, rebooting it
* 13:41 logmsgbot: reedy synchronized php-1.17/lib/geshi 'bug 24383'
* 13:31 logmsgbot: reedy synchronized php-1.17/lib/geshi 'bug 24383'
* 13:17 logmsgbot: reedy synchronized php-1.17/lib/geshi 'bug 24383'
* 13:13 logmsgbot: reedy synchronized php-1.17/lib/geshi 'bug 24383'
* 13:11 logmsgbot: reedy synchronized php-1.17/lib/geshinew 'Test pushing symlink'
* 12:35 Ryan_Lane: shutting down owa1-3.tesla.usability.wikimedia.org
* 12:24 Ryan_Lane: adding maerlant.esams.wikimedia.org as a https/ipv6 proxy node
* 10:06 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled enotif for user talk pages on all wikis'
* 09:54 logmsgbot: reedy synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28895'
* 09:43 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28612. Added the autopatrolled group to elwiktionary'
* 09:28 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'Fill in template for bug 28727 (Portal namespace creation for am.wikipedia)'
* 09:16 logmsgbot: tstarling synchronized php-1.17/includes/MessageCache.php 'removed message stats patch'
* 08:51 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28479. Added the patroller group to plwikiquote'
== May 13 ==
* 17:33 Ryan_Lane: yvon and gurvin configured as https servers
* 17:26 RobH: gurvin os installed pupppet not run
* 17:14 RobH: moved gurvin vlan form 2 to 101
* 17:06 RoanKattouw: Uncommented srv196 in the node list and ran sync-common on it. It was giving me colorless diffs on CodeReview
* 17:03 RobH: yvon reinstalled, wont run puppet initial run due to error from new https service entry
* 16:51 RobH: updating dns for hosts that didnt have public ips (gurvin, some corrections for yvon)
* 16:28 logmsgbot: catrope ran sync-common-all 
* 16:22 RoanKattouw: Running sync-common-all because sync-docroot didn't work
* 16:12 RoanKattouw: Syncing docroot
* 16:12 RoanKattouw: Adding xml/api directory to mediawiki.org's docroot, with an index.html similar to the one in the export-0.4 directory
* 16:02 logmsgbot: reedy synchronized php-1.17/RELEASE-NOTES 
* 12:14 logmsgbot: robh synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28347'
* 10:48 logmsgbot: robh synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 26037'
* 10:31 Tim: deployed a patch to gather i18n usage stats for siebrand
== May 11 ==
* 22:09 mark: Reloaded csw5-pmtpa on existing code base
* 01:16 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'Temporarily disabling UploadWizard per Neil K while a bug is being worked out.'
* 01:15 logmsgbot: demon synchronized php-1.17/extensions/UploadWizard/SpecialUploadWizard.php 'r87865'
* 01:14 logmsgbot: demon synchronized php-1.17/extensions/UploadWizard/UploadWizard.i18n.php 'r87865'
* 00:10 notpeter: puppet back to normal on all boxes that are up except for knsq5.
== May 10 ==
* 23:36 Ryan_Lane: fixing torrus. why oh why must you corrupt yourself constantly, torrus?
* 23:06 notpeter: distributing new puppet certs to all boxes managed by puppet
* 22:51 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'ED 1.17'
* 21:34 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling Article Feedback dashboard on enwiki'
* 21:30 logmsgbot: awjrichards synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.php 'r87844'
* 21:29 logmsgbot: awjrichards synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.i18n.php 'r87844'
* 21:28 logmsgbot: awjrichards synchronized php-1.17/extensions/ArticleFeedback/populateAFStatistics.php 'r87844'
* 21:28 logmsgbot: awjrichards synchronized php-1.17/extensions/ArticleFeedback/SpecialArticleFeedback.php 'r87844'
* 21:02 Ryan_Lane: powercycling srv281 since it's dead
* 21:01 Ryan_Lane: powercycling srv247 since it's dead
* 20:56 Ryan_Lane: mobile has stablized, moving weights back to normal
* 20:50 Ryan_Lane: lowering weight on mobile1 and mobile2
* 20:45 Ryan_Lane: restarted apache on mobile2.
* 19:16 mark: Moved all inter-subnet routing from csw5-pmtpa to csw1-pmtpa
* 19:16 mark: Reconfigured pybal on lvs3 to BGP connect to csw1-sdtpa instead of csw5-pmtpa, and restarted it
* 17:53 mark: Moved vlan 2 routing from csw5-pmtpa:ve2 to csw1-pmtpa:ve2
* 16:27 apergos: or at least srv254.
* 16:25 apergos: nagios is whining about some services that are actually in operation (eg srv250)
* 16:07 Ryan_Lane: or not, mgmt interface isn't letting me in
* 16:06 Ryan_Lane: powercycling srv250
* 13:56 Tim: removed myself from SMS notification in nagios, since the configuration is so broken that it sends me a text every 10 minutes day or night
* 13:49 RobH: nagios sms pages seem to be delayed from the actual service outage by a long, long delay
* 13:10 Tim: did the above in both puppet and nagios directly since puppet seems to be still broken
* 13:07 Tim: increased the number of retries of payments.wikimedia.org before sending an SMS from 1 to 20
* 02:02 notpeter: accidentally deleted sockpuppet:/var/lib/puppet/ssl am regnerating keys and will begin signing soon
== May 9 ==
* 23:36 notpeter: so annoying. doubling puppet freshness check timing to 4 hours.
* 22:01 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disabling dashboard on enwiki pending fixes to data collection script'
* 21:20 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87774: 
* 21:20 RoanKattouw: Running scap to deploy UploadWizard changes
* 20:54 RoanKattouw: Running extensions/ArticleFeedback/populateAFStatistics.php on enwiki
* 20:49 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/populateAFStatistics.php 'r87770'
* 20:49 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r87770'
* 20:30 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling ArticleFeedback dashboard on testwiki'
* 20:28 RoanKattouw: Running populateAFStatistics.php on testwiki
* 19:39 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'Dummy sync to push config change through in ResourceLoader'
* 19:39 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Re-enable AFT click tracking at 2%, and bump tracking version'
* 19:32 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'Force config changes to update in startup module'
* 19:21 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to the target value of 2.7%'
* 19:03 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 2%'
* 18:41 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 1%'
* 18:39 RobH: upgraded Reedy's cluster access to allow him to deploy and do shell requests, will take awhile to push out to the cluster
* 18:25 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Raise AFT lottery odds to 0.5% for real this time'
* 18:21 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump AFT tracking version, needed to actually change the click tracking percentage'
* 18:16 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28495'
* 18:14 Ryan_Lane: adding a shell account for asher (Asher Feldman), and adding him to roots
* 18:13 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Temporarily disable click tracking for AFT'
* 18:11 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bump AFT lottery odds to 1%'
* 18:10 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable AFT dashboard on enwiki'
* 18:09 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump ArticleFeedback tracking version'
* 18:01 RoanKattouw: Running schema change (adding index of aa_timestamp field on enwiki.article_feedback table) on s1 by directly running it on the master. Table size is ~275k rows
* 17:56 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Set $wmgArticleFeedbackLotteryOdds to 0.005 on enwiki. This enables AFT on 0.5% of all enwiki main namespace pages'
* 17:44 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87742: 
* 17:43 RoanKattouw: Running scap to deploy ArticleFeedback changes
* 16:49 RoanKattouw: Deploying ArticleFeedback changes to testwiki
* 14:21 apergos: restarting amssq39 squid back-end (memory hog)
* 13:41 apergos: restarting squid back-end on amssq40 (memory hog)
* 13:27 logmsgbot: catrope synchronized php-1.17/resources/mediawiki/mediawiki.js 'r87712'
* 13:27 logmsgbot: catrope synchronized php-1.17/includes/OutputPage.php 'r87712'
* 13:26 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderContext.php 'r87712'
* 13:26 RoanKattouw: Deploying r87712, which should fix the ResourceLoader breakage in IE
* 13:12 apergos: restarted squid back-end on amssq37 (hogging memory)
* 08:06 apergos: rebooting srv198 for testing
* 07:52 apergos: rebooting srv281 for testing
== May 8 ==
* 22:46 RobH: fixed deep linking issue for uploads from old techblog. merge did not copy, had to manually pull from singer and also add additional rewrite rule to apache via puppet
* 22:40 RobH: copied over techblog upload data, which upon further review was not copied by the merge plugin as expected
* 18:29 RobH: updated categories on blog to remove testblog url entry
* 18:20 RobH: updating dns to revert bad change on blog propogation, split wikimediafoundation template files to themselves, were soft linked to wiktionary.org
* 18:05 RobH: blog redirects are in place for projects, not sure why or what added them, as they do not belong. fixed hooper via puppet to handle blog.wikimediafoundation.org
* 15:46 RobH: added exim simple mail sender to puppet entry for hooper
== May 7 ==
* 20:31 RobH: hooper running back on puppet, with all config files being updated normally
* 18:11 RobH: tinkering with redirects on hooper. disabled puppet daemon while doing local apache config changes, once tested will push up to puppet and reenable it on hooper
* 18:08 RobH: blog dns updated enough for both myself and guillom to test basic blog function. all basic use seems fine. still testing more advanced use cases
* 16:58 RobH: updated dns for blog and techblog. may take an hour or so to propogate, cannot update new blog until it resolves for authors to it. (old blogs will slowly cycle out of dns)
* 16:40 RobH: blog migration moving along, all data migrated, basic rewrites and such are in place, final review and cleanup in progress before changing testblog.w.o to blog.w.o in database and updating DNS
* 16:03 RobH: testblog.w.o currently now running 'master' database for the wmf blog. online as testblog, can do quick db replace when its ready to go online
* 15:57 RobH: starting the copy of the main blog, any changes to blog.w.o after this time will be lost in the migration to the new host
* 15:53 RobH: updating blog software on hooper, installing all extensions before connecting to database copy of newsblog (not yet copied)
* 15:37 RobH: wiping out testblog on hooper and getting basic software installed for new blog deployment. database not yet copied, old blogs still 'primary'
* 12:55 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'raise account throttle for el wiki for workshop'
* 09:20 rainman-sr: deleted some old logs on search11 to free up space
== May 6 ==
* 22:46 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28773. Modified wgAddGroups and enabled subpages for stewardwiki'
* 20:39 Ryan_Lane: added apaches::service to imagescalers class in puppet to ensure that sync-common will run before apache is started on the imagescalers
* 18:10 awjr: enabling drupal/civicrm logging on civicrm.wikimedia.org (forgot to turn this back on post-upgrade)
* 17:41 notpeter: killing the shit out of puppet to clear its cache. 
* 17:38 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28791. Fixed logo for piwiki'
* 17:19 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28689 - enabled NewUserMessage on thwiki'
* 13:44 RobH: installed and activated wordpress importer on testblog
* 08:58 RoanKattouw: Ran sync-common on srv281 and restarted Apache. It was complaining about missing directories in docroot
* 08:55 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87486: 
* 08:55 RoanKattouw: Uncommenting srv281 in /etc/dsh/group/mediawiki-installation and scapping again
* 08:53 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87486: 
* 08:52 RoanKattouw: Running scap because srv281 is out of sync. I thought we had measures to prevent out-of-sync servers from being pooled?
* 05:03 apergos: powercycled srv281 (again)
* 01:55 Tim: stopped xinetd on fenari, running svn up to add REL1_17 branch to extdist snapshot
== May 5 ==
* 17:32 notpeter: taking down nfs1 to upgrade to lucid and upgrade udplog
* 05:47 logmsgbot: tstarling synchronized php-1.17/img_auth.php 'r87486'
* 05:47 logmsgbot: tstarling synchronized php-1.17/includes/WebRequest.php 'r87486'
* 05:46 logmsgbot: tstarling synchronized php-1.17/includes/User.php 'r87486'
== May 4 ==
* 23:11 awjr: all CiviCRM-related cron and hudson jobs re-enabled
* 22:19 awjr: CiviCRM upgrade complete, putting civicrm.wikimedia.org back on-line
* 22:17 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php 'r87462'
* 22:17 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap.php 'r87462'
* 22:16 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap.alias.php 'r87462'
* 21:33 awjr: performing CiviCRM upgrade (3.1.6->3.4.0) on grosley
* 21:24 awjr: upgrade to drupal 6.20 complete for civicrm.wikimedia.org
* 21:10 awjr: replacing /srv/org.wikimedia.civicrm dir on grosley with svn co from new deployment branch in wikimedia repo (/branches/deployment/fundraising-civicrm/d620c34)
* 20:51 awjr: disabling civicrm and drupal-related cron/hudson jobs
* 20:49 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87452: 
* 20:43 logmsgbot: catrope synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php 'Fix GNSM, r87488'
* 20:37 awjr: backing up 'civicrm' database from db9
* 20:34 awjr: backing up 'drupal' database from db9
* 20:30 awjr: taking civicrm.wikimedia.org offline for drupal and civicrm upgrade
* 20:14 RoanKattouw: Deploying new UploadWizard code to testwiki
* 18:42 Ryan_Lane: created a restricted account for neilk
* 15:33 logmsgbot: demon synchronized php-1.17/extensions/CentralAuth/SpecialWikiSets.php 
* 14:29 hcatlin: deploying mobile language update
== May 3 ==
* 22:38 logmsgbot: pdhanda synchronized php-1.17/wmf-config/abusefilter.php 'Bug 28502 - custom config for thwiki'
* 22:37 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28502 - enabled abusefilter for thwiki'
* 21:21 Ryan_Lane: powercycling srv281
* 21:12 logmsgbot: catrope synchronized php-1.17/README 'Test sync to see which hosts are borked'
* 21:04 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 28257. Added requested NamespaceAlias to orwiki'
* 20:32 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87355: 
* 20:31 RoanKattouw: Running scap to deploy 1.17wmf1 merges (r87353)
* 16:24 awjr: re-upgrading dev.civicrm.wikimedia.org from civicrm 3.1.6 to 3.4.0
* 15:59 RobH: testblog ops work done, no other services impacted
* 15:13 RobH: working on testblog on hooper and db9, no other services should be affected
* 03:30 RobH: issue of singer network spiking and plummeting, along with swap death, at ~3am gmt
* 03:25 RobH: singer blocked on db9, flushed hosts to allow to reconnect
* 01:30 logmsgbot: demon synchronized php-1.17/extensions/GoogleNewsSitemap/GoogleNewsSitemap_body.php 'r87311'
* 01:29 logmsgbot: demon synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'r87311'
* 00:44 tomaszf: killing long running CiviReport query on db9 due to watchdog being off for upgrade
== May 2 ==
* 20:02 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 'Adding Research namespace to meta, per bug #28742'
* 17:52 Ryan_Lane: powercycling srv281
* 17:50 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabled LabeledSectionTransclusion for mediawiki - requested by guillom'
* 17:28 RobH: fixing issue with bad serial assignment leading to bad network config on ps1-a1/2-eqiad, restarting both power strips
* 15:47 mark: Powercycled amssq35
* 04:21 logmsgbot: tstarling synchronized php-1.17/includes/Wiki.php 'r87235'
* 03:58 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 03:34 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 
* 03:31 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 
* 02:13 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27891 log'
* 02:00 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 'r87234'
* 01:22 logmsgbot: demon synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'r87233'
* 01:21 logmsgbot: demon synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.php 'r87233'
* 00:09 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'wgBlockDisablesLogin on checkuserwiki'
== May 1 ==
* 21:14 Ryan_Lane: ran flush host on db9 to bring blog.wikimedia.org back
== April 30 ==
* 11:38 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryAllUsers.php 'Attempt to fix auprop=blockinfo bug'
* 11:15 Ryan_Lane: (make that blog.wikimedia.org ;) )
* 11:14 Ryan_Lane: ran flush hosts on db9 to bring blogs.wikimedia.org back up
== April 29 ==
* 23:27 ^demon: all of that was to deploy GoogleNewsSitemap to testwiki for the enwikinews folks.
* 23:23 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 
* 23:22 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 23:18 logmsgbot: demon synchronizing Wikimedia installation... Revision: 87149: r87149
* 19:43 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Fix config bug that caused tracking to fail for AFT, and bump tracking version'
* 19:18 mark: powercycled ersch
* 15:40 RobH: typo asw-b8
* 15:39 RobH: swapping msw1-eqiad fan tray with asw-a8-eqiad fan tray
* 05:58 logmsgbot: midom synchronized php-1.17/wmf-config/db.php 'adding srv170 after hw reset'
* 05:55 logmsgbot: midom synchronized php-1.17/wmf-config/db.php 
== April 28 ==
* 23:46 Tim: doing hard reset of alsted, it's not sending ssh version strings and not allowing logins by serial
* 22:17 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disabled option to allow blocked users to edit their talk pages on itwiki. See Bug 9073'
* 21:02 Ryan_Lane: created Ambassador-announce-l list
* 20:52 richcole1: replacing memory in srv281
* 20:49 RobH: srv281 shutdown for memory replacement
* 20:41 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable by default, option to enable blocked users to edit their own talk page'
* 20:34 richcole1: replacing bad drive in srv284
* 20:08 Ryan_Lane: powercycling srv281
* 20:03 RobH: ms4 is powered down, was already offline and unresponsive to console.
* 18:19 logmsgbot: demon synchronizing Wikimedia installation... Revision: 87091: r87090, r87091
* 18:02 mark: Restarted apache on srv235, probably corrupted apc cache
* 17:57 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Re-enabling EmailCapture on enwiki'
* 17:54 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 87084: 
* 17:47 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Preparing conf for ArticleFeedback with way less click tracking to prevent cluster breakage'
* 17:40 RobH: etherpad.proxy to 000-etherpad.proxy on hooper apache2 enabled, should fix
* 17:34 RobH: hooper vhost for testblog back online
* 17:17 RobH: rolled back testblog vhost from hooper
* 17:01 RobH: testblog enabled for initial setup on hooper without disrupting etherpad service (on either of its url redirects)
* 16:56 RobH: updating etherpad.proxy apache config to include eiximenis serveralias and enabling the testblog vhost
* 16:48 Reedy: Manually pinged CR update script, had got stuck - 2 revisions. Let me know if it does it again
* 16:21 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 87085: 
* 16:21 RoanKattouw: Rolling back ArticleFeedback deployment
* 16:11 RoanKattouw: API cluster was briefly dead just now after enabling EmailCapture, disabling it seems to be bringing it back
* 16:09 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable EmailCapture, maybe this will bring our API Apaches back'
* 16:04 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enabling EmailCapture on enwiki'
* 16:01 RoanKattouw: (that was for the ArticleFeedback deployment)
* 15:58 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 87083: 
* 14:08 nelson: Upped the max_connections for rsyncd from 2 to 5 across the test cluster.
* 13:56 nelson: Upped the concurrency of swift-object-replicator from default of 1 to 5
* 13:38 mark: Restored I/O scheduler deadline for device /dev/sda on alsted
* 13:34 mark: Temporarily set I/O scheduler cfq for device /dev/sda on alsted
* 12:29 mark: powercycled harmon
* 01:00 awjr: upgraded dev.civicrm.wikimedia.org to CiviCRM 3.4.0
== April 27 ==
* 20:24 nimish_g: nimishg enabling CustomUserSignup extension on enwiki
* 20:20 logmsgbot: nimishg synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling'
* 01:44 logmsgbot: nimishg synchronized php-1.17/extensions/ClickTracking/modules/ext.UserBuckets.js 'setting cookie expiry to 90 days'
* 01:44 logmsgbot: nimishg synchronized php-1.17/extensions/ClickTracking/ClickTracking.hooks.php 'setting cookie expiry to 90 days'
* 01:43 logmsgbot: nimishg synchronized php-1.17/extensions/CustomUserSignup/CustomUserSignup.hooks.php 'setting cooky expiry to 90 days'
* 01:03 Tim: updated sudoers in puppet for r83000
* 00:25 Tim: reniced sshd on sockpuppet to -2 so that I can log in while catalog updates are running
== April 26 ==
* 23:40 logmsgbot: nimishg synchronizing Wikimedia installation... Revision: 86895: 
* 23:33 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 
* 23:26 logmsgbot: root synchronizing Wikimedia installation... Revision: 86895: 
* 23:15 Tim: disabled CustomUserSignup, was causing fatal error on mediawiki.org due to incorrect path
* 23:15 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 
* 23:13 logmsgbot: nimishg synchronized php-1.17/wmf-config/InitialiseSettings.php 'turning'
* 23:00 logmsgbot: nimishg synchronized php-1.17/wmf-config/InitialiseSettings.php 'turning'
* 22:55 logmsgbot: nimishg synchronized php-1.17/extensions/CustomUserSignup/CustomUserSignup.hooks.php 'r86994'
* 22:49 notpeter: stopping nagios for a wee sec and then restarting.
* 22:09 logmsgbot: nimishg synchronizing Wikimedia installation... Revision: 86895: 
* 22:08 nimish_g: nimishg pushing CustomUserSignup code to cluster, running on mediawikiwiki only
* 21:28 logmsgbot: awjrichards synchronized php-1.17/includes/CategoryPage.php 'r86988'
* 20:38 logmsgbot: pdhanda synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled CustomUserSignup extension for testwiki'
* 20:37 logmsgbot: pdhanda synchronized php-1.17/wmf-config/CommonSettings.php 'enabled CustomUserSignup extension for testwiki'
* 20:32 awjr: disabled wmf_premiums module on civicrm.wikimedia.org - it is currently not being used and potentially causes queue consumption to choke with certain contribution messages
* 19:51 logmsgbot: catrope ran sync-common-all 
* 19:50 RoanKattouw: Running sync-common-all to deploy UploadWizard changse
* 17:52 pdhanda: Running maintenance/populateParentId.php on all wikis
* 08:21 Andrew: sync-common-all worked. scap still broken
* 08:21 logmsgbot: andrew ran sync-common-all 
* 08:21 Andrew: trying sync-common-all
* 08:19 Andrew: syncs are broken, log littered with XXX: [sudo] password for andrew:
* 08:12 Andrew: re-scapping, typo in extension-list
* 08:12 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 86895: 
* 08:11 Andrew: Scapping to enable DisableAccount extension
* 08:11 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 86895: 
* 08:02 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 86895: 
* 08:02 Andrew: running scap to deploy the code itself
* 08:01 Andrew: deploying DisableAccount extension to checkuserwiki, stewardwiki, arbcom_enwiki since the special page was removed without consulting Philippe
* 02:15 logmsgbot: robh synchronized php-1.17/wmf-config/InitialiseSettings.php 'adding settings for checkuser and steward wikis'
== April 25 ==
* 23:33 Ryan_Lane: added python-mwclient to lucid repo
* 21:36 RobH: storage2 still offline, wont boot into os, but is remotely accessible
* 21:20 RobH: trying to fix storage2
* 20:16 notpeter: actually adding everyone on ops to watchmouse service... didn't know this had not already been done.
* 20:02 RobH: updated csw1 to removed labels and move to default vlan ports 11/12, 11/14, 11/19, & 11/21. old connection ports for dataset2, tridge, ms1, and ms5
* 19:53 RobH: the datacenter is looking awesome.
* 19:45 RobH: ms1 moved from temp network to permanent home, no downtime, responding fine
* 19:42 RobH: ms5 connection moved, no downtime, responds fine, less than 4 seconds
* 19:40 RobH: updated csw1-sdtpa 15/1,15/2 from vlan 105 to vlan 2, 15/3 and 15/4 from vlan 105 to 101
* 18:52 RobH: snapshot4 relocated to new home, ready for os install
* 18:42 RobH: db19 and db20 back online (not in services as they have other issues)
* 18:39 RobH: db19 and db20 powering back up
* 18:25 RobH: virt4 experienced an accidental reboot when rebalancing power in the rack, my fault, not the hardware
* 18:12 RobH: rack b2 power rebalanced
* 18:01 RobH: db19 set to slave, depooled in db.php, no other services evident, shutting down (mysql stopped cleanly)
* 18:00 RobH: db20 shutdown
* 18:00 RobH: didnt log that i setup ports 11/38-40 for db19, db20, and snapshot4 on csw1-sdtpa. tested out fine and all my major configuration changes on netowrk should be complete
* 17:56 RobH: ok, db20 and db19 are coming offline to relocate their rack location due to power distro issues
* 15:47 RobH: delay, not coming down yet, need more cables
* 15:46 RobH: db19 is coming down as well, it is depooled anyhow
* 15:46 RobH: db20 is coming down, ganglia aggregation for those hosts may be delayed until it is back online.
* 15:21 RobH: relocating snapshot4 into rack c2, it will be offline during this process
* 15:20 RobH: db43-db47 network setup, sites not down, yay me
* 15:10 RobH: being on csw1 makes robh nervous.
* 15:09 RobH: labeling and setting up ports on 11/33 through 11/37 on csw1-sdtpa for db43 through db47
* 14:47 RobH: fixed storage2 serial console (set it to higher rate, magically works, or it just fears me) and also confirmed its remote power control is functioning
* 14:42 RobH: stealing dataset1's known good scs connection to test storage2. dataset1 service will remain unaffected.
== April 24 ==
* 21:30 Ryan_Lane: restarting apache on mobile1
* 15:35 RobH: swapping bad disk in db30, hotswap, should be fine
* 14:36 RobH: swapping out the management switch in c1-sdtpa. msw-c1-sdtpa will be offline, so the mgmt interfaces of servers in that rack will be offline. all normal services will remain unaffected.
== April 23 ==
* 22:31 RobH: required even.
* 22:31 RobH: no drives display error leds, futher investigation requried
* 22:27 RobH: ms2 is having bad drive investigated. if we do this right, it wont go down. if we don't it will. is a slave es server.
* 22:00 RobH: singer returned to operation, blog, techblog, survey, and secure returned to normal operation
* 21:52 RobH: singer is once again coming back down for drive replacement. This will take offline blog.wikimedia.org, techblog.wikimedia.org, survey.wikimedia.org, and secure.wikipedia.org. Service will be returned as soon as possible. 
* 21:19 RobH: singer back online, for awhile, will come back down for further repair shortly.
* 21:05 RobH: singer going down, blogs will be offline, so will secure, system will return to service as soon as possible
* 21:00 RobH: preparing to fix the dead drive in singer, this will offline secure, blog, techblog, and survey during the drive replacement process
* 19:49 mark: Upgrading mr1-pmtpa to junos 10.4R3.4
* 17:49 RobH: migrating searchidx1 & search1-search10 to new ports in same rack. moving one at a time and ensuring link lights between moves. (already tested with search10)
* 14:11 RobH: db19 is back online, seems to not have any mysql setup done.
* 14:02 RobH: restarting db19
* 14:02 RobH: arcconf checks out all drives on db19 are indeed working as rich found earlier
* 12:47 mark: Added (x121Address=1) condition to the LDAP query of the ldap_aliases router on mchenry's exim
* 00:32 hcatlin: Mobile: Deploying fix to an issue that kept the standard-style Main_Page from displaying on mobile
* 00:25 Ryan_Lane: restarting memcached on all of the mobile servers
* 00:23 Ryan_Lane: repooling mobile3, since mobile will die without it (fun!!)
* 00:17 Ryan_Lane: depooling mobile3
* 00:13 Ryan_Lane: restarting apache on mobile3
* 00:10 Ryan_Lane: puppet was broken on mobile1, reinstalled it
== April 22 ==
* 23:56 domas: detached gdb from srv193 apache, apparently it was used for something
* 23:14 notpeter: restarting nagios (again)wq
* 22:43 notpeter: restarting nagios
* 19:23 apergos: shot all stopped rsyncs on ms5 (that were copying from ms4 about two weeks ago), changed all perms on the directories they had reached so thumbs can be served/read from them.. oh. not me, someone else must have done it, I'm not here :-P
* 19:02 RobH: ms4 shutting down for memory troubleshooting
* 18:52 RobH: ms4 troubleshooting, disragrd bounces]
* 18:51 notpeter: restarting nagios
* 12:41 hcatlin: Restarting mobile cluster with April code update.
* 00:49 notpeter: restarting nagios. hopefully now with more sms!
== April 21 ==
* 23:32 logmsgbot: midom synchronized php-1.17/includes/ImagePage.php 
* 20:54 logmsgbot: pdhanda synchronized live-1.5/404.php 
* 19:02 domas: bumped up maxclients/serverlimit on singer to 350 (up from 150), set maxrequestsperchild to 30 to avoid heap blowup (down from 0), all governed via apache2/conf.d/maxrequests
* 18:25 Ryan_Lane: restarting apache on singer
* 18:19 Ryan_Lane: applying system patches to raskin
* 17:47 Ryan_Lane: restarting apache on singer
* 17:04 logmsgbot: pdhanda synchronized live-1.5/404.php 
* 16:52 rainman-sr: copying stuff from /home/ariel/searchidx/ to searchidx2 as it didn't get copied
* 15:32 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Fixing typo to properly check for wgUseEmailCapture'
* 15:29 logmsgbot: awjrichards ran sync-common-all 
* 15:06 logmsgbot: awjrichards synchronized php-1.17/wmf-config/InitialiseSettings.php 'Added section for EmailCapture, enabling on testwiki'
* 15:05 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Added section for EmailCapture'
* 14:50 logmsgbot: awjrichards synchronizing Wikimedia installation... Revision: 86464: 
* 14:49 awjr: running scap on fenari to dark-deploy EmailCapture extension
* 03:44 Ryan_Lane: uninstalled python-json on wikitech, and installed python-simplejson
* 01:45 Ryan_Lane: testing statusnet library again
* 01:45 Ryan_Lane: testing statusnet library again
* 01:44 Ryan_Lane: testing statusnet library again
* 01:35 Ryan_Lane: testing a statusnet library for morebots that has a sane license
* 01:20 Ryan_Lane: installing ca-certificates on wikitech
* 01:11 Ryan_Lane: installing curl on wikitech
* 01:04 Ryan_Lane: installing python-json on wikitech
* 00:11 Ryan_Lane: added a sweet, special message to morebots
* 00:03 notpeter: moving some searchidx files into puppet's sphere of influnece.
== April 20 ==
* 23:10 rainman-sr: search indexing is up and running again, might take a couple of days for things to catch up though
* 23:07 rainman-sr: doing a slow restart of the whole search cluster (2 minute intervals between host restarts)
* 22:52 rainman-sr: starting search indexer on searchidx2
* 16:29 mark: Squids finished rebuilding, removed iptables rules
* 16:21 mark: Removed iptables rules on sq77/78
* 16:09 mark: Downgraded sq65
* 16:06 mark: removed iptables rule on sq66
* 16:05 Ryan_Lane: downgrading squid on knsq27, knsq28, knsq29, knsq30, knsq26, knsq25, knsq24
* 16:02 Ryan_Lane: downgrading squid on amssq31
* 16:01 Ryan_Lane: downgrading squid on amssq38
* 16:01 Ryan_Lane: downgrading squid on amssq39
* 16:01 Ryan_Lane: downgrading squid on amssq40
* 16:01 Ryan_Lane: downgrading squid on amssq41
* 16:00 Ryan_Lane: downgrading squid on amssq37
* 15:59 Ryan_Lane: downgrading squid on amssq36
* 15:59 Ryan_Lane: downgrading squid on amssq32
* 15:59 Ryan_Lane: downgrading squid on amssq33
* 15:58 Ryan_Lane: downgrading squid on amssq34
* 15:58 Ryan_Lane: downgrading squid on amssq35
* 15:57 mark: Downgrading Squid on sq71-78
* 15:57 Ryan_Lane: downgrading squid on amssq42
* 15:56 Ryan_Lane: downgrading squid on amssq43
* 15:56 Ryan_Lane: downgrading squid on amssq45
* 15:54 Ryan_Lane: downgrading squid onamssq45
* 15:53 Ryan_Lane: downgrading squid on knsq23
* 15:33 RobH: lowered maxclients more on singer, its going to slow down secure, but hopefully keep it online longer.
* 15:04 Ryan_Lane: downgraded sq71 to squid-2.7.7
* 14:51 mark: Pulled squid 2.7.9 packages from the lucid-wikimedia reprepro APT repository, reinstated 2.7.7
* 14:34 mark: Downgrading squid on sq66
* 14:15 mark: Stopping squid on sq72
* 14:14 RobH: reduced singer apache maxclients from 400 to 200, hopefully will reduce singer crashes
* 14:09 RobH: singer died, rebooting.
* 14:07 RobH: secure, blogs on singer, bouncing due to singer cpu max
* 13:53 RobH: singer threw invalid cert warning, cpu spike was occuring, restarting apache and load is normalizing
* 11:34 logmsgbot: catrope ran sync-common-all 
* 11:33 RoanKattouw: Running sync-common-all to deploy r86464
* 11:11 logmsgbot: catrope synchronized php-1.17/includes/Exif.php 'Fix for bug 28615 (fatal on image description page)'
* 00:41 awjr: deployment of r86441 to payments servers complete
* 00:37 awjr: deploying changes to send full donor address information to PayflowPro (r86441) to all payments servers
* 00:31 awjr: re-deploying changes to send full donor address information to PayflowPro on payments2.wikimedia.org (suspected cause of previous issue: local browser cacheing)
* 00:14 logmsgbot: midom synchronized php-1.17/wmf-config/db.php 
== April 19 ==
* 23:24 richcole1: rebooting db11 to check raid
* 22:57 awjr: reverting to r84367 of deployment branch on payments2.wikimedia.org due to bad address information appearing in civicrm
* 22:53 awjr: deploying changes to send full donor address information to PayflowPro payments2.wikimedia.org
* 22:20 Ryan_Lane: virt1-4 installed
* 20:54 Ryan_Lane: powercycling singer
* 20:53 Ryan_Lane: restarting apache on spence
* 20:52 mark: powercycled singer
* 20:35 Ryan_Lane: powercycling srv196
* 17:31 Ryan_Lane: finished ams and kn text squids
* 17:17 RobH: change live, etherpad still online and functional
* 17:16 RobH: pushing change to hooper so it doesnt kill etherpad when other apache vhosts exist
* 17:16 RobH: updated puppet entry for etherpad vhost
* 16:54 Ryan_Lane: upgrading squid on all squid systems, slowly, over the course of the day
* 15:09 mark: Killed all etherpad-user processes and started etherpad
* 15:09 RoanKattouw: Removing data for the section edit link experiment from the clicktracking table. Data is backed up in /home/catrope/sel . This will delete 8.4M of the 9.6M rows in the clicktracking table
* 15:05 RoanKattouw: Etherpad is broken, serving 500s. Have e-mailed Peter
* 14:06 logmsgbot: demon synchronized php-1.17/includes/diff/DifferenceEngine.php 'r86395'
* 12:50 mark: NICEd bayes processing cron job on mchenry
* 10:14 RoanKattouw: <logmsgbot>	!log catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump $wgArticleFeedbackTrackingVersion to 1'
* 10:14 RoanKattouw: <logmsgbot>	!log catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'r86384'
* 10:14 RoanKattouw: <logmsgbot>	!log catrope synchronized php-1.17/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'r86299'
* 10:14 RoanKattouw: Restarted morebots
* 01:05 Tim: added an entry to /etc/sudoers on fenari and hume allowing wikidev users to run commands as apache
== April 18 ==
* 21:28 mark: Setup PIM-DM on cr1-eqiad:ae1.666 (pseudowire port to pmtpa) and csw5-pmtpa:ve10 (counterpart) to get multicast routing from tampa
* 21:24 mark: Setup PIM-SM on cr1-eqiad and cr2-eqiad, with static RPs and a local RP on cr1-eqiad. PIM is running on all production subnets.
* 16:16 pdhanda: running maintenance/CleanupImages on all.dblist
* 15:58 mark: Rolling restart of pmtpa text frontend squids; one every two minutes
* 14:27 mark: Powercycled db11
* 14:13 logmsgbot: mark synchronized php-1.17/wmf-config/db.php 'depool db11'
* 14:12 Reedy: DB11 related issues. Ops are looking at the issue
* 10:30 logmsgbot: root synchronizing Wikimedia installation... Revision: 86086: 
* 10:05 logmsgbot: root synchronizing Wikimedia installation... Revision: 86086: 
* 08:43 Tim: running /home/wikipedia/bin/l10nupdate to get r86294
== April 17 ==
* 20:27 mark: Setup dirty "pseudowire" between pmtpa and eqiad; a GRETAP tunnel inside a v4-in-v6 tunnel bridged to eth1/eth0.666 on 2 linux boxes (sq71 on pmtpa side). Now running OSPF on it to get full v4 interconnectivity
== April 16 ==
* 08:33 apergos: disabled snaps on searchidx on in rainman's crontab, moving some en.wiki.* files in /a/search/indexes/snapshot off to fenari:/home/ariel/searchidx/.... 
== April 15 ==
* 18:41 mark: Rsynced private thumbs dir from ms4 to ms5
* 15:45 mark: Upgraded re0.cr1-eqiad to JUNOS 10.4R3.4
* 15:27 mark: Switching over routing engines from re0 to re1 on cr1-eqiad
* 15:23 mark: Upgraded re1.cr1-eqiad to JUNOS 10.4R3.4
* 14:44 mark: Upgraded re0.cr2-eqiad to JUNOS 10.4R3.4
* 14:34 mark: Upgraded re1.cr2-eqiad to JUNOS 10.4R3.4
* 09:57 apergos: moving some old (from 2009) dirs from /a/search/indexes/import on searchidx1 to /home/ariel/searchidx1/... on fenari, if I'm told we can toss em outright I'll do that later.
* 06:01 apergos: cleared out space on searchidx1 again. we only have 4.5gb free and it's still at 100%. out of tricks...
* 00:54 notpeter: py making some changes to nagios. subscribing the service to lots of files. also, lots more files being pushed out by puppet.
== April 14 ==
* 23:41 Ryan_Lane: adding reverse mappings for virt1-4
* 23:28 Ryan_Lane: adding dns entries for virt1-4
* 22:28 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Re-enable ArticleFeedback on enwiki, upgrade is complete'
* 22:09 RoanKattouw: Running populateAFRevisions.php on enwiki
* 22:06 RoanKattouw: Creating article_feedback_revisions table on enwiki
* 22:04 RoanKattouw: Clearing message blobs
* 22:03 logmsgbot: catrope ran sync-common-all 
* 22:03 RoanKattouw: Deploying ArticleFeedback changes to the cluster with sync-common-all
* 22:02 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable ArticleFeedback on enwiki for upgrade'
* 21:42 mark: Added vlan 103 to ports e 15/5 to 15/8 (virt1-4 eth1) (tagged) on csw1-sdtpa
* 21:42 mark: Put ports e 15/1 to 15/4 (virt1-4 eth0) in vlan 105 (untagged) on csw1-sdtpa
* 21:40 mark: Created VLAN 105 ("virt-hosts") on csw1-sdtpa, assigned virtual interface ve 9, assigned ip 10.4.16.0/24
* 21:39 mark: Added virtual interface ve 7 ("virt") on csw1-sdtpa, assigned ip 10.4.0.1/24
* 21:39 mark: Removed virtual interface ve 7 ("virt") on csw5-pmtpa
* 21:24 Ryan_Lane: cleaning up searchidx1
* 21:11 RoanKattouw: Running extensions/ArticleFeedback/populateAFRevisions.php on testwiki
* 21:11 RoanKattouw: Creating article_feedback_revisions table on testwiki
* 21:08 RoanKattouw: Updating ArticleFeedback code on test
* 16:18 RobH: singer back online, blogs, survey, and secure should all be back in service
* 16:15 RobH: blogs offline due to singer outage, as well as survey, working on resolution
* 16:14 RobH: powercycling singer, unresponsive to ssh and to serial console
* 16:13 RobH: secure issue is being investigated
* 16:13 pdhanda: running cleanupTitles.php for all wikis in all.dblist
* 16:13 RobH: investigating singer cpu spike
* 15:36 mark: Implemented LDAP lookups on mail relay mchenry
* 14:13 Reedy: Delayed message. Ran repopulateCodePaths on mediawikiwiki
* 14:13 Reedy: Delayed message. Ran populateFollowupRevisions on mediawikiwiki
* 14:04 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/modules/ext.codereview.js 'Disable comment field autofocus'
* 07:54 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/CodeReview.php 'Fix CodeReview fatals'
* 07:50 logmsgbot: catrope ran sync-common-all 
* 07:47 RoanKattouw: Running sync-common-all and clearMessageBlobs.php to deploy UploadWizard changes
* 07:35 RoanKattouw: Updating MediaWiki on fenari to r86034 for UploadWizard deployment. Only pushing changes to testwiki for now
* 07:14 logmsgbot: tstarling synchronized php-1.17/includes/WebRequest.php 'r86031'
* 07:14 logmsgbot: tstarling synchronized php-1.17/img_auth.php 'r86031'
== April 13 ==
* 23:18 notpeter: restarting nagios
* 21:15 notpeter: restarting nagios
* 19:54 awjr: disabling drupal/civicrm cron on contacts.wikimedia.org (singer) as part of an interim solution to civimail problems in CiviCRM 3.4beta1
* 18:31 ^demon: ran code review updates for mediawikiwiki
* 12:54 mark: Restarted pdns on linne
* 12:54 mark: Defined /64 neighbor blocks in 1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa in DNS
* 12:31 mark: Defined /31 neighbor blocks in 154.80.208.in-addr.arpa in DNS
* 09:15 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'increase account creation throttle value for el wiki for editing workshop (soon we have a two week break from these :-P)'
* 01:55 Ryan_Lane: upgrading squid to 2.7.9-5 on sq58 and sq73
* 01:35 Ryan_Lane: upgrading squid to 2.7.9-4 on sq58 and sq73
* 00:42 Ryan_Lane|food: also removing sheepdog from nova-*.tesla
* 00:39 Ryan_Lane|food: removing natty versions of libvirt and friends and re-installing the lucid versions
== April 12 ==
* 20:18 RobH: killed apache vhost for outreachcivi since it appears defunct in favor of contactscivi on singer
* 19:08 Ryan_Lane: upgrading nova-* on nova-*.tesla
* 19:05 Ryan_Lane: updating squid on sq73 to test new squid and patches
* 05:51 Ryan_Lane: purging nova-ajax-console-proxy from nova-compute1/2.tesla and adding to nova-controller.tesla
* 04:57 Ryan_Lane: purging nova-ajax-console-proxy from nova-controller.tesla and adding to nova-compute1/2.tesla
* 02:11 logmsgbot: tstarling synchronized php-1.17/includes/Sanitizer.php 'r85859'
* 01:30 logmsgbot: root synchronizing Wikimedia installation... Revision: 85853: 
* 01:28 Tim: svn up/scap to deploy r85848
* 00:31 Ryan_Lane: installing glance on nova-controller.tesla
== April 11 ==
* 22:13 Ryan_Lane: installing python-nova.adminclient on nova-controller.tesla
* 21:19 notpeter: added lag check to storage3. restarting nagios
* 20:33 Ryan_Lane: pushing squid-2.7.9-1wm1 to wikimedia-lucid repo
* 05:15 Tim: did squid configuration update for bug 28235
* 04:40 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php 'Made categories reviewable on kawiki'
* 03:23 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php 
* 03:21 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'flaggedrevs for kawiki'
* 03:13 AaronSchulz: Enabled FlaggedRevs for Georgian Wikipedia
* 03:11 logmsgbot: aaron ran sync-common-all 
== April 10 ==
* 05:22 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'increase account creation throttle value for el wiki for editing workshop (can't wait for local sysadmins to be able to do this :-P)'
== April 9 ==
* 17:07 logmsgbot: ariel ran sync-common-all 'sync for elwikinews round 2, let's get the import right this time folks cause this is too nerve-wracking'
* 16:40 logmsgbot: ariel synchronized all.dblist 'remove elwikinews, need to drop and recreate after borked import'
* 16:29 logmsgbot: robh synchronized php-1.17/wmf-config/InitialiseSettings.php 'logo updates for a couple wikis for phillipe'
* 10:10 mark: Changed dataset1's clock source to HPET, synced it with ntpdate and restarted ntpd
* 07:58 apergos: power cycling searchidx1, load was at 60, unresponsive to commands after login from mgmt console
* 02:46 RobH: troubleshooting a couple new wikis, had to sync-apaches and restart them gracefully
* 01:01 notpeter: changed my.cnf on storage3 to replicated-do-db= drupal,mysql,civicrm
* 00:50 Ryan_Lane: installing nova-ajax-console-proxy on nova-controller.tesla
== April 8 ==
* 23:19 logmsgbot: laner ran sync-common-all 
* 23:18 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 22:50 Ryan_Lane: that graceful was me
* 22:28 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 22:16 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 22:14 logmsgbot: laner synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 22:07 logmsgbot: laner ran sync-common-all 
* 22:06 Ryan_Lane: gave myself deploy access in svn
* 17:10 notpeter: pushing out new dns zones. forgot to change ptr record for yvon...
* 15:09 RobH: updating dns with testblog info
* 13:36 mark: Added swap on /dev/sdc1 and /dev/sdd1 on ms5
* 13:34 mark: Stopped RAID10 array /dev/md2 again, sync takes too long
* 13:30 mark: Created RAID10 array for swap across first partition of 46 drives on ms5
* 13:21 mark: Stopped all rsyncs to investigate ms5's sudden kswapd system cpu load
* 07:57 apergos: assigned snapshot1 internal ip in dns
* 06:11 apergos: moving snapshot1 to internal vlan etc
* 04:15 notpeter: pushing new dns w/ eixamanius as cname for hooper and yvon as new name for box that was previously eixamanius
* 04:15 notpeter: stopping etherpad
== April 7 ==
* 21:43 notpeter: removed a silly check for hooper that I made and restarted nagios
* 19:06 Ryan_Lane: switching openstack deb repo back to trunk, and upgrading packages on nova-controller, since we are likely to target cactus now
* 15:40 mark: Restarted rsyncs
* 15:26 mark: Created a test btrfs snapshot of /export on ms6
* 15:12 mark: Temporarily stopped the rsyncs on ms5 to test zfs send performance
* 13:56 mark: Reenabled ms6 as backend on esams.upload squids
* 13:11 apergos: replaced ms4 in fstab on fenari with ms5 so we have thumbs mounted there
* 12:08 mark: Restarted rsyncs on ms5
* 12:07 apergos: nginx conf file change to "thumb" added to puppet 
* 12:00 mark: Removed the test snapshot on ms5
* 11:47 apergos: edited in place /etc/nginx/sites-available/thumbs and /export/thumbs/scripts/thumb-handler.php to make thumbs generated on the fly return 200. they were returning 404
* 10:25 logmsgbot: catrope synchronized php-1.17/thumb.php 'Attempted fix for wrong temp/thumb paths'
* 10:20 apergos: (after reports from en vp that the search index has not been updated for over 4 days)
* 10:18 apergos: restarting search indexer on searchidx1 
* 09:35 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialUploadStash.php 'debugging'
* 09:08 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialUploadStash.php 'r85612'
* 08:23 logmsgbot: catrope synchronized php-1.17/extensions/UploadWizard/UploadWizard.php 'Fix fatal due to missing API module'
* 08:17 logmsgbot: catrope ran sync-common-all 
* 08:16 RoanKattouw: I meant srv196, not srv193
* 08:15 RoanKattouw: Deploying UploadWizard for real this time, forgot to svn up first. sync-common-all then clearMessageBlobs.php
* 08:14 RoanKattouw: Commenting out srv193 in mediawiki-installation node list because its timeouts take forever
* 08:10 RoanKattouw: srv196 is not responding to SSH or syncs from fenari (they time out after a looong time) but Nagios says SSH is fine. Should be fixed or temporarily depooled
* 08:08 RoanKattouw: Clearing message blobs
* 08:07 logmsgbot: catrope ran sync-common-all 
* 08:04 RoanKattouw: Scap broke with sudo stuff AGAIN, running sync-common-all
* 08:01 RoanKattouw: Running scap to deploy UploadWizard changes
* 07:11 apergos: turned em off again, started seeing timeouts. bah
* 06:39 apergos: and two more...
* 06:31 apergos: restarted two of the 8 rsyncs on ms5, keeping an eye on them
* 01:31 domas: added nobarrier to xfs mount options on db32 and db37
== April 6 ==
* 20:38 RobH: updated puppet with a svn::client class (rt#721)
* 20:18 RobH: pulled wm09schols, wm10schols, and wm10reg out of enabled sites on singer
* 20:05 apergos: suspended all rsyncs on ms5, we were seeing nfs timeouts on the renderers all of a sudden
* 18:50 apergos: killed morebots and let the restart script start it up again
== April 5 ==
* 23:00 Ryan_Lane: restarting search indexer on searchidx to free space held by deleted logs
* 22:58 Ryan_Lane: clearing up some space on searchidx1
* 22:20 notpeter: crammed an etherpad db into db9's mysql hole.
* 17:57 Ryan_Lane: restarting llsearchd on all search boxes
* 17:45 RoanKattouw: Restarted morebots, running on wikitech as catrope
* 17:45 Ryan_Lane: changing the udp log location for search to emery
* 12:16 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Undo $wgForceUIMsgAsContentMsg change on incubator from last night per DannyB'
== April 4 ==
* 23:28 Ryan_Lane: uploading ircecho package to lucid-wikimedia repo, for nagios irc bot
* 22:22 Ryan_Lane: upgrading wikimedia-task-appserver package on srv281
* 22:22 Ryan_Lane: uploading new version of wikimedia-task-appserver to lucid-wikimedia repo; merges back in 1.17 changes that were missing
* 22:04 RobH: updated noc robots entry in its apache config on fenari
* 21:58 Ryan_Lane: srv281 is acting as a temporary scaling server for testing of lucid imagescalers, and to help with thumbs load.
* 21:27 Ryan_Lane: depooling srv281 from appservers
* 21:21 Ryan_Lane: syncing apaches to get configuration pushed to srv281
* 21:16 Ryan_Lane: rebooting srv281
* 21:01 Ryan_Lane: adding srv281 to rendering cluster in pybal via fenari
* 20:32 Ryan_Lane: uploading a new version of wikimedia-task-appserver fixing a problem with sync-common
* 20:13 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add mainpage to $wgForceUIMsgAsContentMsg for incubatorwiki'
* 19:55 Ryan_Lane: srv281 successfully ran imagescaler puppet class. ready for testing.
* 19:47 Ryan_Lane: adding php5-fss to lucid-wikimedia repo
* 19:11 Ryan_Lane: adding wikimedia-task-appserver to lucid-wikimedia repo
* 18:58 RobH: bugzilla updates complete
* 18:50 RobH: updating bugzilla per rt#718 bz#28409 bz#28402
* 18:42 notpeter: added cname etherpad for hooper.wikimedia.org
* 18:00 Ryan_Lane: added the wikimedia-fonts package to lucid-wikimedia repo
* 17:29 notpeter: adding self to nagios group. rebooterizing nagios.
* 05:58 apergos: cleaned up perms on commons/thumb/a/af, left over from interrupted rsync test last night
* 05:50 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling pool counter on all wikis'
* 04:12 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling PoolCounter on testwiki and test2wiki'
* 01:22 Tim: apache CPU overload lasted ~10 mins, v. high backend request rate, don't know cause, seems to have stopped now
== April 3 ==
* 18:42 apergos: 8 rsyncs of ms4 thumbs restarted with better perms so scalers can write... in screen as root on ms5. If we start seeing nfs timesouts in the scaler logs please shoot a couple
* 17:14 mark: Deployed max-connections on all cache peers for esams.upload squids to their florida parents (current limit 200)
* 17:00 mark: Removed the carp weights on the esams backends again, as the weighting was completely screwed up
* 16:59 mark: Started knsq13 backend
* 14:27 logmsgbot: catrope ran sync-common-all 
* 14:26 RoanKattouw: Running sync-common-all to deploy r85256
* 13:03 apergos: shot rsyncs on ms5, setting 777 dir perms on all thumbnail dirs (eg e/ef/blablah.jpg) so scalers can write into them
* 12:53 apergos: did same for rest of projects and subdirs (777 on hash dirs)
* 12:46 apergos: chmod 777 on commons/thumb/*/* on ms5 so that scalers can create directories in there (mismatch of uid apache vs www-data)
* 11:12 mark: Raised per-squid connection limit to ms5 of 200 to 400 connections
* 11:05 mark: Raised per-squid connection limit to ms5 of 100 to 200 connections
* 10:55 mark: Fixed squid loop, the pmtpa.upload squids were using the esams squids as "CARP parents for distant content"
* 10:29 mark: Fixed puppet on sq42/43
* 09:44 mark: Lowered FCGI thumb handlers from 90 to 60 again, to reduce concurrency
* 08:08 mark: Started 4 more rsyncs (8 total now)
* 07:49 mark: Removed mlocate from ms5, puppetising
* 07:42 mark: Started 4 rsyncs from ms4 to ms5 (--ignore-existing)
* 07:32 mark: increased thumb handler count from 60 to 90
* 07:11 mark: Doubled the amount of fcgi thumb handlers
* 07:08 mark: Turned off logging of 404s to nginx error.log
* 06:50 mark: Restarted Apache on the image scalers
* 06:49 mark: Reconfigured ms5 to use the 404 thumb handler
* 06:48 Ryan_Lane: disabling nfs on ms4
* 06:33 mark: Running puppet on all apaches to fix fstab and mount ms5.pmtpa.wmnet:/export/thumbs
* 06:32 mark: Unmounting /mnt/thumbs on all mediawiki-installation servers
* 06:30 mark: Remounted NFS /mnt/thumbs on the scalers to ms5
* 06:28 Ryan_Lane: bring nfs back up
* 06:28 Ryan_Lane: brought ms4 back up. stopping the web server service and nfs
* 06:20 mark: Setup NFS kernel server on ms5
* 06:18 Ryan_Lane: powercycling ms4
* 05:29 Ryan_Lane: rebooting ms4 with -d to get a coredump
* 05:14 apergos: reanbling webserver on ms4 for testing
* 04:45 apergos: stopping web service on ms4 for the moment
* 04:29 apergos: shot webserver again
* 04:26 apergos: turned off hourly snaps on ms4, turned back on webserver and nfs
* 04:09 apergos: rebooted ms4, shut down webserver and nfsd temporarily for testing
* 02:58 apergos: still looking at kernel memory issues, still rebooting, ryan should be here in a few minutes to help out
* 02:03 apergos: a solaris advisor... also have zfs arch cache max to 2g which is ridiculously low but wtf right?
* 02:02 apergos: set tcp_time_wait_interval to 10000 at suggestion of
* 01:37 apergos: lowered zfs arch max to 2g (someone should reset this later)... will take effect on next reboot
* 00:29 apergos: rebooting with the new zfs arc cache max value, which will reduce the min value as well... dunno if this will give us enough breathing room or not
* 00:24 apergos: set zfs arc cache to ridiculously low value of 4gb, since when it's healthy it's using much less than that (1gb), this will take effect on reboot
* 00:22 Reedy: Still experiencing MS4 issues, thumb service is likely to be problematic for most users
== April 2 ==
* 23:47 apergos: rebooting ms4 from serial console, out to lunch and took the renderers down too 
* 18:42 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Per NeilK, change Category:Uploaded_by_UploadWizard to Category:Uploaded_with_UploadWizard'
* 17:59 mark: Upgrading varnish to 2.1.5
* 17:14 logmsgbot: demon synchronized php-1.17/includes/filerepo/LocalFile.php 'r85200'
* 14:19 mark: Implemented CARP weights for distant CARP parents on squid configurator (used to be all equal before)
* 11:36 mark: Created btrfs filesystem on ms6, striped (raid10 style) over 46 devices - very experimental
* 09:50 mark: Reinstalling ms6 with Ubuntu 10.04
* 09:50 mark: Fixed torrus again
* 06:02 mark: !wikipedia The image thumbnail servers appear stable now
* 04:59 mark: Increased nginx worker processes from 1 to 4, set file limit to 30k
* 04:40 mark: !wikipedia Image Thumbnail server outage, it's being worked on
* 04:34 mark: Power cycling ms4 again
* 04:06 mark: Power cycled ms4 again
* 04:02 mark: Removed ms4 from pmtpa.upload config, sending all thumbs to ms5
* 03:47 mark: Restarted rsyncs ms4->ms5
* 03:25 Ryan_Lane: powercycling ms4 again
* 02:59 Ryan_Lane: rebooting ms4
* 02:46 Ryan_Lane: seems ms4 is totally dead, powercycling it
* 01:09 Ryan_Lane: installing python-pyinotify on spence for an updated ircecho
== April 1 ==
* 21:35 Ryan_Lane: purging some binlogs on db9 to free up space
* 21:35 RobH: bugzilla now version 4
* 21:31 RobH: taking down bugzilla for a quick upgrade
* 18:48 Ryan_Lane: added ctwoo, brion, py, and reedy to the engineering alias
* 18:36 mark: Deployed ms5.pmtpa.wmnet as a special 'apache' for pmtpa squid uploads... now serving a small portion of commons thumbs
* 18:11 RobH: bugzilla back online, CRproxy was affected, and repaired
* 17:30 RobH: bugzilla.wikimedia.org going offline for database backup and upgrade
* 17:13 RobH: beginning upgrade process for bugzilla, it's availability will be in question during this time
* 16:59 mark: Turned off Etag in the webserver7 configuration (/opt/webserver7/https-ms4/config/obj.conf) on ms4
* 16:50 notpeter: rm-ing old binlogs on db9 after confirming that there is no slave lag on db10
* 15:53 mark: Puppetised nginx and htcp purger setup on ms5
* 11:36 apergos: restarted lighty on dataset2 (but why did it die?)
* 00:06 logmsgbot: tstarling synchronized php-1.17/includes/specials/SpecialImport.php 'r85099'
== March 31 ==
* 20:52 notpeter: also, restarting t3h nagioz
* 20:51 notpeter: added self to /etc/nagios/contacts.cfg
* 20:32 logmsgbot: awjrichards synchronized php-1.17/extensions/UserDailyContribs/api/ApiUserDailyContribs.php 'r85088 updates to UserDailyContribs API to retrieve past year edit count'
* 18:58 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialImport.php 'r85078'
* 14:58 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php 
* 14:58 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'same cleanup; ptwikinews (just did sqwiki)'
* 14:49 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php 
* 14:48 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'moved add/removgroups settings from flaggedrevs.php to here'
* 14:44 mark: power cycled singer
* 14:40 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'added auteditor group on bewiki'
* 14:36 mark: Temporarily overriding reload caching request headers on upload squids to ease ms4 load
* 14:35 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php 'added auteditor group on bewiki; removed redundant config settings'
* 14:24 mark: Remounted /export/thumbs with noatime on ms4
* 14:21 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php 'added stablesettings right to sysop on bewiki'
* 14:01 mark: Outage resolved by disabling ehcache
* 13:57 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 13:51 logmsgbot: root synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 13:37 logmsgbot: mark synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable ehcache'
* 11:43 mark: set max_stale 1 month on pmtpa.upload squids
* 11:21 mark: Set refresh_pattern on pmtpa upload squids
* 10:47 mark: Set refresh_pattern on sq85
* 00:21 Ryan_Lane: added drac management script and dependencies to fenari, via puppet
* 00:06 Ryan_Lane: make that python-paramiko
* 00:06 Ryan_Lane: installing paramiko on fenari
== March 30 ==
* 23:15 Ryan_Lane: adding puppet configuration for the collector and adding that class to spence
* 23:09 Ryan_Lane: updating udpprofile package again on spence
* 23:02 Ryan_Lane: upgrading udpprofile package on spence to fix init script
* 22:48 Ryan_Lane: killing the collector on spence, and trying to run via the package
* 22:48 Ryan_Lane: installing udpprofile package on spence
* 22:44 Ryan_Lane: created a package for the udpprofile code in svn; added to the main lucid-wikimedia repo
* 19:33 Ryan_Lane: fixed sanger certificate check for ldaps. warning will go away on next puppet run
* 17:05 Ryan_Lane1: enabled a nagios certificate check on sanger for ldaps
* 16:57 thedj: Maintenance ongoing on image server. Thumbnails on #Wikimedia sites may appear broken; we expect to restore full service shortly. (by Danese)
* 15:23 apergos: s/fine/find/
* 15:22 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'increase account creation throttle value for el wiki for editing workshop (maybe we should fine a better solution?)'
* 13:03 mark: Started 8 parallel rsyncs of ms4 /export/thumbs/wikipedia/commons/thumb/{8..f} to ms5
* 10:50 Tim: restarted ehcache with increased cache size limits
* 10:49 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabling ehcache'
* 10:45 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling ehcache temporarily'
* 10:15 apergos: rebooting ms5, trying to fix up raid config
* 08:52 logmsgbot: catrope ran sync-common-all 
* 08:52 RoanKattouw: Running sync-common-all to push out UploadWizard changes
* 08:30 RoanKattouw: Deploying UploadWizard cahnges to testwiki
* 05:47 Tim: restarted profile-stats-logger on fenari, it died when spence went down
* 05:39 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabled ehcache on all wikis'
* 05:29 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabled ehcache'
* 05:27 Tim: got OOM. Restarted tomcat with substantially more than the default heap maximum of 128MB
* 05:19 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling ehcache, something is wrong'
* 05:05 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enable ehcache on enwiki'
* 05:00 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enable ehcache on enwikiquote'
* 04:59 logmsgbot: tstarling synchronized php-1.17/includes/GlobalFunctions.php 'live hack: send site notice cache to memcached not ehcache'
* 04:58 logmsgbot: tstarling synchronized php-1.17/includes/MessageCache.php 'another one'
* 04:55 logmsgbot: tstarling synchronized php-1.17/includes/Skin.php 'live hack to send sidebar cache to memcached not ehcache'
* 04:50 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling ehcache client on testwiki and test2wiki'
* 04:46 Tim: installed ehcache under tomcat on db40. Documented at Ehcache.
* 03:49 Tim: did apt-get upgrade on db40, now rebooting to get new kernel
* 01:10 mark: Reinstalled ms5 with Ubuntu 10.04, created software RAID10 array /dev/md1 out of 48 drives
== March 29 ==
* 20:44 mark: Moved ms5 to internal vlan 2
* 20:34 mark: Pushed DNS for ms5.pmtpa.wmnet
* 18:56 mark: Changed NFS client params to rsize=8192,wsize=8192 on srv228:/mnt/thumbs
* 15:37 logmsgbot: ariel synchronized php-1.17/wmf-config/CommonSettings.php 'increase account creation throttle value for el wiki for editing workshop'
* 10:41 apergos: rsync on dataset2 for White Cat one last time, please shoot in three days (check with user first)
* 00:42 Ryan_Lane: enabled ssl on RT, all non-http now forces to https for RT
* 00:42 Ryan_Lane: fixed puppet issue on erzurumi by adding the wikidev group.
== March 28 ==
* 23:37 Ryan_Lane: spence died; powercycling it
* 23:06 logmsgbot: awjrichards synchronized php-1.17/extensions/UserDailyContribs/api/ApiUserDailyContribs.php 'Deploying bug fixes to UserDailyContribs API (r84935)'
* 22:41 Ryan_Lane: added a certificate check for ldaps on nfs1/2
* 22:36 Ryan_Lane: pushed the internal CA certificate to spence, so that we can monitor the LDAP certificates
* 21:43 awjr: running sync-comon all to deploy api updates to extension UserDailyContrib
* 21:43 logmsgbot: awjrichards ran sync-common-all 
* 21:38 logmsgbot: awjrichards synchronized php-1.17/extensions/UserDailyContribs/api 'Attempting to create UserDailyContribs/api dir using sync-file'
* 21:07 Ryan_Lane: pushed out a cert check for secure.wikimedia.org
* 19:17 Ryan_Lane: changing weight of mobile1 to 10
* 17:10 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Disable Vector/SectionEditLinks on enwiki'
* 16:58 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryCategoryMembers.php 'r84908'
* 14:58 apergos: was seeing sluggish response from scalers, ms4, running out of swap space 
* 14:54 apergos: restarting web server on ms4... 
* 05:58 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling ehcache'
* 05:57 Tim: ehcache broke, disabling client
* 05:39 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 05:35 Tim: db40 CPU usage is 5% of all cores, disk util <1%, network ~1%, request rate ~58% of weekly peak. Enabling ehcache on all wikis.
* 05:32 Tim: fixing torrus per wikitech instructions
* 05:08 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling ehcache on enwiki'
* 05:07 Tim: resource utilisation on db40 is a nothingth of a percent, moving on to stage 4
* 05:04 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling ehcache for enwikiquote'
* 04:56 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'put ehcache first so it will get more load'
* 04:45 logmsgbot: tstarling synchronized php-1.17/includes/objectcache/EhcacheBagOStuff.php 'curl fix'
* 04:38 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enable ehcache on test and test2'
* 04:36 logmsgbot: tstarling synchronized php-1.17/includes/objectcache/EhcacheBagOStuff.php 'don't use CURLOPT_TIMEOUT_MS'
* 04:28 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'ehcache setup'
* 04:28 Tim: fixed sync on searchidx1
* 04:27 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'test sync'
* 04:27 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'test sync'
* 04:26 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'ehcache setup'
== March 27 ==
* 22:28 Ryan_Lane: restarting apache on singer
* 22:19 Ryan_Lane: pushing new star cert to singer
* 21:55 RoanKattouw: SSL certificate for secure.wikimedia.org expired about an hour ago, no ops around. I've just texted Ryan
* 19:52 logmsgbot: demon synchronized php-1.17/extensions/ProofreadPage/ProofreadPage_body.php 'r84871'
* 14:14 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderModule.php 'r84845, hopefully fixes catchable fatal error about makeCombinedStyles()'
* 10:55 logmsgbot: hashar: on searchidx1 wmf-config path does not exist triggering an error on code sync
* 10:54 logmsgbot: hashar: auto defers SocialProfile, editor_trends & WikiSnaps
* 10:54 logmsgbot: hashar synchronized php-1.17/wmf-config/codereview.php 
== March 26 ==
* 21:41 logmsgbot: ariel synchronized php-1.17/extensions/Narayam/Narayam.i18n.php 'remove BOM from file, was being written to stdout'
* 19:13 apergos: running rsync for White Cat again to pick up en wiki 20110115 hist files, on dataset2 in screen session as root, please shoot after 3 days (check with user first in channel)
* 15:05 logmsgbot: mark synchronized php-1.17/wmf-config/db.php 'Repooled db38 with lower load (100 instead of 400)'
* 14:57 logmsgbot: mark synchronized php-1.17/wmf-config/db.php 'Depooling db38'
* 10:55 logmsgbot: midom synchronized php-1.17/wmf-config/mc.php 'srv206 to srv243 swap'
* 10:53 domas: I misread before. memcached check was broken for 23 days, not 23 hours. duct-taped by creating symlink php-1.17 -> wmf-deployment on spence, not sure for how long :)
* 10:50 domas: added 'pkill -9 -f smssend' hourly crontab on spence...
== March 25 ==
* 15:36 mark: Upgraded csw1-esams and br1-knams to IronWare 2.7.3a
* 15:31 mark: Reloading br1-knams
* 15:09 mark: Reloading csw1-esams
* 15:09 logmsgbot: demon synchronized php-1.17/includes/HTMLForm.php 'r84748'
* 14:53 mark: Upgraded JUNOS to 10.0S10.0 on csw2-esams
* 13:18 mark: Moving esams traffic to pmtpa for upcoming router firmware upgrade/reload
* 04:49 Tim: reniced apaches to nice level 0 again
* 03:53 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 03:51 logmsgbot: aaron synchronized php-1.17/wmf-config/abusefilter.php 'Moved abusefilter rights here from InitialiseSettings.php'
* 03:36 logmsgbot: root synchronizing Wikimedia installation... Revision: 84728: 
* 03:36 Tim: svn up/scap to r84727
* 02:45 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'Added missing autopatrolled group to siwiki'
* 02:29 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'Added missing rollbacker group to siwiki'
* 02:07 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php 'DO NOT PUT ABUSEFILTER SETTINGS IN HERE'
* 02:06 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'Copy abusefilter right settings from flaggedrevs.php to here, were they belong'
* 01:55 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php 
* 01:24 logmsgbot: aaron synchronized php-1.17/wmf-config/flaggedrevs.php 'Simple config cleanups'
* 00:48 hcatlin: Started rolling restarts passenger/ruby cluster on mobile2
== March 24 ==
* 23:50 tomaszf: lowering weight for mobile2 from 45 -> 30 on m.
* 22:51 logmsgbot: tfinc synchronized php-1.17/wmf-config/InitialiseSettings.php 'adding missing redirects for mobile sites'
* 22:33 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 22:27 logmsgbot: aaron synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 28195 - added two namespaces two sqwikinews'
* 20:23 Ryan_Lane: fixing the authorized_keys file on ragweed
* 17:16 rainman-sr: clearing up some space on searchidx1
* 17:04 RobH: hooper switch port vlan updated, system ready for installation
* 16:54 tomaszf: rebalancing mobile1/2/3 traffic load to 15/40/60
* 15:54 logmsgbot: catrope synchronized php-1.17/extensions/Narayam/Narayam.i18n.php 'r84684'
* 15:54 logmsgbot: catrope synchronized php-1.17/extensions/Narayam/Narayam.hooks.php 'r84684'
* 15:54 logmsgbot: catrope synchronized php-1.17/extensions/Narayam/Narayam.php 'r84684'
* 15:40 logmsgbot: catrope synchronized php-1.17/extensions/Gadgets/Gadgets_body.php 'r84683'
* 13:37 thedj: ResourceLoader broken due to bug 28215
* 04:28 logmsgbot: tstarling synchronized php-1.17/skins/common/shared.css 'r84666'
* 02:58 Tim: fixing bug 9204: AllowEncodedSlashes On
== March 23 ==
* 19:37 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'Bump filter cache version'
* 19:37 logmsgbot: catrope synchronized php-1.17/includes/libs/JavaScriptMinifier.php 'r84628'
* 18:34 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderModule.php 
* 18:34 logmsgbot: catrope synchronized php-1.17/includes/AutoLoader.php 
* 18:32 RoanKattouw: Something went wrong with that sync, trying to fix it with individual file syncs
* 18:30 logmsgbot: catrope ran sync-common-all 
* 18:29 RoanKattouw: Running sync-common-all to deploy ResourceLoader changes and some other random things (r84613, r84616, r84620)
* 17:49 RoanKattouw: I've staged some ResourceLoader changes on testwiki. Please DO NOT RUN sync-common-all or scap for the next hour or so
* 17:00 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialUpload.php 'r84605'
* 17:00 logmsgbot: catrope synchronized php-1.17/includes/upload/UploadBase.php 'r84605'
* 17:00 logmsgbot: catrope synchronized php-1.17/extensions/TitleBlacklist/TitleBlacklist.list.php 'r84605'
* 16:46 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesLtg.php 'r84604'
* 16:46 logmsgbot: catrope synchronized php-1.17/languages/Names.php 'r84604'
* 16:46 logmsgbot: catrope synchronized php-1.17/includes/ChangeTags.php 'r84604'
* 16:46 logmsgbot: catrope synchronized php-1.17/extensions/Renameuser/Renameuser.php 'r84604'
* 16:46 logmsgbot: catrope synchronized php-1.17/extensions/Renameuser/Renameuser_body.php 'r84604'
* 16:28 logmsgbot: catrope synchronized php-fatal-error.html 'Update this file from SVN /trunk/tools/php/wmerrors/error.html . This also changes its chmod so people who are not Tim can change this file'
* 16:14 logmsgbot: demon synchronized php-1.17/extensions/FlaggedRevs/forms/RevisionReviewForm.php 'r84599'
* 13:08 apergos: started shovelling some data over to the newly revived dataset1 (from dataset2, screen session as root)
* 12:55 mark: Configured OWA on owa{1,2,3} to send error log emails to nimish, instead of to "empty recipients", and therefore, root
* 12:28 apergos: yeah I wanted sync-common-file and typed all instead. grrrr but apparently no harm done
* 12:15 logmsgbot: ariel ran sync-common-all 'php-1.17/wmf-config/CommonSettings.php raise acct throttle for el wiki for editing workshop (3)'
* 00:25 notpeter: dumped bugzilla db from db9 and imported to db10, skipped some errors, slaving is moving again.
== March 22 ==
* 21:12 Ryan_Lane: changing donation.tesla to nomad.tesla in DNS
* 19:34 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/api/ApiQueryCodeRevisions.php 'r84545'
* 18:44 Ryan_Lane: destroying donation.tesla.usability.wikimedia.org and replacing with nomad.tesla.usability.wikimedia.org
* 17:26 nimish_g: changed owa sql timeouts, restarted apaches on owa machines
* 17:19 RobH: dataset1 back online for testing, apergos will take over later today/tomorrow, will not go into service just yet
* 14:47 logmsgbot: catrope ran sync-common-all 
* 14:46 RoanKattouw: Running sync-common-all to deploy r84467, r84484, r84495 and r84530. Needed because directory additions don't work with sync-file AFAIK
* 06:16 logmsgbot: ariel synchronized docroot/www.wikimedia.org/google01afe16dd444713b.html 'authorize generic tech account for google webmaster tools (for use with google storage)'
* 00:28 Tim: removing old binlogs from db9
== March 21 ==
* 23:30 Ryan_Lane: upgrading nova-* packages on nova-*.tesla
* 23:30 Ryan_Lane: installing ajaxterm on nova-compute1/2
* 21:59 Reedy: dataset1 is getting another beating. You know it deserves it, right?
* 20:42 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28129 - Enable subpages in main namespace on sawikisource'
* 20:37 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28134 - Enable alias "WL" -> "Wikilajme" for Albanian Wikinews'
* 20:18 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27256 - Correcting content page count at en.wikibooks and pt.wikibooks'
* 20:12 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '26281 - Please allow AbuseFilter on Hungarian Wikisource and Wikibooks'
* 20:10 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28160 - AbuseFilter on Kannada Wikipedia'
* 03:44 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'disabling PoolCounter due to bug 28144'
* 01:42 logmsgbot: demon synchronized php-1.17/extensions/FlaggedRevs/FlaggedArticleView.php 'r84437'
* 00:26 rainman-sr: built search indexes for new wikis: eowikisource, ltgwiki, sawikisource, sahwikisource, kbdwiki, sqwikinews
== March 20 ==
* 21:19 logmsgbot: catrope synchronized php-1.17/extensions/WikiEditor/WikiEditor.php 'r84422'
* 21:19 logmsgbot: catrope synchronized php-1.17/extensions/WikiEditor/WikiEditor.hooks.php 'r84422'
* 21:11 logmsgbot: catrope synchronized php-1.17/extensions/WikiEditor/WikiEditor.php 'rm duplicate call'
* 21:11 logmsgbot: catrope synchronized php-1.17/extensions/WikiEditor/WikiEditor.hooks.php 'Fix fatal'
* 21:09 logmsgbot: catrope ran sync-common-all 
* 21:07 RoanKattouw: Deploying r84420 with sync-common-all
* 19:57 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27379 - Please create Wikipedia in Latgalian language'
* 15:18 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add oldwikisource to import sources on sawikisource'
* 14:50 logmsgbot: catrope synchronized php-1.17/resources/jquery/jquery.textSelection.js 'r84385'
* 12:35 RoanKattouw: Clearing and rebuilding LocalisationUpdate cache. Using altered version of /h/w/bin/l10nupdate which pushes updates to Apaches immediately, then clears message blobs
* 09:39 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add English Wikisource to import sources on sawikisource'
* 09:36 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Change sitename too on orwiki, was untranslated'
* 09:35 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Fix previous change - set $wgMetaNamespace, not $wgSitename'
* 09:33 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27948 - Set $wgSitename on orwiki'
== March 19 ==
* 14:51 RoanKattouw: Purging http://eo.wikisource.org/wiki/Speciala%C4%B5o:Preferoj
* 14:51 logmsgbot: hashar: running purge http://eo.wikisource.org (no trailing slash)
* 14:34 logmsgbot: hashar: previous sync was to fix wikisource eo logo
* 14:33 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 13:01 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabling PoolCounter (was disabled during the previous breakage)'
* 12:59 logmsgbot: tstarling synchronized php-1.17/includes/PoolCounter.php 'r84323: do not show error to user on connection refused etc.'
* 12:25 Tim: attempted to restart poolcounterd with a different file descriptor limit. It didn't start up properly, causing every parse request to return an error. Fixed in about 30 seconds.
* 12:23 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 12:15 logmsgbot: tstarling synchronized php-1.17/extensions/PoolCounter/PoolCounterClient_body.php 'timeout fix'
* 12:14 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 're-enabling PoolCounter, now with error logging enabled'
* 12:13 logmsgbot: tstarling synchronized php-1.17/includes/PoolCounter.php 'poolcounter error logging'
* 12:12 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'poolcounter error logging'
* 11:24 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '28114 - sqwikinews logo'
* 07:06 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27663 - Please install Abuse Filter in Bashkir Wikipedia'
* 07:05 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '24466 - Enable Collection extension on hiwiki'
* 07:04 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27926 - Enable AbuseFilter for Outreach Wiki'
* 07:03 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28049 - AbuseFilter in fa.wikiquote'
* 07:01 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27578 - Enable Extension'
* 06:16 logmsgbot: midom synchronized php-1.17/wmf-config/InitialiseSettings.php 'disable poolcounter - it should not be serving errors'
== March 18 ==
* 22:04 logmsgbot: jeluf synchronized php-1.17/cache/interwiki.cdb 'Updating interwiki cache'
* 21:55 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php 'set sqwikinews logo'
* 21:50 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27842 - enable blocked user to edit talk page on hewikiquote'
* 21:48 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27740 - give sysop the abusefilter-modify-restricted right in the Hebrew Wikipedia'
* 21:39 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27840 - enable transwiki import to latin wikisource'
* 21:34 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28092 - Add Portal namespace on Swedish Wikiversity'
* 21:32 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28092 - Add Portal namespace on Swedish Wikiversity'
* 21:28 logmsgbot: jeluf synchronized php-1.17/wmf-config/CommonSettings.php '27896 - Enable Special:Cite at en.wikiversity'
* 21:21 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28022 - Create the Albanian Wikinews'
* 21:13 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '28003 - Create the Kabardian Circassian Wikipedia'
* 21:06 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27557 - Create Sakha Wikisource'
* 20:58 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '27379 - Please create Wikipedia in Latgalian language'
* 20:41 logmsgbot: jeluf synchronized php-1.17/wmf-config/InitialiseSettings.php '26136 - Create Wikisource in Esperanto'
* 20:34 JeLuF: apache-graceful-all reports lots of NTP errors
* 20:14 logmsgbot: jeluf ran sync-common-all 'Created eowikisource, ltgwiki, sawikisource, sahwikisource, kbdwiki, sqwikinews'
* 19:47 domas: fixed autocommit in wmf-config by uncommenting a cron entry in my crontab \o/
* 19:44 logmsgbot: jeluf synchronized langlist 'Added ltg and kbd'
* 19:00 awjr: Archived Hudson build logs for CiviMail-related Hudson jobs to storage3 (/archive/fundraising/hudson_builds) and cleared out related log dirs on Grosley. Hudson CiviMail-related builds now back to normal.
* 10:13 Tim: fixed punctuation in bugzilla quip 1155 by direct DB access
* 03:05 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling PoolCounter on all wikis'
* 03:04 Tim: poolcounterd CPU usage still negligible. Enabling on all wikis.
* 03:00 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling PoolCounter on en.wikipedia.org'
* 02:59 logmsgbot: tstarling synchronized php-1.17/wmf-config/PoolCounterSettings.php 'longer connect timeout'
* 02:53 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'enabling PoolCounter on en.wiktionary.org'
* 02:39 logmsgbot: root synchronizing Wikimedia installation... Revision: 84213: 
* 02:37 logmsgbot: root synchronizing Wikimedia installation... Revision: 84213: 
* 02:37 Tim: added PoolCounter to extension-list
* 02:36 logmsgbot: tstarling synchronized php-1.17/extensions/PoolCounter/PoolCounterClient_body.php 
* 02:34 logmsgbot: tstarling synchronized php-1.17/extensions/Renameuser/Renameuser.php 'increase contribution limit to 50k'
== March 17 ==
* 22:47 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump $wgVectorSectionEditExperiment'
* 22:46 logmsgbot: catrope synchronized php-1.17/extensions/Vector/modules/ext.vector.sectionEditLinks.js 'r84213'
* 22:28 RoanKattouw: Clearing message blobs
* 22:27 logmsgbot: catrope ran sync-common-all 
* 22:27 RoanKattouw: Deploying r84160 and r84210 with sync-common-all
* 21:03 logmsgbot: mark synchronized php-1.17/wmf-config/db.php 'Temp depool srv178-180'
* 20:37 mark: Shutdown srv178-180 for relocation to rack D1
* 19:21 tomaszf: ran purge-varnish MobileRedirect.js to clear out old redirect regex
* 19:06 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 
* 19:06 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js 
* 18:59 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 'Bump style version
* 07:41 logmsgbot: ariel ran sync-common-all 'php-1.17/wmf-config/CommonSettings.php raise acct throttle for el wiki for editing workshop (2)'
* 02:29 mark: Set high performance http sysctl settings on mobile1 (temp, needs puppetising)
* 02:22 tomaszf: changing weight on mobile1 to 15 instead of 30
* 02:20 mark: restarted apache on mobile1
== March 16 ==
* 15:39 logmsgbot: ariel ran sync-common-all 'php-1.17/wmf-config/CommonSettings.php raise acct throttle for el wiki for editing workshop'
* 13:01 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Don't enable Narayam by default'
* 12:38 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable Narayam on mlwiki'
* 12:33 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable Narayam on mlwikisource'
* 12:30 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable Narayam on mlwiktionary'
* 12:28 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable Narayam on mlwikibooks'
* 12:25 RoanKattouw: I meant mlwikiquote of course
* 12:25 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable Narayam on nlwikiquote'
* 12:19 logmsgbot: catrope ran sync-common-all 
* 12:18 RoanKattouw: Running sync-common-all to push Narayam code to the cluster (currently only enabled on testwiki)
== March 15 ==
* 21:18 hcatlin: restarted mobile app on mobile1 due to unexpected load increase beginning at 20:43
* 20:36 RoanKattouw: Clearing message blobs
* 20:36 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/ext.articleFeedback/ext.articleFeedback.js 'r84052'
* 20:35 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/ArticleFeedback.i18n.php 'r84052'
* 19:28 Ryan_Lane: added svg to allowed upload types for wikitech
* 17:51 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Cleaning out cruft from CentralizeNotice config; Updating how https is handled for CentralNotice; PoolCounter settings inclusion in CommonSettings (but not checking in/enabling new Poolcounter settings in InitialiseSettings)'
* 17:00 mark: Replaced root's authorized_keys file
* 06:47 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'pool counter deployment on testwiki only'
* 06:46 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'pool counter deployment on testwiki only'
* 06:42 logmsgbot: root synchronizing Wikimedia installation... Revision: 83954: 
* 06:42 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 83954: 
* 06:41 Tim: running scap to get the PoolCounter extension
* 06:28 Tim: installed poolcounter on tarin
* 06:08 Tim: tarin upgraded to lucid
* 05:15 Tim: tarin just has an ancient test wordpress install on it which hasn't been touched since April 2010. Removing the website and mysql instance, will attempt to upgrade the distro.
* 03:20 logmsgbot: tstarling synchronized php-1.17/includes/HTMLCacheUpdate.php 'disable squid cache invalidation from HTMLCacheUpdate, to hopefully eliminate apache CPU spikes'
* 03:12 Tim: started job runners
* 03:09 Tim: apache CPU spike, stopped job runners
* 03:05 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'removed bug 27891 logging'
* 03:04 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 'removed bug 27891 logging'
== March 14 ==
* 19:57 logmsgbot: catrope synchronized php-1.17/extensions/ArticleFeedback/modules/jquery.articleFeedback/jquery.articleFeedback.css 'r83954'
* 19:33 logmsgbot: catrope synchronized php-1.17/extensions/ClickTracking/modules/ext.clickTracking.js 'r83952'
* 19:33 logmsgbot: catrope synchronized php-1.17/extensions/ClickTracking/modules/jquery.clickTracking.js 'r83952'
* 19:28 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable ArticleFeedback on enwiki'
* 19:23 logmsgbot: catrope ran sync-common-all 
* 19:22 RoanKattouw: Got sudo weirdness on scap again, running sync-common-all
* 19:21 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 83949: 
* 19:20 logmsgbot: catrope synchronized php-1.17/extensions/Collection/Collection.suggest.php 'Syntax error'
* 19:19 RoanKattouw: Running scap to deploy ArticleFeedback (only enabled on test for now, will enable on enwiki after this)
* 18:38 Ryan_Lane: installing sudo schema on nova-controller.tesla and restarting opendj
* 18:14 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/CodeReview.php 'Style version bump'
* 18:14 logmsgbot: catrope ran sync-common-all 
* 18:12 RoanKattouw: Running sync-common-all to deploy r83934
* 09:55 Tim: increased memcached read timeout from 100ms to 500ms. This reduced the read timeout rate from 51/s to 3.4/s.
* 09:50 logmsgbot: tstarling synchronized php-1.17/wmf-config/mc.php 'increasing memcached timeout from 100ms to 500ms while monitoring timeout rate'
* 09:44 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'log memcached timeout errors'
* 09:40 logmsgbot: tstarling synchronized php-1.17/includes/memcached-client.php 'log timeout errors'
* 06:26 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 'more specific logging hack'
* 04:32 logmsgbot: tstarling synchronized php-1.17/includes/Article.php 'r83869'
* 03:42 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 'logging tweak'
* 03:36 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 'logging tweak'
* 03:32 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 'more logging'
* 03:23 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 
* 03:07 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'logging hack for bug 27891'
* 03:06 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 'logging hack for bug 27891'
* 01:52 logmsgbot: tstarling synchronized php-1.17/includes/HTMLCacheUpdate.php 'throttle HTML cache invaldiation'
* 00:27 logmsgbot: midom synchronized php-1.17/includes/Block.php 
* 00:13 domas: reniced sshd to -10, apache2 to 0, to have better colors on graphs :)
* 00:03 logmsgbot: midom synchronized php-1.17/includes/Block.php 'livehack for separate connection'
== March 13 ==
* 19:25 logmsgbot: catrope synchronized php-1.17/skins/vector/main-rtl.css 
* 19:24 logmsgbot: catrope synchronized php-1.17/skins/vector/main-ltr.css 'Put this back but forgot to sync, d'oh'
* 16:16 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '26446 - Change title of Kannada Wikipedia'
* 15:54 logmsgbot: hashar: Have set commonswiki user Wikinger77 email to the same as the dewiki account with the name.
* 15:37 logmsgbot: midom synchronized php-1.17/languages/Language.php 
* 15:24 logmsgbot: hashar synchronized php-1.17/wmf-config/CommonSettings.php 'enable script for proofreadpage extension'
* 15:23 logmsgbot: hashar synchronized php-1.17/wmf-config/proofreadpage.php 'script to configure the proofreadpage extension'
* 14:48 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27255 - Setup Rollback group at ko.wikipedia'
* 14:40 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27268 - New "Rollbacker" group for zh.wikinews'
* 12:15 logmsgbot: hashar: ran a groupby query on db36:enwiki database by mistakes :/
* 11:51 logmsgbot: hashar synchronized php-1.17/wmf-config/CommonSettings.php 'bug 24313 - hide preference minordefault for enwiki'
* 11:29 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27651 - Create WikiProject namespace for ko.wikipedia'
* 11:27 logmsgbot: hashar: committed db.php
* 11:17 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27728 - Add WP as an alias to project namespace in Portuguese Wikipedia'
* 11:11 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27878 - Change project namespace on kuwiktionary'
* 11:02 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27898 - Create Autor namespace at ca.wikisource'
* 10:52 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27898 - Create Autor namespace at ca.wikisource'
* 10:44 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27413 - Allow eswiki bureaucrats to add/remove the confirmed usergroup'
* 09:56 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27998 - Add WP alias on dawiki'
* 01:31 logmsgbot: midom synchronized php-1.17/wmf-config/db.php 'even out a bit'
* 00:07 domas: db16 write-behind cache failed yet again, changing states doesn't help much, battery seems to be dead ( FSA_EM_ENHANCED_BATTERY_CHANGE ? ) - set innodb log sync to 0
== March 12 ==
* 15:47 logmsgbot: midom synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.hooks.php 
* 15:15 logmsgbot: midom synchronized php-1.17/extensions/FlaggedRevs/specialpages/ProblemChanges_body.php 
* 15:04 logmsgbot: midom synchronized php-1.17/extensions/FlaggedRevs/specialpages/PendingChanges_body.php 
* 14:17 logmsgbot: hashar: shell madness is over for today.
* 14:06 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27195 - Grant anonymous users 'createpage' right and enable Extension'
* 14:01 domas: fixed *-filter segfaults on locke, probably :) 
* 13:14 logmsgbot: hashar: Cleaning bugs in the Wikimedia product
* 13:04 logmsgbot: hashar: Reedy made me an admin on bugzilla. Then allowed myself to edit components.
* 12:51 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'whitespaces'
* 12:37 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27954 - Change autoconfirmed parameters on pt.wiki'
* 12:30 logmsgbot: hashar: committed db.php
* 12:27 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27971 - Change the translation of "user" and "user_talk" namespaces on ka.wiki'
* 12:14 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27361 - Update logo for Fiji Hindi Wikipedia'
* 12:05 logmsgbot: hashar: ran namespaceDupes.php on cywiki : Looks good!
* 12:04 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27684 - Create Portal namespace on cy.wikipedia.org'
* 11:55 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27826 - Logo update on Tamil Wikipedia'
* 11:50 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27918 - Enable patrolled edits on Hebrew Wikibooks'
* 11:37 logmsgbot: midom synchronized php-1.17/extensions/ClickTracking/ClickTracking.hooks.php 
* 11:37 logmsgbot: midom synchronized php-1.17/extensions/ClickTracking/ClickTracking.hooks.php 
== March 11 ==
* 23:42 logmsgbot: catrope synchronized php-1.17/extensions/Vector/Vector.php 'r83735'
* 23:42 logmsgbot: catrope synchronized php-1.17/extensions/Vector/Vector.hooks.php 'r83735'
* 23:34 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Set $wgVectorSectionEditLinksExperiment to 1'
* 23:30 logmsgbot: catrope synchronized php-1.17/extensions/ClickTracking/modules/jquery.clickTracking.js 
* 23:12 RoanKattouw: Apache CPU went back to normal at 23:10
* 23:07 RoanKattouw: Apache cluster CPU shot up to ~95% around 23:05 UTC
* 21:43 rainman-sr: freed up some space on searchidx1 which was again out of disc space
* 12:00 RoanKattouw: updateCollation.php finished on enwiki. The script has now run to completion on all wikis
* 04:39 Tim: made myself owner of the mediawiki-cvs mailing list
* 02:25 logmsgbot: tstarling synchronized php-1.17/StartProfiler.php 'separate CLI from other profiling'
* 02:12 logmsgbot: tstarling synchronized php-1.17/maintenance/jobs-loop.sh 'increased number of processes per job runner from 4 to 5'
* 02:10 Tim: installed job runners on srv231-247
* 00:59 ^demon: landing page for svn.wm.o is now php, so it can read latest public dump file if available
* 00:29 Ryan_Lane: patching ec2 api on nova-controller.tesla to return project instead of user in the status
== March 10 ==
* 21:55 Ryan_Lane: updating nova-* packages on nova-*.telsa
* 21:36 logmsgbot: catrope synchronized php-1.17/extensions/Vector/modules/ext.vector.sectionEditLinks.js 'r83672'
* 21:35 logmsgbot: catrope synchronized php-1.17/extensions/Vector/Vector.hooks.php 'r83672'
* 21:35 logmsgbot: catrope synchronized php-1.17/extensions/Vector/Vector.php 'r83672'
* 21:34 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Set $wgVectorSectionEditLinksLotteryOdds to 1%'
* 21:18 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 27918 - Enable patrolled edits on Hebrew Wikibooks'
* 21:10 logmsgbot: hashar: we have pending changes in db.php and CommonSettings.php . Please commit! :p
* 18:52 awjr: restarting dovecot on grosley in feeble attempt to resolve civimail issue
* 18:22 RoanKattouw: Put the 1.16wmf4 versions of skins/vector/main-{ltr,rtl}.css in /h/w/c/php/skins/vector as a temp hack so Squid-cached pages don't point to nonexistent CSS
* 18:18 Ryan_Lane: removing custom built memcached from lucid-wikimedia
* 17:03 Ryan_Lane: adding memcached to lucid-wikimedia repo
* 16:17 RoanKattouw: updateCollation.php has finished on s4 (commonswiki) and s6 (fr+ja+ruwiki). Now only running on s1 (enwiki)
* 11:40 logmsgbot: tstarling synchronized php-1.17/includes/job/JobQueue.php 
* 09:15 hashar: Moved server admin logs from November 1st 2010 till December 31st 2010 to Server admin log/Archive 17
* 05:32 logmsgbot: tstarling synchronized php-1.17/maintenance/jobs-loop.sh 'de-prioritise renameUser due to slave lag'
* 05:28 logmsgbot: tstarling synchronized php-1.17/maintenance/jobs-loop.sh 
* 05:28 logmsgbot: tstarling synchronized php-1.17/maintenance/runJobs.php 
* 05:13 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 'offloading db24 since it is very laggy'
* 04:56 logmsgbot: tstarling synchronized php-1.17/maintenance/jobs-loop.sh 'prioritise renameUser'
* 04:43 logmsgbot: tstarling synchronized php-1.17/maintenance/nextJobDB.php 'testwiki hack'
* 04:24 logmsgbot: tstarling synchronized php-1.17/maintenance/nextJobDB.php 'bug fixes'
* 04:23 logmsgbot: tstarling synchronized php-1.17/maintenance/nextJobDB.php 'bug fixes'
* 04:11 logmsgbot: tstarling synchronized php-1.17/maintenance/nextJobDB.php 
* 02:48 awjr: attempting to restart Hudson on grosley due to it crashing with Java exceptions
* 02:27 logmsgbot: tstarling synchronized php-1.17/includes/job/JobQueue.php 
* 02:20 logmsgbot: tstarling synchronized php-1.17/includes/DefaultSettings.php 
* 02:20 logmsgbot: tstarling synchronized php-1.17/includes/GlobalFunctions.php 
* 02:20 logmsgbot: tstarling synchronized php-1.17/includes/job/JobQueue.php 
* 02:19 Tim: svn up/scap for r83628
* 02:18 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'wgAggregateStatsID'
== March 9 ==
* 21:55 Ryan_Lane: adding imagemagick to lucid-wikimedia
* 20:01 logmsgbot: catrope synchronized php-1.17/extensions/Vector/Vector.php 'r83608'
* 19:55 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable ClickTracking on testwiki too'
* 19:51 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Setting $wmgClickTrackThrottle to -1 on all wikis'
* 19:48 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable ClickTracking on enwiki, needed for VectorSectionEditLinks'
* 19:41 logmsgbot: catrope ran sync-common-all 
* 19:40 RoanKattouw: Running sync-common-all to deploy VectorSectionEditLinks for 0.1% of enwiki users
* 19:12 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 'studid typo'
* 18:57 logmsgbot: catrope synchronized php-1.17/includes/db/LoadBalancer.php 'r83591'
* 18:48 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 
* 18:47 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js 
* 18:05 logmsgbot: hashar: running refreshLinks.php wikimania2011wiki
* 17:44 RobH: survey.w.o upgrade complete
* 17:40 Ryan_Lane: adding php5.3 packages to lucid-wikimedia main repo
* 17:32 RobH: upgrading survey.w.o install, so it may be inaccessible for a few minutes tops.
* 14:50 RoanKattouw: ETAs for still-running updateCollation.php instances: enwiki: 33 hours, commons: 14 hours, s6 (fr+ja+ruwiki): 19 hours
* 14:29 RoanKattouw: updateCollation.php has finished on s2
* 12:41 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 'Style version bump'
* 12:40 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js 'Fix syntax error'
* 12:31 RoanKattouw: updateCollation.php finished on s3
* 11:55 RoanKattouw: Correction, 11:00 UTC
* 11:55 RoanKattouw: updateCollation.php finished on s5 (dewiki) around 12:00 UTC
* 11:52 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 'Style version bump'
* 11:50 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js 'Live hack mobile redirect script to fix JS errors'
* 10:10 RoanKattouw: cl_collation!='uppercase' was causing a full table scan on categorylinks on dewiki and frwiki. FORCE INDEX didn't help, so I live-hacked the script to do cl_collation= instead. This makes the script run noticeably faster across the board
* 10:07 RoanKattouw: updateCollation.php has finished on s7
* 02:36 logmsgbot: tfinc synchronized php-1.17/extensions/CentralNotice/SpecialBannerController.php 
* 02:36 logmsgbot: tfinc synchronized php-1.17/extensions/CentralNotice/CentralNotice.i18n.php 
* 02:35 logmsgbot: tfinc synchronized php-1.17/extensions/CentralNotice/1x1.png 
* 01:04 logmsgbot: catrope synchronized php-1.17/includes/CategoryPage.php 'Live hack to make from=a work in category pages'
* 01:03 logmsgbot: catrope synchronized php-1.17/includes/CategoryPage.php 'Unbreak lowercase parameters like pagefrom=a'
* 00:43 logmsgbot: catrope synchronized php-1.17/extensions/CategoryTree/CategoryPageSubclass.php 'r83651'
* 00:14 logmsgbot: catrope synchronized php-1.17/includes/CategoryPage.php 
* 00:09 RoanKattouw: s2 lag back to normal
* 00:04 RoanKattouw: This fixed s1 replag. s2 replag is still pretty bad, db24 is now at 1560s
* 00:02 RoanKattouw: Fixed wait for slave logic in updateCollation.php , was broken
== March 8 ==
* 23:17 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/backend/CodeRevision.php 'r83551'
* 22:41 logmsgbot: awjrichards synchronized php-1.17/wmf-config/CommonSettings.php 'Adding openZIM as file format for Collection extension'
* 21:47 RoanKattouw: Restarted all collation update jobs with a more efficient SELECT query (with STRAIGHT_JOIN) and a smaller batch size (50 instead of 1000)
* 21:18 RoanKattouw: Restart updateCollation.php on s4 (commons) with a batch size of 50 to see how that plays out
* 21:07 RoanKattouw: Move complete. Aborted updateCollation.php on fenari, now running 7 instances of /home/catrope/collatecluster in a screen on hume
* 21:01 RoanKattouw: Breaking out updateCollation.php run into one process per DB cluster, and running them on hume rather than fenari
* 20:42 logmsgbot: catrope synchronized php-1.17/includes/CategoryPage.php 'Experimental fix by Aryeh for category sorting problems'
* 20:35 logmsgbot: catrope synchronized php-1.17/includes/parser/Parser.php 
* 20:35 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryCategoryMembers.php 
* 20:34 logmsgbot: catrope synchronized php-1.17/includes/Title.php 
* 20:34 logmsgbot: catrope synchronized php-1.17/includes/LinksUpdate.php 
* 20:34 logmsgbot: catrope synchronized php-1.17/includes/CategoryPage.php 
* 20:34 logmsgbot: catrope synchronized php-1.17/extensions/intersection/DynamicPageList.php 
* 20:34 logmsgbot: catrope synchronized php-1.17/extensions/FlaggedRevs/specialpages/UnreviewedPages_body.php 
* 20:34 logmsgbot: catrope synchronized php-1.17/extensions/Collection/Collection.body.php 
* 20:33 logmsgbot: catrope synchronized php-1.17/extensions/CategoryTree/CategoryTreeFunctions.php 
* 20:32 RoanKattouw: Re-syncing a bunch of files without changes, didn't seem to have been picked up right by scap
* 19:38 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryCategoryMembers.php 'Sync without changes, maybe this didn't get picked up right'
* 19:35 logmsgbot: catrope synchronized php-1.17/includes/CategoryPage.php 'Use array_merge instead of + for paranoia'
* 17:46 RoanKattouw: Running updateCollation.php on all wikis
* 17:37 RoanKattouw: Running updateCollation.php on cswiktionary
* 17:36 logmsgbot: robh synchronizing Wikimedia installation... Revision: 83531: 
* 17:24 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 83531: 
* 17:23 RoanKattouw: Deploying category collation code changes (r83254, r83531) sitewide using scap
* 15:16 RoanKattouw: Updating MediaWiki on fenari from SVN to reintroduce category collation changes. For now, this is only for testing, so don't run sync-common-all or scap just yet
* 15:03 RoanKattouw: Looks fine
* 15:02 RoanKattouw: upgrade-1.17wmf1-final.php seems to have finished, running it again to verify it succeeded
* 13:28 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'NamespaceWithSubpages settings for Czech wikis, per Danny B'
* 12:44 mark: Changed 'wrr' scheduler to 'sh' on payments and owas LVS services
* 12:16 logmsgbot: midom synchronized php-1.17/extensions/FlaggedRevs/specialpages/PendingChanges_body.php 'syncing http://p.defau.lt/?9RQEZQiZ_WeMT7_f7TpX6A live too'
* 12:09 logmsgbot: midom synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'syncing livehack http://p.defau.lt/?BQwaKa6fH17X_CLt74khzA'
* 10:55 Tim: running upgrade-1.17wmf1-final.php to upgrade old masters
* 10:53 Tim: master positions: db36-bin.000001:106, db13-bin.000001:106, db31-bin.000001:106, db23-bin.001:79
* 10:53 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 
* 10:52 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 
* 10:52 Tim: switching master for s5: db14 -> db23
* 10:52 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 
* 10:52 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 
* 10:51 Tim: switching master for s4: db22 -> db31
* 10:51 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 
* 10:49 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 
* 10:48 Tim: switching master for s2: db30 -> db13
* 10:48 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 
* 10:48 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 
* 10:48 Tim: switching master for s1: db32 -> db36
* 10:08 logmsgbot: midom synchronized php-1.17/wmf-config/db.php 'adding db33 back with snapshotting disabled - checking if that plus temptables were causing the melt'
* 08:53 logmsgbot: jeluf synchronized php-1.17/wmf-config/db.php 'removed db33 from s4'
* 08:51 JeLuF: db33 down, commons unavailable
* 07:31 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 're-enabled squid purging'
* 07:24 logmsgbot: tstarling synchronized php-1.17/maintenance/jobs-loop.sh 'temp hack: no priority for htmlCacheUpdate'
* 07:22 Tim: it appears more likely that the overload was due to htmlCacheUpdate jobs updating page_touched, in a flood after jobs-loop.sh fixes
* 07:18 Tim: apaches overloaded after I fixed job queue squid purging. Disabled all squid purging and stopped job queue.
* 07:15 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 
* 06:47 logmsgbot: tstarling synchronized php-1.17/maintenance/jobs-loop.sh 
* 06:40 logmsgbot: tstarling synchronized php-1.17/includes/SquidUpdate.php 
* 06:40 logmsgbot: tstarling synchronized php-1.17/includes/Title.php 
* 06:39 logmsgbot: tstarling synchronized php-1.17/includes/DefaultSettings.php 
* 06:06 logmsgbot: tstarling synchronized php-1.17/maintenance/jobs-loop.sh 
* 05:51 Tim: deployed new jobs-loop.sh script which prioritises fast important jobs like email notification. Thousands of emails are currently being sent.
* 05:23 Tim: apt-get upgrade on prototype.wikimedia.org
* 05:19 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'filemover $wgRemoveGroups'
* 05:05 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'added filemover group to enwiki, as on commons, per request on WP:VPR'
* 04:33 Tim: re-running upgrade-1.17wmf1-final.php to do db34, db21 and db18
* 04:31 Tim: new master positions: db39-bin.000001:106, db29-bin.001:79, db37-bin.000001:106
* 04:31 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 
* 04:30 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 
* 04:30 Tim: doing master switch s7: db18 -> db37
* 04:29 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 
* 04:29 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 
* 04:29 Tim: doing master switch s6: db21 -> db29
* 04:28 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 
* 04:25 logmsgbot: tstarling synchronized php-1.17/wmf-config/db.php 
* 04:24 Tim: doing master switch: s3 db34 -> db39
* 02:05 Tim: apt-get upgrade on formey
* 01:47 Tim: apt-get upgrade on grosely and project2
* 01:35 Tim: apt-get upgrade on mchenry
* 01:22 Tim: apt-get upgrade on singer
* 00:53 Tim: manually upgraded python-memcache on several servers from 1.44-1wm1 to 1.44-1ubuntu1wm1~hardy1, which apt thought was a downgrade
* 00:22 logmsgbot: midom synchronized php-1.17/wmf-config/db.php 'sending db33 back to duty'
* 00:20 RobH: filterxsshtml re-enabled on limesurvey, initial testing done
== March 7 ==
* 23:57 Tim: running apt-get upgrade on mediawiki-installation servers, for USN-1085-1
* 23:43 Tim: restarted upgrade-1.17wmf1-final.php, which had stopped due to user_last_timestamp existing on some wikis
* 23:42 Tim: updating schema change scripts to r83491 and restarting
* 23:30 RobH: disabled filterxsshtml in limesurvey install, per awjr
* 23:30 RobH: enabled filterxsshtml in limesurvey install
* 23:13 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryBase.php 'r83490'
* 21:45 logmsgbot: catrope ran sync-common-all 
* 21:44 RoanKattouw: Running sync-common-all to deploy r83487
* 21:41 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php '27875 - Create a new usergroup Patroller with 'autopatrol' and 'patrol' right on simple.wikipedia'
* 21:24 logmsgbot: hashar: subversion working copy is now clean :)
* 21:22 logmsgbot: hashar: committing stuff in wmf-commit since the svn repository is available again
* 19:48 logmsgbot: midom synchronized php-1.17/wmf-config/db.php 
* 12:34 hcatlin: Deploying mobile fix
* 11:02 Tim: removing some duplicate jobs from enwiki.job
* 10:18 Tim: running schema changes using upgrade-1.17wmf1-final.php
* 10:06 logmsgbot: root synchronizing Wikimedia installation... Revision: 83453: 
* 10:06 Tim: running svn up/scap to get my latest schema change scripts
* 01:52 Tim: live-patched XSS vulnerability in mobile
* 00:20 logmsgbot: demon synchronized php-1.17/includes/media/Bitmap_ClientOnly.php 'r83417'
* 00:20 logmsgbot: demon synchronized php-1.17/includes/revisiondelete/RevisionDelete.php 'r83417'
== March 6 ==
* 18:54 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 27902 - Namespace fix on dawiki -- Help page talk too per IRC discussion'
* 18:46 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 27902 - Namespace fix on dawiki'
* 18:12 mark: Powered up sq35
* 17:43 richcole: swapping drive 7 on db15
* 15:02 hcatlin: Deployed to mobile again. Seems a little angry on ganglia.
* 14:28 hcatlin: Mobile deploy complete. Seems stable.
== March 5 ==
* 19:21 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 'Live hack to put mobile script back in <head>'
* 03:53 logmsgbot: andrew synchronized php-1.17/extensions/LiquidThreads/LiquidThreads.php 'Attempt to fix missing NSes on svwiktionary'
* 03:53 logmsgbot: andrew synchronized php-1.17/extensions/LiquidThreads/classes/Hooks.php 'Attempt to fix missing NSes on svwiktionary'
== March 4 ==
* 22:25 logmsgbot: hashar: Fixed outdated page by manually purging it (?purge + confirm). Issue resolved. Mark> you probably want to purge the squid/varnish caches for anything before 1.17 deployment :p
* 22:22 logmsgbot: hashar: page below is dated "Thu, 03 Feb 2011 12:18:28 GMT" HIT from amssq39.esams.
* 22:20 logmsgbot: hashar: got 404 errors on http://fr.wikipedia.org/wiki/H%C3%A9t%C3%A9rogramme : main-ltr.css points to 404 http://bits.wikimedia.org/skins-1.5/vector/main-ltr.css?283-18 , same with jquery.min.js http://bits.wikimedia.org/skins-1.5/common/jquery.min.js?283-18
* 00:12 logmsgbot: demon synchronized php-1.17/extensions/CodeReview/backend/CodeRevision.php 'r83186'
== March 3 ==
* 20:00 rainman-sr: modify lsearch-global-2.1.conf to point to an existing InitaliseSettings.php file (/home/wikipedia/common/php/wmf-config/InitialiseSettings.php). The file it was pointing before was removed a couple of days ago which made the incremental indexer broken.
* 19:23 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 
* 19:20 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js 
* 19:20 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/redirector_test.js 
* 19:19 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js 
* 19:19 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/redirector_test.js 
* 18:22 logmsgbot: catrope synchronized php-1.17/languages/classes/LanguageFi.php 
* 18:22 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialProtectedpages.php 
* 18:22 logmsgbot: catrope synchronized php-1.17/includes/DefaultSettings.php 
* 18:21 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryAllpages.php 
* 18:21 logmsgbot: catrope synchronized php-1.17/includes/filerepo/ForeignDBFile.php 
* 18:21 logmsgbot: catrope synchronized php-1.17/includes/filerepo/LocalFile.php 
* 18:21 logmsgbot: catrope synchronized php-1.17/includes/filerepo/File.php 
* 18:21 logmsgbot: catrope synchronized php-1.17/includes/Title.php 
* 18:19 RoanKattouw: Deploying r83154 with individual syncs
* 18:01 logmsgbot: catrope synchronized php-1.17/resources/jquery/jquery.placeholder.js 'r83152'
* 13:45 logmsgbot: midom synchronized php-1.17/wmf-config/db.php 
== March 2 ==
* 22:50 Ryan_Lane: powercycling loudon, since we are locked out of the system via manual iptables rules, and the management console is non-responsive
* 21:01 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.php 
* 21:01 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/MobileRedirect.js 
* 20:59 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/WikimediaMobile.i18n.php 
* 20:58 logmsgbot: tfinc synchronized php-1.17/extensions/WikimediaMobile/redirector_test.js 
== March 1 ==
* 23:49 Tim: removed "nomcomwiki" rows from the centralauth DB, they were causing fatal errors in Special:CentralAuth
* 22:07 logmsgbot: catrope synchronized php-1.17/includes/Wiki.php 'r83049'
* 22:07 logmsgbot: catrope synchronized php-1.17/resources/mediawiki/mediawiki.js 'r83051'
* 19:36 logmsgbot: catrope synchronized php-1.17/resources/mediawiki/mediawiki.js 'r83037'
* 13:45 logmsgbot: ariel synchronized php-1.17/languages/LanguageConverter.php 'make translate() public so zh abstract dumps run'
* 12:51 logmsgbot: tstarling synchronized php-1.17/includes/parser/ParserCache.php 
* 12:29 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'stats for all wikis'
* 12:21 logmsgbot: tstarling synchronized php-1.17/includes/Setup.php 'removed more wfIncrStats calls'
* 12:14 logmsgbot: tstarling synchronized php-1.17/includes/filerepo/LocalFile.php 'disabling some unnecessary wfIncrStats() calls to reduce collector load'
* 06:41 logmsgbot: tstarling synchronized php-1.17/includes/GlobalFunctions.php 'wfIncrStats() aggregation hack'
* 06:35 logmsgbot: tstarling synchronized php-1.17/includes/Article.php 'profiling'
* 04:58 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'remove commented-out read only files'
* 04:22 Tim: moved liquidthreads.labs.wikimedia.org from all.dblist to deleted.dblist, so that its brokenness will offend people less
* 03:45 Ryan_Lane: changing the proxyfetch url to something non-version specific; should also repool all varnish servers
* 03:43 logmsgbot: tstarling synchronized docroot/bits/pybal-test-file 
* 03:37 logmsgbot: tstarling ran sync-common-all 
* 02:47 logmsgbot: root synchronized php-1.17/wmf-config/InitialiseSettings.php 'sync-file test'
* 02:41 logmsgbot: root synchronizing Wikimedia installation... Revision: 82948: 
* 02:40 logmsgbot: root synchronizing Wikimedia installation... Revision: 82948: 
* 02:39 Tim: running the new scap
* 02:24 Tim: updated the scap scripts in wikimedia-task-appserver, now racing puppet to update the package everywhere
* 02:02 Tim: updated scap and sync-file and related scripts in /home/wikipedia/bin to work with 1.17
* 01:22 ^demon: removed requirement for comment on bug resolution
* 00:56 Tim: fixed /home/wikipedia/bin/mysql-list, it makes a list of mysql servers
* 00:31 Tim: moving a lot of old scripts out of /home/wikipedia/bin to junk
* 00:19 Tim: fixed subversion versioning in php-1.17/wmf-config, checked in 1.17-related changes into the local subversion instance
* 00:10 logmsgbot: root ran sync-common-all 
* 00:09 Tim: sync-common-all
* 00:07 Tim: except fonts
* 00:03 Tim: moved some old junk out of /home/wikipedia/common to /home/wikipedia/junk/common: nagios-fedora-plugins, fonts, lockfiles, proxies
== February 28 ==
* 23:53 Tim: tarred up the old MediaWiki directories in /home/wikipedia/common and moved them out to /home/wikipedia/lazy-backups/nostalgia
* 23:31 Tim: created job-runners group with a list of servers with job runners
* 23:26 Tim: cleaned up /etc/dsh/group somewhat, moved old files to the junk subdirectory
* 22:42 apergos: removed the wm-job-runner package on srv151 through srv190, so job runners don't accidentally get started back up on those hosts
* 21:59 apergos: stopped job runner on hosts that also serve as ext stores (srv151-srv190) to see if that deals with the "Too many connections" issue seen today for connections to srv160 and srv161
* 21:10 Ryan_Lane: repooling sq68
* 21:05 Ryan_Lane: pushing out load.php fix for domains like arbcom.nl.wikipedia.org
* 21:00 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryImageInfo.php 'r82960'
* 20:55 Ryan_Lane: adding another subdomain level for load.php in bits; depooling sq68 for testing
* 20:48 RobH: poking at blogs
* 19:05 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryAllUsers.php 
* 19:05 logmsgbot: catrope synchronized php-1.17/includes/api/ApiBase.php 
* 19:04 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryImageInfo.php 
* 19:04 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryFilearchive.php 
* 19:04 logmsgbot: catrope synchronized php-1.17/includes/Article.php 
* 19:04 logmsgbot: catrope synchronized php-1.17/extensions/Nuke/Nuke_body.php 
* 19:02 RoanKattouw: Deploying Reedy's recent merges (r82714, r82731, r82806, r82820, r82825) using individual file syncs
* 17:07 mark: Reinstalling lvs4
* 13:34 logmsgbot: tstarling synchronized php-1.17/includes/Setup.php 'profiling hack'
* 13:01 Tim: deployed wikimedia-job-runner 1.2
* 11:59 logmsgbot: ariel synchronized php-1.17/maintenance/dumpTextPass.php 'back off on XMLReader() due to performance issues'
* 11:42 mark: Power cycled sq70
* 11:38 mark: Disabled RP_filter on new LVS hosts with puppet
* 06:53 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 06:51 logmsgbot: tstarling synchronized live-1.5/MWVersion.php 'disabling the multi-version hack, will caused test2wiki to go to 1.17'
* 06:39 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 06:38 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 06:33 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'more stats'
* 06:31 logmsgbot: tstarling synchronized php-1.17/includes/GlobalFunctions.php 'reverted 1.17 transition stats patch'
* 06:29 logmsgbot: tstarling synchronized php-1.17/StartProfiler.php 'profiling all wikis'
* 04:43 logmsgbot: tstarling synchronized php-1.17/wmf-config/wgConf.php 'fixing bug 27767 by sending commons requests to localhost'
* 01:45 Tim: added a "mediawiki" group to /svnroot/mediawiki/conf/authz, which allows for shell users to be omitted from MediaWiki r/w access
* 01:35 Tim: unsubscribed amcchord@dattobackup.com from wikitech-l
== February 27 ==
* 21:09 logmsgbot: demon synchronized php-1.17/includes/media/SVGMetadataExtractor.php 'r82912'
* 17:55 domas: added db25 to s3
* 17:54 logmsgbot: midom synchronized php-1.17/wmf-config/db.php 
* 17:53 mark: NFS /home back up
* 17:48 mark: Removed (duplicate) service ip 10.0.5.8 from eth0 on nfs1; nfs2 is current master
* 17:44 richcole: moving lvs4
* 17:27 mark: Migrated text.pmtpa.wikimedia.org from lvs4 to lvs2
* 17:24 mark: Migrated bits.pmtpa.wikimedia.org from lvs4 to lvs2
* 17:19 mark: Migrated m.pmtpa.wikimedia.org LVS service from lvs4 to lvs2
* 16:51 mark: Reinstalled lvs6 with Lucid as well
* 16:39 richcole: moving nfs1
* 16:39 mark: Reinstalling lvs5 with Lucid
* 16:39 mark: Rich renamed transcode2 to lvs5 and transcode3 to lvs6
* 16:30 richcole: moving lvs2
* 15:57 richcole: moving transcode2
* 15:45 richcole: moving transcode3
* 11:38 logmsgbot: midom synchronized php-1.17/wmf-config/db.php 
== February 26 ==
* 15:32 apergos: apt-get clean on srv219 and 222 to keep them from whining as much about disk space
* 09:09 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 27240 - Request for localized logo to be effected at tpi.wikipedia.org'
== February 25 ==
* 22:36 apergos: job queue reported slower than molasses, checked the job runners and there was one host running something. I did not restart the job runners though because the package has not been updated for 1.17 and was using the pre-1.17 deployment codebase. woops.
* 13:59 mark: Powered down transcode3 (preparing for move)
* 13:30 mark: Reinstalled lvs2 with Lucid, preparing as new LVS balancer
* 13:02 mark: Removed BGP session for lvs2 on csw5-pmtpa
* 12:36 mark: Setup csw1-sdtpa:11/20 for lvs2 (untagged vlan 100, tagged vlan 2)
* 12:12 mark: Powercycled amssq33, 35, 45
* 12:05 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Add sidebar messages to $wgForceUIMsgAsContentMsg for mediawikiwiki'
* 11:50 logmsgbot: hashar: verified logos are live on kowikiquote and nnwikiquote. The later need a transparent background, user notified
* 11:47 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'logos for kowikiquote (bug 27548) and nnwikiquote (bug 27555)'
* 11:38 logmsgbot: hashar: running updateArticleCount on ttwiki for bug 27705
* 11:30 logmsgbot: hashar: srv169 & srv284 unreacheable. Maybe they should be removed. srv266 still in timeout :(
* 11:29 logmsgbot: hashar: checked that the three previous logos are correctly deployed
* 11:26 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'logos for knwiki (bug 27657), sawiki (bug 27661) and orwiki (bug 27704). Please note they are png thumbnails of the .svg images'
== February 24 ==
* 21:18 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27495 - adding user namespaces aliases in portuguese for ptwiki. Default is brazilian variant.'
* 19:34 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Another fix: apparently wikieditor-preview and wikieditor-previewDialog both exist, gotta hide both of them'
* 19:32 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Hide preview dialog preference, this feature is unstable. Was hidden before but got exposed because the preference key changed'
* 16:24 RoanKattouw: Restarted morebots as catrope. robh still has a zombie instance running that I can't kill
== February 23 ==
* 21:24 RoanKattouw: Adding skins-1.17 symlink to all docroots that also have skins-1.5
* 21:21 logmsgbot: catrope synchronized php-1.17/extensions/LiquidThreads/classes/View.php 'Also stop LQT from including JUI CSS and add WikiEditor dialog CSS instead'
* 21:18 logmsgbot: catrope synchronized live-1.5/skins-1.17 
* 21:17 logmsgbot: catrope synchronized php-1.17/extensions/LiquidThreads/classes/View.php 'Fix LQT to not include its own JUI copy'
* 21:03 logmsgbot: catrope ran sync-common-all 
* 21:02 RoanKattouw: Running sync-common-all to deploy r82700
* 17:08 mark: Migrated upload.pmtpa.wikimedia.org from lvs2 to lvs1
* 17:03 mark: Migrated owa.wikimedia.org from lvs4 to lvs1
* 16:57 mark: Migrated payments.wikimedia.org LVS to lvs1
* 16:49 mark: Reboot test of lvs1
* 16:49 logmsgbot: catrope synchronized php-1.17/includes/parser/Parser.php 
* 16:48 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/ui/CodeCommentsListView.php 
* 16:48 logmsgbot: catrope synchronized php-1.17/extensions/ProofreadPage/ProofreadPage_body.php 
* 16:48 logmsgbot: catrope synchronized php-1.17/extensions/Collection/Collection.php 
* 16:48 logmsgbot: catrope synchronized php-1.17/skins/common/wikibits.js 
* 16:48 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesPt.php 
* 16:47 RoanKattouw: Deploying r82678 with individual syncs
* 16:30 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Set account creation throttle to 300 for 194.64.239.* on elwiki. Requested by Ariel'
* 15:39 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable $wgHtml5 on all wikis'
* 15:26 logmsgbot: catrope synchronized php-1.17/wmf-config/InitialiseSettings.php 'Enable $wgHtml5 on testwiki. If this goes well I will enable it on all wikis soon'
* 14:51 mark: Fixed puppet for lvs[234]
* 07:13 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'removed hacks to increase memory_limit to 100MB in certain cases'
* 07:12 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'increased memory_limit to 120MB'
== February 22 ==
* 02:33 logmsgbot: tstarling synchronized php-1.17/languages/LanguageConverter.php 
== February 21 ==
* 21:43 logmsgbot: catrope ran sync-common-all 
* 21:41 RoanKattouw: Running sync-common-all to deploy r82576.
* 14:21 logmsgbot: tstarling synchronized php-1.17/extensions/ProofreadPage/ProofreadPage_body.php 
== February 20 ==
* 20:42 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesTt_cyrl.php 'r82535'
* 20:23 logmsgbot: catrope synchronized php-1.17/includes/OutputPage.php 
* 20:18 logmsgbot: catrope ran sync-common-all 
* 20:16 RoanKattouw: Deploying merges in r82533 with sync-common-all
* 18:31 RoanKattouw: Running clearMessageBlobs.php
* 17:34 logmsgbot: catrope synchronized php-1.17/extensions/WikiEditor/modules/ext.wikiEditor.toolbar.js 'Touch this file to force module to regenerate'
== February 19 ==
* 17:23 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesMl.php 'Fix typo in category talk namespace name'
* 14:59 logmsgbot: hashar synchronized php-1.17/wmf-config/InitialiseSettings.php 'Bug 27495 - Change NS_USER and NS_USER_TALK namespaces at ptwiki'
* 14:38 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialListfiles.php 
* 14:38 logmsgbot: catrope synchronized php-1.17/extensions/CentralAuth/SpecialCentralAuth.php 
* 14:38 logmsgbot: catrope synchronized php-1.17/extensions/CentralAuth/ApiQueryGlobalUserInfo.php 
* 14:38 logmsgbot: catrope synchronized php-1.17/extensions/skins/p2wiki/P2wiki.skin.php 
* 14:38 logmsgbot: catrope synchronized php-1.17/extensions/ProofreadPage/ProofreadPage_body.php 
* 14:38 logmsgbot: catrope synchronized php-1.17/extensions/FlaggedRevs/FlaggedArticleView.php 
* 14:37 logmsgbot: catrope synchronized php-1.17/skins/vector/screen.css 
* 14:37 RoanKattouw: Deploying updates in r82454 with individual syncs
* 13:59 RoanKattouw: Syncing php-1.17/cache/l10n/* to all Apaches manually using ddsh+rsync
* 12:47 logmsgbot: catrope synchronized php-1.17/extensions/LocalisationUpdate/LocalisationUpdate.class.php 'r82450'
* 12:09 RoanKattouw: Running LocalisationUpdate with prospective fix patched in
* 11:47 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r82444'
* 11:46 logmsgbot: catrope synchronized php-1.17/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'r82444'
* 11:36 logmsgbot: hashar: running eval wfMsg( 'file-info-size') on all wiki for bug 27472
* 02:04 logmsgbot: LocalisationUpdate failed
== February 18 ==
* 21:08 RoanKattouw: Restarted import of Geograph images. Running in a screen on hume (as import.sh) with 30-min sleeps between batches
* 17:19 mark: Created mgmt.eqiad.wmnet and 65.10.in-addr.arpa DNS zones/ORIGINs
* 14:29 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'Bump filter cache version'
* 14:29 logmsgbot: catrope synchronized php-1.17/includes/libs/JavaScriptDistiller.php 'r82402'
* 05:55 logmsgbot: tstarling synchronized php-1.17/includes/resourceloader/ResourceLoader.php 
* 03:48 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'bug 27367'
* 02:12 logmsgbot: LocalisationUpdate failed
* 00:35 Tim: fixed various cron jobs on hume
* 00:30 logmsgbot: tstarling synchronized php-1.17/maintenance/jobs-loop.sh 'fix directory'
* 00:01 logmsgbot: tstarling synchronized php-1.17/wmf-config/secure.php 'wgStylePath fix'
== February 17 ==
* 23:57 logmsgbot: tstarling synchronized php-1.17/includes/DefaultSettings.php 
* 23:54 logmsgbot: tstarling synchronized php-1.5/includes/DefaultSettings.php 'commenting out the $wgStylePath reference assignment, to try to fix the canary mismatch errors'
* 23:42 Tim: fixed logrotate on nfs2 and ran it with -f
* 23:16 logmsgbot: catrope synchronized php-1.17/includes/api/ApiQueryPageProps.php 'r82363'
* 23:16 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialContributions.php 'r82363'
* 23:16 logmsgbot: catrope synchronized php-1.17/includes/specials/SpecialNewpages.php 'r82363'
* 23:08 logmsgbot: catrope synchronized php-1.17/includes/MessageCache.php 'r82362'
* 22:45 logmsgbot: catrope synchronized php-1.17/extensions/WikiEditor/WikiEditor.hooks.php 'r82360'
* 22:38 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'Increment RL minifier cache version again'
* 22:37 logmsgbot: catrope synchronized php-1.17/includes/libs/JavaScriptDistiller.php 'r82358'
* 22:25 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'r82355. Will expire the entire ResourceLoader minifier cache'
* 22:08 logmsgbot: catrope synchronized php-1.17/includes/libs/JavaScriptDistiller.php 'Deploy minfier fixes'
* 21:37 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump style version appendix'
* 21:36 logmsgbot: catrope synchronized php-1.17/extensions/LiquidThreads_alpha/jquery/plugins.js 'r82349'
* 21:35 logmsgbot: catrope synchronized php-1.17/extensions/LiquidThreads/jquery/plugins.js 'r82349'
* 16:06 logmsgbot: mark synchronized php-1.5/../php-1.17/includes/media/SVGMetadataExtractor.php 
* 15:55 mark: Disabled apparmor profile for apache2 on srv178, which was spamming the log
* 15:09 mark: Temporarily disabled wikidiff on srv151 (puppet will reenable soon)
* 14:53 mark: Cleaned up srv178
* 04:39 logmsgbot: demon synchronized php-1.17/includes/parser/Preprocessor_DOM.php 'Sync wfErrorLog removals, was filling up /home'
* 04:15 apergos1: killed udp2log on nfs2 and restarted it via init script after removing preprocessorfatal.log again, got back 160 gb, we had gone down to 6.5 gb free on /home on fenari 
* 02:47 Tim: re-enabled snaprotate, bug 27471 does not involve data loss
* 02:22 logmsgbot: LocalisationUpdate failed
* 01:59 Tim: redirected /home/wikipedia/common/php symlink to php-1.17
* 01:36 Tim: disabled snaprotate.pl temporarily while I investigate bug 27471. This has to be reverted within a few days or else the snapshots will fill
* 00:04 Tim: rebooting spence, its load is too high to let me get a terminal
== February 16 ==
* 21:17 logmsgbot: catrope synchronized php-1.17/wmf-config/db.php 'Depool srv178 from ES'
* 21:09 Ryan_Lane|dc: on nfs2
* 21:09 Ryan_Lane|dc: killing udp2log and starting it via init script
* 21:08 Ryan_Lane|dc: restarting udp2log on nfs2
* 21:07 Ryan_Lane|dc: restarting rsyslog on nfs2
* 21:05 Ryan_Lane|dc: restarting rsyslog on fenari
* 21:00 RoanKattouw: Removing /h/w/logs/preprocessorfatal.log , was 146GB
* 20:59 logmsgbot: catrope synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Remove debug logging, 146 GB of the same error message over and over again is enough'
* 20:39 RobH: srv178 powered down, full disk, preventing syncs, needs to be powered backup and fixed later
* 20:12 hashar: Freenode seems down. I can connect using IPv6 though :)
** hashar: Most probably the IPv6 server had a netsplit.
* 20:10 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderWikiModule.php 'r82287'
* 20:07 RoanKattouw: Changed /home/wikipedia/bin/l10nupdate (L10Update cron job) to use php-1.17 instead of wmf-deployment
* 20:04 RoanKattouw: Fixed redirect loop on mlwiki a few minutes ago by changing the namespace names for NS_CATEGORY and NS_CATEGORY_TALK in MessagesMl.php
* 20:04 logmsgbot: catrope synchronized php-1.17/includes/Wiki.php 'Remove debugging code'
* 20:03 RoanKattouw: Restarted morebots as me on wikitech
* 17:35 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesKaa.php 'r82262'
* 17:35 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesCa.php 'r82262'
* 17:28 logmsgbot: catrope synchronized php-1.17/includes/Skin.php 'Fix stupid fatal'
* 17:21 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderStartUpModule.php 'r82260'
* 17:21 logmsgbot: catrope synchronized php-1.17/includes/Skin.php 'r82260'
* 16:36 logmsgbot: catrope synchronized php-1.17/languages/Language.php 'r82256'
* 16:36 logmsgbot: catrope synchronized php-1.17/includes/MessageCache.php 'r82255'
* 15:17 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Re-enable MWSuggest now that it can coexist with SimpleSearch'
* 15:07 logmsgbot: catrope synchronized php-1.17/extensions/WikiEditor/modules/jquery.wikiEditor.iframe.js 'r82251'
* 15:06 logmsgbot: catrope synchronized php-1.17/extensions/WikiEditor/modules/jquery.wikiEditor.js 'r82251'
* 14:49 logmsgbot: catrope synchronized php-1.17/resources/startup.js 'Touch this file so changes to ResourceLoaderStartupModule.php take effect'
* 14:37 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderStartUpModule.php 'r82248'
* 14:34 logmsgbot: catrope synchronized php-1.17/includes/OutputPage.php 'r82248'
* 14:10 logmsgbot: catrope synchronized php-1.17/extensions/ProofreadPage/ProofreadPage_body.php 'Debugging'
* 14:01 logmsgbot: catrope synchronized php-1.17/extensions/CategoryTree/CategoryPageSubclass.php 'r82243'
* 13:58 mark: Added hyphen to character set in load.php matching regex on bits
* 13:52 logmsgbot: tstarling synchronized php-1.17/extensions/OpenSearchXml/ApiOpenSearchXml.php 'revert'
* 13:29 logmsgbot: tstarling synchronized php-1.17/extensions/OpenSearchXml/ApiOpenSearchXml.php 'debugging'
* 13:17 logmsgbot: catrope synchronized live-1.5/MWVersion.php 'Attempted fix for wikis with dashes in their name; they were still on 1.16'
* 13:15 logmsgbot: tstarling synchronized php-1.17/extensions/OpenSearchXml/ApiOpenSearchXml.php 'debugging'
* 13:12 logmsgbot: catrope synchronized php-1.17/languages/messages/MessagesKo.php 'r82237'
* 13:10 logmsgbot: tstarling synchronized php-1.17/extensions/OpenSearchXml/ApiOpenSearchXml.php 'debugging'
* 12:28 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'wgStyleSheetPath dereference'
* 12:25 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'wgStyleSheetPath dereference'
* 12:23 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'wgStyleSheetPath dereference'
* 12:21 RoanKattouw: Finished running fixUsabilityPrefs2.php a long time ago, forgot to log
* 12:04 logmsgbot: tstarling synchronized php-1.17/includes/DefaultSettings.php 'heap overflow part 2'
* 11:59 mark: Shutting down sq35 for bad drive
* 11:57 logmsgbot: demon synchronized php-1.17/extensions/CentralAuth/CentralAuthUser.php 'r82234'
* 11:55 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'possible fix for heap overflows'
* 11:54 RoanKattouw: Running fixUsabilityPrefs2.php again, now with wikieditor-% and usenavigabletoc fixes
* 11:52 mark: Restarted squid-frontend on sq35
* 11:48 RoanKattouw: Running maintenance/wmf/fixUsabilityPrefs2.php on all wikis
* 11:34 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'fix wfGetIP call'
* 11:30 Tim: all wikis -> 1.17
* 11:19 Tim: enwiki -> 1.17
* 10:59 mark: Restoring traffic to bits.esams
* 10:42 mark: Moving bits.esams traffic to bits.pmtpa
* 10:36 mark: Installing 2.6.37-12 kernel on knsq4, rebooting
* 10:14 mark: Deploying fix: User-agent sending in Varnish backend checks
* 10:11 Tim: enwiki -> 1.16
* 10:01 Tim: enwiki -> 1.17
* 09:18 Tim: whacking canary error moles
* 08:49 logmsgbot: catrope synchronized php-1.17/languages/classes/LanguageEo.php 'Fix cx->cxx problem on Esperanto wikis'
* 08:42 Tim: doubled APC cache size to 240MB
* 08:27 apergos: restarted apache on srv247, was seeing lots of segfaults
* 08:25 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 're-enable wmerrors'
* 08:23 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 'disable wmerrors'
* 08:09 mark: Having puppet install screen and gdb on all machines (in base.pp)
* 07:56 logmsgbot: catrope synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.php 'Style version'
* 07:54 logmsgbot: catrope synchronized php-1.17/extensions/FlaggedRevs/client/flaggedrevs.css 'Fix for FR RTL issue'
* 07:43 logmsgbot: demon synchronized php-1.17/extensions/FlaggedRevs/specialpages/ProblemChanges_body.php 'Stop fatal'
* 07:40 logmsgbot: root ran sync-common-all 
* 07:35 logmsgbot: demon synchronized php-1.17/includes/User.php 'r82222'
* 07:20 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'Turning NewUserMessage back on for 1.17 wikis'
* 07:01 Tim: frwiki -> 1.16
* 06:51 logmsgbot: catrope synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'Fix FlaggedRevs path bug'
* 06:48 Tim: zhwiki -> 1.17
* 06:46 Tim: enwiktionary -> 1.17
* 06:45 Tim: commonswiki -> 1.17
* 06:43 Tim: jawiki -> 1.17
* 06:38 Tim: dewiki, frwiki to 1.17
* 06:30 Tim: nlwiki to 1.17
* 06:23 Tim: ran sync-common on srv298
* 06:22 Tim: eowiki to 1.17
* 06:21 mark|coffee: Cleaned up space on srv219
* 06:12 Tim: running fixUsabilityPrefs.php
* 02:53 Tim: updated wmerrors in apt to version 1.1.2
== February 15 ==
* 22:03 RoanKattouw: Uncommented srv298 in mediawiki-installation node list. Comment says it's not installed, but it's pooled
* 20:59 logmsgbot: catrope synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.class.php 'Fix extension path in FlaggedRevs'
* 20:59 logmsgbot: catrope synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.php 'Fix extension path in FlaggedRevs'
* 18:58 mark: Singer has a bad HD /dev/sda, marked it as faulty in the RAID1 arrays
* 18:51 mark: powercycled singer
* 14:12 mark: powercycled amssq32
* 10:25 logmsgbot: tstarling ran sync-common-all 
* 10:14 Tim: deploying latest version of 1.17wmf1 to the wikis that use it
* 08:53 Tim: added wmerrors requirement to puppet, with a configuration file that sends logs to UDP on fenari
* 08:38 Tim: set up temporary UDP log receiver for wmerrors on fenari, logging to /var/log/mw/fatal.log
* 07:01 Tim: added package php5-wmerrors to apt, and installed it on srv224
* 05:24 logmsgbot: aaron synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.php 'deployed r82159'
== February 14 ==
* 21:54 RoanKattouw: Running expensive queries on db39 to diagnose a database issue causing preferences bugs in 1.17
* 19:32 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'and codereview-associate too'
* 19:26 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Restrict CodeReview signoffs to coders'
* 17:55 ^demon: cleaned up bogus code_tags on mediawikiwiki containing commas from before we comma-delimited them
* 17:44 logmsgbot: demon synchronized php-1.17/extensions/CodeReview/ui/CodeTagListView.php 'Add some newlines between links'
* 16:57 logmsgbot: catrope synchronized php-1.5/wmf-config/ExtensionMessages.php 'This accidentally got 1.17-ified on Feb 2nd'
* 16:32 RoanKattouw: Running sync-common on srv298 because it was spewing errors about commented-out code
* 15:01 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/backend/CodeRevision.php 'Attempting to fix CR notif e-mail issue'
* 14:56 logmsgbot: catrope synchronized php-1.17/extensions/CodeReview/CodeReview.i18n.php 
* 14:48 logmsgbot: catrope synchronized live-1.5/MWVersion.php 'Previous sync fixed mediawikiwiki for HTTP, was already working for HTTPS. Now fixed same bug for foundationwiki and oldwikisource so those will work once switched over'
* 14:46 logmsgbot: catrope synchronized live-1.5/MWVersion.php 
* 14:45 logmsgbot: catrope synchronized live-1.5/MWVersion.php 'Attempted bugfix for mediawikiwiki'
* 14:41 RoanKattouw: Switching mediawikiwiki to 1.17
* 10:32 Tim: committed Mark's version of 1.16 CommonSettings.php to svn
* 10:14 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 
* 10:13 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 
* 10:11 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 
* 10:08 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Fix ptwikibooks (autothrottle broken)'
== February 13 ==
* 22:20 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'Turning NewUserMessage back off for 1.17 wikis'
* 22:04 logmsgbot: demon synchronized php-1.17/wmf-config/CommonSettings.php 'Turning NewUserMessage back on for 1.17 wikis, with fixes'
* 21:59 logmsgbot: demon synchronized php-1.17/resources/Resources.php 'r82084'
* 00:06 logmsgbot: catrope synchronized php-1.17/resources/Resources.php 'r82044'
== February 12 ==
* 23:42 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'r82043'
* 23:38 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderFileModule.php 'r82041, the right file this time'
* 23:32 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderModule.php 'r82041'
* 22:02 logmsgbot: catrope synchronized live-1.5/resources 'Point this symlink to the 1.17 tree'
* 21:47 RoanKattouw: Fixed accidental svn1.6 contamination of php-1.17 by checking it out again with svn1.4, and copying modified, ignored and unversioned files. Kept the original tree around in /h/w/c/old-php-1.17-with-svn-1.6 in case something is missing, although diff -rq | grep -v .svn didn't think so
* 21:07 logmsgbot: demon synchronized php-1.17/extensions/NewUserMessage/NewUserMessage.class.php 'r82032'
* 21:07 logmsgbot: demon synchronized php-1.17/extensions/NewUserMessage/NewUserMessage.i18n.php 'r82032'
* 20:53 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Set $wgUseRCPatrol to true for test2wiki...one last time'
* 18:59 ^demon: scratch that, I should've waited for mark's fix to go live
* 18:58 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Set $wgUseRCPatrol to true for test2wiki...again'
* 17:38 Ryan_Lane: upgrading libvirt packages on nova-*.tesla for sheepdog support
* 08:47 domas: killed few more smssend processes...
* 00:43 logmsgbot: demon synchronized php-1.17/wmf-config/InitialiseSettings.php 'Set $wgUseRCPatrol to true for test2wiki'
== February 11 ==
* 22:49 Ryan_Lane: rebooting nova-controller.tesla
* 21:36 Ryan_Lane: changing nova-* packages to the bexar release version on nova-*.tesla
* 16:15 mark_: made Puppet catalog compiles fast
* 14:35 domas: disabled newusermessage 
* 14:34 logmsgbot: midom synchronized php-1.17/wmf-config/CommonSettings.php 
* 14:34 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 
* 12:20 logmsgbot: demon synchronized php-1.17/includes/ChangesList.php 'Fix warning, $classes should be an array for insertTags()'
* 12:18 logmsgbot: catrope synchronized php-1.17/extensions/ParserFunctions/ParserFunctions.i18n.magic.php 'Touching this file so the magic words fix takes effect'
* 12:12 logmsgbot: catrope synchronized php-1.17/includes/parser/Preprocessor_DOM.php 
* 12:11 logmsgbot: tstarling synchronized php-1.5/includes/GlobalFunctions.php 'remove nlwiki and eowiki from 1.17 profiling'
* 12:10 logmsgbot: tstarling synchronized php-1.5/StartProfiler.php 'remove nlwiki and eowiki from 1.17 profiling'
* 12:09 logmsgbot: tstarling synchronized php-1.17/StartProfiler.php 'remove nlwiki and eowiki from 1.17 profiling'
* 12:00 logmsgbot: catrope synchronized php-1.17/extensions/CentralAuth/SpecialWikiSets.php 'r81956'
* 11:59 logmsgbot: catrope synchronized php-1.17/includes/LocalisationCache.php 'r81960'
* 11:06 logmsgbot: demon synchronized php-1.17/extensions/CentralAuth/SpecialWikiSets.php 'Fixing fatal without breaking SpecialWikiSets'
* 11:04 logmsgbot: demon synchronized php-1.17/extensions/CentralAuth/SpecialWikiSets.php 'Fixing fatal'
* 11:01 logmsgbot: catrope synchronized live-1.5/MWVersion.php 'Unbreak test on secure'
* 10:47 logmsgbot: catrope synchronized php-1.17/includes/parser/Preprocessor_DOM.php 'Trying to get a backtrace on preprocessor fatals'
* 10:28 logmsgbot: catrope synchronized php-1.17/languages/classes/LanguageEo.php 'r81954'
* 10:18 Tim: switched eowiki and nlwiki back to 1.16
* 10:13 Tim: switched the following wikis to 1.17: hewikisource eowiki enwikiquote enwikinews enwikibooks betawikiversity nlwiki hewikisource
* 10:11 logmsgbot: catrope synchronized php-1.17/resources/mediawiki.special/mediawiki.special.preferences.js 'r81953'
* 09:49 logmsgbot: catrope synchronized php-1.17/resources/jquery/jquery.client.js 
* 09:48 logmsgbot: catrope synchronized php-1.17/resources/jquery/jquery.client.js 'Attempting to fix JS error'
* 09:45 mark: Deploying /etc/php5/conf.d/wikidiff2.ini using puppet
* 09:43 logmsgbot: tstarling synchronized php-1.17/wmf-config/CommonSettings.php 're-enable wikidiff2, use extension_loaded() to test for existence instead of file_get_contents()'
* 09:40 logmsgbot: catrope synchronized php-1.17/includes/EditPage.php 'r81947'
* 09:38 Tim: enabling wikidiff2 on all servers by manually creating wikidiff2.ini
* 09:29 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderUserModule.php 'r81944'
* 09:28 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoaderSiteModule.php 'r81944'
* 09:16 logmsgbot: catrope synchronized php-1.17/extensions/CentralAuth/SpecialGlobalGroupMembership.php 'Trying to fix GlobalUserRights->UserRights redirect'
* 09:01 logmsgbot: catrope synchronized php-1.17/extensions/PrefSwitch/PrefSwitch.hooks.php 'r81939'
* 09:01 logmsgbot: catrope synchronized php-1.17/extensions/PrefSwitch/PrefSwitch.php 'r81939'
* 08:55 logmsgbot: tstarling synchronized php-1.17/StartProfiler.php 
* 08:54 logmsgbot: tstarling synchronized php-1.17/includes/GlobalFunctions.php 
* 08:54 Tim: adding hewikisource to the relevant phase1 lists
* 08:54 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 
* 08:53 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Disable PrefSwitch links on 1.17 wikis'
* 08:50 Tim: upgraded metawiki and hewikisource to 1.17
* 08:48 logmsgbot: catrope synchronized php-1.17/extensions/LiquidThreads/classes/View.php 'Attempting to fix JS error by loading in dialogs module'
* 08:41 logmsgbot: catrope synchronized php-1.17/wmf-config/liquidthreads.php 'Fix another broken LQT extensions path'
* 08:36 logmsgbot: catrope synchronized php-1.17/extensions/LiquidThreads/LiquidThreads.php 'Fix LQT path to use $wgExtensionAssetsPath'
* 08:33 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'r81938'
* 08:22 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Bump style version appendix'
* 08:20 logmsgbot: catrope synchronized php-1.17/extensions/LiquidThreads/lqt.js 'Testing fix for JS error'
* 07:58 RoanKattouw: Truncating module_deps on all wikis
* 07:48 Tim: switching strategy.wikimedia.org, simple.wikipedia.org and simple.wiktionary.org to 1.17
* 07:20 logmsgbot: catrope synchronized php-1.17/includes/MessageBlobStore.php 'r81933: Disable clear()'
* 07:19 logmsgbot: tstarling synchronized php-1.17/includes/MessageBlobStore.php 
* 06:48 logmsgbot: catrope synchronized php-1.17/includes/MessageBlobStore.php 'Wrap clear() in a transaction to hopefully make stuff break less badly'
* 06:34 mark: Migrating srv248 and srv249 to bits appservers
* 06:24 logmsgbot: root ran sync-common-all 
* 06:16 logmsgbot: tstarling synchronized php-1.17/extensions/CommunityVoice/Modules/Ratings.php 
* 05:59 Tim: moving usabilitywiki to 1.17
* 05:19 logmsgbot: tstarling synchronized php-1.17/StartProfiler.php 'Synchronising StartProfiler.php with the 1.16wmf4 version'
* 05:14 logmsgbot: tstarling synchronized php-1.17/StartProfiler.php 'Synchronising StartProfiler.php with the 1.16wmf4 version'
* 05:10 logmsgbot: tstarling synchronized php-1.17/includes/GlobalFunctions.php 'pcache stats patch'
* 03:58 logmsgbot: ariel synchronized php-1.17/wmf-config/InitialiseSettings.php 'wgLogAutocreatedAccounts enabled for test2wiki'
* 02:22 logmsgbot: tstarling synchronized php-1.5/includes/GlobalFunctions.php '1.17 stats attempt 3'
* 02:18 logmsgbot: tstarling synchronized php-1.5/includes/GlobalFunctions.php '1.17 stats attempt 2'
* 02:14 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'stats for phase 1 deployment'
* 02:11 logmsgbot: tstarling synchronized php-1.5/includes/GlobalFunctions.php 'configurable profile ID for wfIncrStats()'
* 01:29 logmsgbot: tstarling synchronized php-1.5/StartProfiler.php 'set up profiling section for the first batch of wikis which will be moved to 1.17, for baseline'
* 00:48 logmsgbot: tstarling synchronized php-1.5/includes/parser/ParserCache.php 'remove age stats'
* 00:45 Tim: made http://noc.wikimedia.org/cgi-bin/report.py a redirect to http://noc.wikimedia.org/cgi-bin/ng/report.py
== February 10 ==
* 23:10 Ryan_Lane: rebooting nova-compute1 and nova-compute2
* 22:36 mark: bits.wikimeda.org on the backend is now only served by a special application server cluster (srv191, srv192), load balanced by Varnish
* 22:06 Ryan_Lane: upgrading qemu, kvm, qemu-kvm, qemu-common, purging kvm-pxe, installing libjpeg62 on nova-compute1
* 22:05 Ryan_Lane: installing libjpeg62 on nova-compute2, as it is a requirement for qemu-common
* 22:04 Ryan_Lane: upgrading those packages on nova-compute2
* 22:03 Ryan_Lane: upgrading qemu, kvm, qemu-kvm, qemu-common, purging kvm-pxe
* 19:58 mark: Depooled sq67 for testing
* 19:45 mark: Depooled srv192 as normal app server, converted to bits app server (in puppet)
* 19:41 Ryan_Lane: deploying a new squid config to all squids
* 19:35 mark: Depooled srv191 as normal app server, converted to bits app server (in puppet)
* 19:24 Ryan_Lane: restarting squid processes on sq34
* 19:09 Ryan_Lane: restarting gmetad on spence
* 19:07 Ryan_Lane: made a new ganglia group for api squids, made sq59 and sq60 aggregators for text squids, made sq31 and sq32 aggregators for api squid
* 19:07 Ryan_Lane: moved sq34 and sq35 to api squids
* 19:07 Ryan_Lane: depooling sq34, sq35
* 18:40 Ryan_Lane: adding sq34 and sq35 to api squids
* 18:35 Ryan_Lane: powering off sq32 - has bad disks
* 18:26 Ryan_Lane: powercycling sq32, it's dead, again
* 18:13 Ryan_Lane: powercycling sq32
* 16:41 Ryan_Lane: cleaning up logs on ekrem, and restarting apache
* 15:30 ^demon: made the 1.17.dblist available with the others on noc.wikimedia.org
* 12:22 logmsgbot: tstarling synchronized php-1.17/load.php 
* 12:20 logmsgbot: tstarling synchronized php-1.17/wmf-config/InitialiseSettings.php 'fix wgCacheDirectory'
* 11:57 logmsgbot: tstarling synchronized php-1.17/includes/resourceloader/ResourceLoaderFileModule.php 'profiling fix'
* 11:53 logmsgbot: tstarling synchronized php-1.17/includes/resourceloader/ResourceLoader.php 
* 11:52 logmsgbot: tstarling synchronized php-1.17/extensions/Gadgets/Gadgets_body.php 'profiling fix'
* 11:46 logmsgbot: tstarling synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'profiling fix?'
* 11:41 logmsgbot: tstarling synchronized php-1.17/includes/resourceloader/ResourceLoader.php 'profiling fix?'
* 07:09 logmsgbot: tstarling synchronized php-1.17/load.php 
* 07:03 logmsgbot: root ran sync-common-all 
* 06:56 logmsgbot: tstarling synchronized php-1.17/StartProfiler.php 'splitting out test2 with 1:1 sample rate for profiling testing'
* 04:34 Ryan_Lane: installing scsi-tools on nova-*.tesla
* 04:29 Ryan_Lane: adding 50GB of storage to each nova-*.tesla vm
* 04:05 Ryan_Lane: installing corosync on nova-*.tesla
* 04:04 Ryan_Lane: installing sheepdog on nova-*.tesla
* 03:35 RobH: mailing lists restarted
* 03:27 RobH: correction, halting mail LIST servers, not all mail servers.
* 03:27 RobH: halting mail servers to clean out an email from archives, will resume normal service shortly
* 02:53 Tim: restarting all apaches to get apc.ini change
* 02:31 Tim: deploying new version of php5-apc package to all apaches, to change a single line in /etc/php5/conf.d/apc.ini
* 00:12 logmsgbot: catrope synchronized php-1.17/StartProfiler.php 'Put 1.17 wikis in their own profiling category'
* 00:10 logmsgbot: catrope synchronized live-1.5/MWVersion.php 'Define ONE_SEVENTEEN for wikis running 1.17'
== February 9 ==
* 23:46 RoanKattouw: Het deploy is now working on secure
* 23:45 RoanKattouw: Syncing docroot
* 23:43 logmsgbot: catrope synchronized php-1.17/wmf-config/secure.php 'Fix $wgLoadScript on secure'
* 23:41 logmsgbot: catrope synchronized live-1.5/MWVersion.php 'Trying to make het deploy work on secure'
* 23:17 logmsgbot: tstarling synchronized live-1.5/apc-sma-info.php 
* 23:16 logmsgbot: tstarling synchronized live-1.5/apc-sma-info.php 
* 22:25 logmsgbot: catrope synchronized live-1.5/MWVersion.php 'Switch 1.17 stuff to a dblist'
* 22:18 logmsgbot: catrope synchronized php-1.17/includes/resourceloader/ResourceLoader.php 
* 22:18 logmsgbot: catrope synchronized php-1.17/includes/parser/ParserOptions.php 
* 22:17 logmsgbot: catrope synchronized php-1.17/extensions/LiquidThreads_alpha/classes/View.php 
* 21:18 logmsgbot: catrope synchronized php-1.17/includes/db/Database.php 
* 21:18 logmsgbot: catrope synchronized php-1.17/includes/MessageCache.php 'r81786'
* 21:14 logmsgbot: catrope synchronized php-1.17/extensions/Vector/modules/ext.vector.collapsibleNav.js 
* 21:07 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Change paths to skins-1.17 and extensions-1.17'
* 21:00 logmsgbot: catrope synchronized live-1.5/extensions-1.17 'Sync extensions symlink too'
* 20:57 Ryan_Lane_: pushing out 1.17 aliases in wikimedia.conf file
* 20:19 logmsgbot: catrope synchronized live-1.5/skins-1.17 'Add skins-1.17 symlink'
* 20:07 logmsgbot: catrope synchronized php-1.17/LocalSettings.php 'Update path here too'
* 20:02 logmsgbot: catrope synchronized live-1.5/MWVersion.php 'Implement experimental hack to put 1.17 on test2wiki. Should not affect other wikis'
* 19:57 logmsgbot: catrope synchronized php-1.17/wmf-config/CommonSettings.php 'Changes for 1.17 on test2wiki experiment'
* 19:53 mark: Reconfigured exim on mchenry and sanger to accept +whatever suffixes to e-mail address local parts
* 19:13 apergos: importing an old simplewiki dump to test2wiki, via importDump.php, running in screen on fenari. 35 000 pages, 200K revisions. 
* 14:10 ^demon: granted myself +sysop +bcrat on test2wiki b/c Roan said I could
* 13:29 logmsgbot: catrope synchronized php-1.5/extensions/CentralAuth/CentralAuthUser.php 'Workaround for CentralAuth exceptions'
* 11:59 Tim: created wiki at test2.wikipedia.org
* 11:58 logmsgbot: tstarling ran sync-common-all 
* 11:51 Tim: adding DNS entry for test2.wikipedia.org
* 11:31 mark: Dovecot on sanger restarted with doubled max memory size of 512 MB
* 02:46 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'test finished'
* 02:36 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'setting $wgCacheEpoch to test theory on 1.17wmf1 failure'
* 01:03 logmsgbot: tstarling synchronized php-1.5/includes/parser/ParserCache.php 'cache hit age instrumentation formatting'
* 01:01 logmsgbot: tstarling synchronized php-1.5/includes/parser/ParserCache.php 'cache hit age instrumentation'
== February 8 ==
* 22:26 logmsgbot: midom synchronized php-1.5/includes/GlobalFunctions.php 'msg profiling'
* 22:09 logmsgbot: midom synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling stats for enwiki'
* 22:00 logmsgbot: midom synchronized php-1.5/wmf-config/InitialiseSettings.php 'udpstats on testwiki'
* 19:16 RoanKattouw: Running maintenance/eval.php calling $wgMessageCache->clear(); on all wikis. This should fix the stuck messages issue
* 17:48 mark: Power cycled singer
* 17:23 logmsgbot: mark synchronized php-1.5/load.php 
* 17:17 RoanKattouw: Servers going back to normal, second deployment attempt failed. Same symptoms as the first: pcache miss rate in the 40s
* 17:12 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 81334: 
* 17:12 RoanKattouw: Rolling back 1.17 deployment again
* 17:04 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEo.php 
* 16:56 logmsgbot: catrope synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Attempting to shut up fatals for good'
* 16:52 logmsgbot: catrope synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Another fatal workaround'
* 16:49 logmsgbot: catrope synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Workaround for preprocessor fatal'
* 16:45 logmsgbot: catrope synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Revert debug change'
* 16:44 logmsgbot: catrope synchronized php-1.5/includes/parser/Preprocessor_DOM.php 'Debugging'
* 16:41 logmsgbot: catrope synchronized php-1.5/wmf-config/secure.php 'Do not send load.php through bits on secure'
* 16:37 logmsgbot: catrope synchronized php-1.5/extensions/SiteMatrix/SiteMatrix.php 'Fix class SiteMatrix now found fatal'
* 16:32 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 81755: 
* 16:32 RoanKattouw: Running scap
* 16:30 RoanKattouw: Switching cluster to 1.17
* 16:30 RoanKattouw: Switching /home to 1.17
* 16:29 logmsgbot: catrope ran sync-common-all 
* 16:29 RoanKattouw: Running sync-common-all to push changes to php-1.17
* 16:28 RoanKattouw: Switching test back to 1.16 in preparation for sync-common-all
* 15:13 RoanKattouw: test is now running 1.17wmf1 again
* 14:42 RoanKattouw: Forgot to log that I put testwiki back on 1.16 about 30 mins ago
* 14:29 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesJa.php 'Testing to see if this will fix the message caching issue'
* 14:00 RoanKattouw: Moving directories on /home again, putting 1.17 on test so we can diagnose the cause of the overload
* 13:41 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 81334: 
* 13:41 RoanKattouw: Running scap
* 13:40 RoanKattouw: 1.17-rollback done
* 13:39 RoanKattouw: Running 1.17-revert to move directories back cluster-wise
* 13:38 RoanKattouw: Moving directories back on /home
* 13:38 RoanKattouw: Rolling back 1.17 deployment
* 13:31 logmsgbot: catrope synchronized php-1.5/includes/parser/Parser.php 'r81738, fixes registerWatcher fatal'
* 13:30 logmsgbot: catrope synchronized php-1.5/extensions/NewUserMessage/NewUserMessage.class.php 'r81737, fixes fatal'
* 13:08 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 81732: 
* 13:07 RoanKattouw: Running scap to make sure the move takes effect properly, caches are purged, etc.
* 13:07 RoanKattouw: Moving wmf-deployment to w-d.old and php-1.17 to wmf-deployment on all Apaches. Run 1.17-rollback to undo this
* 13:06 RoanKattouw: Moving wmf-deployment to w-d.old and php-1.17 to wmf-deployment on /home
* 13:05 logmsgbot: catrope ran sync-common-all 
* 13:04 RoanKattouw: Rerunning sync-common-all
* 13:03 mark: Upgraded rsync to a version with higher tcp backlog on nfs2
* 13:01 RoanKattouw: sync-common-all failed for a few servers due to NFS problems. Mark is fixing it
* 12:58 logmsgbot: catrope ran sync-common-all 
* 12:57 RoanKattouw: Moved old wmf-deployment/ and php-1.17/ back on /home , now syncing that out to the cluster
* 10:38 mark: Power cycled amssq37, 43, 49
* 10:33 mark: Power cycled amssq35
* 10:15 mark: Deploying special case for test.wikipedia.org (=> srv193) on bits.pmtpa (varnish)
* 10:12 logmsgbot: catrope synchronized live-1.5/load.php 'Revert failed hack'
* 10:01 logmsgbot: catrope synchronized live-1.5/load.php 'Temp hack to make load.php work on test'
* 09:28 logmsgbot: root ran sync-common-all 
* 08:58 logmsgbot: root ran sync-common-all 
* 08:41 mark: Unstuck SMS modem
* 07:50 Tim: restarted apache on srv174
* 07:22 mark: Started Nagios bot
* 06:15 ^demon: fixed svn permissions for wmf-deployment group
== February 7 ==
* 22:20 Ryan_Lane: upgrading nova-* on nova-*.tesla
* 20:02 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '27195 - Grant anonymous users 'createpage' right and enable Extension'
* 19:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '27209 - Request Extension of Quiz Addon to English Wikibooks'
* 17:18 logmsgbot: demon synchronized php-1.5/wmf-config/CommonSettings.php 'Removing proof-of-concept MWReleases extension'
* 17:18 logmsgbot: demon synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disabling proof-of-concept MWReleases extension'
* 13:18 Tim: running schema updates on all old masters. db29 already done.
* 13:17 Tim: new master positions: db22-bin.000001:106, db18-bin.000001:106, db34-bin.000001:106
* 13:16 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 13:16 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 13:15 Tim: doing master switch on s3 from db39 to db34
* 13:15 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 13:14 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 13:13 Tim: doing master switch on s7 from db37 to db18
* 13:13 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 13:13 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 13:12 Tim: doing master switch on s4 from db31 to db22
* 13:11 Tim: new positions: db21-bin.001:79, db32-bin.000001:106
* 13:07 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 13:06 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 13:06 Tim: doing master switch on s1 from db36 to db32
* 13:03 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 13:03 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 13:02 Tim: doing master switch of s6 from db29 to db21
* 08:02 Tim: running upgrade-1.17wmf1-initial.php again to upgrade db13 and db23
* 07:52 Tim: s5: New position: db14-bin.001:79
* 07:51 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 07:51 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 07:51 Tim: switching master on s5 from db23 to db14
* 07:41 Tim: s2: New position: db30-bin.000001:106
* 07:41 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 07:40 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 07:39 Tim: switching master on s2 from db13 to db30
* 04:35 apergos: running rebuild of the first of 4 history dump pieces on snapshot3, root, screen session
== February 6 ==
* 21:27 apergos: spliting up stubs into pieces in prep for resume of truncated history dumps fo en wiki; running on snapshot3 in screen session as root (see "part1,8,9,10.sh" in /backups-atg for what's running)
* 14:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'testwiki: Revert'
* 14:48 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'testwiki: Use unifont-5.1.20080907.ttf as default font for EasyTimeline'
* 14:43 logmsgbot: jeluf synchronized fonts/unifont-5.1.20080907.ttf 'GNU Unifont for timeline
* 13:57 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php '21497 CommonSettings.php'
* 13:38 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '27191 - Re-point to local Wiki.png for kowiki'
* 13:35 logmsgbot: jeluf synchronized fonts/FreeSansWMF.ttf 'Merged FreeSerif into FreeSans (latest version). Provides Malayalam characters'
* 13:04 logmsgbot: jeluf synchronized fonts/FreeSansWMF.ttf 'Merged FreeSerif into FreeSans (latest version). Provides Malayalam characters'
* 13:02 logmsgbot: jeluf synchronized fonts/FreeSansWMF.ttf 'Merged FreeSerif into FreeSans (latest version). Provides Malayalam characters'
* 10:38 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'testwiki: Use FreeSansWMF as default font for EasyTimeline'
* 10:38 logmsgbot: jeluf synchronized fonts/FreeSansWMF.ttf 'Merged FreeSerif int FreeSans. *should* now provide Malayalam characters'
* 07:54 apergos: filled up root filesystem on snapshot3 by accident running an xml dump repair job, cleared out the file and restarted the job writing elsewhere, we whould be back to normal on that host now
* 06:49 apergos: four of the newly created en wikipedia history bz2 files are incomplete; they will be resumed after the 1.17 deployment has been pushed and is proven not to impact the dumps
== February 5 ==
* 22:35 logmsgbot: jeluf synchronized fonts/FreeSans.ttf 'New version from freefont-ttf-20100919'
* 22:33 logmsgbot: jeluf synchronized fonts/FreeSerif.ttf 'New version from freefont-ttf-20100919'
* 21:06 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'testwiki: Use FreeSerif as default font for EasyTimeline'
* 21:01 logmsgbot: jeluf synchronized fonts/FreeSerif.ttf '21497 - Change default font for EasyTimeline on ML projects to something that actually has glyphs for Malayalam characters'
* 20:04 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php '27172 CommonSettings.php'
* 20:04 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '27172 - Increase $wgAccountCreationThrottle for Portuguese Wikibooks'
* 19:53 Ryan_Lane: adding the star certificate to lily, and changing lighttpd to use it
* 19:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26739 - wgLogo-variable points to wrong path'
* 12:44 logmsgbot: hashar: Marked bug 24078 as wontfix, wikis we have to request locallogo one by one
* 12:39 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'revering back to commons logo since some wiki have upload disabled (per IRC)'
* 12:20 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php '26739 - wgLogo-variable points to wrong path'
== February 4 ==
* 23:42 Ryan_Lane: disabling port forwarding on formey
* 22:38 Tim: resuming upgrade-1.17wmf1-initial.php ,stopped on usabilitywiki due to a schema idiosyncrasy there
* 22:22 Ryan_Lane: powercycling singer; it's non-responsive
* 17:04 Ryan_Lane: running apache-graceful-all
* 16:36 Ryan_Lane: pushing mod_expires fix to all apaches
* 16:33 Ryan_Lane: restarting apache on srv193 to force load of configuration
* 16:25 Ryan_Lane: testing mod_expires fix on test.wikipedia.org
* 04:49 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'adding a max lag so that servers executing schema changes will automatically be skipped'
* 04:43 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 04:41 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 04:37 Tim: running schema upgrades in a screen on fenari using upgrade-1.17wmf1-initial.php
== February 3 ==
* 23:25 logmsgbot: tfinc synchronized php-1.5/extensions/WikimediaMobile/MobileRedirect.js 
* 23:00 Tim: fixed refresh-dblist to use db.php instead of clusters.dblist for cluster assignments, moved clusters.dblist to junk
* 21:37 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25438 - Enable FlaggedRevs for Belarussian Wikipedia'
* 21:35 logmsgbot: jeluf synchronized flaggedrevs.dblist '25438 - Enable FlaggedRevs for Belarussian Wikipedia'
* 21:34 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '25438 - Enable FlaggedRevs for Belarussian Wikipedia'
* 20:48 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26985 - New namespace alias "WP'
* 20:41 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26817 - Create "Subject" namespace for Bengali Wikibooks'
* 12:08 RoanKattouw: Had to svn switch --relocate the tree from svn+ssh:// to http:// in order to be able to do so without knowing Priyanka's SSH key passphrase
* 12:07 RoanKattouw: Updated /srv/org/wikimedia/prototype/wikis/rc and its extensions on prototype
== February 2 ==
* 21:01 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'dpl nlwiki reverted to false'
* 20:55 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 8240 and 8563'
* 20:52 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 7952 dpl eswiktionary iswiktionary'
* 20:47 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 6163 DPL on nlwiki'
* 20:27 logmsgbot: pdhanda synchronized php-1.5/wmf-config/InitialiseSettings.php 'updated per bug 26062 - enabled abusefilter on ptwikisource'
* 19:31 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '27069 - Define $wgUploadNavigationUrl for ptwikibooks'
* 18:55 Ryan_Lane: adding mlqt.tesla.usability.wikimedia.org to DNS
* 18:02 logmsgbot: hashar: srv169 srv266 and srv284 do not answer to ssh connection triggering error messages on sync :(
* 18:01 logmsgbot: hashar synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache for bug 26376'
* 18:01 logmsgbot: hashar synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache for bug 26376'
* 15:49 Ryan_Lane: running flush hosts on db9
* 14:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable subpages in API namespace on mediawikiwiki'
* 14:29 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set $wgSitename to Wikcionario on eswiktionary per IRC discussion and existing use'
* 13:54 mark: Unblocked some remoteLoaders in squid conf
* 00:48 logmsgbot: tstarling synchronized docroot/wikibooks.org/googled73abdc60a3b46a4.html 
* 00:48 logmsgbot: tstarling synchronized docroot/wikipedia.org/googled73abdc60a3b46a4.html 'authorizing de.wikipedia.org for google webmaster tools'
* 00:46 logmsgbot: tstarling synchronized docroot/wikipedia.org/googled73abdc60a3b46a4.html 'authorizing de.wikipedia.org for google webmaster tools'
* 00:42 Tim: reset password for noc@wikimedia.org google web account
== February 01 ==
* 23:13 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '27080 - Create an API'
* 23:10 Ryan_Lane: installing git-core on nova-controller.tesla
* 23:10 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26320 - Set project namespace of Portuguese Wikipedia to "Wikip??dia"'
* 22:57 logmsgbot: tstarling synchronized php-1.5/languages/Language.php 
* 22:44 logmsgbot: tstarling synchronized php-1.5/languages/Language.php 
* 22:41 logmsgbot: root synchronizing Wikimedia installation... Revision: 81334: r81334
* 21:05 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'Take live hack back out. Cause found, it's FlaggedRevs'
* 21:04 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'Live hack for debugging of noindex,nofollow issue'
* 02:26 river: subscribed all public mailing lists to the newsgate.tcx.org.uk NNTP gateway via Mailman add_member
== January 31 ==
* 21:37 logmsgbot: tstarling ran sync-common-all 'FlaggedRevs on trwiki, bug 26943'
* 19:40 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php '26366 - Thai Wikiquote project name & logo'
* 19:07 Ryan_Lane: purging bin logs to mysql-bin.001858 on db9
* 18:33 RobH: updated puppet to move user pdhanda into mortals group, able to push apache config changes
* 18:18 Ryan_Lane: uninstalling php-openid from nova-controller.tesla
* 17:14 Ryan_Lane: installing php-openid on nova-controller.tesla
* 09:06 tomaszf: installed php5-dev and PECL hash on grosley for david
* 09:06 tomaszf: installed php-mhash on grosley
== January 30 ==
* 17:32 Ryan_Lane: changing otrs to use the star cert
* 17:18 Ryan_Lane: adding star cert to williams
* 14:53 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26363 - Please Add Patroller User Group to fawikinews'
== January 29 ==
* 20:31 apergos: one more time for en wiki history dumps (looks like they might actually work this time). in screen on snapshot3, from /backups-atg.
* 08:46 apergos: starting enwiki history dumps (finally) with latest guesses at page ranges so we get more even no. of revisions per chunk... in screen on snapshot3, out of /backups-atg
* 08:11 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26988 - Changing the logo for Buginese Wikipedia'
* 08:00 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '27003 - enable wgEnotifWatchlist for frrwiki'
* 07:47 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '27015 - Change sitename for gdwiki'
* 07:40 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26451 - Install ReaderFeedback extension on ptwikibooks'
* 07:21 logmsgbot: ariel synchronized wmf-deployment/maintenance/dumpTextPass.php 'multiple prefetch files for xml dumps'
* 07:20 logmsgbot: ariel synchronized wmf-deployment/maintenance/backupPrefetch.inc 'multiple prefetch files for xml dumps'
== January 28 ==
* 21:52 Ryan_Lane: changing out bugzilla's certficiate and key
* 21:10 Ryan_Lane: adding mortals to emery, and adding sudoers.d entry for nimish
* 19:58 logmsgbot: catrope synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php 'r81158, r81159'
* 19:57 logmsgbot: catrope synchronized php-1.5/extensions/ProofreadPage/proofread.js 'r81156'
* 19:05 logmsgbot: catrope synchronized live-1.5/load.php 'Apparently this needs to be a wrapper'
* 18:57 logmsgbot: catrope synchronized live-1.5/load.php 'Add load.php entry point here as well, hopefully this will work'
* 18:50 logmsgbot: catrope synchronized php-1.5/load.php 'load.php rewrite rule testing'
* 18:23 Ryan_Lane: added url rewriting for load.php to bits
* 18:23 Ryan_Lane: repooling sq67
* 17:01 Ryan_Lane: depooling sq67
* 15:26 rainman-sr: searchidx1 is again out of disc space, freeing some space by deleting index snapshots etc, but we need more disc space soon!
* 13:05 ^demon: tweaked "no commit summary" hook on formey to be informative
== January 27 ==
* 22:46 Ryan_Lane: setting carp weight of sq63 to 30
* 22:43 Ryan_Lane: starting squid and puppet services on sq63
* 21:58 richcole: switching network cable out sq63
* 07:16 Ryan_Lane: stopping squid and disabling puppet on sq63
* 07:08 Ryan_Lane: changing carp weight of sq63 to 0
* 06:35 Ryan_Lane: restarting squid processes on sq63
* 02:55 Ryan_Lane: restarting pdns on ns2, it was hung
* 00:56 Ryan_Lane: adding mdale to engineering alias
== January 26 ==
* 21:38 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix typo'
* 18:44 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26915 - Modify $wgNamespaceRobotPolicies to noindex User and User talk namespaces on the French Wikipedia (fr.wikipedia.org)'
* 18:40 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26897 - Enable flag 'flood' in Spanish Wikinews (es.wikinews)'
* 18:37 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26946 - Create new group on Russian Wikipedia'
* 17:50 RobH: updated dns for harmon
* 17:23 mark: Moved ersch, alsted and harmon into the sandbox vlan
* 17:23 mark: Replaced ganglia aggregators for MySQL group
* 14:54 RobH: emery shutdown, needs to have drives swapped back to default
* 00:51 tomaszf: reset Renflauf pass on wikitech
* 00:13 richcole: thistle, ixia,lomaria going down for decommissioning a
== January 25 ==
* 22:55 richcole: db1-4 going down for decommission and wipe
* 22:20 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26889 - Set WB namespace alias to NS_PROJECT in Bengali wikibook'
* 22:14 RobH: project2 moved and back online
* 22:12 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26916 - Enable subpages in Template namespace on wikimania (and for future editions)'
* 22:08 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26913 - Please enable patrolled edits on the Hebrew Wikiquote (he.wikiquote.org)'
* 22:01 RobH: project2 (bugzilla testing) coming down for migration
* 21:39 logmsgbot: jeluf synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache for new wikitech entry'
* 21:22 Ryan_Lane: deleting review.tesla.usability.wikimedia.org, and removing from DNS
* 21:17 Ryan_Lane: removing public IP from analytics.tesla.usability.wikimedia.org and removing it from dns
* 20:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26932 - Add 'suppressredirect' right to 'eliminator' group on Portuguese Wikipedia'
* 18:02 RobH: srv281 repair working being done
* 16:38 Ryan_Lane: adding exim packages to nova-controller.tesla for mail sending from MediaWiki
== January 24 ==
* 23:53 Ryan_Lane: upgrading nova-* packages on nova-*.tesla
* 23:22 Ryan_Lane: rebooting nova-compute1/2
* 21:20 Ryan_Lane: rebooting nova-controller.tesla
* 18:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Kill another Nogomatch remnant, commented-out experiment from 2005'
* 18:49 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Remove Nogomatch, experimental since 2005 and doesn't seem to log anywhere'
* 18:04 RobH: sq86 still above normal ms4 connection thresholds, leaving it at increased rate for now
* 17:24 mark: Applied ipv4 route-map to ipv6 AS 6908 BGP session on br1-knams, so IPv6 routes via 6908 2828 are preferred, and IPv6 connectivity between pmtpa and esams is working again
* 17:11 RobH: sq86 is back online with lucid, set ms4 maxcon to 800 while it catches up
* 16:59 mark: Rebooting spence
* 16:27 mark: AMS-IX migration complete, all BGP sessions back up again
* 16:17 mark: Shutting down all AMS-IX sessions for SFP+ ER test migration
* 16:01 RobH: sq86 coming down for upgrade to lucid
* 15:27 mark: Shutdown project1 for decommissioning
* 13:33 logmsgbot: demon synchronized php-1.5/wmf-config/CommonSettings.php 'Reverting codereview-proxy removal, did not work'
* 13:31 logmsgbot: demon synchronized php-1.5/wmf-config/CommonSettings.php 'Removing codereview-proxy step from CR'
* 13:05 mark: Purged payments1/payments2 resources from the exported Puppet database on db9
* 12:57 mark: Restarted ntpd on srv236
* 12:55 mark: Added puppet classes ntp::client and ganglia to snapshot1-3
* 12:50 mark: Unstuck APT on knsq29, and fixed puppet
* 12:47 mark: Fixed puppet on Kaulen
* 12:46 mark: Removed APT pin -10 for Wikimedia repository on Kaulen - this has been fixed
* 12:44 mark: Unstuck APT on knsq30, and fixed puppet
* 12:38 mark: Finished upgrade of Kaulen
* 12:28 mark: Unstuck APT on knsq8, and fixed puppet
* 12:12 mark: Starting distribution upgrade on Kaulen
* 12:03 mark: Patched Bugzilla on Kaulen
== January 23 ==
* 15:19 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php '26886 - Move toggle for DoubleWiki to InitialiseSettings.php'
* 15:18 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26886 - Move toggle for DoubleWiki to InitialiseSettings.php'
* 14:41 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix logo of ruewiki'
* 14:34 JeLuF: created Rusyn Wikipedia, ruewiki
* 14:14 logmsgbot: jeluf ran sync-common-all '25924 - Create Wikipedia in Rusyn'
* 13:51 JeLuF: addwiki.php broken. SQL error: "1283: Column 'si_title' cannot be part of FULLTEXT index (10.0.6.49)"
* 11:58 logmsgbot: hashar: I did add eowikisource yesterday. Will write a report to wikitech-l asap
* 11:48 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php '18479 - Set $wgUseDynamicDates = false for the English Wikipedia'
* 01:28 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26323 - Requestin ReaderFeedback be enabled on ru.wikinews'
== January 22 ==
* 21:39 RobH: set sq83, sq84, and sq85 ms4 maxconn back to 100, all were well below and no longer needed increased max.
* 19:42 apergos: removing eowikisource, which was added to this file today but apparently doesn't exist; this breaks SUL-related things
* 19:42 logmsgbot: ariel synchronized all.dblist 
* 18:29 logmsgbot: hashar synchronized php-1.5/wmf-config/wgConf.php 'indentation'
* 16:59 logmsgbot: 26853 - Local upload disabled at Swedish Wikisource
* 16:59 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 14:19 hashar: ran rsync manually on srv287 after receiving this message from scap "rsync error: some files could not be transferred (code 23) at main.c(1385) [generator=2.6.9]"
* 14:10 logmsgbot: hashar ran sync-common-all 'Bug 16878 - Disable Special:CrossNamespaceLinks'
* 12:07 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 11776 - Showing also articles in the CategoryTree of categories in he-wiki'
* 11:49 logmsgbot: hashar synchronized langlist 'get ride of closed-zh-tw'
* 11:38 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26142 - really enable abusefilter'
* 11:20 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26584 - Add Portal namespace to Tarandine Wikipedia'
* 01:03 Ryan_Lane: powercycling storage3 - it's dead
== January 21 ==
* 23:11 logmsgbot: catrope synchronized php-1.5/extensions/CentralAuth/CentralAuthHooks.php 'Back out live hack, may be causing issues'
* 22:33 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26453 - Enable Collection extension on Romansh (rm) Wikipedia'
* 22:17 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26812 - Create namespace end namespaceAliases for Chechen (ce) Wikipedia'
* 22:11 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26855 - Create new Book namespace for gl.wikipedia'
* 22:02 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '25740 - Enable Abuse Filter in gl.wp'
* 21:57 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26776 - Enable $wgUseRCPatrol for siwiki (si.wikipedia.org)'
* 21:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26777 - Enable AbuseFilter in si.wiki'
* 21:49 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 80711: 
* 21:48 RoanKattouw: Running scap to deploy r80711 (i18n updates)
* 20:03 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26817 - Create "Subject" namespace for Bengali Wikibooks'
* 19:57 RobH: upload squids re-normalizing after sq85 resinstallation. sq83/84 set to 400 maxconns, sq85 set to 800, will change back when normalized
* 19:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26779 - Please create "Wikijunior" namespace for Bengali wikibook'
* 19:46 RobH: sq85 back online, frontend false, backend rebuilding, set to 800 maxconnection during rebuild
* 19:26 RobH: raised maxconn back to 400 on sq83, sq84, as they started to max out too often
* 19:18 RobH: upload services times back to normal, all upload squids online. taking down sq85 for upgrade to lucid.
* 19:06 RobH: reverting max_connections for sq83 & sq84 from 800 back to 100 as they have normalized well below the limit.
* 17:32 RobH: sq83/84 having empty caches on reinstall are maxing out ms4 connections for themselves, raising limits to 800 for them to catch up
* 16:53 RobH: sq85 coming down for lucid reinstallation
* 16:46 RobH: sq83 back in rotation
* 16:01 RobH: sq84 coming down for reinstallation to lucid
* 15:47 RobH: sq83 coming down for reinstallation to lucid
* 15:47 RobH: morebots died, restarted it on wikitech under my user
* 01:22 awjr: re-enabling automatic donation queue consumption in Hudson for queue2civicrm and recurring donation processing modules
== January 20 ==
* 23:42 logmsgbot: catrope synchronized php-1.5/wmf-config/ExtensionMessages.php 'Regenerated ExtensionMessages.php after removing unused extensions from extension-list and alphabetizing'
* 23:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove $wmgUseGeoLite'
* 23:36 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Remove GeoLite, enabled on test and meta but not actually used'
* 23:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove $wmgUseHoneypotIntegration'
* 23:18 logmsgbot: catrope synchronized php-1.5/wmf-config/secret-projects.php 'Remove HoneypotIntegration code'
* 23:18 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Remove HoneypotIntegration code, disabled everywhere'
* 23:16 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Remove commented-out code for ExternalPages, FixedImage and MiniDonation'
* 23:12 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove $wmgUseDrafts'
* 23:12 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Remove Drafts code; disabled on all wikis, breaks LiquidThreads'
* 23:10 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove $wmgUseConfigure'
* 23:10 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Remove Configure code, was disabled two years ago'
* 22:12 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Disable AjaxTest, it's ancient and useless'
* 21:46 logmsgbot: demon synchronized php-1.5/wmf-config/InitialiseSettings.php 'And let bcrats grant membership'
* 21:46 logmsgbot: demon synchronized php-1.5/wmf-config/CommonSettings.php 'Adding new svnadmins group for repository administration.'
* 21:42 awjr: deploying and enabling 'recurring' module on CiviCRM
* 21:41 awjr: temporarily setting the 'Days before considering transaction too old to automatically thank' setting to 50 (from 14) for thank_you module in CiviCRM in order to automatically send thank-you notes for old recurring payment messages that have not yet been consumed
* 21:40 awjr: temporarily suspending automatic donation queue consumption (done by queue2civicrm module) in Hudson to live-test the new recurring payment handling module
* 20:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/mc.php 'replace failed server'
* 19:14 Ryan_Lane: Patch python-nova on nova-controller.tesla for lp705195, lp705205, and lp705223
* 19:01 RobH: sq81 takes change, still running everything correctly, and updates disk permissions correctly now
* 19:00 RobH: committed puppet updates for udev rules on squids to templates, testing on sq81
* 18:38 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26314 - Creating new namespace in ca.wikinews'
* 17:02 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26252 - Enable autopatrolled and patroller groups on it.wikibooks'
* 16:16 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26615 - Set $wgBlockAllowsUTEdit to true for mediawikiwiki'
* 16:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26598 - Change wgNoticeProject on specieswiki'
* 16:04 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26545 - Enable NewUserMessage extension on Meta-Wiki'
* 16:02 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26532 - Enable Collection extension on Estonian (et) Wikipedia.'
* 15:29 JeLuF: Added vhost stats.wikipedia.org on spence. Provides a redirect to stats.wikimedia.org
* 15:03 JeLuF: added DNS alias stats.wikipedia.org -> stats.wikimedia.org
* 11:30 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '16410 - Enable e-mail notification for watchlist and minor edits on portuguese Wikibooks'
* 10:47 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26193 - Enable e-mail notification for watchlists on pt.wikiversity'
* 10:45 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26771 - Enable e-mail notification for movement roles wiki'
* 10:36 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26769 - Enable subpages on movement roles wiki'
* 10:28 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '26783 - $wgEnotifWatchlist for WMPL wiki'
* 10:18 logmsgbot: jeluf synchronized docroot/commons/favicon.ico '#23917 - Commons favicon broken on Safari & Chrome (Mac) // Also distorted on other browsers'
* 04:05 Tim: set up IMAP account for Mark Hershberger
* 03:05 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable Drafts, causes issues in combination with LQT'
* 01:56 RoanKattouw: Restarted morebots
* 01:52 awjrichards: renamed fundcore_log table to queue2civicrm_log on drupal and dev_drupal databases on db9 in preparation for re-implementing fundcore module's logging feature in its replacement module, queue2civicrm
* 01:32 kaldari: deployed update to CentralAuth with new getCookieDomain method
* 01:22 logmsgbot: tfinc synchronized php-1.5/extensions/CentralAuth/CentralAuthUser.php 
* 00:37 robla: added commit access for appy (Apekshit Sharma)
* 00:19 Tim: removed pilot.wikimedia.org, was an old insecure MediaWiki instance
== January 19 ==
* 23:26 Ryan_Lane: added agarrett to engineering alias
* 21:45 Ryan_Lane: upgrading nova-* packages on nova-*.tesla
* 21:41 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 26787 - English Wikisource: Have Template: namespace converted to be subpages'
* 20:47 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 26126 - New "Collection" namespace for en.wikiversity.org'
* 20:45 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'reorder wgExtraNamespace + folding'
* 20:25 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 26565 - Please enable upload on Persian Wikinews'
* 20:15 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 26572 - Enable subpages in main namespace on svwikiversity'
* 20:09 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'put db34 back'
* 19:44 hashar: frwiktionary WT was a namespace alias for NS_CATEGORY from 19:01 to 19:03 roughly. Now NS_PROJECT. Some page might show unwanted categories, issue can be solved by purging the page
* 19:03 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 26761 - was NS_PROJECT not 14!'
* 19:01 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 26761 - New namespace alias WT for project namespace on frwiktionary'
* 18:19 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'sync comments enabling vim folding (no code change)'
* 18:03 hcatlin: Mobile deploy complete and stable
* 18:02 RoanKattouw: Ganglia's fine
* 18:00 hcatlin: Ganglia seems to be down for me. "fsockopen error: Connection refused"
* 17:57 hcatlin: deploying mobile updates
* 17:44 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'remove db34 to dump s3'
* 17:38 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'put db16 back'
* 14:29 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'remove db16 to dump s7'
* 01:24 Ryan_Lane: fixing torrus
== January 18 ==
* 23:58 Ryan_Lane: upping CARP weight of sq79 to 30
* 23:50 Ryan_Lane: those last two sq49 logs were really sq79
* 23:50 Ryan_Lane: upping CARP weight for sq80-82 to 30, upping sq49 to 20
* 23:45 Ryan_Lane: upping CARP weight for sq80-82 to 20, upping sq49 to 15
* 23:28 Ryan_Lane: upping CARP weight for sq80-82 to 15
* 23:27 richcole: db 15 reboot
* 23:10 RobH: db25 & db27 shutdown
* 23:07 Ryan_Lane: upping CARP weight on sq79-82 to 10
* 22:40 mark: Increased max-conns to ms4 on sq80 from 100 to 400
* 21:55 Hashar: that one was adding code folding
* 21:55 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 21:24 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 20:36 RobH: sq79-sq82 reinstalled lucid, returned to service
* 19:52 RobH: sq79-sq82 down for reinstallation to lucid
* 19:52 RobH: updated puppet for udev rules on squid aufs exec.
* 19:52 RobH: had to shoot morebots and restart
* 11:23 mark: Changed subnet size, reenabled AMS-IX BGP sessions
* 11:21 mark: Shutdown all AMS-IX BGP sessions
* 11:19 mark: Starting AMS-IX interface subnet size change
* 06:29 apergos: restarted morebots, it was having issues
== January 17 ==
* 21:20 hashar: morebots is dead. Logged bug 26782
* 02:47 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 02:32 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 01:35 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'removing db16 to be my pillow'
== January 16 ==
* 20:31 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 19:59 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 19:57 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php '26633 - Give commonswiki group 'Image-reviewer' right to Add groups: Image-reviewer'
* 18:49 logmsgbot: hashar synchronized php-1.5/wmf-config/codereview.php 'codereview: fix repo_name case'
* 16:30 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'frwikibooks, allow bureaucrats to add/remove users from abusefilter'
* 16:30 logmsgbot: hashar synchronized php-1.5/wmf-config/abusefilter.php 
* 16:16 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 25019 - trwikinews, create Yorum namespace'
* 15:34 logmsgbot: hashar synchronized php-1.5/wmf-config/CommonSettings.php 'Bug 23494 - Setup $wgCodeReviewDeferredPaths for Mediawiki.org'
* 15:32 logmsgbot: hashar synchronized php-1.5/wmf-config/codereview.php 'add file for mediawikiwiki $wgCodeReviewDeferredPaths'
* 12:54 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 11:40 logmsgbot: hashar synchronized closed.dblist 'bug 26569 - Close usability.wikimedia.org to editing'
* 11:11 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php '26643 - Allow sysops to add/remove a bunch of groups on siwiki'
* 10:15 Tim-away: removed /tmp/gs_* on srv219-224, root partition was full
== January 15 ==
* 01:53 apergos: starting en wikipedia xml dump, one phase at a time. from screen session on snapshot3, this will be a parallelized run
== January 14 ==
* 23:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set $wgLogo to $stdlogo on cswiki'
* 23:15 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set enwiki logo back to $stdlogo so local sysops can set up a special logo for WP day'
* 20:05 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26634 - allow bureaucrats to add/remove for group abusefilter'
* 15:03 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php '26634 - Activation of AbuseFilter on Romanian language Wikipedia'
* 15:02 logmsgbot: hashar synchronized php-1.5/wmf-config/abusefilter.php '26634 - Activation of AbuseFilter on Romanian language Wikipedia'
* 14:55 RobH: rebooting dataset1, it was left in webbios and thus not controllable by serial
* 14:24 logmsgbot: hashar synchronized php-1.5/wmf-config/InitialiseSettings.php '26361 - wgLocaltimezone for koquote'
* 06:53 Tim: on all squids, turning off debug output on 44,1, which was filling up cache.log with "Failed to select source" warnings when the backend was down
* 06:42 Tim: backend squid on sq41-48 all crashed at once, because their root partitions filled up some time ago, and then something HUP'd them all at once. Deleted cache.log and restarted squid.
* 03:45 Tim: commented out the $wgCentralAuthCookieDomain = '' in CentralNotice.php
* 03:45 logmsgbot: tstarling synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 
* 03:33 mark_: db37 binlog: master_log_file='db37-bin.000028', master_log_pos=927356414;
* 03:29 mark_: db37 start pos was: master_log_pos=927356414;
* 03:29 mark_: domas: change master to master_host='db39', master_user='repl', master_log_file='db39-bin.000050', master_log_pos=720582706;
* 03:26 mark_: domas switched s3 master to db39 (position?), and s7 master to db37 (pos?)
* 03:24 mark_: domas depooled db25 from s3 and s7
* 03:18 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 03:18 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 's3 is now two-server 5.1 cluster'
* 03:08 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 02:08 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'read-only on s3+s7'
* 01:55 Ryan_Lane: stopping mysql on db17
== January 13 ==
* 22:35 RobH: sq50 done with testing, bringing it back into service
* 22:26 richcole: shutting down dataset1 for recabling
* 20:31 RobH: pulling sq50 out of rotations, killing puppet and squid processes, and using as my testing for new udev rules on squids
* 20:05 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/backend/RepoStats.php 'r80187'
* 20:05 logmsgbot: catrope synchronized php-1.5/extensions/CodeReview/CodeReview.php 'r80187'
* 16:41 logmsgbot: catrope synchronized php-1.5/extensions/CentralAuth/CentralAuthHooks.php 'Live hack that tries to fix login issues'
* 12:57 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26697 - Set $wgForceUIMsgAsContentMsg for tenwiki'
* 05:36 Ryan_Lane: repooling sq49 and sq50
* 04:25 Ryan_Lane: depooling sq50 as well
* 04:09 Ryan_Lane: starting squid on sq49
* 04:08 Ryan_Lane: stopping squid on sq49
* 04:06 Ryan_Lane: stopping rsyslog, cleaning logs, and reenabling syslog in sq49
* 04:02 Ryan_Lane: depooling sq49
* 01:47 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialHideBanners.php 
* 01:31 RobH: sq51-sq54 back online
* 01:29 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding AllowdHideBanenrs for test wiki'
* 01:24 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding AllowdHideBanenrs for foundation wiki'
* 01:23 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding AllowdHideBanenrs for foundation wiki'
* 01:20 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialHideBanners.php 
* 01:20 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 
* 01:19 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'swapping out down servers'
* 01:13 Ryan_Lane: fixing torrus
* 01:10 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 
* 01:07 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.i18n.php 
* 01:06 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/1x1.png 
* 01:06 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialHideBanners.php 
* 00:47 RobH: sq55-sq58 reinstalled lucid, back online, sq51-54 coming back down
* 00:09 RobH: sq55-sq58 coming down for reinstallation to lucid
* 00:05 RobH: sq51-sq54 reinstalled lucid, pushing back online
== January 12 ==
* 23:32 RobH: sq51-sq54 coming down for reinstallation to lucid
* 23:31 RobH: sq49, sq50 reinstalled lucid, online
* 23:27 RobH: sq46-sq48 reinstalled lucid and back online
* 22:53 RobH: sq46-sq50 coming down for reinstallation to lucid
* 22:51 RobH: sq42-sq45 reinstalled lucid, online
* 22:13 RobH: sq42-sq45 down for reinstallation to lucid
* 20:43 RobH: sq41 coming down for reinstallation to lucid
* 18:42 apergos: that file needs to be left there for awhile apparently (say the google instructions) so please don't just toss it yet
* 18:40 apergos: this is about domain verification for google storage.
* 18:39 logmsgbot: ariel synchronized docroot/www.wikimedia.org/google126853a33948578b.html 
* 17:48 RobH: all text squids in pmtpa/sdtpa have been upgraded to lucid
* 17:48 RobH: sq75-sq78 back in service, lucid
* 17:45 logmsgbot: mark synchronized php-1.5/wmf-config/db.php 'Remove db15 from rotation'
* 17:22 RobH: sq75-sq78 coming down for reinstall to lucid
* 17:19 RobH: sq71-sq74 reinstalled to lucid and pushed back into service
* 16:39 mark_: Increased CARP weight of amssq31 to 30
* 16:33 mark_: Powercycled amssq38
* 16:30 RobH: sq71-sq74 coming down for reinstallation to lucid
* 16:27 RobH: sq65-sq66 reinstalled to lucid and online
* 16:20 mark_: Lowered CARP weight of amssq* non-SSD text squids from 10 to 8 to relieve disk and memory pressure
* 16:20 apergos: running cleanup on index.html and md5sums for the XML dump jobs that recorded bogus "recombine" jobs that weren't actually run
* 16:18 mark_: Stopped old puppet instance and restarted squid-frontend on amssq37
* 16:16 RobH: amsq31 upgraded to lucid and back in service
* 16:05 RobH: both srv182 and srv183 were not responsive to serial console, rebooted both.
* 15:42 RobH: sq65 & sq66 coming down for reinstallation to lucid
* 15:40 RobH: sq62-sq64 installed and online lucid
* 15:31 RobH: reinstalling amssq31
* 15:11 RobH: sq62-sq64 down for reinstall
* 15:06 RobH: sq59-sq61 online as lucid hosts
* 14:59 RobH: updated wmf repo, copying package for squid from karmic to lucid as well (seems to have been lost from other changes, as frontend was still there)
* 14:02 RobH: sq59-sq61 offline for reinstall
* 12:03 mark_: Enabled multicast snooping on csw1-sdtpa
== January 11 ==
* 23:39 Ryan_Lane: adding cron on nova-controller.tesla to svn up /wiki hourly
* 22:27 RobH: have to reinstall the squids, wrong version written into lease
* 22:24 awjr: archived hudson build files for 'donations queue consume' and emptied builds directory on grosley to allow the jobs to continue running (donations queue consume job was failing due to hitting max # of files in a dir for ext3 fs)
* 22:05 RobH: correction, sq65, sq66, & sq71
* 22:04 RobH: sq62-sq64 back online, sq65-sq67 coming down for reinstall
* 21:48 Ryan_Lane: adding exim::simple-mail-sender to owa1-3
* 21:33 RobH: sq59-sq61 in service, sq62-sq64 reinstalling
* 21:27 Ryan_Lane: powercycling amssq31
* 20:52 RobH: sq59-61 reinstalled and online, pooled, partially up
* 20:42 rainman-sr: enwiki.spell index somehow got corrupt, investigating and rebuilding it now on searchidx1
* 18:57 RobH: sq59-sq61 coming back offline, bad partitioning in automatied install, need to update squid configuration for these hosts
* 18:47 RobH: sq59 having reinstall issues, skipping it and moving on
* 18:36 RobH: sq61 reinstalled and back online
* 18:30 RobH: sq60 reinstalled, back in service
* 18:03 RobH: sq59-sq61 depooled for upgrade
* 16:15 RobH: sync-docroot run to push updated tenwiki favicon
* 14:27 mark_: Depooled amssq31 and amssq32 for SSD install
== January 10 ==
* 21:01 Ryan_Lane: patching python-nova on nova-compute*.tesla (see bug #lp700015)
* 20:59 Ryan_Lane: err make that #lp681164 for ldap driver
* 20:59 Ryan_Lane: patching ldap driver on nova-* (see bug #lp681030)
* 20:57 Ryan_Lane: patching ec2 api on nova-controller.tesla (see bug #lp701216)
* 19:56 RobH: updated dns with virt!-virt4 mgmt info
* 16:10 RobH: singer was crashed, investigating why it suddenly had issues. It was pulling down db9, but it halted before it could damage anything.
* 16:04 RobH: secure server, as well as blogs, offline, investigating server issue on singer
* 16:02 RobH: db9 having issues and singer is as well, taking singer down since its already crashed
== January 9 ==
* 22:04 Ryan_Lane: powercycling srv217
* 22:03 Ryan_Lane: powercycling srv271
* 22:01 Ryan_Lane: powercycling srv262
* 19:21 apergos: rebooting amssq61 hoping to clear up its problem. I guess puppet restarts squid instances every so often *sigh*
* 18:50 apergos: stopping squid front and back end instances on amssq61, has network issue
* 14:15 mark: Reduced Varnish max worker threads from 8000 to 2000 per threadpool
* 13:52 mark: Pooled knsq6 and knsq7 as bits.esams
* 13:43 mark: Converted knsq6 and knsq7 into bits.esams machines
* 13:16 mark: Stopping squid instances on knsq6 and knsq7
* 13:16 mark: Depooled knsq6 and knsq7 from the knams.text pool
== January 8 ==
* 16:14 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 16:14 logmsgbot: catrope synchronized php-1.5/skins/common/edit.js 'r79866'
== January 7 ==
* 23:42 davidstrauss: Cloning civicrm -> dev_civicrm on Fundraising infrastructure.
* 23:33 Ryan_Lane: installed dnsmasq on nova-controller.tesla
* 22:47 Ryan_Lane: rebooting nova-*
* 22:45 Ryan_Lane: reconfiguring network on nova-* to remove bridges (FlatDHCPManager handles that)
* 22:10 Ryan_Lane: rebooting nova-controller.tesla
* 22:01 Ryan_Lane: changing to FlatDHCPManager for networking driver on nova-controller.tesla
* 22:00 Ryan_Lane: installing bridge-utils on nova-controller.tesla
* 21:52 RobH: srv217 offline, but shows no errors in drac log, rebooted
* 19:03 Ryan_Lane: disabling apparmor on nova-compute1/2 to ensure it isn't causing instance creation to fail
* 16:40 Ryan_Lane|sleep: adding CNAME svn.mediawiki.org -> svn.wikimedia.org per bug 26474
* 14:44 RoanKattouw: Installed xdebug on prototype so I can turn it on when debugging fatals
* 00:51 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 
* 00:31 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.php 
* 00:30 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/centralnotice.js 
* 00:30 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php 
* 00:30 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialBannerController.php 
* 00:29 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/CentralNotice.i18n.php 
== January 6 ==
* 22:21 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Set $wgCodeReviewRepoStatsCacheTime to 1 hour, requested by Sam'
* 22:18 Ryan_Lane: added awjrichards to storage3
* 16:29 Ryan_Lane: upgrade nova-* packages on nova-*.tesla, as the current branch packages are broken
* 12:37 mark: Powercycled amssq33
* 04:04 Ryan_Lane: powercycling mobile1, that is
* 04:04 Ryan_Lane: restarting mobile1
* 03:36 Ryan_Lane: updating nova* packages on nova-controller.tesla, this version of trunk packages is simply broken
== January 5 ==
* 22:22 Ryan_Lane: repooling payments1 and payments2
* 22:05 Ryan_Lane: depooling payments1 and payments2
* 21:35 Ryan_Lane: adding LVS realserver package and configuring it via puppet for payments1-2
* 20:33 Ryan_Lane: powercycling sq80
* 20:32 Ryan_Lane: powercycling knsq14
* 20:30 Ryan_Lane: powercycling amssq56
* 20:29 Ryan_Lane: powercycling amssq51
* 20:29 Ryan_Lane: powercycling amssq47
* 20:28 Ryan_Lane: powercycling amssq46
* 20:27 Ryan_Lane: powercycling amssq41
* 20:18 Ryan_Lane: fixing torrus
* 18:37 Ryan_Lane: enabling reporting for store and log on puppetmaster, and enabling reporting on spence for puppet
* 17:22 logmsgbot: demon synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26580 - try different interwiki prefix'
* 17:18 logmsgbot: demon synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26580'
* 16:18 Ryan_Lane: powercycling srv182
* 02:03 Ryan_Lane: uncommenting payment1 and payments1 in pybal config
* 02:02 Ryan_Lane: commenting out payments1 and payments2 to try to force a pybal check
* 01:23 Ryan_Lane: pooled payments1 and payments2
== January 4 ==
* 23:23 Ryan_Lane: moving payments2 from internal to squid vlan
* 23:23 Ryan_Lane: restarting network services on payments2
* 23:18 Ryan_Lane: restarting nginx on payments1
* 23:12 Ryan_Lane: switching IP address info for payments2
* 23:11 Ryan_Lane: moved payments1 from the internal to the squid vlan
* 22:12 Ryan_Lane: restarting network services on payments1
* 21:52 Ryan_Lane: changing IP address for payments1
* 20:07 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'changed tenwiki anon edit block to anon page creation block'
* 16:53 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Remove hack disallowing anon edits on tenwiki - moved to its proper place in InitialiseSettings.php'
* 16:52 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disallow anon edits on tenwiki'
* 16:50 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 26554 - Set $wgAutoConfirmAge to 0 on tenwiki'
* 16:43 Ryan_Lane: removing login.tesla from DNS
* 15:42 apergos: snapshot1 removed from mediawiki-installation for now til set up again
* 15:35 Ryan_Lane: dropping nova database on nova-controller.tesla, letting nova recreate it
* 15:29 Ryan_Lane: changing ldap schema on nova-controller.tesla to newer (v2) openstack schema
* 14:24 logmsgbot: robh synchronized php-1.5/wmf-config/CommonSettings.php 
* 14:12 Ryan_Lane: powercycling mobile3
* 12:17 RoanKattouw: srv182, srv183 seem to have gone into swap death
* 12:11 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'r79572'
* 09:54 apergos: restarted lsearcdh on search7 (in theory the one host in the search-pool3 ) to see if that helps
* 06:28 logmsgbot: root synchronizing Wikimedia installation... Revision: 79563: 
* 06:27 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 79563: 
* 06:26 Tim: svn up/scap to r79563 for clickjacking defences
* 02:47 Ryan_Lane: disabling the puppetmaster service, and enabling the opendj service on nova-controller.tesla
* 02:45 Ryan_Lane: clearing mediawiki database for nova-controller.tesla, and clearing out most of the LDAP data
* 02:31 Ryan_Lane: rebooting nova-*.tesla
* 02:21 Ryan_Lane: updating nova-* on nova-controller.tesla
== January 3 ==
* 23:19 Ryan_Lane: deleting login.tesla vm
* 23:05 Ryan_Lane: adding reedy account, putting samreed account back in, and ensuring the ssh key is absent for samreed
* 23:00 Ryan_Lane: adding rfaulk to storage3
* 22:58 Ryan_Lane: renaming samreed to reedy in admins.pp
* 22:51 Ryan_Lane: adding demon to kaulen
* 22:51 Ryan_Lane: giving robla access to modify-ldap-user, add-ldap-user, and delete-ldap-user on formey
* 18:19 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'db15 going live'
* 17:44 apergos: reboot snapshot1 for install of lucid
* 16:18 logmsgbot: robh synchronized php-1.5/wmf-config/mc.php 'replacing srv266 since its down'
* 16:16 RobH: rebooting srv266
* 14:34 mark: Site should be stable again
* 14:08 mark: Depooled text squids again
* 13:59 mark: Adding text.esams squids into the bits.esams LVS pool
* 13:43 mark: Killed pybal on amslvs1
* 13:34 mark: Depooled knsq1 for installing debug symbols for oprofiling
* 10:55 logmsgbot: midom synchronized php-1.5/wmf-config/mc.php 
== January 2 ==
* 23:12 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'db16 going live for s7 duty'