Server Admin Log/Archive 15

== June 30 ==
* 21:44 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 68749
* 21:44 Andrew: deploying new version of LiquidThreads to labs
* 17:57 Andrew: running thread_pending_relationship schema update for LiquidThreads on liquidthreads_labswikimedia
* 17:50 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'New logos for fawiki, arwiki, arzwiki'
* 17:44 tomaszf: adding generic wikimedia svn depot for display wall project
* 17:30 RoanKattouw: Rolled back switchover on ukwiki on request
* 17:29 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Whitespace change for sync-dblist'
* 17:19 RoanKattouw: Messed up by not doing stuff in the right order; switchover complete now
* 17:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Whitespace change'
* 17:12 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Whitespace change'
* 17:11 RoanKattouw: Syncing dblist to make switchover work
* 17:09 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Switch all wikis in switchover-jun30.dblist to Vector, UsabilityInitiative, new thumb size and new logo'
* 17:09 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Switch all wikis in switchover-jun30.dblist to Vector, UsabilityInitiative, new thumb size and new logo'
* 15:57 domas: domas' oprofile killed db40's mysqld :)
* 15:56 Rob: removing out of band connection from scs-ext and moving to mr1
* 04:54 logmsgbot: jeluf synchronizing Wikimedia installation... Revision: 68749
* 02:44 river: stopping ES replication on ms2 to dump for ES backup
== June 29 ==
* 22:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 22:20 logmsgbot: catrope synchronized php-1.5/skins/vector/main-rtl.css 'r68749'
* 22:13 logmsgbot: catrope synchronized php-1.5/skins/vector/main-ltr.css 'r68749'
* 21:45 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump UI CSS style version'
* 21:43 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 21:41 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 68745
* 21:41 RoanKattouw: Running scap to deploy UsabilityInitiative and Vector fixes, for real this time
* 21:27 RoanKattouw: Strike that, putting the code on test first
* 21:26 RoanKattouw: Running scap to deploy UsabilityInitiative and Vector fixes
* 20:46 Fred: removed A record for volunteer.w.o since it was pointed out that it linked to a winamp playlist.
* 19:14 Fred: removed Naoko from lists to prevent bounces :(
* 18:46 Fred: removed cron entry from isidore for bz-reporter -> installed mail on Kaulen which will now handle the weekly reports.
* 17:54 Rob: connected mr1-pmtpa port 0/1 to csw5 port 7/1
* 17:48 Rob: and msw1-pmtpa:prt24 to msw-b2 (port unimportant)
* 17:47 Rob: new mgmt connections: mgmt-gateway: serial to scs-ca-pmtpa:port1; mgmt-gateway:0/0 to msw1-pmta:port23, msw1-pmtpa:mgmt to scs-c1-pmtpa port2
* 16:23 Rob: dataset1 bad disk swapped and array shows rebuilding
* 15:37 Rob: srv89 resurrected and playing nicely with the other server kids
* 15:15 Rob: srv146 updated (puppet,sync-docroot,sync-apache), working, and back in lvs cluster
* 15:09 Rob: fixing srv146, removed from lvs until fixed
* 08:19 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 
* 08:14 logmsgbot: midom synchronized php-1.5/includes/db/DatabaseMysql.php '68713 live'
* 08:13 logmsgbot: midom synchronized php-1.5/includes/DefaultSettings.php '68713 live'
* 08:08 logmsgbot: midom synchronized php-1.5/includes/GlobalFunctions.php 'removing msg profiling'
* 06:00 apergos: restarted morebots on wikitech
== June 28 ==
* 17:43 Rob: ignore sq33 flapping, its disabled in lvs and I am working on it.
* 16:43 Fred: restarted apache on mobile1. Had locked up for some reason (probably too many threads).
* 15:57 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/forms/RevisionReviewForm.php 'Deployed r68663'
* 15:54 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.class.php 'Deployed r68663'
== June 27 ==
* 08:12 domas: locke locked up for no apparent reason :( 
== June 25 ==
* 18:13 Rob: srv104 disk failure, shutting it down (via ticket to the noc)
* 15:01 Rob: srv284 has E1410 System Fatal Error, will place ticket later today
* 14:59 Rob: db28 is borked, needs more replacement memory, reopened ticket, will remain offline for now.
* 14:59 Rob: fixed srv281 issues, needs reinstallation later today
* 14:58 Rob: tested db34, LOM is doa, calling in later today
* 14:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Switch private wikis to Vector per Erik M'
* 13:53 Rob: ignore db28 flapping, working on it.
* 09:43 domas: updated password formats for wikiuser on s1 and s2
* 08:20 logmsgbot: midom synchronized php-1.5/wmf-config/mc.php 'swapping out srv104, putting in srv242'
* 07:38 JeLuF: enabled PHP modules on fenari so that noc.wikimedia.org deliveres syntax highlighted config files.
* 00:10 logmsgbot: andrew synchronized php-1.5/extensions/AbuseFilter/SpecialAbuseLog.php 
== June 23 ==
* 22:18 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 
* 22:11 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Finishing touch'
* 21:58 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Testing'
* 21:49 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Testing'
* 21:42 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Merge r68488'
* 14:23 domas: 'fixed' abusefilterlog performance issue with a livehack, hehehe: http://p.defau.lt/?55XRFbMETCGrQHRI5a5PDQ 
* 14:22 logmsgbot: midom synchronized php-1.5/extensions/AbuseFilter/SpecialAbuseLog.php 
* 13:49 domas: reduced db9 killing threshold from 30s to 10s :)
== June 22 ==
* 22:55 tomaszf: switching cname of download. from storage2 to dataset1
* 14:47 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Reenable CollapsibleNav on officewiki per Guillaume'
* 14:38 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Reenable EditWarning'
* 14:35 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Modules/EditWarning/EditWarning.i18n.php 'r68406'
* 14:32 Tim: on mayflower: disabled sshd TCP forwarding
* 14:40 river: dumping ES to dataset1
* 14:05 river: stopped MySQL on clematis to move it to slauerhoff
* 14:03 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'disabled editwarning'
* 14:03 Tim: upgrading libtiff4 on all apaches due to USN-954-1
* 13:35 logmsgbot: midom synchronized php-1.5/includes/GlobalFunctions.php 'msg profiling hook'
* 13:17 mark: (and domas) Restarted DNS on ns0 and ns2
* 10:21 mark: Depreffed _1299_16265_43821_ route
== June 21 ==
* 18:56 tomaszf: sync'ing public dir from storage2 to dataset1
* 18:29 tomaszf: importing prod civi database onto dev
* 12:36 domas: so easy to earn a beer!
* 12:32 domas: powercycled pdf1, OOM
== June 20 ==
* 16:27 JeLuF: lighttpd issue on lily, it only listens for port 443 on IPv4, not on IPv6, server.use-ipv6 = "enable" is set, it listens to port 80 on both v4 and v6.
== June 19 ==
* 20:42 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24042 - Namespace Aliases for English Wikibooks'
* 19:10 logmsgbot: catrope synchronized php-1.5/includes/StringUtils.php 'Revert live hack'
* 18:57 logmsgbot: catrope synchronized php-1.5/includes/StringUtils.php 'Live hack for debugging OOM in it:Template:Particella subatomica'
* 18:53 JeLuF: fixed broken wikimedia-task-appserver package on fenari. PHP maintenance scripts were no longer running.
* 08:47 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticleView.php 
== June 18 ==
* 23:07 atglenn: add pdhanda to core svn committers 
* 22:22 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 68239
* 22:14 AaronSchulz: deployed r68251
* 22:14 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 68239
* 20:52 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 20:52 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r68239'
* 17:26 logmsgbot: catrope synchronized php-1.5/includes/DefaultSettings.php 'r68222'
* 17:25 logmsgbot: catrope synchronized php-1.5/skins/vector/main-rtl.css 'r68222'
* 17:25 logmsgbot: catrope synchronized php-1.5/skins/vector/main-ltr.css 'r68222'
* 16:40 Rob: updated dns with analytics and donation ip assignments on the tesla dev box
* 04:44 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version'
* 03:18 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Merge r68206'
* 03:13 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Merge r68204'
* 03:05 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Fix column ambiguity not found on my local install'
* 03:03 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Merge r68202'
* 02:48 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Merge r68200'
* 02:34 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Merge r68198'
* 02:21 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Deploy r68196'
* 02:20 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy.php 'Deploy r68196'
== June 17 ==
* 21:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 21:22 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r68180'
* 21:09 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 21:09 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r68180'
* 20:33 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '24010 - id.wikipedia requesting FlaggedRevs'
* 20:27 logmsgbot: jeluf synchronized flaggedrevs.dblist '24010 - id.wikipedia requesting FlaggedRevs'
* 20:06 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling mobile redirect for eo,ko,krc,xal,ca,mr,ja wiki'
* 19:34 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling mobile redirect for th,vi,zh,id,hsb,dsb,bn,huw wiki'
* 19:24 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling mobile redirect for sv,da,et,el,ml,ar wiki'
* 19:15 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling mobile redirect for cs,eo,he,nn,it wiki'
* 19:12 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling mobile redirect for pt,nl,fi wiki'
* 16:33 domas: did a horrible hack to please philippe in LQT.
* 16:32 logmsgbot: midom synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php 
== June 16 ==
* 21:25 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'adding mobile redirect for es,ru,fr"'
* 20:47 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'adding mobile redirect for de'
* 16:20 logmsgbot: jeluf synchronized docroot/wikibooks.org/favicon.ico '17049 - New favicon for Wikibooks projects'
* 15:50 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24001: Make Vector, usability features default on ukwikimedia'
* 15:23 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 24001: Make Vector, usability features default on officewiki'
* 14:43 logmsgbot: midom synchronized php-1.5/wmf-config/flaggedrevs.php 
* 06:17 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '21451 - Change favicon on outreach.wikimedia.org'
* 05:30 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23988 - Switch the right that crats are able to give and take away to Transwiki Importer instead of Importer'
* 04:48 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23918 - Enable 'researcher' permission on testwiki'
* 01:31 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/maintenance/wikimedia-periodic-update.sh 
* 01:29 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'captcha whitelist, bug 23982'
* 01:27 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'captcha whitelist, bug 23982'
* 01:26 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 01:02 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticleView.php 'deployed r68112'
== June 15 ==
* 23:47 Tim: on enwiki master, was seeing 14 threads queued for up to 50 seconds waiting to do Title::invalidateCache() in response to FlaggedArticleView::addToDiffView(). Disabled that caller, fixed now
* 23:43 logmsgbot: tstarling synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticleView.php 'disabling cache update'
* 23:26 AaronSchulz: enabled Pending Changes on enwiki
* 23:23 logmsgbot: aaron ran sync-common-all 
* 23:16 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Get enwiki settings here first'
* 21:24 AaronSchulz: deployed r68107
* 21:22 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
* 18:40 logmsgbot: tfinc synchronized php-1.5/extensions/WikimediaMobile/WikimediaMobile.php 
* 17:51 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 23978: Enable subpages in Outreach namespace on officewiki'
* 14:57 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Fixed $wgFlaggedRestrictions on hewikisource. Changed editor -> review; there is no editor right. Probably broken for ages.'
* 14:41 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Made autoreview right work without review right present.'
* 14:25 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed clearly redundant $wgFlagRestrictions values'
* 12:45 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedArticleView.php 'deployed r68068'
* 12:44 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'deployed r68068'
* 12:44 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevision.php 'deployed r68068'
* 11:51 domas: expect higher purging rate for now, as I had to drop corrupted dampen db :)
* 11:49 domas: restarted dampen via separate udp2log process (the communication between main udp2log and dampen is now async) 
* 11:26 logmsgbot: midom synchronized php-1.5/wmf-config/flaggedrevs.php 'removing talk namespaces from hebrew wikisource flaggedrevs configuration, doh'
* 11:15 domas: disabled dampen.py/vector purge on locke, as it was blocking on i/o and stopping whole logging pipeline
* 06:09 JeLuF: fixed "16994 - Normalize *.org portal page URLs to www.*.org"
* 05:03 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '23948 - Remove autoreview bit from editor usergroup on enwikinews'
* 03:58 AaronSchulz: deployed r68058
* 03:58 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
* 03:10 logmsgbot: tstarling synchronized live-1.5/404.php 'more errors'
* 03:09 logmsgbot: tstarling synchronized 404.html 'fixed spelling error'
* 03:03 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.php 'deployed r68056'
* 01:08 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
* 00:55 logmsgbot: aaron synchronized php-1.5/wmf-config/CommonSettings.php 'removed labs msg load hack'
* 00:54 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'removed wmgUseFlaggedRevsAlpha'
* 00:53 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed labs hacks, uses same FR branch as other sites now'
== June 14 ==
* 23:50 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 
* 23:39 logmsgbot: aaron ran sync-common-all 
* 23:38 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'switch to _alpha again'
* 23:28 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
* 23:28 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'reverted previous switch'
* 23:25 atglenn: ran /home/wikipedia/bin/l10nupdate for flaggedrevs alpha rolllout
* 23:06 tomaszf: killing running xml snapshots on snapshot2
* 23:04 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
* 23:04 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'switched all wikis to _alpha'
* 21:36 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23961 - Change Project namescape in Abkhazian wikipedia'
* 21:21 JeLuF: "apt-get remove mysql-server-5.0" on srv169
* 21:13 JeLuF: fixed dpkg --configure problems on ~100 servers caused by /var/log/gmetricd/gmetricd.log being owned by root
* 21:10 tomaszf: rsync enwiki 20100514 from dataset1 to storage2
* 21:08 tomaszf: installing lighttpd on dataset1
* 21:00 JeLuF: "apt-get update" was hanging on ca 10 servers, killed them and run puppetd --test afterwards
* 20:58 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedArticleView.php 'deployed r68032'
* 20:58 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevs.hooks.php 'deployed r68032'
* 20:57 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevs.class.php 'deployed r68032'
* 20:49 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 20:49 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'r68027'
* 20:47 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/js2stopgap/jui.combined.min.js 'r68027'
* 20:47 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r68027'
* 20:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r68027'
* 20:45 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/images/wikiEditor/toolbar/format-bold-ru.png 'r68027'
* 20:43 logmsgbot: catrope synchronized php-1.5/skins/vector/main-rtl.css 'r68027'
* 20:43 logmsgbot: catrope synchronized php-1.5/skins/vector/main-ltr.css 'r68027'
* 20:43 logmsgbot: catrope synchronized php-1.5/skins/common/wikibits.js 'r68027'
* 20:43 logmsgbot: catrope synchronized php-1.5/skins/common/shared.css 'r68027'
* 20:42 logmsgbot: catrope synchronized php-1.5/skins/monobook/main.css 'r68027'
* 20:41 logmsgbot: catrope synchronized php-1.5/skins/Vector.php 'r68027'
* 20:09 JeLuF: srv217: dpkg reported broken gmond package, refused further updates. Fixed. Run puppetd --test.
* 17:35 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22616 - Create aliases for Romanian namespaces'
* 16:47 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '23948 - Remove autoreview bit from editor usergroup on enwikinews'
* 16:44 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23953 - Add rollback permission to the patrollers group in the French Wikisource'
* 15:28 AaronSchulz: deployed r67999 (Flaggedrevs_alpha)
* 15:27 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
* 07:31 domas: established 90-day purge policy on locke's archives via /etc/cron.daily/purge-archive
* 04:04 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevs.hooks.php 'deployed r67978'
* 02:49 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'added NS_PROJECT to review namespaces on flaggedrevs_labswikimedia'
* 02:35 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/forms/RevisionReviewForm.php 'deployed r67978'
* 02:34 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevs.class.php 'deployed r67978'
* 02:04 logmsgbot: tstarling synchronized php-1.5/includes/Html.php 'r67975, disable HTML 5 form validation'
* 01:48 AaronSchulz: Deployed 67968,67973 (flaggedrevs_alpha)
* 01:47 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
== June 13 ==
* 23:30 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove autopatrol permission from reviewer group on enwiki'
* 15:39 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23939 - Enable Collection extension on ka.wikipedia.org'
* 13:39 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Merge r67943'
* 13:32 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 'Merge r67941'
* 13:32 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy.php 'Merge r67941'
* 12:15 RoanKattouw: Had JeLuF change l10nupdate to run as catrope rather than brion, should shut up the SSL error and make it easier for me to fix in the future
* 08:03 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'add reviewer group back to flaggedrevs_labswikimedia'
* 02:01 logmsgbot: LocalisationUpdate failed
== June 12 ==
* 21:33 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23935 - Autopatroller group for Swedish Wikisource (svwikisource)'
* 21:28 JeLuF: #23928 - change nyc.wikimedia.org redirect target in redirects.conf
* 20:19 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23918 - Enable 'researcher' permission on testwiki'
* 20:00 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22663 - Add some interwiki links in Special'
* 19:36 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23644 - Create new user group to disable new article creation'
* 18:47 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23644 - Create new user group to disable new article creation'
* 18:26 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23644 - Create new user group to disable new article creation'
* 18:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23644 - Create new user group to disable new article creation'
* 17:53 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23644 - Create new user group to disable new article creation'
* 14:13 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 23922: Add reviewer group on enwiki'
* 04:38 JeLuF: (bug 23912)
* 04:37 JeLuF: added "linux-libertine" font package to the requirements for image scalers in puppet
* 02:02 logmsgbot: LocalisationUpdate failed
== June 11 ==
* 17:54 Rob: working on db28, ignore its alerts, it does not serve any production traffic presently
* 17:08 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23833 - Request for creation/activation of portal namespace'
* 17:06 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23874 - Change usability wiki sitename to "Wikimedia Usability Initiative"'
* 17:05 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23883 - Add rights to Reviewer in Testwiki'
* 16:53 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23892 - double entries in config file for arbcom_enwiki'
* 13:11 Rob: got email from johannes, pdf2 needed reboot done
* 06:24 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'en wiktionary flood group tweaks again'
* 05:57 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'en wiktionary flood group tweaks'
* 05:29 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'add flood group to en wiktionary, bug 23832'
* 05:04 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'add commons to en wiktionary import sources, bug 23900'
* 02:02 logmsgbot: LocalisationUpdate failed
== June 10 ==
* 23:23 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'adding thank you banners to tracking page'
* 18:00 mark: Setup live purging of all requests for vector enabled wikis, to remove some broken pages of this morning from the cache 
* 14:52 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'fixing wgCollectionArticleNamespaces'
* 14:51 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'fixing wgCollectionArticleNamespaces'
* 09:03 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/PrefSwitch.i18n.php 'r67789'
* 08:59 mark: Purge done.
* 08:34 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-it.cache 'Rebuilt'
* 08:33 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-ja.cache 'Rebuilt'
* 08:33 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-fr.cache 'Rebuilt'
* 08:33 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-de.cache 'Rebuilt'
* 08:33 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-pt.cache 'Rebuilt'
* 08:33 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-pl.cache 'Rebuilt'
* 08:33 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-ru.cache 'Rebuilt'
* 08:32 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-es.cache 'Rebuilt'
* 08:32 mark: Florida squid purge done, esams cache dump done. Started purging esams pages.
* 08:22 RoanKattouw: Rebuilding LU cache for es, ru, pl, pt, de, fr, ja, it, (nl already done) to fix weird PrefSwitch messages
* 08:20 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-nl.cache 'Rebuilt, hopefully this works'
* 08:05 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-nl.cache 'Attempt at making i18n behave'
* 07:54 logmsgbot: LocalisationUpdate failed
* 07:53 RoanKattouw: Running l10nupdate on hume for real now
* 07:47 mark: Started purging the Florida squid caches, while the esams squids are still dumping their caches
* 07:45 RoanKattouw: Running LocalisationUpdate hoping that'll resolve weirdness
* 07:34 RoanKattouw: Creating prefswitch_survey DB table on all wikis
* 07:27 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/PrefSwitch.i18n.php 'resync'
* 07:23 mark: Dumping Squid caches for top-9 wikipedias
* 07:23 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Vector rollout, logo change, thumbnail size change on top 10 wikis sans enwiki'
* 07:23 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Vector rollout, logo change, thumbnail size change on top 10 wikis sans enwiki'
* 07:04 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Disable expandablesearch'
* 06:42 RoanKattouw: Applying extensions/UsabilityInitiative/ClickTracking/patch-additional_info.sql on all wikis
* 06:35 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump CSS style version'
* 06:34 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 06:30 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 
* 06:30 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 66620
* 06:29 RoanKattouw: Gonna run scap to deploy UsabilityInitiative and Vector updates
* 02:49 Tim: re-enabled apache and mod_php5 on fenari so that noc.wikimedia.org will work
* 02:27 Tim: installed wikimedia-task-appserver on fenari, this needs to be present for mediawiki to work correctly
* 01:03 logmsgbot: tfinc synchronized php-1.5/extensions/WikimediaMobile/MobileRedirect.js 
== June 9 ==
* 22:05 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 22:00 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'tweak wgCollectionNamespaces setting again'
* 21:49 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'added grants on metawiki to wgCollectionNamespaces'
* 17:55 Fred: mobile1 reimaged / tested. Putting back in rotation.
* 17:35 Fred: mobile1 going down for some lucid goodness
* 17:18 Fred: briefly depooling mobile1 while it get upgraded to Lucid.
* 01:28 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling WikimediaMobile on enwiki'
* 01:20 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'fixed mobile redirect again'
* 01:10 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling WikimediaMobile on plwiki'
* 01:00 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'fixed error in $wgWikimediaMobileUrl'
* 00:49 AaronSchulz: deployed r67676, r67677 (Flaggedrevs_alpha)
* 00:46 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
== June 8 ==
* 23:27 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 'Adding global support page to tracking'
* 23:05 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionTrackingStatistics_body.php 'adding range support to Special:ContributionTrackingStatistics'
* 22:36 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'turning off fr redirect till fix is in place'
* 22:08 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabling mobile redirect on FRwiki'
* 21:34 Fred: mobile LVS setup. Traffic to mobile now balanced between mobile1 (30%) and mobile2 (70%)
* 19:58 Rob: updated civicrm certificate file, had to apache2ctl restart on grosley
* 19:03 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Add .omniplan to extension whitelist for private wikis. Requested by Pete'
* 18:56 Fred: new wikimedia-task-dns-auth deployed to nameservers. langlist regenerated. *lang*.m.wikipedia.org is now cnamed to m.wikimedia.org
* 16:41 Rob: pushed updates to wordpress plugins on singer
* 16:40 Fred: I foobared. need to redirect traffic to mobile2 so mobile1 can be re-imaged. 
* 15:20 Rob: updated singer and dns records for austin's setup of wm10reg
* 01:29 Tim: running throughput test on mobile2
== June 7 ==
* 23:12 Fred: modified puppet receipe for mobile to allow for easier passenger gem update.
* 22:04 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 22:04 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r67568'
* 20:55 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'rollback for de_labs editors'
* 19:46 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php 'Picking up fix for tmp_id'
* 19:40 logmsgbot: catrope synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php 'r67558'
* 19:12 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php 'Picking up pagination from r67553'
* 18:56 logmsgbot: tfinc synchronized php-1.5/extensions/CentralNotice/SpecialNoticeText.php 'Picking up pagination from r67553'
* 16:57 Fred: Limiting message size to 5M for lists.
* 16:43 Tim: moved mediawiki-cvs archive files away to prevent further archiving
* 16:26 Tim: disabled archiving on problematic list mediawiki-cvs
* 14:07 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 
* 14:04 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 
* 13:49 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 
* 13:48 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 
* 13:46 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 
* 07:43 Tim: on lily: fixed broken sources.list and upgraded python, installed python symbols
* 07:25 Tim: on lily: restarted mailman (including ArchRunner)
* 07:07 Tim: on lily: there are three instances of ArchRunner which are in a tight loop using CPU but not doing any syscalls, presumably dead. Renicing them, will try to make backtraces
* 03:41 Tim: srv281 back up and resynced. However it crashed after being up only 3 days and shows more machine check errors. Suggest RMA.
* 03:25 Fred: setup ganglia for lily. 
* 03:07 Tim: srv281 down for 41 hours, trying reboot
* 02:54 Fred: upgraded gmond on lily. Modified gmond.conf to handle new modular arch. 
* 02:46 Tim: on lily: experimentally disabling broken bayes feature in spamassassin
* 02:35 Tim: on lily: restarting spamd and fixing lock file errors
* 01:25 Tim: on mobile1: Reduced PassengerMaxRequests to 5000 to reduce memory leakage
== June 6 ==
* 11:33 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'r67456'
* 11:33 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'r67456'
* 07:33 Tim: on mobile1: use PassengerMaxRequests instead of PassengerPoolIdleTime, to avoid oscillation between 200 and 30 processes, once every half an hour due to unknown slowdown
* 03:10 Tim: serve a 404 error for requests to the mobile server for domains other than *.m.wikipedia.org. DNS points here for lots of domains.
* 02:14 Tim: installed a redirect from en.m.wikipedia.com to .org
* 01:18 Tim: changed the mobile1 apache access log format to something more useful (and squid-like)
* 00:47 Tim: on mobile1: installed logrotate script for apache2 (via puppet)
== June 5 ==
* 21:21 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'fr_labs config cleanup'
* 21:15 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 
* 20:55 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'fr_labs config cleanup'
* 03:29 Tim: fixed mobile1 in nagios, added mobile2
* 02:09 Tim: restarting gmond on all miscellaneous cluster servers, to make mobile2 reappear in ganglia (broken by IP renumber)
== June 4 ==
* 22:52 tomaszf: starting webstats with new binary
* 22:50 tomaszf: stopping webstats in prep for update to track mobile stats
* 19:30 atglenn: moved bad snapshots (apr 11 through may 6 2010) to /mnt/dumps/public/bad so public index shows only good dumps and so there will be no prefetch against them
* 18:47 Fred: moved mobile2 to squid vlan / re-ip'ed / dns changed. mobile1 => 115 mobile2 => 116
* 18:35 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix. Gotta kill this thing some time'
* 18:35 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r67355'
* 18:34 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r67355'
* 12:11 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'WikimediaMobile'
* 11:37 Tim: mobile down for 15 minutes, possibly apache threads exhausted, restarting apache
* 09:56 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/SpecialContact.php 'r67333'
* 09:56 domas: deployments manage to kill apache processes sometimes
* 09:50 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 66620
* 09:50 Tim: pushing out WikimediaMobile (r67331) in preparation for deployment on testwiki
* 08:44 domas: decreased keepalivetimeout and timeout on mobile1
* 08:35 Tim: on mobile1: reduced max passenger pool size to 200, Domas and I think it's about right, shouldn't exceed allowable memory, should give us close to 100% CPU.
* 08:26 Tim: on mobile1: domas fixed file limit, now 50k
* 08:10 Tim: increasing MaxClients on mobile1 to 1500
* 05:01 Fred: Added apache2.conf, memcached.conf to puppet receipe for mobile. 
* 03:43 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23784 - Modify add/remove rights for bureaucrats on officewiki'
* 02:46 Tim: mobile1: increased ServerLimit to 1500 and reduced MaxClients to 500
* 02:35 Tim: on mobile1: increased memcached memory limit from 64M to 5000M
* 02:15 Tim: switched mobile1 over from apache2-mpm-worker to apache2-mpm-prefork (via puppet)
* 01:03 Tim: set ganglia host_dmax to 1 day
== June 3 ==
* 21:57 Fred: mobile1 re-imaged and puppetized. Changed subnet for mobile1. Changed DNS for mobile1. m pointing to newly imaged mobile1 (until transition is completed)
* 20:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23689 - Enable Collection extension on Thai Wikipedia'
* 20:22 AaronSchulz: deployed r67296 FlaggedRevs_alpha
* 20:21 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
* 19:39 mark: Moved mobile1 switchport from vlan 101 to 100
* 19:36 mark: Reverted DNS change of mobile1, back to .157
* 17:21 Fred: mobile1 going to be unreacheable while re-ip'ing
* 14:05 logmsgbot: midom synchronized php-1.5/wmf-config/InitialiseSettings.php 'timezone change for bat-smg'
* 11:53 mark: Made m.wikipedia.org CNAME m.wikipedia.org, m.wikipedia.org A to mobile1/2 in RR
* 10:57 hcatlin: mobile2 has been rebuilt and is featuring the new apache/mobile stack taking 40% of all mobile traffic. pls help monitor on ganglia.
* 09:04 Tim: cleaning COSS on sq45, resynced its configuration, will start squid when done
* 08:58 Tim: kernel reports degraded RAID on sq33, sq34, sq35, sq37, sq38, sq40
* 08:39 Tim: checked all serial consoles, all nonresponsive, rebooted all
* 08:23 Tim: sq33, sq34, sq35, sq37, sq38, sq40, sq45 have been down for 16-28 days, apparently for no good reason, can't find any log or DT entries. Will try reboots.
* 07:56 Tim: added new squids to nagios
* 06:36 Tim: cleaning cache directories on sq56 to avoid resurrection of expired content
* 06:35 Tim: adding monitoring for rather important service IPs: upload.esams and text.esams
* 06:22 Tim: sq56 not responding to ping or serial console (for 4 days), nothing in racadm getsel, rebooting
* 06:07 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabling ClickTracking due to CR r58099'
* 05:24 Tim: started apache on srv216, was stopped for some reason
* 03:57 Fred: shutting down mailman on list for a few minutes while exim and spamd catch up
* 01:42 Tim: adding forward and reverse DNS for mobile.tesla.usability.wikimedia.org, 208.80.152.245
== June 2 ==
* 23:48 Fred: re-imaging mobile2 again ;p
* 21:12 logmsgbot: robh ran sync-common-all 
* 20:57 Rob: srv134 has temp errors and multiple bad fans, out of warranty, decommissioning and removing from nagios/lvs/dsh
* 20:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23756 - Enable "Rollbacker" group on itwiki'
* 20:44 Rob: how many ops does it take to bring srv281 back online? one + a vounteer to point out he did it wrong.
* 20:42 logmsgbot: robh ran sync-common-all 
* 20:36 logmsgbot: robh ran sync-common-all 
* 20:26 logmsgbot: jeluf synchronizing Wikimedia installation... Revision: 66620
* 20:15 Rob: srv281 back online and apache files updated
* 20:08 Rob: lomaria is down, because it was throwing crap around and generally broken, so its shutdown now.
* 19:59 Rob: had to swap power cable on msw-b5-sdtpa
* 14:02 Rob: stay off db28, fixing its firmware.
* 13:44 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 
* 13:39 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 
* 13:38 Tim: ArchRunner on lily was stuck in a busy loop of some kind, had to restart it with kill -9
* 13:36 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 
* 13:35 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 
* 13:32 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 
* 13:32 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ParserFunctions.php 
* 13:31 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy.i18n.php 
* 13:31 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy.php 
* 06:49 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/backend/Subversion.php 'r67213'
* 06:45 Tim: updated codereview-proxy on isidore to r67211
* 06:13 Tim: attempting to set up CodeReview for pywikipedia
* 05:48 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23705 - Change namespaces in ja.wikipedia'
* 05:33 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '23742 - make category namespace flaggable on en.wikinews'
* 05:29 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23572 - Imagemap cannot accept dynamic links.'
== June 1 ==
* 23:10 Fred: mobile2 re-installed and puppetized
* 22:44 Fred: re-imaging mobile2 for total lucid puppetization.
* 20:17 Rob: mobile1 decided no more ssh server, its reinstalled and defined in puppet so it is ok now.
* 19:42 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesCkb_arab.php 'r67198'
* 14:26 logmsgbot: catrope synchronized php-1.5/extensions/CategoryTree/CategoryTreeFunctions.php 'r67183'
* 14:10 domas: livehacked the CategoryTreeFunctions.php to fix a fatal error :)
* 14:09 logmsgbot: midom synchronized php-1.5/extensions/CategoryTree/CategoryTreeFunctions.php 
* 12:02 logmsgbot: tstarling synchronizing Wikimedia installation... Revision: 66620
* 12:01 Tim: updating CategoryTree to r67181
* 05:56 logmsgbot: midom synchronized php-1.5/includes/db/Database.php 
* 05:52 logmsgbot: midom synchronized php-1.5/extensions/CategoryTree/CategoryTreeFunctions.php 
== May 31 ==
* 20:45 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23646 - Flagged Revisions on Tamil Wikinews (ta.wikinews.org)'
* 20:44 logmsgbot: jeluf synchronized flaggedrevs.dblist 
* 20:39 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23705 - Change namespaces in ja.wikipedia'
* 19:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23728 - Local time for Croatian Wiktionary'
* 19:45 JeLuF: sync-apache doesn't update /etc/apache2/apache2.conf on all hosts, two different versions are currently in use. Probably due to missing /home NFS mount on some apaches
* 19:17 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23737 - Enabling extension AbuseFilter for Upper Sorbian and Lower Sorbian Wikipedia'
* 19:13 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '15063 - wpSpamRegex entry for large image tables'
* 19:08 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23728 - Local time for Croatian Wiktionary'
* 09:36 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable $wgEnotifWatchlist on outreachwiki'
* 06:57 Tim: installed NRPE on db22
* 06:53 Tim: installed NRPE on db14
* 06:51 Tim: installed NRPE on db16
* 06:24 Tim: db16 has the same problem, killed 6 times today, taking the same action (27GB to 24GB)
* 06:16 Tim: mysql on db14 has been killed 4 times today by the OOM killer, reducing innodb_buffer_pool_size from 28GB to 24GB
* 06:13 Tim: mysql on db14 killed by kernel OOM killer, recovered without intervention
== May 30 ==
* 16:03 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r67084'
* 16:02 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.php 'r67084'
* 16:02 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'r67084'
* 16:02 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.php 'r67084'
* 15:48 logmsgbot: catrope synchronized php-1.5/wmf-config/ExtensionMessages.php 'Regenerated'
* 15:47 RoanKattouw: Removed nonexistent ActiveTaskForces.php from extension-list to unbreak it
* 15:46 logmsgbot: catrope synchronized php-1.5/extensions/ContactPageFundraiser/SpecialContact.php 'r67081'
* 15:45 logmsgbot: catrope synchronized php-1.5/extensions/ContactPageFundraiser/ContactPage.php 'r67081'
* 15:36 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/ContactPage.i18n.php 'Resync, didn't seem to be picked up right'
* 15:32 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22741: Enable $wgContactIncludeIP on nlwiki'
* 15:17 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/SpecialContact.php 'r67075'
* 15:16 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/ContactPage.i18n.php 'r67075'
* 15:16 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/ContactPage.alias.php 'r67075'
* 15:15 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/ContactPage.php 'r67075'
* 08:38 Tim-away: restarted memcached on srv194
== May 29 ==
* 23:17 apergos: started third worker on snapshot3. I'm going to stop logging these now.
* 16:15 apergos: started second worker doing xml dumps on snapshot3 (same screen session as root)
* 03:57 apergos: xml dumps resumed: running one copy of "worker" on snapshot3, in a screen session as root, along with "monitor" modified in place to not meddle with lock files (so that it leaves the enwiki dump, in progress on snapshot2, undisturbed)
== May 28 ==
* 23:37 atglenn: taking snapshot3 out of sync updates for a few days (commented out in mediawiki-installation node file) 
* 21:55 JeLuF: creating new captcha images on hume
* 16:54 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23486 - Enable Collection on gdwiki'
* 13:35 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php 
* 13:30 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ParserFunctions.php 'Missing svn add'
* 13:27 logmsgbot: andrew synchronizing Wikimedia installation... Revision: 66620
* 13:27 Andrew: running scap to deploy strategywiki extension updates
* 13:23 logmsgbot: andrew synchronized php-1.5/includes/specials/SpecialRevisiondelete.php 
* 09:00 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'moving over db16 to commons duty, db14 to dewiki, reenabling db4 and db5'
* 07:53 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'removing db5 and db4 for copies'
* 07:11 logmsgbot: tstarling synchronized php-1.5/includes/Sanitizer.php 'r66994'
* 06:31 domas: upgrading LOM/BIOS/OS on db14,db16,db22 to 3.0/10.4
* 02:41 Tim: srv92 locked up for a few minutes with syslog showing SATA errors. May be near death.
* 01:17 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 're-enabled CategoryTree, unless there is some emergency, you can't just disable extensions like this'
== May 27 ==
* 21:35 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable CollapsibleNav on outreachwiki'
* 21:22 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 
* 21:09 AaronSchulz: deployed r66980 (Flaggedrevs_alpha)
* 21:07 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
* 20:24 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable Vector and usability features on outreachwiki and foundationwiki'
* 20:17 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable CollapsibleNav for beta users on all wikis'
* 20:17 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enable SimpleSearch for beta users on all wikis'
* 11:59 domas: opened a new sub-categories related bug at https://bugzilla.wikimedia.org/show_bug.cgi?id=23682 ;-)
* 11:30 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'disable category tree'
* 04:13 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'vector as default skin for flaggedrevs_labswikimeida'
* 02:40 Tim: ran namespaceDupes.php with suffix on jawiki to fix talk namespace duplicates
== May 26 ==
* 23:59 Rob: updated blog to vector earlier today, didnt log... just pushed another update to the theme, blog is still happah
* 21:43 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'updated foundationwiki default skin from monobook to vector'
* 17:02 Tim: on mobile1 nginx: set proxy_next_upstream=off to avoid having the site go down regularly for 10s at a time
* 16:30 Tim: changing mobile1 nginx log format
* 06:39 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23667 - Local time for Macedonian Wikipedia'
* 03:39 Tim: set up SSLCertificateChainFile on williams (otrs) per bug 23631, used the cut down version of gd_bundle.crt posted on the bug report
* 02:08 logmsgbot: andrew synchronized php-1.5/includes/LogEventsList.php 
* 02:07 logmsgbot: andrew synchronized php-1.5/includes/specials/SpecialRevisiondelete.php 
* 02:07 logmsgbot: andrew synchronized php-1.5/languages/messages/MessagesEn.php 
* 02:05 Andrew: Deploying r66793 and r66823, first to test.wikipedia.org for final testing, then to all sites.
== May 25 ==
* 17:02 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Re-add style version appendix now that borked usability JS was purged from Varnish'
* 12:51 domas: what happened - server goes down, configuration gets updated, server comes up, builds new cache out of old conf, sync has timestamp older than cache, mw has stale config
* 12:47 domas: cleaned /tmp/mw-cache for some servers - apparently sync-common is not enough to get stuff in sync anymore :) we have now monobook pages in caches, I guess. 
* 12:26 rainman-sr: freed up some space and restarted indexing on searchidx1, indexing was stuck because the disk was full
* 11:16 mark: Downgraded puppet on db36
* 06:56 domas: resynced srv206/srv193 - seems to have lost critical configuration updates while they were down. Again, why servers are not resynced after downtime? 
* 01:24 Tim: fixed redirect from http://racktables to https://racktables, was broken due to IP address change
* 01:15 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'removed cruft from wgStyleVersion'
* 01:12 logmsgbot: tstarling synchronized php-1.5/skins/common/wikibits.js 'r66861'
* 00:19 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed Editor group from flaggedrevs_labswikimedia'
* 00:02 AaronSchulz: unhide the username of edits by "The Thing That Should Not Be" after accidental account locking (and unlock didn't work)
== May 24 ==
* 15:01 Rob: sq39 will stay offline for a bit, have a configuration concern with replacement disk performance.
* 15:01 Rob: pushed updates to srv206 and srv193, they should be back online and ok now.
* 14:54 Rob: sq39 has disk replaced and reinstalled, but its failing squid backend due to configuration. pulling out of rotation to look at later today.
* 14:51 Rob: working on srv206 and srv193
* 14:21 Rob: replaced bad disk in sq39, reinstalling.
* 12:16 domas: db32, db36, db38 and db40 are now serving enwiki (+ db12 and db26 )
* 12:15 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 09:25 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'db22 being removed too'
* 09:23 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'removing db14, db16 from enwiki duty, will host different clusters, db26 upgrade'
* 09:17 domas: master switch db16->db36, new master position - db36-bin.000001:106
* 09:16 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 09:16 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 09:09 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 09:06 domas: srv141 was in prod, but not in mw_inst
* 07:24 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 07:09 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 07:06 domas: srv134 was having i/o issues, probably bad disk, rebooted for it to stop flapping, sorry Jens :)
== May 23 ==
* 15:12 apergos: turned off overcommit on bayes (left overcommit ratio at default = 50, let's see how that goes) 
* 14:57 apergos: rebooted bayes after OOM
* 13:18 domas: forward-ported (cp from hardy to wikimedia repo ;-) libstdc++5 to unbreak wikimedia-raid-utils and few other things. \o/
* 12:40 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 08:01 JeLuF: increased size of filesystem /var/backup on mchenry by 10 GB
* 06:44 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23545 - Rights changes for frwiktionary'
* 06:39 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 06:00 domas: reinstalled bunch of new DBs with lucidlynx
== May 21 ==
* 22:53 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23545 - Rights changes for frwiktionary'
* 19:36 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Use separate Apple Touch icon for testwiki'
* 18:17 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 18:16 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r66729'
* 16:33 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 16:33 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r66726'
* 16:18 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 16:17 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Resync, seems this didn't get updated properly last time'
* 09:53 RoanKattouw: Deployed new favicon and apple toch icon on test; overwrote the latter locally on srv124, will be overwritten by scap until we change the wikipedia.org-wide default
== May 20 ==
* 21:31 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump CSS style version'
* 21:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 21:02 logmsgbot: catrope synchronizing Wikimedia installation... Revision: 66620
* 21:01 RoanKattouw: Running scap to deploy r66708
* 19:50 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23596 - Add the 'Vikiprojekts' namespace and a related 'Vikiprojekta diskusija' namespace to the Latvian Wikipedia'
* 19:36 AaronSchulz: deployed r66704 (Flaggedrevs_alpha)
* 19:28 logmsgbot: aaron synchronizing Wikimedia installation... Revision: 66620
* 19:02 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/PrefSwitch.i18n.php 'r66701'
* 17:25 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '20074 - Add an Apple Touch icon for MediaWiki.org'
* 17:21 logmsgbot: jeluf synchronized docroot/mediawiki/Apple-touch-icon.png 
* 13:55 mark: Unshut BGP session to 13030, and reenabled announcement of our prefixes on the session
* 13:41 mark: Changed IP address of text.esams.wikimedia.org (i.e. the main IP address of our wiki farm at esams) from 91.198.174.2 to 91.198.174.232
* 13:14 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveTaskForces/ActiveTaskForces_body.php 
* 13:11 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveTaskForces/ActiveTaskForces_body.php 'Testing'
* 13:09 logmsgbot: andrew synchronized php-1.5/extensions/StrategyWiki/ActiveTaskForces/ActiveTaskForces_body.php 'ActiveTaskForces: Merge r66686'
* 13:06 mark: Added new LVS service IPs to the older esams text squids
* 10:53 mark: Removed /etc/cron.hourly/mw-cleanup-tmp on all appservers
== May 19 ==
* 21:34 mark: Manually updated countries.nerd.dk databases with new versions on the auth dns servers
* 20:16 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23568 - Allow bureaucrats to grant and remove patroller, autopatroller, rollbacker on eswiktionary'
* 20:11 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23592 - Disable creating new articles for IP-users in ckbWikipedia'
* 20:07 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23576 - Alias of Potal talk namespace on jawiki, jawikinews'
* 15:39 Rob: mchenry's drive replaced, it will start auto mirroring shortly
* 15:33 Rob: pulled mchenry's hot swap spare to put in a 1.5TB for migration
* 15:33 Rob: welcome back morebots
== May 18 ==
* 20:46 Rob: got distracted, but mailman is back online now.
* 20:43 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23545 - Rights changes for frwiktionary'
* 20:23 Rob: have to pull a message from a list, stopping mailmain on lily to edit the mbox (will restart and log its restart shortly)
* 19:50 logmsgbot: catrope synchronized php-1.5/includes/DefaultSettings.php 'r66618'
* 19:50 logmsgbot: catrope synchronized php-1.5/skins/vector/main-rtl.css 'r66618'
* 19:49 logmsgbot: catrope synchronized php-1.5/skins/vector/main-ltr.css 'r66618'
* 19:41 atglenn: added jeluf to core committer group on mayflower svn (why wasn't he there? who knows)
* 18:22 RoanKattouw: Started morebots, didn't come back up when wikitech did
* 18:21 RoanKattouw: <logmsgbot> robh ran sync-common-all 
* 18:21 RoanKattouw: <logmsgbot> catrope synchronized php-1.5/languages/messages/MessagesRo.php 'r66615'
* 14:32 Andrew: 13:50 <+logmsgbot> !log andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Activate single-revision deletion for admins on all projects'
* 03:45 logmsgbot: tstarling synchronized php-1.5/maintenance/addwiki.php 
== May 17 ==
* 23:38 logmsgbot: aaron synchronized php-1.5/StartProfiler.php 'gave flaggedrevs_labswikimedia a profiling ID'
* 20:48 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23569 - Change name of Portal talk namespace on jawiki, jawikinews'
* 20:40 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'touch'
* 20:40 logmsgbot: jeluf synchronized php-1.5/wmf-config/abusefilter.php '22008 - Abusefilter-log-detail for Arabic Wikipedia'
* 20:35 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23562 - New Namespace for ko.wikisource'
* 20:20 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23404 - Create new namespaces on trwikibooks'
* 20:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23328 - New namespace requested on fr.wikt'
* 20:10 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22921 - New namespaces and aliases for ml.wikibooks'
* 19:50 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23244 - Enable FlaggedRevs/patrol extension on mkwiki'
* 19:42 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23351 - Please remove the 'reviewer' user group from en wikinews if possible'
* 19:41 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '23351 - Please remove the 'reviewer' user group from en wikinews if possible'
* 19:29 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23488 - Add rename user group on Commons'
* 19:22 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23545 - Rights changes for frwiktionary'
* 16:52 logmsgbot: catrope synchronizing Wikimedia installation...
* 16:51 RoanKattouw: Running scap to deploy r66569
* 09:17 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Reenable SimpleSearch in config (still disabled in JS), bump style version appendix'
* 09:16 RoanKattouw: Commented out srv141 in /etc/dsh/mediawiki-installation to stop sync-file from hanging forever
* 09:15 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'Disable SimpleSearch in JS with if(false)'
* 08:41 domas: banned a sucker via badbadip
* 08:22 domas: site had some overload for ~15mins, two memcached/app servers didn't recover afterwards (I rebooted srv181, srv141 is still in oom or some other limbo) 
* 08:13 logmsgbot: midom synchronized php-1.5/wmf-config/mc.php 
* 05:24 logmsgbot: ariel synchronized wmf-deployment/wmf-config/InitialiseSettings.php 'enable lqt on en wikt, per bug 23417'
== May 16 ==
* 18:25 JeLuF: fixed doxygen cronjob on mayflower. Also added logfile rotation
* 15:04 mark: sq39 died due to missing drive /dev/sdb
* 11:11 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23543 - Add download as PDF in ru.wikiversity'
* 10:45 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'typo'
* 06:54 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23396'
* 06:43 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23481 - Add MediaWiki, Wikimania wikis to the "wikimedia" project in CentralNotice'
* 06:26 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23531 - Turn on $wgVectorShowVariantName for zhwiki'
== May 15 ==
* 21:44 JeLuF: added nyc.wikimedia.org to DNS and redirects.conf
* 19:32 mark: Changed IPv4 address of upload.esams.wikimedia.org from 91.198.174.3 to 91.198.174.234
* 18:02 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'r66486'
* 16:50 mark: Emergency switch of upload LVS (91.198.174.3) to amslvs4
* 16:37 mark: Removed amssq61 from the LVS pool as "it's weird"
* 13:53 mark: Pooled new upload squids amssq47-62 with normal LVS weight 30
* 13:37 mark: Pooled new upload squids amssq47-62 with normal LVS weight 12
* 13:32 mark: Pooled new upload squids amssq47-62 with low LVS weight 1
* 13:20 mark: Pooled new upload squids amssq47-62 with low CARP weight 1
* 12:01 mark: Fixed an issue with puppet on all hosts, trying to import ssh keys with an empty title
* 11:56 mark: Upgraded server sockpuppet to 10.04 LTS
* 10:20 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Disable SimpleSearch temporarily until bugs are resolved'
* 05:50 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23502 - Install Quiz on French Wiktionary'
== May 14 ==
* 15:22 mark: Installed Lucid on amssq47..62
* 12:14 river: restarted pdns on ns1
* 11:54 domas: (and mark) fixing derailed squid conf for ~16 servers that went down
== May 13 ==
* 22:09 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 22:08 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r66404'
* 22:08 logmsgbot: catrope synchronized php-1.5/skins/Vector.php 'r66404'
* 21:31 mark: Setup amslvs2 and amslvs4, created fenari:/home/w/conf/pybal/esams/ for having a central location where PyBal can fetch realserver lists from
* 19:49 logmsgbot: catrope synchronized php-1.5/skins/Vector.php 'r66381'
* 19:38 mark: Downgraded puppet on server alsted (running 10.04 LTS); the puppet in 10.04 is too new
* 17:43 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix for reports of weirdness'
* 17:24 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Hide template pref for real'
* 17:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Remove template collapsing pref'
* 14:38 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevsXML.php 
* 14:32 mark: Copied the standard Squid error messages over the Wikimedia ones on API squids sq31 and sq32, until we upgrade them
* 14:22 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevsXML.php 
* 13:55 AaronSchulz: deployed r66342 (Flaggedrevs_alpha)
* 13:52 logmsgbot: aaron synchronizing Wikimedia installation...
* 12:18 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/PrefSwitch.i18n.php 'r66336'
* 12:17 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 12:17 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r66336'
* 12:16 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r66336'
* 10:41 mark: Increased purge sample rate to 1
* 10:33 mark: Increased purge sample rate to 1/5
* 08:46 RoanKattouw: refreshCategoryCount.php running nicely on Commons now, should finish in about 2 hours
* 08:29 RoanKattouw: Running refreshCategoryCount.php on Commons again. Hacked script to print more progress markers, hacked Category.php to not do LOCK IN SHARE MODE
* 08:23 RoanKattouw: Nope, that didn't work
* 08:22 RoanKattouw: Attempting to run refreshCategoryCounts.php on Commons, last attempt failed with DB deadlock almost immediately
* 08:04 mark: Sample rate of purging increased to 1/10
* 08:02 mark: First stage of purging (all squid objects with known URL) done, now enabling live purging of requested enwiki pages, with 1/100 sample rate
* 07:38 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/PrefSwitch.i18n.php 'r66327, r66329'
* 07:10 mark: Upped speed of purge rate limiting to approx 1500 purges/s (running on a screen on locke)
* 07:02 mark: Started purging enwiki squid pages
* 06:58 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'r66325'
* 06:57 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'r66325'
* 06:52 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Change logo URL for enwiki to point to new logo'
* 06:45 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 06:14 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable Vector, usability features by default on enwiki'
* 06:08 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enable SimpleSearch on Vector-default wikis, bump style version appendix'
* 06:05 mark: Restarted apache on srv159
* 06:02 logmsgbot: catrope synchronizing Wikimedia installation...
* 06:01 RoanKattouw: Running scap to deploy Vector&UsabilityInitiative updates
* 05:34 mark: /dev/sdb on knsq2 has 20 uncorrected errors, about to die
* 05:28 RoanKattouw: Deploying r66322, r66323 on test.wikipedia.org
== May 12 ==
* 18:44 Rob: new apaches seem to be handling things just fine
* 18:34 Rob: new apaches are online, pooled one with no issues, so just pushed the rest (srv258, then srv259-srv289)
* 12:09 JeLuF: There are reports about issues with .de domains, DENIC seems to have problems with the DNS server for .de
* 07:01 Tim: running SecurePoll voter list script for checkuser and oversight elections, in screen on hume
== May 11 ==
* 22:37 Rob: adding new hosts to nagios, prepare for flapping from them (but they are not in service so can be ignored for now)
* 21:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22139 - Install Labeled Section Transclusion on wikt'
* 21:17 logmsgbot: robh synchronized php-1.5/wmf-config/CommonSettings.php 'added back with proper syntax'
* 21:12 logmsgbot: robh synchronized php-1.5/wmf-config/CommonSettings.php 'revert'
* 21:11 domas: unbreaking Rob's change. 
* 21:10 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'lol'
* 21:05 logmsgbot: robh synchronized php-1.5/wmf-config/CommonSettings.php 'adding file extensions for foundationwiki per cary'
* 20:09 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 
* 20:08 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23462 - please allow Crats to add/remove the Transwiki importer right on SimpleWiki'
* 19:55 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 
* 19:43 RoanKattouw: Running refreshCategoryCounts.php on the remaining wikis, 6 parallel instances on hume
* 19:38 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 
* 19:34 AaronSchulz: cleared out de_labswikimedia pages
* 19:12 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 
* 17:32 mark: Moved bits.esams.wikimedia.org to 91.198.174.233 in DNS. In case of problems, move back to .2
* 16:50 mark: Allocated 4 out-of-subnet service IPs for LVS, to test whether this confuses the foundry RXes less
* 13:21 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 23374'
* 13:02 mark: Set up an LVS service for bits.esams.wikimedia.org (91.198.174.8), and changed it in DNS. Currently served by new server amslvs1
* 12:18 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'configured de_labswikimedia like dewiki'
* 10:24 mark: Installed Lucid on amslvs1-4
* 01:20 tomaszf: starting new logstream of pagecounts to dataset1
== May 10 ==
* 23:05 atglenn: power cycled pdf2 from the drac, was not responding to ping
* 22:17 RoanKattouw: pdf2 down, pinging from fenari yields Host unreachable
* 22:06 RoanKattouw: Stopped refreshCategoryCounts.php run about an hour ago, will resume tomorrow
* 20:32 JeLuF: running 4 instances of populateRevisionLength.php in parallel
* 20:17 RoanKattouw: Now running 8 parallel instances of refreshCategoryCounts.php on hime
* 20:04 RoanKattouw: Running refreshCategoryCounts.php on all wikis on hume, two instances in parallel
* 19:12 JeLuF: running populateRevisionLength.php on all wikis (bug 12188)
* 18:15 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 18:14 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r66149'
* 11:30 hcatlin: deployed new UDP logger for mobile. someone needs to keep an eye out on the parser!
* 09:36 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix Portail namespace on frwikisource'
* 07:44 mark: Trippled CARP weight on the new ams squids
* 05:21 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22663 - Add some interwiki links in Special:Import of fr.wiktionary, fr.wikibooks'
== May 9 ==
* 22:41 mark: Pooled amssq31-46 in LVS with weight equal to the older squids
* 22:23 mark: Pooled amssq31-46 in LVS with weight 1
* 20:25 mark: Installed Lucid on amssq32-46
* 17:12 mark: Pooled amssq31 in CARP with low weight
* 16:38 mark: Pooled amssq31 in LVS
* 15:30 apergos: added respository dumps link to index.html on mayflower
* 15:25 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Add amssq31-62 to the proxies list'
* 12:43 mark: Setup OSPF on the esams management network
* 09:44 apergos: added directory for demon's weekly svn dumps to apacke config on mayflower 
== May 8 ==
* 23:14 mark: Setup quick hack on brewster to proxy puppet connections from esams to sockpuppet
* 20:04 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 23317 - Enable subpages for Template namespace at English Wikibooks'
* 19:57 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23251 - Turn on subpages in main namespace of Wikimedia Norge's wiki'
* 19:31 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23389 - Disable anonymous edits on chapterwiki no.wikimedia'
* 19:24 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23390 - Namespace alias VP -> NS_PROJECT on eowiki'
* 19:17 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23392 - Allow bureaucrats to de-sysop on English Wikinews'
* 19:10 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23409 - Add Portail namespace for French Wikisource'
* 19:04 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23418 - Change English Wiktionary logo'
* 18:59 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23410 - Set logo on Bhojpuri Wikipedia'
* 18:54 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23423 - Enable Collection extension in Upper Sorbian and Lower Sorbian Wikipedia'
* 18:51 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23434 - Activation of the extension Collection on eowiki'
* 18:49 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23437 - Collection extension on Ukrainian Wikipedia'
* 18:44 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23357 - Disable Login for German ArbCom wiki'
* 10:30 RoanKattouw: Upped $wgMaxShellMemory to 200M (from 100M) on wikitech, unbreaks ImageMagick
* 02:12 Andrew: Forced query page update on strategywiki, added ActiveTaskForces to extension-list
* 02:11 logmsgbot: andrew synchronizing Wikimedia installation...
* 02:08 logmsgbot: andrew synchronizing Wikimedia installation...
* 01:02 logmsgbot: andrew synchronizing Wikimedia installation...
* 01:02 Andrew: Per Eugene's request, installing ActiveTaskForces on strategywiki
== May 7 ==
* 22:15 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 22:09 RoanKattouw: My name is morebots and I'm alive and well
* 22:05 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 21:58 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 21:47 RoanKattouw: Switched default skin on wikitech to Vector again
* 21:46 RoanKattouw: Upgraded mwclient on wikitech for login CSRF fix
* 20:42 RoanKattouw: Testing morebots 
* 20:23 RoanKattouw: Installed JPEG delegate for imagemagick on wikitech, now reinstalling imagemagick
* 20:21 RoanKattouw: Installed JPEG delegate for imagemagick on wikitech, now reinstalling imagemagick
* 20:21 RoanKattouw: Enabled PdfHandler on wikitech
* 20:11 RoanKattouw: Enabled PdfHandler on wikitech
* 20:07 RoanKattouw: Enabled PdfHandler on wikitech
* 19:57 RoanKattouw: Switching wikitech to 1.16wmf4, was running wmf-deployment
* 19:56 RoanKattouw: Set $wgDefaultSkin = 'vector'; on wikitech
* 17:56 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 17:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r66031'
* 17:36 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 16:57 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 16:52 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 16:48 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 16:41 RoanKattouw: Running svn cleanup again after patching some permissions issues (created catrope-owned copies of affected dirs)
* 16:14 RoanKattouw: Running svn cleanup in /mnt/upload6/private/ExtensionDistributor/mw-snapshot/branches/REL1_15/extensions
* 08:23 logmsgbot: mark synchronized php-1.5/wmf-config/db.php 'Repooling db12'
* 05:25 domas: running test build on db40, serving enwiki duty
* 05:24 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'normalizing load a bit hehe'
* 05:15 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 05:14 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 03:38 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'preparing for a copy'
* 01:46 logmsgbot: tstarling synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php 'more profiling'
* 01:32 logmsgbot: tstarling synchronized php-1.5/includes/parser/Parser.php 'profiling tweak'
* 01:29 logmsgbot: tstarling synchronized php-1.5/includes/parser/Parser.php 'profiling tweak'
== May 6 ==
* 22:49 mark: STOPped mysqldump on snapshot1 (holding table locks) and killed the currently running mysql query. db12 is catching up again
* 22:25 logmsgbot: mark synchronized php-1.5/wmf-config/db.php 'depool db12 (as opposed to setting load to 0)'
* 21:53 logmsgbot: mark synchronized php-1.5/wmf-config/db.php 'depool db12'
* 21:43 RoanKattouw: db12 lag at 1464 seconds and rising
* 20:00 AaronSchulz: deployed r65996 (Flaggedrevs_alpha)
* 20:00 logmsgbot: aaron synchronizing Wikimedia installation...
* 19:42 mark: Shutdown BGP session to AS 13030
* 18:38 Rob: had wrong setting in name for dns on new scs, updated dns to correct
* 18:18 Rob: updated dns for scs-b1-pmtpa
* 17:14 schmir: schmir drinks beer
* 17:08 Rob: project2 had two bad fans. if it fails a lot more, it may have system damage from the overheating
* 16:55 Rob: replicated package installation from pdf2 to pdf3
* 16:14 Rob: pdf3 online and setup with identical users to pdf1/2, handed off to pediapress for mwlib magics
* 14:59 Rob: updated dns due to linne and gilman having same IP in zone file, but linne was not listed in the reverse dns file. changing gilman IP info since its not online and in cluster just yet.
* 14:39 Rob: updated dns for pdf3 server
* 14:08 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable PDF creation for anons on enwiki'
* 00:54 logmsgbot: tstarling synchronizing Wikimedia installation...
* 00:53 Tim: updating collection extension to trunk
* 00:38 logmsgbot: tstarling synchronized php-1.5/includes/parser/Parser.php 'r65974'
== May 5 ==
* 21:56 logmsgbot: catrope synchronized php-1.5/includes/DefaultSettings.php 'r65966'
* 21:56 logmsgbot: catrope synchronized php-1.5/skins/vector/main-rtl.css 'r65966'
* 21:56 logmsgbot: catrope synchronized php-1.5/skins/vector/main-ltr.css 'r65966'
* 19:49 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable CollapsibleNav on commons'
* 19:48 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump CSS style version'
* 19:47 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 19:45 logmsgbot: catrope synchronizing Wikimedia installation...
* 19:45 RoanKattouw: Running scap to deploy r65959 (Vector, UsabilityInitiative fixes)
* 19:05 RoanKattouw: Deploying r65959 on testwiki
* 12:52 mark: DRAC of server multatuli not working - aborts ssh login with error
* 04:32 river: adding vlan 302 for toolserver web cluster private interconnect
* 01:48 logmsgbot: tstarling synchronizing Wikimedia installation...
* 01:47 Tim: deploying r65931, MIME type fixes especially .oga
* 01:26 Tim: installed ca-certificates on fenari to fix bug 23179
* 00:45 Tim: deploying oggvideotools r168 (large page size fix, bug 23402) to all image scalers
== May 4 ==
* 21:54 logmsgbot: catrope synchronized php-1.5/RELEASE-NOTES 'Dummy sync in an attempt to find problematic servers'
* 01:30 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 01:29 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 
* 01:12 Tim: installed oggThumb on remaining image scalers. Deployed r65889 with oggThumb enabled.
* 01:12 logmsgbot: tstarling synchronizing Wikimedia installation...
== May 3 ==
* 15:11 logmsgbot: catrope synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php 'r65858'
* 15:10 logmsgbot: catrope synchronized php-1.5/extensions/ProofreadPage/proofread_index.js 'r65857'
* 10:09 RoanKattouw: Syncing srv99 in response to comments on bug 23373
* 09:28 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Threads.php 'r65846'
* 09:27 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Threads.php 'r65846'
* 09:20 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Threads.php 'r65843'
* 09:20 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Threads.php 'r65843'
== May 1 ==
* 21:35 RoanKattouw: Apache not running on srv246
* 20:25 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialUserlogin.php 'r65761'
* 20:24 logmsgbot: catrope synchronized php-1.5/includes/templates/Userlogin.php 'r65761'
* 13:05 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedArticleView.php 
* 13:02 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedArticleView.php 
* 08:42 AaronSchulz: deployed r65728 (Flaggedrevs_alpha)
* 08:42 logmsgbot: aaron synchronizing Wikimedia installation...
== April 30 ==
* 23:39 Tim-away: installed the new oggvideotools on srv224, source install from /root/src
* 20:00 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21375: Allow bcrats to remove sysop, bcrat on arbcom_fiwiki'
* 16:51 Rob: ryan rebooted prototype vm on tesla, it was giving oom errors, software on system needs to be adjusted, as the system has 4GB dedicated to that virtual host.
* 05:38 logmsgbot: tstarling synchronized php-1.5/includes/media/Bitmap.php 'r65682'
== April 29 ==
* 20:59 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'updating logo imports language for bug 19609'
* 20:06 atglenn: restarted (sort of) morebots
== April 28 ==
* 03:25 AaronSchulz: deployed r65602 (Flaggedrevs_alpha)
* 03:25 logmsgbot: aaron synchronizing Wikimedia installation...
* 00:57 logmsgbot: midom synchronized php-1.5/includes/User.php 'getAllCallers for user::invalidate'
* 00:33 logmsgbot: midom synchronized php-1.5/includes/Title.php 'verbose fname hook for invalidation'
* 00:07 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'removing srv169 from cluster9 ES config - seems to be freshly imaged server without the datastore atm, needs resync'
== April 27 ==
* 22:47 Tim: removed overly-broad block of all user agents containing "download" from common-acls.conf
* 16:36 Rob: tridge coming down for disk array addition
* 14:26 Rob: replaced disk 16 and 25 in dataset1
== April 26 ==
* 21:16 logmsgbot: ariel synchronized wmf-deployment/wmf-config/InitialiseSettings.php 'remove ip throttle exception for en pedia new user seminar, ended'
* 15:55 apergos: shutting down back end squid on knsq2, sdb bad 
* 12:51 ^demon: Setup cron on mayflower for /home/demon/svn-dump-update every saturday at 1am.
* 07:45 mark: Removed sendmail from mobile1, so it stops spamming with cron messages. Why was it installed?
* 02:51 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
== April 25 ==
* 23:19 logmsgbot: andrew synchronized php-1.5/extensions/TorBlock/TorBlock.class.php 'MFT r65535'
* 23:13 logmsgbot: andrew synchronized php-1.5/includes/HttpFunctions.php 'Deploy r65532'
* 22:56 Andrew: Deployed r65529, HttpFunctions fix.
* 22:56 logmsgbot: andrew synchronized php-1.5/includes/HttpFunctions.php 
* 17:24 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'taking out db14 for some migrations'
* 11:58 JeLuF: added image/vnd.djvu to the mime.types file of ms7, refreshed webserver (bug 23318)
* 10:46 logmsgbot: catrope synchronized php-1.5/extensions/GlobalBlocking/SpecialGlobalBlockStatus.php 'r65520'
* 09:46 JeLuF: deleted oprofiled.log on mobile1 (17GB)
== April 24 ==
* 19:03 apergos: so on ombile1 there is now an "extra" error log in /var/log/nginx/error.log.toobig.gz (it needed to be compressed, sorry)
* 18:51 apergos: cleared some space from mobile1 (hd full), trying restarts again
* 17:52 Fred: mobile1 went oom again. Rebooting the server. 
* 16:58 Rob: unracked csw-a4-pmtpa
* 16:48 Rob: new msw1-sdtpa switch is now working on serial console (scs was set to 57600, changed to 9600)
* 13:18 mark: Set up all sdtpa row B ServerTech PDUs 
* 10:50 RoanKattouw: Rerunning refreshLinks for svwikisource
* 06:17 apergos: stopping back end squid on knsq11, sdb is bad
== April 23 ==
* 23:15 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' 23259 - Disable Extension:NewUserMessage on kawiki'
* 22:38 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23233 - rollbacker group on farsi wikipedia'
* 22:35 Fred: manually starting merb on mobile1. 
* 22:25 Fred: mobile1 went OOM and shot itslef in the foot. Rebooted.
* 22:24 JeLuF: running refreshLinks for svwikisource
* 22:23 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23287 - $wgCategoryPrefixedDefaultSortkey on Swedish Wikisource'
* 22:20 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23244 - Enable FlaggedRevs/patrol extension on mkwiki'
* 22:17 logmsgbot: jeluf synchronized flaggedrevs.dblist '23244 - Enable FlaggedRevs/patrol extension on mkwiki'
* 22:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23299 - Allow imports from Wikimedia Commons to Hebrew Wikipedia'
* 22:12 Fred: mobile1 went dead. Looking into it.
== April 22 ==
* 17:55 Rob: added wikidev group to snapshot servers, which includes all roots and also specifically included andrew
* 17:18 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turn LiquidThreads on on wikimania2010wiki'
* 01:45 AaronSchulz: deployed r65418 (Flaggedrevs_alpha)
* 01:37 logmsgbot: aaron synchronizing Wikimedia installation...
* 00:00 atglenn: trying to run memtest on dataset1... little issue with the serial port line speed :-P
== April 21 ==
* 23:47 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set oldChangeTagsIndex var to false on flaggedrevs_labswikimedia'
* 22:30 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'reenabling ixia and db30 - both got disabled weeks ago by someone...'
* 22:11 domas: installed hardy on db31-db40 sans db34
* 14:55 logmsgbot: tstarling synchronized php-1.5/includes/Article.php 'r65394 fix pagetitle-view-mainpage'
* 14:25 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Turn on live preview on testwiki'
* 14:25 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Turn on live preview on testwiki'
* 14:23 logmsgbot: andrew synchronized php-1.5/includes/Preferences.php 'r65392'
* 13:56 logmsgbot: andrew synchronized php-1.5/skins/common/preview.js 'MFT r65388'
* 13:49 logmsgbot: andrew synchronized php-1.5/skins/common/preview.js 'MFT r65384'
* 11:00 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 10:59 logmsgbot: catrope synchronized php-1.5/skins/vector/main-rtl.css 'r65374'
* 10:59 logmsgbot: catrope synchronized php-1.5/skins/vector/main-ltr.css 'r65374'
* 10:17 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'r65369'
* 10:17 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'r65369'
* 10:08 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/NewMessagesController.php 'r65367'
* 10:07 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/NewMessagesController.php 'r65367'
* 06:29 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'enabled abusefilter for flaggedrevs_labswikimedia'
* 02:07 domas: how did I install db31? 
== April 20 ==
* 18:18 Rob: see that, me syncing files, not like you cheaters fred and ariel!
* 18:18 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 23231'
* 16:55 Rob: replaced bad disk on db5
* 13:19 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Touch this file to purge config cache'
* 11:53 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Merge r65305'
* 11:52 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r65305'
* 11:49 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r65302'
* 11:49 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Merge r65302'
* 11:28 ^demon: svn dump successfully completed. Dumped revs 1-65293 to /home/demon/svn/mediawiki-20100418-dump-with-deltas. Took up ~1.1G
* 11:13 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style vers'
* 11:13 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/lqt.css 'Deploy r65297'
* 11:11 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.css 'Deploy r65297'
* 10:40 logmsgbot: andrew synchronized php-1.5/extensions/NewUserMessage/NewUserMessage.class.php 'Deploy r65295'
* 10:27 ^demon: Running 'svnadmin dump --deltas' in a screen on Mayflower. Writing to storage2 Tomasz mount�ed at /home/demon/svn. Already to ~r30000
== April 19 ==
* 23:14 atglenn: one more time with elwiki, cswiki and ltwikt worker threads (shooting parent processes as well this time) on snapshot3
* 23:08 domas: started slow query killer on db9 - it was constantly alerting in #wikimedia-tech 
* 22:32 atglenn: killed fetchText.php on snapshot3 for ltwikt, cswiki, thwiki, all hung on writes (nfs issue ? they were stuck from April 16 on... coincidentally when cname change went out for download.wm -> dumps.wm) Dumps seem to be running normally now
* 19:56 Tim: upgraded libavcodec and libavformat on image_scalers node group due to USN-931-1
* 00:18 Fred: made room on McHenry to resume backups from lily and sanger.
== April 18 ==
* 07:37 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '21243 - Request for PDF Version & Page collection'
* 07:27 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' 23218 - Add CAT: as namespace alias for Category: on enwikinews'
== April 16 ==
* 23:02 JeLuF: 16412 - Create a redirect from hu.wikimedia.org to http://wiki.media.hu
* 21:24 JeLuF: running cleanupTitles.php on *.wiktionary.org
* 21:01 JeLuF: updated site_stats:ss_images on plwiki
* 20:48 JeLuF: Redirect download.wikipedia.org and other aliases to dumps.wikimedia.org
* 20:07 JeLuF: 17008 - Redirecting from eo.wikisource.org
* 19:37 JeLuF: enabled on-demand SVG compression on upload.wikimedia.org
* 19:06 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22157 - Enable Extension:NewUserMessage on ka.wiki'
* 18:46 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '20947 - Enable RevisionDelete on Hungarian Wikipedia (hu.wikipedia)'
* 18:31 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22342 - Need a new namespace for the thesaurus in Bengali Wiktionary'
* 18:27 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23048 - Change name of Portal talk namespace on Tamil Wikipedia'
* 18:21 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php '23166 - New namespace for pl.wikisource'
* 18:18 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23166 - New namespace for pl.wikisource'
* 18:06 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22426 - Change eo.wikt user namespace to "Uzanto"'
* 18:05 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22426 - Change eo.wikt user namespace to "Uzanto"'
* 17:36 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22319 - $wgBlockDisablesLogin = true; for otrs_wikiwiki and chapcomwiki'
* 17:32 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#23197 need to have Nominating Committee wiki deleted '
* 17:31 logmsgbot: jeluf synchronized all.dblist 'Deleting nomcomwiki'
* 17:31 logmsgbot: jeluf synchronized closed.dblist 'Deleting nomcomwiki'
* 11:55 Tim: running cleanupImages.php on mlwiki
* 09:46 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Change Czech talk namespace names per Danny B'
* 09:34 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesCs.php 'r65113 Change talk namespaces names'
* 08:57 hcatlin: changed mobile1 so logrotate is now run hourly instead of daily. should stop the disk from overflowing
== April 15 ==
* 11:55 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialRandompage.php 'r65056'
* 11:16 RoanKattouw: Running l10nupdate expecting it to fail, testing failure logging
* 10:31 RoanKattouw: Running l10nupdate by hand to debug issue reported at translatewiki
* 09:27 logmsgbot: tstarling synchronized php-1.5/extensions/CentralAuth/SpecialCentralAuth.php 'r65048'
* 09:26 logmsgbot: tstarling synchronized php-1.5/extensions/CentralAuth/CentralAuth.i18n.php 'r65048'
== April 14 ==
* 20:04 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'Add group IP to wgRateLimitsExcludedIPs for a wp for new users conf. To be removed Apr 20 2010'
* 18:27 atglenn: (yesterday's news) dataset1 back up with smaller ext3 partition replacing the xfs one, for testing
* 15:42 logmsgbot: catrope synchronized php-1.5/extensions/CentralAuth/CentralAuthUser.php 'r65033'
* 15:08 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialUpload.php 'r65031'
* 14:18 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEn.php 'r65028'
* 14:17 logmsgbot: catrope synchronized php-1.5/includes/UserMailer.php 'r65028'
* 13:59 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialUpload.php 'r65026'
* 13:09 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 13:08 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r65023'
* 12:58 logmsgbot: catrope synchronized php-1.5/includes/DefaultSettings.php 'r65022'
* 12:57 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialUpload.php 'r65022'
* 12:57 logmsgbot: catrope synchronized php-1.5/skins/common/upload.js 'r65022'
* 11:14 logmsgbot: catrope synchronized php-1.5/skins/Vector.php 'r65016'
* 11:14 logmsgbot: catrope synchronized php-1.5/skins/MonoBook.php 'r65016'
== April 12 ==
* 22:31 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22630 BlockDisablesLogin enabled on arbcom-nl'
* 20:46 atglenn: reboot hangs on dataset1 (probably due to same lock issue on xfs we see in logs), power cycle will have to be done from the dc tomorrow
* 20:13 atglenn: removed /data from fstab on dataset1, rebooting
* 18:20 Rob: sq49 shiny
* 17:52 Rob: sq50 shiny
* 17:25 Rob: updated squid upload-setttings removing the specific server settings of sq49 & sq50, defaulting those hosts back to pmtpa setttings
* 17:07 Rob: sq48 shiny, sq50 reinstall
* 16:44 Rob: sq47 shiny, sq49 reinstalling
* 16:01 Rob: sq45 shiny, sq46 reinstalling
* 15:46 Rob: sq44 shiny, sq46 reinstalling
* 15:25 Rob: sq43 shiny, sq45 reinstalling
* 15:05 Rob: sq42 back online, sq44 reinstalling
* 14:33 Rob: sq43 reinstalling
* 14:02 Rob: sq42 being reinstalled
== April 11 ==
* 22:33 domas: s/DRAC/ILOM/
* 22:32 domas: setting up new DB machine DRAC with following configuration: http://p.defau.lt/?azoQvnt25YpR4Sa7hIwXOg
* 20:20 RoanKattouw: Fixing incomplete rename on frwiktionary, CaBot -> CaBot Senior, 4641 edits
* 20:11 mark: Unshut e8/1 on csw5-pmtpa
* 20:05 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22625 - reviving deleted projects, restored to status "closed"'
* 20:05 logmsgbot: jeluf synchronized all.dblist 
* 19:56 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22625 - reviving deleted projects, restored to status "closed"'
* 19:53 logmsgbot: jeluf synchronized all.dblist 
* 19:52 logmsgbot: jeluf synchronized all.dblist 
* 19:49 logmsgbot: jeluf synchronized all.dblist 
* 18:48 mark: Restored DNS scenario to normal: European traffic back to esams
* 18:45 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix, reports of empty CSS files'
* 18:07 mark: For reference: cable SMF00292 can be cleaned up (MPLS link)
* 15:44 mark: Switched traffic back to Florida ;-)
* 15:44 river: shut e8/1 HGTN on csw5-pmtpa because it was saturated
* 15:44 mark: River switched traffic back to Europe due to port congestion problem
* 15:14 mark: Moving all European traffic to Florida for a core router reload
* 15:14 mark: Moving all 
* 12:18 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r64929/r64930 LanguageConverter fixes'
* 12:18 logmsgbot: catrope synchronized php-1.5/includes/parser/Parser.php 'r64929/r64930 LanguageConverter fixes'
* 12:18 logmsgbot: catrope synchronized php-1.5/includes/Article.php 'r64929/r64930 LanguageConverter fixes'
* 12:18 logmsgbot: catrope synchronized php-1.5/languages/Language.php 'r64929/r64930 LanguageConverter fixes'
* 12:18 logmsgbot: catrope synchronized php-1.5/languages/LanguageConverter.php 'r64929/r64930 LanguageConverter fixes'
* 10:33 mark: Enabled RSTP on Toolserver vlans 102, 300, 301
* 10:24 mark: Implemented BFD between br1-knams and csw2-esams
* 09:40 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialAllmessages.php 'r64926 Fix Special:Allmessages sorting'
* 09:29 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Add researcher group on enwiki'
* 07:56 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23143 - Enable flagged revisions on sr: wikinews'
* 07:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23143 - Enable flagged revisions on sr: wikinews'
* 07:55 logmsgbot: jeluf synchronized flaggedrevs.dblist 
* 03:18 Tim: restarted morebots
* 03:17 Tim: running cleanupMl.php in a screen on fenari, to fix the mess left by cleanupTitles.php
* 01:16 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 
* 01:12 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/api/ApiFeedLQTThreads.php 
* 01:09 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/pages/ThreadPermalinkView.php 
* 01:09 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Thread.php 
* 01:08 Andrew: Deployed trunk revision r64920
* 01:08 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 
* 00:22 Tim: running cleanupTitles.php on all wiktionaries
== April 10 ==
* 22:28 mark: Enabled Graceful Routing Engine Switchover (GRES) on csw2-esams
* 20:42 mark: Implemented IPv6 gateway failover using IPv6 DAD (Duplicate Address Detection)
* 20:23 mark: Implemented VRRP between csw1-esams and csw2-esams for VLANs 100, 101, 102 and 301
* 17:34 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/SpecialGlobalUsage.php 'r64890'
* 15:20 RoanKattouw: Deleting about 10k redundant redirects on mlwiki using deleteBatch.php
* 14:29 logmsgbot: catrope synchronized php-1.5/includes/EditPage.php 'r64882 for bug 23139'
* 12:52 mark: Set up BGP prefix advertisements through summarization with aggregate addresses, instead of iBGP announcement of the summary addresses of AS43821
* 12:11 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/PrefSwitch/PrefSwitch.i18n.php 'r64869'
* 11:32 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 11:31 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r64864'
* 11:31 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r64864'
* 11:13 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialEmailuser.php 'r64863'
* 10:09 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialEmailuser.php 'r64861 Shut up fatal'
* 09:53 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 23128 Rename Project namespace on outreachwiki'
* 03:43 logmsgbot: tstarling synchronized php-1.5/includes/specials/SpecialContributions.php 
* 03:27 logmsgbot: tstarling synchronized php-1.5/includes/specials/SpecialUpload.php 'r64847: fix $1 in uploadtext'
* 02:44 logmsgbot: tstarling synchronized php-1.5/includes/specials/SpecialContributions.php 'r64845, diff/hist link swap'
* 01:27 Tim: running cleanupTitles.php on ml wikis (starting with ml.wiktionary)
== April 9 ==
* 22:35 logmsgbot: catrope synchronized php-1.5/includes/parser/Parser.php 'r64839 Language converter bugfix'
* 22:32 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Set initialCaptial to true for Commons ForeignFileRepo setups. Chad made me do it'
* 22:18 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialUserlogin.php 'r64838 Fix fatal'
* 20:46 Rob: sockpuppet back online and shiny
* 20:40 Rob: tired of seeing pls reboot msgs on sockpuppet, rebooting it
* 20:39 Rob: byebye 412 days of uptime on sq41, coming down for OS reinstallation
* 19:53 logmsgbot: catrope synchronized php-1.5/includes/parser/Parser.php 'r64824 Recover lost namespace names'
* 19:31 logmsgbot: catrope synchronized php-1.5/includes/parser/Parser.php 'r64821 Language converter bugfix'
* 19:00 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#20045 - Logo change for pih.wikipedia'
* 18:16 logmsgbot: catrope synchronized php-1.5/includes/parser/Parser.php 'r64815 Language converter bug'
* 17:15 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug23118 Grant Bureaucrat all rights removal on outreach wiki'
* 16:52 logmsgbot: catrope synchronized php-1.5/includes/api/ApiRollback.php 'Live hack for bug 23117'
* 15:59 logmsgbot: jeluf synchronized php-1.5/wmf-config/abusefilter.php '23099 - Enable EditFilter on Spanish Wikipedia'
* 15:47 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23099 - Enable EditFilter on Spanish Wikipedia'
* 14:45 logmsgbot: catrope synchronized php-1.5/includes/diff/DifferenceInterface.php 'r64803 for bug 23113'
* 14:20 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesPt.php 'r64801 Restore name of User namespace in pt'
* 12:45 logmsgbot: tstarling synchronized php-1.5/includes/Title.php 'revert'
* 12:43 logmsgbot: tstarling synchronized php-1.5/includes/Title.php 'r64793'
* 12:42 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesPt.php 'r64791 restore pt specialpage aliases to 1.16wmf3 state'
* 12:17 RoanKattouw: Creating tables for FlaggedRevs on bswiki, were missing and causing DB errors
* 12:16 Tim: added /sr variant alias, apparently this is the default now on srwiki and the lack of it was causing a redirect loop
* 12:07 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEs.php 'r64787 misspelled alias'
* 11:55 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEs.php 'r64784: Restore ORDERNAR: alias'
* 10:39 logmsgbot: catrope synchronized php-1.5/includes/specials/SpecialUserlogin.php 'r64783'
* 09:43 logmsgbot: tstarling synchronized php-1.5/languages/messages/MessagesMl.php 
* 09:35 RoanKattouw: Touched MessagesEn.php on srv252 to shut up "not a valid magic thingie" errors
* 09:34 logmsgbot: catrope synchronized php-1.5/includes/api/ApiBase.php 'Deploy r64778 to shut up PHP warning'
* 07:33 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'all repooled'
* 07:20 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 07:19 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'read/write'
* 07:18 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'depooling servers still lagged'
* 06:55 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 
* 06:54 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'all wikis read-only'
* 06:48 Tim: preloading centralauth.globaluser into RAM on s3 slave servers
* 06:37 logmsgbot: tstarling synchronized php-1.5/wmf-config/ExtensionMessages.php 
* 06:36 logmsgbot: tstarling synchronized php-1.5/wmf-config/ExtensionMessages.php 
* 06:33 logmsgbot: root synchronizing Wikimedia installation...
* 06:33 logmsgbot: root synchronizing Wikimedia installation...
* 06:33 Tim: deploying MW 1.16wmf4
* 05:33 Tim: svn switch svn+ssh://svn.wikimedia.org/svnroot/mediawiki/branches/wmf/1.16wmf4
* 05:05 JeLuF: 15357 - Redirect dk.wiktionary and dk.wikibooks to da.wiktionary and da.wikibooks respectively
* 04:53 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#11766 - Install Labeled Section Transclusion extension on fr.wikipedia'
* 00:01 Tim: pilot updated for r64769
== April 8 ==
* 23:50 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php 
* 23:50 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Thread.php 
* 23:49 Andrew: Deploying r64767
* 23:46 Tim: on kaulen: svn up 1.16wmf4
* 23:38 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/specialpages/Stabilization_body.php 'deployed r64765'
* 23:06 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Thread.php 
* 23:06 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 
* 23:05 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Thread.php 
* 23:05 Andrew: Deploying r64762
* 22:16 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesAce.php 'r64760'
* 20:14 mark: Set up p2p link (vlan) between csw1-esams and csw2-esams over the 2x10G trunk and configured OSPF
* 19:41 mark: Made csw1-esams and br1-knams originate default routes (v4/v6) in OSPF
* 19:39 logmsgbot: aaron synchronizing Wikimedia installation...
* 19:39 AaronSchulz: deployed r64754 (FlaggedRevs_alpha)
* 19:34 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#23038 - Set sitename of Belorussian Wiktionary'
* 19:24 atglenn: missing.php update was copy paste from meta in accordance with bug 11125
* 19:23 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22974 - Enable FlaggedRevs on bswiki'
* 19:23 logmsgbot: jeluf synchronized flaggedrevs.dblist '#22974 - Enable FlaggedRevs on bswiki'
* 19:23 logmsgbot: ariel synchronized php-1.5/wmf-config/missing.php 
* 18:22 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '23077 - Extension:Collection for English Wikiversity'
* 18:19 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php '23081 - Remove "makesysop" right from bureaucrats and stewards groups'
* 17:42 mark: Implemented OSPFv3 on AS43821
* 17:00 Rob: db31-db40 racked and lom setup (except db34 which has an unresponsive bmc fresh from factory.
* 16:10 mark: Rebooted br1-knams after it got confused about OSPF area 0 vs 0.0.0.0
* 15:10 mark: Changed OSPF reference bandwidth from 100 Mbps to 100 Gbps on all AS43821 routers
* 06:25 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 
* 06:25 river: removing lomaria to dump s5 for TS
* 06:22 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 
* 06:22 river: removing db30 to dump s2 for TS
* 05:53 The_Thing: Can I test this?
* 02:59 logmsgbot: andrew synchronized php-1.5/includes/media/Bitmap.php 
* 02:58 logmsgbot: andrew synchronized php-1.5/includes/DefaultSettings.php 
* 02:58 Andrew: Seems to work correctly, deploying fix to all wikis.
* 02:55 Andrew: Deploying r64727 on testwiki for testing GIF scaling fixes.
* 01:43 Tim: on kaulen: reset the root mysql password and saved it in /root/.my.cnf for future reference
* 01:05 Tim: installing PHP on kaulen and various MW dependencies
== April 7 ==
* 21:56 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 21:55 Andrew: Sticking LiquidThreads on officewiki, by Erik's request.
* 20:52 Fred: ganglia shut down for a couple minutes while I grow the tmpfs.
* 18:13 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Change PrefStats tracking for wikis with Vector/toolbar enabled by default'
* 08:54 logmsgbot: catrope synchronized php-1.5/includes/api/ApiLogin.php 'r64695'
* 02:47 logmsgbot: tstarling synchronizing Wikimedia installation...
* 00:16 logmsgbot: tstarling synchronizing Wikimedia installation...
* 00:16 Tim: deploying r64679, CSRF fix, see bug 23076
== April 6 ==
* 22:44 mark: Implemented OSPF on AS43821
* 16:22 mark: Disabled the automatic purging of Commons pages
* 16:00 apergos: turned off profiling on db22, removed stuff in samples, disk was full
* 08:17 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Set 220px as default thumbnail size on Commons'
* 08:16 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Add 220px thumbnail size option for Commons'
* 07:43 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable Vector, toolbar by default on Commons'
* 07:43 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enable dialogs too when $wmgUsabilityEnforce set'
* 07:36 logmsgbot: catrope synchronized php-1.5/wmf-config/ExtensionMessages.php 'Added PrefSwitch'
* 07:29 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable PrefSwitch on test'
* 07:29 mark: Removed ldap from nsswitch.conf on zwinger and browne
* 07:24 mark: With srv1 dead, we're without an LDAP server. Removed zwinger from the mediawiki-installation node group
* 07:24 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix, add conditional code for PrefSwitch'
* 07:22 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r64652'
* 07:18 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 
* 07:13 mark: RIP: srv1. It reached the respectable uptime age of 1447 days. It will be sorely missed, especially by its twin brother, srv2.
* 07:06 mark: Started apache on srv240
* 07:03 Andrew: Running ~andrew/populateGIFMetadata.php on all wikis in a screen on fenari. Feel free to kill it if it causes any problems.
* 07:01 mark: Restarted apache on srv228, srv235, srv241, 
* 06:57 mark: Started apache on srv252
* 06:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/images/wikiEditor/toolbar/button-sprite.png 'Forgotten file'
* 06:51 mark: Restarted apache on srv158
* 06:48 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump CSS style version'
* 06:47 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 06:46 mark: Set up purging of all commons hits, removing duplicates using 'dampen.py' in locke:/root
* 06:45 logmsgbot: catrope synchronizing Wikimedia installation...
* 06:45 RoanKattouw: Scapping to deploy new version of UsabilityInitiative extension
* 04:39 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'GIF scaling working correctly on testwiki, pushing to other wikis again'
* 04:20 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Turn GIF thumbnailing off except on testwiki for further investigation'
* 04:16 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 
* 04:15 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#19831 - Add an icon to Wikinews for iPhone's & iPod Touch's'
* 04:13 Andrew: Trying re-activating GIF scaling for images in the limits
* 04:11 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 04:10 Andrew: setting $wgMaxAnimatedGifArea back to its default to prevent animated GIFs from being scaled as single frames.
== April 5 ==
* 22:01 Fred: modified site.pp to change ganglia squids aggregators.
* 21:10 mark: Changed purge sample rate to 1/10
* 21:06 tomaszf: lowering xml snapshot workers count down to three to prepare for storage server move
* 20:56 mark: Set up automatic purging of 1 out of every 100 Commons requests, to test for the migration tomorrow
* 20:44 tomaszf: removing the 20091128 xml snapshot dataset to free up space.
* 18:18 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#23050 - Change ab.wikipedia site logo'
== April 3 ==
* 19:59 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'null edit, to update langlist cache'
* 19:58 logmsgbot: jeluf synchronized langlist 'remove tokipona'
* 19:23 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22912 - Change sitename, add namespace alias on ukwikisource'
* 19:05 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#21068 - Some wikis have missing logo images on secure server'
* 18:40 logmsgbot: jeluf synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache for new wikitech entry'
* 18:24 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22994 - Logo switch on vi.wikiquote'
* 18:11 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#23032 - Enable NewUserMessage on ro.wikinews.org'
== April 2 ==
* 21:05 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 
* 21:05 river: repooled db5
* 20:34 mark: connected scs1-esams serial console to knsq16's com1
* 16:59 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 
* 16:59 river: repooled db1
* 16:26 river: depooling db5 to dump s4
* 16:26 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 
* 09:49 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 
* 09:49 river: repooled db27
* 09:02 river: removing db1 to dump s6 for TS
* 09:02 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 
== April 1 ==
* 22:02 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 22:00 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins/jquery.wikiEditor.html 'r64505'
* 21:30 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r64464'
* 21:27 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r64464'
* 20:33 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 
* 20:33 river: removing db27 from s3 for TS dump
* 20:08 AaronSchulz: deployed r64498 for FlaggedRevs_alpha
* 20:07 logmsgbot: aaron synchronizing Wikimedia installation...
* 18:25 IShadowed: DOMAS OMG WIKI DOWN
== March 31 ==
* 21:16 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Don't totally disable collapsiblenav, keep it as a hidden pref'
* 20:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 20:30 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'r64455'
* 20:30 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'r64455'
* 20:29 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r64455'
* 20:29 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r64455'
* 19:55 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Disable collapsiblenav explicitly in wake of usability deployment'
* 18:19 RoanKattouw: Manually corrected time on prototype, was 6 hours behind. Guess we need NTP there too
* 16:56 Rob: rebooted srv119
* 16:31 Rob: srv89 had temp warnings and failure to respond to console, replaced two bad fans in chassis with fans from srv127
* 16:22 Rob: srv98 was unresponsive to console, rebooted and appears to be back online and working just fine for now.
* 16:20 Rob: srv127 decomissioned. dead hard disks, dead fans, temp warnings, no lom, and out of warranty. will be parted out to other servers
* 16:08 Rob: srv127 throwing both hard disk errors and temp warnings
* 15:00 Rob: db28 is not in any current use to to memory issue, investigating memory issue now for hardware warranty replacement
* 05:33 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Tim says this helps'
* 05:02 logmsgbot: jeluf ran sync-common-all 
* 04:59 logmsgbot: jeluf ran sync-common-all 
* 00:50 Tim: increased apache TimeOut setting on singer from 5 to 30 seconds, to hopefully fix multiple reports of high error rates on secure.wikimedia.org
== March 30 ==
* 20:18 Rob: new server gilman racked for future secure gateway use and dns updated for server fqdn
* 20:17 Rob: back to work morebots
* 17:22 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22650 - bureaucrats should only be allowed to remove abusefilter and bot, not the other stuff'
* 09:21 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version'
* 09:20 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/lqt.css 'Deploy r64384'
* 09:20 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.css 'Deploy r64384'
* 05:28 logmsgbot: jeluf ran sync-common-all 
* 04:39 logmsgbot: tstarling synchronized php-1.5/extensions/InputBox/InputBox.classes.php 
* 04:29 Tim: removed the ex-fedora instances from srv122, srv124, srv142 from ms1
* 00:29 Tim: copying half the data back from ms1 to ms2 so I can have a backup, with half the data on each server
== March 29 ==
* 23:57 Tim: deleting obsolete ex-fedora instances from ms3 and ms2
* 23:50 Tim: shutting down all ex-fedora mysql instances except cluster1/2
* 20:32 atglenn: removed dead upload5 mounts from hume, fenari, snapshot1, should be the last traces 
* 19:50 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 19:50 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r64357'
* 14:29 RoanKattouw: prototype.wikimedia.org VM seems to have crashed, prototype down
== March 28 ==
* 21:25 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Add a new blocked email address'
* 18:49 RoanKattouw: Ganglia monitoring of Flordia Squids broken, reports all boxes down
* 16:33 apergos: upped maxclients and turned keepalive back on on singer (these were changed last week during the site outage) 
== March 27 ==
* 18:21 JeLuF: upgraded ViewVC to 1.1.4
* 07:51 JeLuF: roled out new apache config with a catch-all for www.*.(wikisource|wiktionary|wikiversity).org and www.(meta|commons).wikimedia.org, see bug 1698
* 00:47 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 
* 00:42 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'cleaned up testwiki config'
* 00:36 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed redundant configs'
* 00:26 AaronSchulz: added plwikisource flaggedrevs tables
== March 26 ==
* 21:46 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22919 - Enable FlaggedRevs for pl.wikisource'
* 21:45 logmsgbot: jeluf synchronized flaggedrevs.dblist 
* 21:44 logmsgbot: jeluf synchronized php-1.5/wmf-config/flaggedrevs.php 
* 21:41 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 21:41 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins/jquery.wikiEditor.html 'r64223'
* 21:39 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22965 - wgUploadNavigationUrl for nl.wikipedia'
* 20:40 Fred: restarted DNS server on ns2
* 20:34 Rob: added dns support for new misc server tarin
* 19:44 Rob: ignore nagioswarnings for sq11-sq30, they are being decommissioned
* 18:49 Fred: removed srv217 from pybal (api) since the server is not ssh-accessible.
* 18:48 Fred: removed srv89 from pybal since the server is not ssh-accessible.
* 18:44 Rob: removing sq11-sq30 from nagios and node lists
* 18:41 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 13:14 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 13:13 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r64216'
* 13:10 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'More fixes'
* 12:59 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r64215'
* 07:34 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionListView.php 'more path'
== March 25 ==
* 23:10 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bump style version appendix'
* 23:08 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins/jquery.wikiEditor.html 'r64195'
* 23:08 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'config for r63650'
* 21:54 atglenn: added nimish to wmf-deployment committers for current usability testing sessions
* 20:28 JeLuF: commons and meta now also use mediawiki-based robots.txt generation, based on robots.php (#22040)
* 20:23 AaronSchulz: deployed r64173 (flaggedrevs_alpha)
* 20:00 JeLuF: set up svn in /home/wikipedia/conf/httpd
* 17:36 Rob: copied meta favicon to outreachwiki
* 13:35 Rob: pdf2 become unaccessible, required reboot
* 12:07 RoanKattouw: Deployed r64154 around 11:30 UTC
* 12:07 RoanKattouw: Andrew restarted morebots, was down
== March 24 ==
* 23:11 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 23:08 RoanKattouw: Running scap to deploy UsabilityInitiative updates
* 22:10 Tim: moved mayflower:/home/mwdocs/phase3 to /var/mwdocs to reduce space usage in /home. Updated apache conf and crontab.
* 22:07 mark: Sending European traffic back to esams
* 22:01 Rob: sq46 was unresponsive and rebooted, now is working fine
* 21:30 Rob: secure.wikimedia.org back online
* 20:47 mark: Renamed rr.wikimedia.org to text.wikimedia.org
* 19:38 mark: Brought esams Sun V20z servers back up - restored rr.esams and upload.esams LVS
* 19:24 mark: Bringing lily back up
* 19:07 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22924 - Enable the Collection extension at el.wikipedia'
* 18:40 logmsgbot: jeluf synchronized live-1.5/robots.php 
* 18:39 domas: disabled keepalive on singer
* 18:37 river: delegated 1.24.10.in-addr.arpa to ha-dns-auth.toolserver.org
* 18:17 Rob: techdetail: lowered maxclients on singer apache2 and disabled secure site in apache configuration, do not rollback changes to normal until after esams is back online
* 18:16 Rob: techlbog back up, secure is NOT up, and is not expected to come back up, until the esams outage is resolved
* 18:01 Rob: disabled secure on singer, as everyone trying to use it overloaded both secure and blogs.
* 18:01 Rob: Secure is offline
* 17:51 mark: http://techblog.wikimedia.org/2010/03/global-outage-cooling-failure-and-dns/
* 17:50 mark: Reduced Negative TTL setting in SOA record of wikimedia.org from 1H to 10M
* 17:40 domas: NXDOMAINs with 1h TTL served on all records for short moment
* 17:40 Rob: globally (robs bad)
* 17:40 Rob: pending techblog post regarding the issue with accessibility in EU
* 17:25 mark: DNS scenario dir esams-down was empty, all zone files missing. Fixed.
* 17:18 mark: esams down due to overheating issues; moving traffic to Florida
* 14:18 mark: csw2-oe12-esams connected to iris's serial console (temp)
* 04:24 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 
* 04:21 Andrew: Running scap to update LiquidThreads beta
* 04:17 Andrew: Preparing to update LiquidThreads beta (production) to alpha version.
== March 23 ==
* 22:16 apergos: latest patches and reboot for ms5
* 22:13 Rob: sq40 back online, sq41 reinstalling
* 21:50 Rob: reinstalling sq40 & sq41
* 21:27 Rob: sq38 & sq39 reinstalled and online
* 20:43 Rob: taking down sq38 & sq39 for reinstallation
* 20:43 Rob: sq36 & sq37 back online and in rotation
* 19:39 Rob: sq36 & sq37 being reinstalled
* 19:35 logmsgbot: jeluf synchronized live-1.5/robots.php 'change charset to utf8'
* 19:32 Rob: sq34 and sq35 (not 25)
* 19:32 Rob: sq34 and sq25 back online and in rotation
* 19:27 logmsgbot: jeluf synchronized all.dblist 
* 19:24 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.hooks.php 'r64086'
* 19:24 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.php 'r64086'
* 19:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Add vector-collapsiblenav as a hiddenpref'
* 19:21 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'removed more group permission for closed wikis'
* 18:34 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove all simplewikibooks configs from Initialisesettings'
* 18:32 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'REVERT Test case for simplewikibooks'
* 18:29 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'Test case for simplewikibooks'
* 18:28 Rob: sq34 & sq35 coming down for reinstallation
* 18:23 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'REVERT Test case for simplewikibooks'
* 18:22 logmsgbot: jeluf synchronized php-1.5/wmf-config/CommonSettings.php 'Test case for simplewikibooks'
* 18:19 Rob: sq33 is reinstalled and doing quite fine. 
* 16:02 Rob: puppet stopped on sq34-sq50, DO NOT RESTART PUPPET. These hosts need reinstallation before puppet can be restarted, or squid changes can be deployed to them.
* 15:52 Rob: changed the setting for pmtpa to change partitioning schema and up the cache size. Do not deploy old squids with this information until Rob finishes reinstalling them
* 15:52 Rob: changed squid config in text squids for hard disks update on reinstalls
* 12:01 logmsgbot: jeluf ran sync-common-all 
* 11:59 logmsgbot: jeluf synchronized closed.dblist 
== March 22 ==
* 21:02 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#21213 - logo for ace wikipedia'
* 18:16 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22657 updating logo'
== March 21 ==
* 20:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22882 - Add a patroller group to fr.wikisource'
* 20:51 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22454 - Patroller and autopatrolled flags on es.wikipedia'
* 16:48 domas: db28 has more memory errors: http://p.defau.lt/?IvvVKaNdi05v1yfwLiHWVw
* 16:44 domas: dist-upgraded db28 to karmic 
* 12:14 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'bgwiki group permissions fixed'
* 11:37 logmsgbot: jeluf synchronized closed.dblist 
* 11:04 JeLuF: removed tokipona wikis from all.dblist
* 11:02 logmsgbot: jeluf ran sync-common-all 
* 10:25 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22446 - Change the WT namespace to an alias for Wiktionary, for Swedish Wiktionary'
* 10:03 logmsgbot: jeluf ran sync-common-all 
* 09:53 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'UNDO: 22446 - Change the WT namespace to an alias for Wiktionary, for Swedish Wiktionary - pages still exist in these namespaces'
* 09:40 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#16857 - Request to switch over logo for sanskrit language wikipedia localisation'
* 09:34 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22283 - Add "Author" namespace to Telugu Wikisource'
* 09:26 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22329 - Redirect WV to Wikiversity space for English Wikiversity'
* 09:22 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22446 - Change the WT namespace to an alias for Wiktionary, for Swedish Wiktionary'
* 09:13 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22481 - Namespace alias at mwl.wikipedia.org'
* 09:06 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '22481 - Namespace alias at mwl.wikipedia.org'
* 06:57 JeLuF: bugzilla.wikimedia.org
* 06:57 JeLuF: Changed kaulen config to include ServerAlias bugzilla.wikipedia.org for
== March 20 ==
* 21:21 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed sysop access to stabilization except on wikis already have config rows. This page should not be enabled by default.'
* 19:49 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22549 - Enable email on watchlist change on fr.WV'
* 19:42 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22699 - Autopatroler at cs.wikipedia'
* 18:17 logmsgbot: jeluf synchronized php-1.5/wmf-config/mc.php 'srv98 down'
* 18:13 logmsgbot: jeluf synchronized php-1.5/wmf-config/mc.php 'srv119 down'
* 14:53 apergos: tossed a few files from tridge to make space, a very short-term solution
* 14:13 JeLuF: Bug 22681: migrated users on bgwiki dawiki enwiktionary hewiki hrwiki itwiktionary from groups autopatrol and autopatroller to group autopatrolled
* 14:12 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22681 - Clean up/unify autopatrol/autopatrolled/autopatroller group names'
* 13:38 JeLuF: created Italian planet at planet.it.wikimedia.org
* 12:57 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22909 revert'
* 12:15 JeLuF: restarted morebots
* 12:15 JeLuF: added it.planet.wikimedia.org to DNS
* 12:15 JeLuF: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22669 - Logo change for az.wikiquote.org'
* 12:15 JeLuF: jeluf synchronized php-1.5/wmf-config/abusefilter.php
* 12:15 JeLuF: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22909 Liquidthreads for huwiki'
* 12:15 JeLuF: jeluf synchronized php-1.5/languages/messages/MessagesKrc.php 'MessagesKrc from trunk, for bug:22657'
== March 19 ==
* 17:53 logmsgbot: tfinc synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'iframe display bugfix'
* 17:53 logmsgbot: tfinc synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.js 'iframe display bugfix'
* 17:52 logmsgbot: tfinc synchronized php-1.5/extensions/UsabilityInitiative/js/plugins/jquery.wikiEditor.js 'iframe display bugfix'
* 17:44 Fred: getting BUFFER IO ERRORS on srv169. JFS refuses to fix itself. 
* 17:39 Fred: srv169 is having problems with sda6 (/a)... working on it and trying to salvage.
* 07:36 logmsgbot: tstarling synchronized php-1.5/includes/Revision.php 'bug 22624 hack'
* 07:05 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 're-adding cluster1+2 for bug 22624'
== March 18 ==
* 23:36 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/language/FlaggedRevs.i18n.php 'deployed r63938'
* 23:35 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedArticleView.php 'deployed r63938'
* 22:27 logmsgbot: root ran sync-common-all 
* 21:11 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedArticleView.php 'deployed r63929'
* 20:34 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'update flaggedrevs_labs config'
* 20:31 AaronSchulz: deployed r63927 (flaggedrevs_alpha)
* 16:14 Fred: security updates of apaches in progress
* 06:22 logmsgbot: tstarling synchronized php-1.5/includes/StubObject.php 'reverted profiling hack, I assume whatever information you wanted to get out of this, you've got now'
* 06:19 logmsgbot: tstarling synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics.css 'revert whitespace-only change'
* 05:57 logmsgbot: tstarling synchronized php-1.5/extensions/CentralNotice/SpecialNoticeText.php 'revert whitespace-only change'
* 05:55 logmsgbot: tstarling synchronized php-1.5/extensions/LiquidThreads_alpha/pages/ThreadPermalinkView.php 'reverted debugging hack'
* 00:33 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Adding override for strategy and usability wiki for wmgNoticeProject'
* 00:05 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Changing mapping of commosn under wmgNoticeProject to just the string commons'
* 00:03 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Commenting out commons from wmgNoticeProject to fix loading'
== March 17 ==
* 22:54 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump style version'
* 22:51 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 22:27 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r63888'
* 22:20 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r63887'
* 22:02 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/OptIn/OptIn.i18n.php 'r63885'
* 22:00 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bump style version appendix'
* 22:00 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r63883'
* 21:41 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 21005'
* 21:30 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix for usability deployment'
* 21:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enable experimental Usability features but add them to $wgHiddenPrefs'
* 21:08 RoanKattouw: Running scap
* 21:05 RoanKattouw: Updating UsabilityInitiative to trunk state cluster-wide
* 20:39 RoanKattouw: Pushing UsabilityInitiative software update to testwiki
== March 16 ==
* 20:55 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'removed ex-fedora clusters, should all be unreferenced now due to RCT completion'
* 19:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 22650 - Enable the AbuseFilter on the French Wikipedia'
* 19:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/abusefilter.php ' Bug 22650 - Enable the AbuseFilter on the French Wikipedia'
* 17:19 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed redundant tab settings'
* 08:05 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version'
* 08:05 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js 'Reactivate LiquidThreads live preview'
* 05:01 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22691 - some fixes on de_wikiquote'
== March 15 ==
* 23:49 Fred: upgraded puppet on ms7 to the recent version that doesn't have scary stupid bugs in it! 
* 21:58 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22663 - Import sources for frwiktionary'
* 21:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22694 - definition of import sources for mg.wiktionary'
* 21:34 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 22817 - Create Artikeldata namespace on dawiki'
* 21:31 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 22825 - Set $wgNamespacesWithSubpages[NS_MAIN] = true on sewikimedia'
* 21:25 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22833 - Enable multipage upload form for English Wikiversity'
* 21:01 logmsgbot: jeluf synchronized live-1.5/robots.php 
* 20:48 logmsgbot: jeluf synchronized docroot/meta/favicon.ico 
* 20:22 logmsgbot: jeluf synchronized docroot/incubator/favicon.ico 
* 20:08 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22648 - Add portal namespace to kw wikipedia'
* 20:03 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 22674 - Update Special:Import options at English Wikibooks'
* 19:55 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php ' Bug 22406 - Enable Extension:Quiz on zhwiki'
* 19:52 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22615 - Import sources for sv.wikibooks'
* 19:49 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22787 - TZ for Estonian projets'
* 19:43 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22831 - wgBlockAllowsUTEdit for kowiki'
* 19:39 logmsgbot: jeluf synchronized php-1.5/wmf-config/InitialiseSettings.php '#22827 - Logo change for pcdwiki'
* 17:53 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Resyncing because I suspect some boxes didn't pick this up'
* 17:00 hcatlin: [mobile] code push complete!
* 16:52 hcatlin: [mobile] pushing code update for adding ext, cz, mr, ca, bn, kl, mn, scn, si, and ta langauge codes
* 16:29 logmsgbot: robh synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22834'
* 15:51 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22473'
* 15:27 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22791'
== March 14 ==
* 04:52 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Make Vector and usability enhancements default on liquidthreads_labswikimedia'
* 04:46 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bumping style version'
* 04:43 Andrew: Scapping to deploy LiquidThreads alpha update
* 04:40 Andrew: Preparing for LiquidThreads alpha update to trunk state
== March 13 ==
* 18:42 Rob: sanger is back online and working
* 18:33 Rob: sanger moved, coming back online
* 18:21 Rob: thats sanger, not singer.
* 18:21 Rob: taking down singer to move it up in the rack, during this downtime (approx 15 min), wikimedia.org email will be unavailable.
* 18:20 Rob: mchenry is back online and dealing with its 10 minute mail backlog.
* 18:14 Rob: mchenry moved, bringing it back online
* 18:01 Rob: mchenry host coming down to move its rack location. Downtime will be approx. 15 minutes. (all mailing list relays may not work and thus new mailing list entries will back up until this move is completed.)
== March 12 ==
* 19:23 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling .odg extension for uploads'
* 18:49 Fred: moved stats.w.o from Zwinger to Spence
* 12:00 RoanKattouw: storage2 (download.wikimedia.org) was down between 11:40 and ~11:53 UTC, coincides with load spike
* 07:43 apergos: really edited the right copy of worker.py this time (the copy in /backups :-P)
* 06:19 apergos: editd worker.py in place on snapshot3 to remove --opt and replace all except lock options individually (it was running with that on db12 again, with high lag). shot that worker process, lag and thread count going down now
* 02:49 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 02:49 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'reducing load on db12 again, seems to be lagging again'
* 02:38 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'increased general load on db12'
* 02:34 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 02:32 Tim: db12 caught up, repooling
* 02:15 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'subpages for Grants namespace on meta (bug 22810)'
* 02:02 tomaszf: killing 20100312 xml snapshot run for enwiki due to unnecessary locking of pagelinks table on db12
* 01:43 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'moving db12 back into the "dump" query group so we don't kill db26 too'
* 01:41 Tim: note: mysqldump query for pagelinks table is running on db12, presumably that is the cause of the slowness and network spike
* 01:37 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 
* 01:32 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'brought db26 into rotation as enwiki watchlist/contribs server, to replace db12 after cache warming'
* 01:30 Tim: network spike on db12 at 01:00. Going to depool it in case it's going to try the same trick as last time
* 01:21 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'resyncing for bug 22810 (stupid ssh-agent :-P)'
* 00:47 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'Grants namespace on meta, bug 22810'
* 00:17 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Removing foundation wiki from wmgDonationInterface'
== March 11 ==
* 22:03 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22669 rollback'
* 22:02 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22669'
* 21:36 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22404'
* 21:19 logmsgbot: ariel synchronized php-1.5/wmf-config/InitialiseSettings.php 'subpages for templates on foundationwiki (#22484)'
* 20:37 atglenn: re-enabled replication on media server (ms7 -> ms8), we are back in business
* 16:43 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22089'
* 12:27 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version'
* 11:22 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Deploy r63591, fixes for LiquidThreads comment digestion'
* 11:21 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Deploy r63591, fixes for LiquidThreads comment digestion'
* 01:34 Tim: reinstalled wikimedia-task-appserver on fenari to fix symlinks /apache and /usr/local/apache/common
* 00:26 atglenn: playing catchup on replication now from ms7-ms8 zfs send/recvs running as root
== March 10 ==
* 22:59 atglenn: patched and rebooted ms8, working on snapshot issue
* 21:07 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/ContactPage.i18n.php 'r63575'
* 21:07 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/ContactPage.php 'r63575'
* 21:07 logmsgbot: catrope synchronized php-1.5/extensions/ContactPage/SpecialContact.php 'r63575'
* 18:54 Fred: removed duplicate ACL for 208.80.152.157 from text-squids
== March 9 ==
* 23:32 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Merge r63523 into LiquidThreads production'
* 23:26 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r63524 into LiquidThreads alpha'
* 23:26 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r63523 into LiquidThreads production'
* 23:24 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r63524 into LiquidThreads alpha'
* 23:24 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r63523 into LiquidThreads production'
* 23:15 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/View.php 'Merge r63154 into LiquidThreads production'
* 23:13 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/View.php 'Merge r63154 into LiquidThreads alpha'
* 20:22 rainman-sr: this morning's search api outage seems to be connected with index update on search8 which triggered high GC load and stuck threads, removing enwp spell index from search8
* 15:37 hcatlin: message before was referring to mobile site.
* 15:37 hcatlin: deployed new languages. cs homepages.
* 14:58 domas: forgot to log - yesterday's API fail was repeated today too, all API nodes were blocked on talking to lucene, and search cluster was idle too, and... 
* 14:54 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEt.php 'force recache to solve l10ncache issue'
* 14:51 logmsgbot: catrope synchronized php-1.5/includes/LocalisationCache.php 'r63417'
* 00:41 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevs.hooks.php 'deployed r63447'
* 00:40 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/FlaggedRevs.class.php 'deployed r63447'
== March 8 ==
* 23:59 logmsgbot: tstarling synchronized php-1.5/includes/Sanitizer.php 'r63426'
* 23:57 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 
* 23:52 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'removed labs config cruft'
* 23:32 AaronSchulz: deployed updates MFT to FlaggedRevs_alpha (r63444)
* 23:30 Fred: updated search nodefile to contain search[13..20]. 
* 23:10 AaronSchulz: finally got labs msgs working it seems
* 23:09 logmsgbot: aaron synchronized php-1.5/wmf-config/CommonSettings.php 'do not override flaggedrevs_alpha msg dirs for lab sites'
* 22:40 logmsgbot: tstarling synchronized php-1.5/thumb.php 'r63432'
* 22:04 logmsgbot: aaron synchronized php-1.5/extensions/FlaggedRevs_alpha/language/OldReviewedPages.i18n.php 
* 21:54 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r63423'
* 21:41 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 
* 21:38 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 21:36 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'disable LU on labs sites'
* 21:09 logmsgbot: catrope synchronized php-1.5/includes/LocalisationCache.php 'r63417'
* 20:57 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'revert, no effect on msg issue'
* 20:53 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'settings labs cache dir'
* 20:31 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'Switched labs sites to FlaggedRevs_alpha'
* 20:30 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'custom cache dir config for labs sites'
* 19:18 logmsgbot: tstarling synchronized wmf-deployment/cache/trusted-xff.cdb 
* 18:15 Fred: disabled regular http check for mobile1 (leaving the longer timeout one in place)
* 13:58 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-et.cache 'Sync file *after* touching it'
* 13:55 logmsgbot: catrope synchronized php-1.5/cache/l10n/l10nupdate-et.cache 'Sync LU cache file'
* 13:52 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEt.php 'Try again'
* 13:48 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesNds_nl.php 'Fix message that didn't get picked up'
* 13:46 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesEt.php 'See if this works'
* 13:20 RoanKattouw: Rebuilding all l10n caches to prep debugging a LocalisationUpdate issue
* 13:16 domas: something was wrong with sq31/sq32 - cpu/network shot through roof, can't correlate load
* 10:42 domas: *sigh*, squid conf: added highPrecedence to defaults, so it is picked up by configuration builder. also, checked in files into RCS (had to take over locks from mr.Starling, sorry!). unbroke test.wikipedia API
* 09:43 domas: where did test api,php fix go?! 
* 04:38 Fred: removed Bart's ACL in squid cache conf since it was a duplicate ACL. 
== March 6 ==
* 22:59 mark: Increased membufs on all backend squids
* 06:58 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabled FlaggedRevs_alpha due to outdated messages issues'
* 05:40 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php '$wgFlaggedRevsStylePath for labs'
* 05:35 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 
* 05:33 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Set labs site to use flaggedrevs_alpha'
* 05:26 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set labs site to use flaggedrevs_alpha'
* 05:10 logmsgbot: aaron ran sync-common-all 
* 00:18 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22728 - create new user group on Greek Wiktionary'
== March 5 ==
* 18:48 Fred: removed duplicate ACL from squid backend config. 
* 18:11 Rob: restarted npre daemon on ms1 since it wasnt transmitting to nagios.
* 00:51 logmsgbot: aaron ran sync-common-all 
* 00:23 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Restore Monobook for nl.wikimedia.org'
== March 4 ==
* 19:44 Tim: installed NRPE and RAID utils on ms1
* 19:40 Tim: removed db19 from the mysql node group and nagios since it is being used for varnish now
* 19:29 Tim: fixed prototype.wikimedia.org (nagios critical), was waiting 10s on every request due to memcached being down
== March 3 ==
* 16:16 apergos: reenables gmetric on ms8 a little bit ago, scrub finished with no errors (yeah right( 
* 06:06 apergos: replication off again tonight, zfs nicely borked on ms8, zfs destroy snapshot X gives message "cannot destroy X, dataset already exists" . time for sun contract and call them
* 05:53 apergos: gmetrics turned off again on ms8, running zpool scrub export 
* 05:40 apergos: reset SYs on ms8 worked (slowly). gmetrics reenabled. (and I thought morebots was just being a bad bot)
* 05:39 apergos: ( :-P ) starting catchup for the last day's lost replication ms7 -> ms8 
* 05:22 apergos: reboot failed, bootadm hung probably waiting on zfs. stop SYS claims to work but subsequent show SYS shows power state on.
* 05:14 apergos: rebootiing ms8 to clear zfs deadlock 
== March 2 ==
* 23:32 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/OptIn/SpecialOptIn.php 'r63191'
* 23:31 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/OptIn/OptIn.php 'r63191'
* 23:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Tweak OptIn settings'
* 22:36 RoanKattouw: Rebuilding localisation cache
* 20:15 Tim: on hume: started recompressTracked.php for all wikis, to fix any newly-tracked pointers from fixBug20757.php and to complete enwiki
* 03:48 atglenn: leaving replication and gmetrics turned off on ms8 for thenight, will go back to looking at scat and mdb output tomorrow. 
* 02:27 atglenn: turning off gmetric on ms8, the dfs are piling up behind a deadlocked zfs list
* 02:13 atglenn: turn of ms7 replication to ms8 temporarily to work out a zfs deadlock issue on ms8
== March 1 ==
* 12:53 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 
* 12:51 river: removing db17 from rotation for ts dump
* 12:51 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 
* 11:48 domas: recovered locke, ran log collection with per-process limit of 200MB
* 11:39 domas: locke was killed by awks using gigabytes of memory, mwahaha. 
== February 27 ==
* 02:09 Tim: rewrote domas's api.php fix and deployed my version
* 01:56 logmsgbot: midom synchronized php-1.5/includes/LocalisationCache.php 'removing profiling hooks'
* 01:42 domas: fixed api.php on test/srv124
== February 26 ==
* 23:40 Tim: disabled the temporary squid purge script, I'm assuming it's done its job now
* 19:43 logmsgbot: catrope synchronized php-1.5/includes/api/ApiBase.php 'r63014, r63015'
* 18:24 Fred: bugzilla now residing on Kaulen... Updating server roles. 
* 18:07 Fred: switching bugzilla cname to new server. (both servers are pointing to the same DB, so switch should be transparent to most)
* 17:28 logmsgbot: midom synchronized php-1.5/includes/LocalisationCache.php 'profiling'
* 17:26 logmsgbot: midom synchronized php-1.5/includes/diff/DifferenceInterface.php 'removing profiling hook'
* 17:23 logmsgbot: midom synchronized php-1.5/includes/StubObject.php 'tracehooks'
* 17:21 logmsgbot: midom synchronized php-1.5/includes/StubObject.php 'tracehooks'
* 14:38 RoanKattouw: logmsgbot is missing in action AGAIN, starting it as me on fenari
* 14:37 RoanKattouw: Synced r63008
* 12:17 domas: *sigh*, recovered locke after OOM, added few entries to limits.conf
== February 25 ==
* 23:48 Tim: running the new fixBug20757.php on all wikis
* 21:48 RoanKattouw: Removed empty line from the end of all.dblist, broke account creation ^^
* 20:18 RoanKattouw: Removed those closed wikis again, db lists back to normal
* 20:08 RoanKattouw: Temp adding some closed wikis to all.dblist to be able to dump them (again)
* 19:01 Rob: removed old servers will, bart, etc from DNS
* 18:32 Rob: pushing more dns changes for LOM on new servers.
* 00:59 Tim: made the squid purge script on locke also purge redirects from /w/index.php?, for bug 22639
* 00:32 Tim: installed ack-grep on fenari
== February 24 ==
* 20:48 Rob: IP conflicts on alsted/ersch/formey/kaulen. I must have typo'd something in one of their DRAC configurations because they are all messed up. will go into DC and physically review settings via console.
* 16:10 Rob: had formey and alsted ip info mixed up in dns, updating dns
* 13:13 RoanKattouw: Adding optin_survey table to wikis that didn't have it
* 01:38 atglenn: ganglia 3.1 packages up on ms7 and ms8 (see atg for solaris packages for other boxen)
* 01:20 domas: that was fun, let's do it again!
* 01:17 mark: Implemented udp2log filter on locke that pipes redirected URLs to a modified version of purgeList to clean the squid caches
* 00:59 Rob: added austin to the mortals user category for shell access on puppet, updating singer's account list
* 00:30 mark: Fixing wikipedia.be vhost ServerAlias setting in redirects.conf
* 00:14 domas: redefined tiertwo acl in squid to allow fenari/zwinger purges, purged special:random, restarted mobile1 stuff
== February 23 ==
* 23:41 domas: stopped nginx on en.m until we resolve special:random issue
* 20:53 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22628 - Logo of Chamorro Wikipedia'
* 17:23 Rob: drac setup on new servers ersch, formey, alsted, and kaulen. still need port allocation.
* 16:47 Rob: forgot to add mgmt, updating dns
* 16:43 Rob: updating dns for new servers ersch, formey, alsted, & kaulen
* 15:37 Rob: FINALLY got in the last RMA drive for storage1, arrays were toast (losing 1/2 the drives in a raid10 and having the unlucky chance of them being paired), reinstall in progress
* 12:50 RoanKattouw: Cleaned up my dblist voodoo, everything should be back to normal now
* 12:41 RoanKattouw: Fixed sync-dblist, was broken
* 12:31 RoanKattouw: Syncing pmtpa.dblist anyway, seems I have to
* 12:24 RoanKattouw: Temporarily adding wikis listed in bug 20325 to pmtpa.dblist so I can export their contents. Please don't sync this
== February 22 ==
* 23:40 Tim: restarted replication on ms2
* 23:39 Tim: ms1 is up with a full copy of ms3 data, now replicating cluster22/rc1
* 21:09 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Revert'
* 20:30 logmsgbot: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'Bug 22522 - Allow bureaucrats to remove sysop flag on Simple English Wikitionary'
* 20:09 Rob: finished the setup of srv217 (just needed puppet fixed and files updated)
* 20:02 logmsgbot: root ran sync-common-all 
* 20:01 logmsgbot: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Set vector as default skin on nlwikimedia'
* 19:44 logmsgbot: root ran sync-common-all 
* 18:10 Fred: ganglia is going to hiccup a little for the next few minutes while I restore data... "Don't panic"
* 07:53 Tim: rebooted srv206 via drac
* 07:48 logmsgbot: tstarling synchronized php-1.5/wmf-config/mc.php 'removing srv206, is down'
* 06:19 Tim: increased multicast TTL from 1 to 3 in gmond_template.erb, to fix ganglia on ms*
* 05:54 Tim: started ms2 -> ms1 copy using nc+tarpipe
* 05:30 Tim: shutting down mysql on ms2 for copy to ms1
* 05:09 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 'depooling ms2 for copy to ms1'
== February 21 ==
* 02:43 Fred: puppettized user gmetric
== February 20 ==
* 22:37 mark: Installed Ubuntu 8.04 on ms1 - now ready for ES deployment
== February 19 ==
* 18:48 Fred: removed memcached node srv193 and replaced with srv233
* 16:52 Rob: updated InitiliseSettings with bug 22585
* 01:31 Fred: puppetized gmond.conf generation using templates
== February 18 ==
* 23:51 RoanKattouw: Fixed search script on yongle, removed UA check exemption for yongle
* 23:51 RoanKattouw: logmsgbot is broken again
* 04:07 Tim: re-running storageTypeStats.php on enwiki to identify rows with old_flags='object,utf-8', which need to be handled properly in fixBug20757.php
== February 17 ==
* 17:25 Rob: pushed updates to wordpress installations
* 17:04 RoanKattouw: Synced wmf-config/checkers.php 'Allow missing UA from yongle'
* 17:04 RoanKattouw: Restarted logmsgbot
* 16:47 Fred: test
* 07:24 Fred: there is a problem with ganglia. It is being worked on.
* 00:15 RoanKattouw: Resynced srv151, was returning empty responses
== February 16 ==
* 23:31 Fred: upgrading gmond to 3.1.2 everywhere. However, due to the newish module structure, there is a potential that ganglia will hickup while puppet does its job...
* 22:03 logmsgbot_: mark synchronized php-1.5/wmf-config/checkers.php 'Exceptions'
* 20:30 Rob: srv127 is online, but not in LVS. Its cert was accepted on sockpuppet, but puppetd --test results in a cert failure on retrieval from sockpuppet.
* 20:17 Rob: rebooting srv127
* 20:15 mark: Increased membufs to 40 per COSS dir on the pmtpa upload backend squids
* 19:51 mark: Increased membufs per COSS dir from 10 to 20 on the new pmtpa squids
* 18:10 apergos: but documentation can save people precious time when things are on fire
* 18:02 mark: Documentation is not a substitution for thinking
* 16:44 mark: Fixed puppet on most servers
* 16:23 domas: anyone knows why mysqldump on snapshot3 is locking tables? maybe --single-transaction could work better?!!!?
* 16:22 RoanKattouw: Strike my last, I hear it'll fix itself in a day
* 16:21 RoanKattouw: Oops, meant to type srv187-189
* 16:21 RoanKattouw: Ganglia not picking up data for srv1987-189, 193, 214-218, 250-253, 257 even though the boxes are up and gmond is running; been like this for 3 days
* 15:21 mark: Removing all puppet certs/private keys on all machines
* 14:55 mark: Puppetmaster is screwed up by a wrong command that deleted all files under /var/lib/puppet
* 14:25 logmsgbot_: midom synchronized php-1.5/includes/diff/DifferenceInterface.php 'instrumenting costs by revision pairs'
* 13:46 logmsgbot_: root synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22444 and 22330 enabling collection on dawiki and ruwiki'
* 13:22 domas: oh wait, it does work, something else....
* 13:16 domas: apparently my squid rule for Apple browsers didn't really work, hence the continuing spikes
* 03:16 domas: blocked MacOSX Atom syndication on the edge, this will save terabytes of diskspace on unsuspecting Safari user computers :-)
* 02:13 logmsgbot_: tstarling synchronized php-1.5/wmf-config/checkers.php 
== February 15 ==
* 23:42 logmsgbot_: midom synchronized php-1.5/includes/StubObject.php 'uninstrumenting'
* 23:09 logmsgbot_: midom synchronized php-1.5/wmf-config/checkers.php 
* 23:09 domas: banned all UA-less requests
* 22:53 logmsgbot_: midom synchronized php-1.5/wmf-config/checkers.php 'un-disabling UA check on API'
* 22:14 RoanKattouw: Resynced srv114, was returning empty responses
* 22:10 logmsgbot_: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Test with 220px default thumb size on enwiki (bug #21117)'
* 22:10 logmsgbot_: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'Test with 220px default thumb size on enwiki (bug #21117)'
* 22:02 logmsgbot_: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 21:58 logmsgbot_: catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js 'r62553'
* 21:39 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiMain.php 'r62550'
* 21:30 logmsgbot_: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 'raised wgSearchSuggestCacheExpiry from 1200 to 86400'
* 21:27 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiOpenSearch.php 'r62548'
* 21:27 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiMain.php 'r62548'
* 21:27 logmsgbot_: catrope synchronized php-1.5/includes/DefaultSettings.php 'r62548'
* 20:54 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiBase.php 'r62546'
* 20:54 logmsgbot_: catrope synchronized php-1.5/api.php 'r62546'
* 20:53 domas: Roan and me are doing some perf engineering on opensearch & related stuff. \o/
* 20:40 logmsgbot_: midom synchronized php-1.5/includes/StubObject.php 'readding stub trace'
* 20:34 logmsgbot_: midom synchronized php-1.5/wmf-config/checkers.php 'perf comment action'
* 20:30 logmsgbot_: midom synchronized php-1.5/includes/StubObject.php 'removing trace hooks'
* 20:28 logmsgbot_: midom synchronized php-1.5/includes/StubObject.php 
* 20:24 logmsgbot_: catrope synchronized php-1.5/includes/api/ApiOpenSearch.php 'r62543'
* 19:32 mark: Allowing API requests to be sent to the regular app servers temporarily
* 19:27 mark: Restarted sq31 and sq31 API backend squids with slightly reduced memory settings
* 19:18 logmsgbot_: mark synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 17:51 Rob: set srv127 to false on lvs3, will re-enabled once in DC to reboot it manually
* 17:49 logmsgbot_: root synchronized php-1.5/wmf-config/mc.php 'srv127 down taking out of mc rotation'
* 16:02 RoanKattouw: Resynced srv134 to fix HTTP 500
* 15:13 RoanKattouw: Resynced srv232, was serving HTTP 500s
* 02:02 mark: Fixed stale PHP file removal apache cron job in Puppet
== February 14 ==
* 19:25 mark: srv206 spontaneously rebooted
* 17:12 mark: Added sq32 to the API backend squid group (with an empty cache)
* 16:38 mark: Confined all API requests to sq31 backend Squid
* 12:54 RoanKattouw: logmsgbot is broken
== February 13 ==
* 17:44 mark: Made 64 bit java exception for search13 in puppet
* 17:37 mark: Converted srv214-218 and srv86-87 into API servers
* 17:02 mark: Throwing all API requests at the new API app server cluster
* 16:49 mark: Converted srv187-srv189 and srv250-srv253 into API app servers
* 15:46 mark: Created new Ganglia group "API application servers", deploying with Puppet
* 15:31 mark: Created API app servers group in Puppet, deploying on srv254-257 to start with
* 15:24 mark: Created API LVS service on lvs3 and restarted PyBal
* 15:10 mark: Added DNS entry for api.svc.pmpta.wmnet (LVS ip)
* 14:18 mark: Enabled more verbose logging on all squids: debug message level 1 for all message categories
* 14:18 mark: Disabled cache digests on all squids
* 13:58 mark: Removing sq30 and lower from the Text CARP pool
* 13:51 mark: Reducing persistent_connection_timeout and request_timeout from 1 min to 20s on all frontend squids, to reduce FD usage
* 12:50 mark: Restarting knsq* frontend squids with FD limit raised from 32k to 64k
* 12:23 mark: Grown filesystem /dev/data/ES (/a) on ms3 by 90G, leaving another 98G free
* 00:17 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Sync CSS style versions'
* 00:16 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 00:10 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62403'
* 00:02 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r62400'
* 00:01 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62400'
== February 12 ==
* 23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Try deploying r62397 again'
* 23:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'r62397'
* 23:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62397'
* 23:22 mark: Deployed sq86
* 22:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 22:30 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62388'
* 21:12 mark: Implemented new routing policy for AS43821
* 21:01 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 20:59 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'r62382'
* 19:15 Rob: sq67 has both disks installed now (had to get one replaced) and has no OS installation yet. holding off on install, not sure if mark wants this up as a squid or for varnish development
* 18:22 atglenn: resstarting transfer of data (one chunk) from ms8->ms5->ms6, running in screen as root on all three hosts using nc
* 16:18 Rob: setup project1 and project2 boxes for flaggedrevsdevelopment and project mangement suite testing
* 15:12 mark: Upgraded our AMS-IX connection from 2x 1G to 10G along the way
* 15:12 mark: Reloaded br1-knams twice to fix a CAM partitioning problem; not sufficient IP next hops for 8-path routes
* 11:34 mark: Filtering all prefixes to/from AMS-IX peers
* 02:55 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 're-enabling WikiEditor'
* 02:04 Tim: installed openssh-server on bayes, Erik Z apparently uninstalled it with aptitude
* 01:37 logmsgbot: tstarling synchronized php-1.5/wmf-config/CommonSettings.php 'disabled WikiEditor due to complaints about bug 22428'
* 01:06 atglenn: shoveling over some data from ms8 to ms6 via ms5 :-/ 
* 00:48 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/lqt.js 'Disable live preview entirely, broken by CentralNotice and DismissableSiteNotice'
* 00:37 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable DismissableSiteNotice for lqt.labs, breaks LiquidThreads live preview'
* 00:17 logmsgbot: andrew synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version'
* 00:07 Andrew: Running scap
* 00:02 Andrew: Updating LiquidThreads alpha to trunk state, deploying with scap in a few minutes
== February 11 ==
* 22:06 Fred: implemented disk space check for tridge
* 21:10 mark: Pooled sq71-85 with full CARP weight (30)
* 21:06 mark: Pooled sq79-85 frontends in LVS with full load (30)
* 20:59 mark: Pooled sq79-85 frontends in LVS with load 1
* 20:51 mark: Pooling sq79-85 in CARP on all frontends, with low CARP weight 10 instead of 30
* 20:30 mark: Pooling sq79-85 in CARP on frontend sq51 to seed the cache
* 19:39 atglenn: replication every 15 minutes enabled from ms7 to ms8
* 19:31 mark: Removed sq16-30 from the text frontend squid pool in LVS - these servers will soon be decommissioned
* 19:28 mark: Pooled sq71-78 frontend squids in LVS with full load (30)
* 19:21 mark: Pooled sq71-78 frontend squids in LVS with low load (1) to seed caches
* 18:30 mark: Pooling sq71-78 backend squids on all frontends, with lower CARP weight (10 instead of 30) to seed the caches
* 18:04 mark: Deployed new squid config to sq58-66 frontend squid, to seed the caches of backend squids sq71-78
* 17:27 mark: Deployed new squid config to sq66 frontend squid, to seed the caches of backend squids sq71-78
* 17:24 mark: Deployed Squid with correct configuration on sq71-78
* 11:08 RoanKattouw: Load spike on 4CPU Apaches ended, rr.knams back up, downtime seems to be over
* 11:05 RoanKattouw: Another load spike on the 8 CPU Apaches, approx 11:02-11:-05 UTC
* 11:00 RoanKattouw: Apache load spike started around 10:42 UTC and coincides with a load dip followed by a monitoring blackout on the Kennisnet Squids and a load spike on the Tampa Squids
* 10:58 RoanKattouw: 8CPU Apaches stopped going bonkers all of a sudden according to Ganglia, 4CPU ones still have high CPU usage
* 10:56 RoanKattouw: Site went down, all Apaches have high CPU usage
== February 10 ==
* 22:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix to deploy usability changes sitewide'
* 22:22 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62275 to test'
* 21:28 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62264 for real. Running svn up first helps'
* 21:19 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62264'
* 21:03 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 21:02 Andrew: Switching strategywiki to opt-out LiquidThreads, rather than opt-in
* 20:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'Deploy r62262'
* 20:46 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploy r62262'
* 19:22 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 19:21 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62257'
* 19:05 Rob: storage1 disks replaced, except for the newly dead one, rma placed.
* 18:39 Rob: shutting down storage1 to replace its bad OS mirror disk
* 16:52 mark: Fixed MySQL permissions on srv186 as well
* 16:15 mark: Rigged puppet to deploy a PHP mail.ini file to the apaches, which sets it to call sendmail with -f <> (sending mail with empty envelope sender, like bounces)
== February 9 ==
* 22:06 Andrew: LiquidThreads production software updates completed successfully.
* 21:58 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/Dispatch.php 
* 21:57 mark: Firewalled one ip on singer which was overloading Apache, looked like a Wordpress attack
* 21:50 mark: Lowered MaxClients setting to 250 on singer
* 21:47 Andrew: Running scap to deploy LiquidThreads updates
* 21:45 Andrew: Updating LiquidThreads production deployments to the alpha version in the next few minutes.
* 21:44 mark: Power cycled singer
* 21:19 mark: Fixed MySQL system user & permissions on srv151-srv185
* 21:17 mark: Fixed MySQL instance on srv183
* 20:47 mark: Fixed MySQL instance on srv156
* 19:39 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 19:29 logmsgbot: catrope synchronized php-1.5/skins/common/edit.js 'Deploy r62190. Only for test for now, thank God for style versions'
* 17:01 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 17:01 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62187'
* 16:52 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 16:51 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62184'
* 14:19 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 14:19 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js 'Deploying r62181'
* 02:14 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js 'Merge r62158'
== February 8 ==
* 23:25 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix for r62041. Varnish needs this or Tampa will serve stale JS off bits'
* 23:06 mark: Deploying Exim on all application servers as well
* 22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/OptIn/SpecialOptIn.php 'Deploy r62141'
* 21:51 mark: Added search3 and search9 to search_pool_1 on lvs3
* 21:48 mark: Deploying Exim as a simple mail sending-only MTA (with queuing enabled) on squids, image scalers, search servers and core databases
* 21:38 Andrew: LiquidThreads wikinews rollout complete, though there was some buggage related to replication that had to be resolved.
* 21:29 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php 
* 21:29 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php 
* 21:27 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php 
* 21:24 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php 
* 21:22 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php 
* 21:21 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php 
* 21:18 logmsgbot: andrew synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 21:18 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php 
* 21:13 Andrew: Starting LiquidThreads deployment on enwikinews.
* 20:00 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Track NTOC and dialogs as well in PrefStats'
* 13:04 Tim: running fixBug20757.php on enwiki, in a screen on hume
* 12:51 Tim: locke went into swap half an hour ago and is now dead. Trying reboot.
* 07:38 Tim: stopped recompressTracked.php for now due to bug 20757
* 07:04 logmsgbot: tstarling synchronized php-1.5/includes/Revision.php 'r62120'
* 06:18 Tim: cleared out db9 root partition (which had 370MB free) by moving 4GB of SQL files from /root and /home/tfinc to /a/backup/junk
* 06:01 logmsgbot: tstarling synchronized php-1.5/wmf-config/db.php 're-added ms2 as an ES slave for rc1,cluster22 and the ex-fedora clusters, was depooled for ~6 months'
* 05:55 Tim: added grant for root@fenari to ms3/2
* 05:31 Tim: started apache on srv213, srv161 and srv240, was stopped for no apparent reason
* 04:52 logmsgbot: andrew synchronized php-1.5/includes/ChangesList.php 'Merge r62117'
* 04:37 Andrew: Deploying LiquidThreads alpha updates with scap
* 04:23 Andrew: Planning to deploy LiquidThreads alpha (liquidthreads.labs and test) in the next few minutes
* 01:15 Tim: added mysql and gmetric users on srv155 temporarily, to get those services back up
* 00:42 Tim: ran apt-get upgrade on srv155. Doing reboot test for new kernel.
== February 7 ==
* 23:36 Andrew: Clarification: Somebody should fix it, not I think it's fixed now.
* 23:35 Andrew: The issue of centralnotice pulling notices from the wrong place should be fixed now, though
* 23:34 Andrew: Moved /mnt/upload5/centralnotice/centralnotice.js and /mnt/upload5/centralnotice/wikipedia/centralnotice.js to centralnotice-old.js, should stop fundraising banners from appearing.
* 23:33 Andrew: Issue with fundraising banners appearing on random pages caused by clients with the URL http://upload.wikimedia.org/centralnotice/wikipedia//centralnotice.js?257z6
* 22:11 Andrew: Reports that fundraising banners are appearing on random pages for some users.
* 21:32 domas: we're running on single ES copy.
* 21:32 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'disabling broken storage1'
* 20:05 domas: removing all fixes I've been doing to recover mail. WMF staff will handle this tomorrow, or the problems will be gone by then.
* 19:00 domas: blackholed few thousand IPs on mchenry via OUTPUT chain (using -j REJECT action)
== February 6 ==
* 19:55 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 
* 01:09 logmsgbot: andrew synchronized php-1.5/extensions/CentralAuth/CentralAuth.php 'Merge r61717, because Casey Brown asked me nicely.'
* 01:08 logmsgbot: andrew synchronized php-1.5/extensions/CentralAuth/ApiQueryGlobalUserInfo.php 'Merge r61717, because Casey Brown asked me nicely.'
* 00:58 Andrew: Scapping to deploy r62055 'For usability initiative: merge r62041, r62043'
== February 5 ==
* 06:00 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php 'Re-activate LiquidThreads email notifications, DoS has passed'
* 05:22 Tim: freed up a little space on ms3 by reducing expire_logs_days from 14 to 7, and running flush logs
* 02:45 apergos: cleared out /tmp on srv169...
* 00:23 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump style version for jQuery UI stylesheet'
* 00:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 'Bump style version for combined.min.css'
* 00:03 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css 'Resync this, doesn't seem to have been picked up right'
== February 4 ==
* 23:55 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js 'r62002'
* 23:50 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version appendix'
* 23:37 RoanKattouw: Running scap to deploy new UsabilityInitiative code
* 22:55 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version appendix'
* 22:54 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js 'r61995'
* 22:28 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable UsabilityInitiative_alpha on usabilitywiki'
* 22:20 logmsgbot: tstarling synchronized php-1.5/includes/api/ApiUpload.php 'disabled chunk upload'
* 22:09 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version appendix'
* 22:07 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php 
* 22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php 
* 22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/css/combined.min.css 
* 22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js 
* 22:06 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js 
* 22:05 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/images/wikiEditor/dialogs/insert-link-external-rtl.png 
* 22:04 RoanKattouw: Deploying r61993 with individual sync-files
* 21:15 atglenn: turned off hourly snaps on ms7, we'll have daily plus the standard replication
* 18:49 atglenn: cleared /tmp on srv175 blah blah 
* 18:36 atglenn: cleared out php* files from /tmp on srv163 (need cron job)
* 10:35 mark: Removed sq11-15 from the upload LVS pool
* 10:30 mark: Removed sq11-15 from the upload CARP pool
* 03:28 apergos: added /usr/local/apache/common/php to include path of php on fenari 
== February 3 ==
* 23:56 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version appendix'
* 23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php 
* 23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins/jquery.wikiEditor.html 
* 23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js 
* 23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js 
* 23:55 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/css/combined.min.css 
* 23:54 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php 
* 23:54 RoanKattouw: Deploying r61959 using individual sync-files
* 22:35 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Explicitly enable EditWarning by default here. Previously done in extension file'
* 17:32 mark: Set root password on all servers using Puppet
* 17:26 Rob: loaded sq68 full with 6 ssd, had to 'borrow' drives from the unracked squids in pmtpa awaiting the rail issue resolution.
* 17:26 Rob: connected sq67 and sq68 to secondary network ports as requested
* 16:43 Rob: had to update gmetad and add sq14 and sq17 into puppet as ganglia aggregator hosts. Currently not showing in ganglia, but should when puppet updates their files shortly.
* 16:37 Rob: updated gmetad_pmtpa in NFS store, as well as on zwinger and spence.
* 16:20 Rob: !log cleaned sq1-sq10 certificates off sockpuppet
* 16:10 mark: Rebooting sockpuppet for security upgrades
* 16:06 Rob: updating DNS to remove lvs3.wikimedia.org, as its technically lvs3.pmtpa.wmnet
* 16:03 Rob: removed sq1-sq10 in dsh groups, server roles, pybal (already done, just doublechecked), and now starting wipe and removing their network connections.
* 15:37 Rob: updated video plugin on techblog, had to comment out sumotv support line due to errors.
* 13:30 mark: Removed sq1-10 from the upload backend Squid pool, preparing for decommissioning
* 12:27 mark: Removed eiximenis from the text backend pool
* 12:21 mark: Restarted sq51 with hyperthreading disabled
* 11:31 Tim-away: exim was flooding its mainlog with errors like "User 'exim' has exceeded the 'max_user_connections' resource (current value: 150)". OTRS is apparently broken as a result. Set the limit to infinity to fix it.
* 11:10 Tim-away: debugging mysql connection errors from mchenry to db9
* 11:02 Tim-away: restarting exim4 on mchenry
* 10:57 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.hooks.php 'r61922'
* 10:47 RoanKattouw: Mail delivery reported broken again; last mail delivery to mediawiki-cvs was 3h22m ago
* 10:14 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump UsabilityInitiative_alpha style version'
* 10:13 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins/jquery.wikiEditor.html 
* 10:13 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php 
* 10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js 
* 10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.hooks.php 
* 10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.combined.min.js 
* 10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php 
* 10:12 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js 
* 10:09 RoanKattouw: Deploying r61919 with individual sync-files
* 07:45 Tim: killed eximstats on mchenry, was sending the machine into swap
* 06:02 Andrew: Running scap to deploy LiquidThreads alpha updates
* 05:09 Andrew: Planning to update LiquidThreads alpha to trunk state in the next few minutes.
* 03:31 Tim: removing refreshLinks2 jobs from the enwiki job queue with namespace=0, 100k of them put there by a nasty biography template
* 00:34 domas: restarted varnish on db19
== February 2 ==
* 23:26 mark: PyBal got into a confused state due to a duplicate LVS realserver entry. Set up LVS manually/statically for 30 mins to debug the problem. PyBal is now active again.
* 23:04 Tim: pmtpa text squids down
* 22:33 mark: Pooled sq59-68 in LVS (Text)
* 22:27 apergos: restarted apache on srv170 (corrupted apc cache)
* 22:25 mark: Increased CARP weight of the new Upload squids sq51-58 from 10 to 30
* 22:17 mark: Added sq59-66 to the Text pool of Squids, with low CARP weight (10) to seed the caches
* 21:44 logmsgbot: andrew synchronized php-1.5/wmf-config/liquidthreads.php 
* 21:42 Andrew: temporarily disabling $wgLqtEnotif to mitigate mail server DoS impact on LiquidThreads post/reply speed.
* 21:42 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/classes/NewMessagesController.php 'Deploy r61879'
* 21:42 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/classes/NewMessagesController.php 'Deploy r61879'
* 21:04 mark: Removed sq2-10 from the LVS pool
* 20:46 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Bump style version addition for usability alpha'
* 20:45 logmsgbot: catrope synchronized php-1.5/skins/common/jquery.min.js 'Deploing r61700'
* 20:38 mark: Added sq52-58 to the backend Squid pool with low CARP weight, to seed the caches
* 20:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22351: Enable Vector and toolbar on srwikinews'
* 20:36 mark: Increased Exim load average threshold for delivery on mchenry
* 19:09 Fred: rebooted singer (went down for unknown- at this time- reasons).
* 18:25 Fred: OS-installed sq52-ssq58
* 18:14 mark: OS-installed sq59-sq66
* 17:23 mark: Thrown sq66 backend squid in the Squid pool, CARP weight 30
* 06:01 Tim: deploying r61846 via scap
* 04:25 Andrew: Schema change successfully applied.
* 04:24 Andrew: Applying schema change thread_signature.sql to mediawikiwiki, strategywiki, testwiki
* 04:22 Andrew: Applying schema change thread_signature.sql to liquidthreads_labswikimedia
* 01:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Fix for UsabilityInitiative_alpha config'
* 01:18 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js 'r61842'
* 01:13 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Append to $wgStyleVersion when running usability alpha'
* 01:10 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r61841 for real this time'
* 01:09 logmsgbot: catrope ran sync-common-all 
* 01:07 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r61841'
* 01:05 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r61839'
* 00:51 RoanKattouw: Running scap
* 00:48 logmsgbot: catrope synchronized php-1.5/includes/OutputPage.php 'r61837'
* 00:40 RoanKattouw: Synced extensions/UsabilityInitiative_alpha by hand
* 00:07 logmsgbot: catrope synchronized php-1.5/skins/common/jquery.min.js 
* 00:07 RoanKattouw: Syncing some CSS and JS files for usability deployment so bits.wikimedia.org will pick them up
* 00:06 logmsgbot: catrope synchronized php-1.5/skins/common/wikibits.js 'Sync changes to wikibits.js in r61558'
== February 1 ==
* 23:30 RoanKattouw: Testing new usability code on test
* 23:27 mark: Pooled sq51 frontend squid in the upload LVS cluster, weight 30
* 23:03 mark: Thrown sq51 backend squid in the Squid pool, CARP weight 30
* 20:50 Rob: srv217 shutdown until checked
* 20:50 Rob: srv217 not responsive to network, but LOM works and system is jut not working online, possible bad cable, will check in DC tomorrow
* 20:43 logmsgbot: mark synchronized php-1.5/wmf-config/CommonSettings.php 'Added 36 new Squid servers, removed bart'
* 20:39 mark: Deployed new wikimedia-base package that no longer manages sysctl.conf post Hardy, letting puppet handle it
* 20:10 atglenn: restarted webserver on ms7 to read new conf (fix bug 22321)
* 20:09 mark: Granted cary access to the OTRS db from fenari
* 19:27 RoanKattouw: srv9 reported down, pinging yields Host Unreachable. srv9 is not in Nagios but seems to be in use
* 18:51 Fred: swapped memcache node from srv145 to srv196
== January 31 ==
* 21:18 domas: db28 has a failed dimm
== January 30 ==
* 22:29 RoanKattouw: Scheduling more batches for import overnight using at(1), see http://commons.wikimedia.org/wiki/Commons:Batch_uploading/Geograph#Progress
* 22:13 RoanKattouw: Running image imports overnight on hume, three parallel import scripts started each five hours, see http://commons.wikimedia.org/wiki/Commons:Batch_uploading/Geograph for details
* 22:12 apergos: restarted morebots, must have died after last night's freenode irc move
* 04:40 Tim: fixed broken /etc/rc.local on wikitech
* 04:18 Tim: restarting mysqld on db12
* 04:00 Tim: analysis: mysqld on db12 hit a bug at 02:45 and froze, most threads in futex. The mysql client failed to set a read timeout, leading to db12 sucking up all available apache threads. Several squids became overloaded, presumably due to the large size of our 503 error messages.
* 03:21 Tim: restarted all apaches, thereby killing the reads which started before 3:11, possibly as early as 02:45. Site service resumes immediately. ps -lL on db12: http://p.defau.lt/?Jw1EBE0fnV4Rpxe9OP8yAw
* 03:18 Tim: finally thought to answer the question "are the apaches waiting for something or idle?" Strace confirms that they are waiting for db12 in read(). 
* 03:11 Tim: Having trouble believing db12 could be responsible for the CPU spike on the squids. Depooled db12 just in case. 
* 02:56 Andrew phones Tim
* 02:45 retrolog: db12 ganglia graphs go flat. Apache CPU goes down to near zero. CPU saturates on some of the text squids: sq16, sq17, sq18, sq21, sq22, sq23, sq25, sq26. Frontend squids serve 503s.
* 02:00 retrolog: Network spike on db12
== January 29 ==
* 23:21 domas: restarted sq37 for being in bad shape. 
* 22:25 tomaszf: starting xml snapshots back up on snapshots3
* 21:31 atglenn: cleared out cruft from /tmp on srv158 (how about a cron job? :-P )
* 18:39 logmsgbot: catrope synchronized php-1.5/maintenance/importImages.php 'Deploy r59908'
* 16:20 Rob: db28 mainboard replaced, updated LOM to work, updated dhcp with new mac address info
* 02:50 apergos: started copy of media data from ms7 to ms8 (nc running in screen as root on both hosts)
* 01:51 logmsgbot: andrew synchronized php-1.5/cache/interwiki.cdb 'Updating interwiki cache for new wikitech entry'
== January 28 ==
* 20:15 Rob: racked sq51-sq68, all drac accessible, wired, ready for install
* 19:05 Rob: updated dns for management ranges on new squid servers
* 13:56 mark: Added DNS entries for 36 new Squid servers
* 13:28 mark: Storage1 went down due to I/O failure
* 13:00 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Remove aliases made obsolete by r61632'
* 12:55 logmsgbot: catrope synchronized php-1.5/languages/messages/MessagesMl.php 'Deploying r61632'
* 10:57 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'supressredirect for rollbackers on ruwikisource'
* 09:30 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'Removed redundant huwiki settings'
== January 27 ==
* 18:04 mark: Fixed ExtensionDistributor, which broke due to the image server migration
* 18:02 logmsgbot: mark synchronized php-1.5/extensions/ExtensionDistributor/svn-invoker.php 's/upload5/upload6/'
* 16:10 mark: Added a timeout to puppet's apt-get update execution; hopefully this will prevent things from getting stuck for weeks
* 16:08 mark: Shutdown sq20 once again, please decommission this broken box
* 16:05 mark: Fixed puppet on db7
* 16:05 mark: Killed stuck apt-get update processes on all servers
* 15:58 mark: Made puppet manage NFS mounts /mnt/upload6, /mnt/thumbs and the absence of /mnt/upload5 on all application servers and image scalers
* 13:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix Category talk alias on mlwiki'
* 11:57 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Also add a Unicode 5.1 alias for Category talk on mlwiki'
* 11:52 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22286: Unicode 5.1 namespace alias for Category on mlwiki'
* 11:46 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryBacklinks.php 'Deploying r61571'
* 08:12 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads/lqt.js 
* 08:11 logmsgbot: andrew synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js 
== January 26 ==
* 23:44 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Enable file uploads.'
* 23:34 Fred: restarted morebot
* 23:31 atglenn: turning off web server on ms1 (previously: moved uploads to ms1, mounted ms7 everywhere, updated squid conf)
* 21:10 atglenn: copying uploaded data since last replication, from ms1 to ms7
* 21:05 atglenn: turned off file uploads temporarily
* 21:04 logmsgbot: ariel synchronized wmf-deployment/wmf-config/CommonSettings.php 
* 20:50 Rob: rebooted srv125 per dc task entry by fred
* 20:35 tomaszf: enabling global file uploads notice
* 20:29 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage.php 'Deploying r61534'
* 20:29 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsageHooks.php 'Deploying r61534'
* 18:38 Rob: shutting down pdf2 to relocate its rack position.
* 18:37 Rob: !log mobile2 port not moved yet, as its port assignment on wikitech is invalid
* 18:08 Rob: shutting down mobile2 to relocate its rack location
* 18:07 Rob: bayes back online
* 18:00 Rob: bayes management lom had ip issues, resolved, still working on host
* 17:52 Rob: bayes is not coming back online properly, checking.
* 17:47 Rob: bayes rack location moved and updated in racktables, powering back up
* 17:34 Rob: shutting down bayes to move its rack location from b2-pmtpa to a2-pmtpa
* 14:42 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22274: Set $wgAutoConfirmCount to 10 on itwiktionary'
* 00:56 Tim: on hume: increased RCT proc count, reduced max lag to 1s, increased lag check frequency
== January 25 ==
* 22:36 Tim: ms3 is reporting "table full" for rc1 enwiki.blobs. Running ALTER TABLE.
* 19:33 Fred: stopped apache on srv225
* 19:01 Fred: adjusted apache.conf on srv225 to make it work again...
* 16:07 Rob: powering srv224 & srv225 back up
* 16:01 Rob: shutting down srv225 & srv224 to swap their power cables from single to Y split.
* 15:45 Rob: srv196 memory replaced, back online and in service
* 15:45 logmsgbot: root synchronized php-1.5/wmf-config/mc.php 'pushing out change to two hosts that were down a few minutes ago'
* 15:40 Rob: shutting down db28 to swap memory around in diagnostics (bad memory in system)
* 15:25 logmsgbot: root synchronized php-1.5/wmf-config/mc.php 'took down srv196 for hardware replacement'
* 15:20 Rob: shutting down srv196 to replace faulty memory dimm
* 12:19 Tim: started recompressTracked.sh on hume
== January 23 ==
* 00:16 aZaFred_: implemented ganglia detailled mysql stats for db10.
* 00:16 aZaFred_: implemented DB Lag detection script for DB10
== January 22 ==
* 23:08 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/ClickTracking/ClickTracking.hooks.php 'Deploy r61401'
* 23:07 logmsgbot: catrope synchronized php-1.5/extensions/UsabilityInitiative/CollapsibleTabs/CollapsibleTabs.hooks.php 'Deploy r61403'
* 21:29 brion: freed some space on srv218 removing old upload temp files from /tmp but / is still pretty tight (445m free)
* 20:19 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'Deploying r61387'
* 19:05 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22201: Add WZ: alias on itwiktionary'
* 15:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable $wgEnotifWatchlist on usabilitywiki'
* 11:38 mark: Deployed new gmond 3.0.3-2 package that conflicts with the incompatible ganglia 3.1 packages in Ubuntu, to avoid problems in our upcoming upgrade
== January 21 ==
* 14:13 mark: Added srv210 to LVS
* 14:11 mark: Reenabled srv245 in LVS
* 13:58 mark: Restarted Apache on srv207, srv210, srv233, srv257, srv245
* 09:30 mark: Replaced srv121 (decommissioned) by search13 in smokeping
* 07:59 AaronSchulz: Removed empty ct_tag rows from code_tags
* 07:47 Tim: cleaned up /tmp on srv180 (was out of disk space)
== January 20 ==
* 20:36 logmsgbot: root synchronized php-1.5/wmf-config/mc.php 
* 16:38 Fred: killed long running queries created by civicrm on db9.
* 13:56 domas: did lots of changes on wikitech wiki. :) 
* 02:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging on enwiki'
* 01:20 logmsgbot: tfinc synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling api logging for enwiki'
* 01:16 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disabling API request logging on dewiki'
* 00:03 RoanKattouw: Strike my last, was actually *enabling* it on dewiki
* 00:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging test on dewiki'
* 00:02 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable API logging test on frwiki'
== January 19 ==
* 21:37 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling API request logging on frwiki'
* 21:36 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling API request logging on frwiki'
* 20:34 Fred: bugzilla upgrade completed. It might be up to an hour before attachments can be added due to DNS propagation.
* 19:41 Fred: Bugzilla as you know it is going down. Be back shortly.
* 18:21 Rob: updated blog.wikimedia.org wp-stats plugin to 1.6.1
* 16:03 Rob: ssds installed in db28, during boot it displays a memory error, investigating.
* 15:20 Rob: db28 offline while disks are swapped to solid state for testing.
== January 18 ==
* 05:02 Tim: removed some old xff logs to free up space for logrotate which I ran with the -f flag
* 05:00 Tim: fixed logrotate on nfs1, broken due to duplicated entries between /etc/logrotate.d/rsyslog and /etc/logrotate.d/syslog-ng. Left the syslog-ng ones in for now.
* 04:02 RoanKattouw: NFS /home is full
== January 16 ==
* 14:55 mark: Started backend squid on sq50
* 14:26 mark: sq19 has bad drive /dev/sdc
* 14:22 mark: dist-upgrade & reboot on sq19
* 14:14 mark: Shutdown sq20, bad disk /dev/sda
* 14:13 mark: Reenabled sq24 frontend in PyBal
* 14:03 Tim-away: disk space critical on srv167, cleaned up /tmp
* 06:07 apergos: "recovery", my *ss... cleaned up /tmp on srv181 to get some space back
== January 15 ==
* 23:53 tomaszf: starting test dewiki snapshot on snapshot2
* 17:17 Fred: modified gmetad config on zwinger and spence to reflect new apache 4cpu aggregator
* 17:16 Fred: added srv149 as a gmond aggregator in puppet.
== January 14 ==
* 17:28 domas: load-tested and fixed db19 to handle full bits workload (~22k/s), now again serving just tampa part. 
* 01:32 apergos: moved bits to .2 again, all nameservers seem to be up and reflect the change (bits unresponsive again)
== January 13 ==
* 18:30 Andrew: [andrew@zwinger ~]$ sync-file wmf-config/CommonSettings.php 'Disable GIF scaling again, due to issues reported on village pump, bug 22041'
* 17:05 Rob: updated InitialiseSettings for bug 21174 and 21077
* 16:24 Rob: updated InitialiseSettings.php Bug 20508 Please enable Extention:NewUserMessage on en.Wikinews
* 11:50 domas: started overflow-watchdog ( http://p.defau.lt/?4_Y_Lrl9tVKS6fEYwkswKA ) on db19, sent bits load at it again
* 01:23 atglenn: using .2 for bits.pmtpa, did authdns-update, let's see what happens (bits was failing to respond)
== January 12 ==
* 19:52 rainman-sr: could someone please put search3,9 into search_pool 1 (with search1,4) on lvs3
* 19:37 Fred: Puppet: set spence as a ganglia aggregator for Misc tree.
* 19:23 Rob: usability/prototype linode was crashed, had to reboot
* 17:39 Fred: set wgDefaultSkin back to monobook on wikitech since vector is not operational.
* 16:08 mark: Arcor clients appear to have problems reaching our sites, traffic to Arcor over AMS-IX has been low since midnight UTC
* 15:52 Rob: srv222 & srv223 power reestablished, booting.
* 15:47 Rob: shutting down srv223 & srv222 to change out power cords.
* 15:46 Rob: db10 moved and back online.
* 15:36 Rob: db10 moved to sdtpa a2, powering up.
* 14:50 Rob: taking down db10 to relocate from pmtpa-b1 to sdtpa-a2
* 14:50 Rob: fixed issues with transcode2 and transcode3, completing base installation.
* 05:48 logmsgbot: tstarling synchronized php-1.5/includes/HTMLCacheUpdate.php 
* 05:48 logmsgbot: tstarling synchronized php-1.5/includes/BacklinkCache.php 
* 05:47 Tim: deploying r60962
* 01:17 Tim: on streber: removed a corrupt torrus DB file so it could be rebuilt, torrus should be working now
* 00:57 Tim: killed frozen torrus cron jobs and ran "torrus compile --tree=Network --force"
* 00:51 Tim: maybe torrus collector is still broken, trying /etc/init.d/torrus-common force-reload
* 00:46 Tim: with mpm-prefork managed to debug it fairly easily. Moved away permanently locked DB file render_cache.db, torrus.wikimedia.org is now fixed
* 00:39 Fred: restarting pdns on ns1
* 00:38 Tim: switching streber to apache2-mpm-prefork, can't work out why it's not working
* 00:22 Tim: trying "apache2 -X" on streber
* 00:00 Tim: restarting apache on streber
== January 11 ==
* 23:38 domas: logging the fact that we had cache layer meltdown at some point in time during the day
* 22:30 domas: leaving bits.pmtpa on db19's varnish, in case of troubles - uncomment bits.pmtpa .2 record in /etc/powerdns/templates/wikimedia.org and run authdns-update 
* 19:43 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc.php 'Swapped memcached from srv125 to srv232'
* 19:06 Rob: new apaches srv255, srv257 deployed. Updated node groups and synced nagios
* 19:03 Rob: new apache server srv254 deployed
* 18:24 atglenn: copy backlog of image data from ms1 to ms7 (running in screen as root on both boxes)
* 14:43 mark: Rebooting fuchsia, locked up again
* 14:24 mark: Increased load on knsq16-22 by upping lvs weight from 10 to 15
== January 10 ==
* 23:02 logmsgbot: midom synchronized php-1.5/wmf-config/lucene.php 'rainman asked, rainman guilty, hehehe'
* 23:01 logmsgbot: midom synchronized php-1.5/wmf-config/secure.php 
* 17:36 rainman-sr: search limit raised to 500 again, interwiki search re-enabled for "other" wikis 
* 16:07 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'take ixia back out'
* 16:06 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 'put ixia back'
* 16:06 rainman-sr: restarting search cluster to deploy search13-19
* 15:58 domas: all bits serving switched back to text cluster, we have problems with all threads blocking on write(): http://p.defau.lt/?dOBxveiHj_ukjzupEBX3rA
* 15:19 rainman-sr: configuring search13-19, will leave search20 as spare
* 14:22 domas: apparently varnish worker threads are blocking on network output, ... :) 
* 12:40 domas: full bits pmtpa load sent to sq1
* 12:04 domas: sending half of bits load to sq1
* 12:02 domas: set up separate geo balancing for bits via bits-geo.wikimedia.org
* 10:45 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 'setting extension asset path to bits.wm'
* 10:42 logmsgbot: midom synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php 
* 10:29 logmsgbot: midom synchronized php-1.5/includes/Setup.php 
* 10:29 logmsgbot: midom synchronized php-1.5/includes/DefaultSettings.php 
* 09:36 domas: moving over static assets to 'bits.wikimedia.org'
* 09:11 logmsgbot: midom synchronized php-1.5/wmf-config/CommonSettings.php 
* 09:09 logmsgbot: midom synchronized php-1.5/wmf-config/secure.php 
* 08:26 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 
* 08:25 river: taking ixia out of rotation to dump commons
== January 9 ==
* 17:31 mark: Upgraded pdns-recursor to 3.1.7.2 on dobson, mchenry, lily
* 15:32 mark: Temporarily filtering all prefixes from 1299 on br1-knams, due to some balanced link blackholing issue
* 13:10 logmsgbot: midom ran sync-common-all 
* 12:45 domas: restarted, that is :)
* 12:44 domas: fixed ns1, was deadlocked
== January 8 ==
* 21:48 Rob: nagios is flapping errors for esams hosts, but they are still up and functional. Perhaps due to new transit setup earlier today.
* 20:10 Rob: pushing dns update for flaggedrevssandbox project
* 16:09 Rob: finished mobile2 initial setup, gave hcatlin sudo rights to server
* 15:46 mark: Brought up transit session to 1257 on br1-knams
* 14:06 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 11:42 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
== January 7 ==
* 20:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22038: Enable Collection extension on skwiki'
* 20:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19418: Fix $wgUploadNavigationUrl on bnwiki'
* 20:16 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Enable patrolling on frwikiversity'
* 19:54 Rob: base install done on mobile2, pdf2, transcode1, transcode2
* 16:55 Rob: srv254-srv257 racked, drac online, network attached. needs installs
* 15:46 Rob: racked srv254-srv257. DRAC setup, cables run. Network not plugged in until mark provisions the ports.
* 02:51 Tim: restarted apache2 on wikitech to fix swapping
* 02:27 Fred: rebooted pascal as it was once again hung
* 01:57 mark: Prepended AS 43821 once on incoming prefixes from AS 1299
== January 6 ==
* 22:30 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Disable OptIn on trwikimedia'
* 17:17 Rob: updated dns for tr.wikimedia.org
* 14:47 Rob: updated techblog software to newest stable revision
* 14:01 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22027: Enable AbuseFilter and CAPTCHA on trwiki'
* 13:27 river: restarted slave on db26
* 12:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19418: Set $wgUploadNavigationUrl on bnwiki'
* 12:37 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'remove debugging code from yesterday'
* 12:34 RoanKattouw: Resuming GlobalUsage rebuild
* 08:48 logmsgbot: kate synchronized php-1.5/wmf-config/db.php 
* 08:47 river: removing db26 from s1 to dump
== January 5 ==
* 23:57 RoanKattouw: GlobalUsage refresh script properly handling replag now. Aborted it, will resume in the morning (CET)
* 23:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/refreshGlobalimagelinks.php 'remove debugging code'
* 23:54 RoanKattouw: Resuming GlobalUsage refresh script
* 23:53 logmsgbot: catrope synchronized php-1.5/includes/db/LoadBalancer.php 'Deploying r60705'
* 23:53 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'Deploying r60705'
* 22:21 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'Debugging'
* 22:11 RoanKattouw: Aborting GU refresh *again*, somehow it's not detecting slave lag right
* 22:07 RoanKattouw: Resuming GlobalUsage refresh script
* 22:04 logmsgbot: catrope synchronized php-1.5/includes/GlobalFunctions.php 'Deploy r60695 (wfWaitForSlaves fix)'
* 21:44 RoanKattouw: Running refreshGlobalimagelinks.php on all wikis from the smallest up
* 20:50 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/SpecialGlobalUsage.php 'Deploy r60692 (GlobalUsage fixes)'
* 20:14 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'added outreach namespace to officewiki'
* 19:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Add patroller group on frwikiversity'
* 19:28 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable GlobalUsage on all public wikis'
* 17:10 logmsgbot: fvassard synchronized php-1.5/wmf-config/mc.php 'Swapping srv110 for spare srv231.'
* 16:41 Fred: restarted memcached on srv110
* 16:10 mark: Installed search13-20 as new search servers
* 15:50 mark: Installed karmic on search19
* 15:45 Rob: updating dns for mobile2 and pdf2 servers
* 12:10 mark: dist-upgrade and reboot on sockpuppet
* 12:01 river: copied old wmvids to ~kate/wmvids
* 11:37 mark: Installed OS on search18
* 07:25 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/ui/CodeRevisionView.php 
* 07:25 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/api/ApiCodeDiff.php 
* 07:24 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/CodeReview.i18n.php 
* 07:24 logmsgbot: tstarling synchronized php-1.5/extensions/CodeReview/CodeReview.php 
* 06:48 Tim: restarted apache on srv199, APC cache corruption
* 02:33 tomaszf: compressing support-requests on locke to free up space
* 02:14 logmsgbot: tstarling synchronized php-1.5/wmf-config/InitialiseSettings.php 'disabled $wgCopyUpload and $wgHTTPProxy on commons and test, breaks Lucene Search'
== January 4 ==
* 22:15 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 22002: Add Opinions namespace on bgwikinews'
* 21:51 RoanKattouw: Running rebuildLocalisationCache.php in an attempt to shut up exceptions thrown by srv218
* 21:29 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 20312: Re-enable GIF scaling on all wikis'
* 21:09 RoanKattouw: srv205 is throwing fatal errors because the geoip PHP module is not installed
* 20:51 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix typo'
* 20:45 RoanKattouw: srv218 has a full disk
* 20:45 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'plus proxy setting'
* 20:44 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20512: Enable $wgCopyUploads on test and commons'
* 20:44 mark: OS-installed Karmic on search13, search14, search16, search17, search20
* 20:40 RoanKattouw: Restarting refreshGlobalimageusage.php on Commons after it mysteriously disappeared
* 20:00 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling GIF scaling on testwiki'
* 19:39 RoanKattouw: Running refreshGlobalimagelinks.php on Commons again
* 19:38 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling GlobalUsage on commons again'
* 19:35 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Turning on GlobalUsage on testwiki'
* 19:33 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsageQuery.php 'Seems I forgot this file'
* 19:20 RoanKattouw: Running rebuildLocalisationCache.php in an attempt to kill "MagicWordArray::parseMatch: parameter not found" error
* 19:08 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Disable GlobalUsage again, blank pages on commons'
* 19:06 RoanKattouw: Running rebuildGlobalimagelinks.php on Commons
* 19:03 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Re-enabling GlobalUsage'
* 18:57 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage.i18n.php 'r60611'
* 18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/SpecialGlobalUsage.php 'r60611'
* 18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage.php 'r60611'
* 18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsageHooks.php 'r60611'
* 18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/GlobalUsage_body.php 'r60611'
* 18:56 logmsgbot: catrope synchronized php-1.5/extensions/GlobalUsage/ApiQueryGlobalUsage.php 'r60611'
* 18:55 RoanKattouw: Deploying r60611 (GlobalUsage update) with individual sync-file's
* 18:49 RoanKattouw: Truncating globalimagelinks table on commonswiki
* 18:48 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Temp disable GlobalUsage'
* 18:46 RoanKattouw: Disabling GlobalUsage temporarily for software update
* 18:41 logmsgbot: catrope synchronized php-1.5/wmf-config/ExtensionMessages.php 'Enabling WikimediaLicenseTexts on Commons'
* 18:40 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling WikimediaLicenseTexts on Commons'
* 18:31 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enabling WikimediaLicenseTexts on testwiki'
* 18:31 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'Enabling WikimediaLicenseTexts on testwiki'
* 18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php 'r60608'
* 18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.i18n.php 'r60608'
* 18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaMessages.php 'r60608'
* 18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.php 'r60608'
* 18:24 logmsgbot: catrope synchronized php-1.5/extensions/WikimediaMessages/WikimediaGrammarForms.php 'r60608'
* 18:23 RoanKattouw: Deploying r60608 (WikimediaMessages update) using sync-file
* 17:47 Rob: search13-search20 racked and ready for install
* 17:17 Rob: updating dns for new search servers
* 14:57 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace downed memcached box srv84'
* 14:45 Rob: moved power for asw-b3-pmtpa, srv86 and srv87 may flap.
* 14:43 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace downed memcached boxes srv120, srv121'
* 14:31 Rob: decommissioned srv120, srv121 to make room for new search servers.
* 14:29 Rob: pulling srv34 from rack to decommission, need the space for new search servers.
* 14:09 logmsgbot: mark synchronized php-1.5/wmf-config/mc.php 'Replace memcached on just decommissioned host srv82'
* 13:57 Rob: srv82, srv83, srv84 decommissioned to make room for new search servers in rack
* 12:56 Tim: started squid-frontend on knsq8, knsq10, knsq11, knsq13, knsq14, knsq15, all crashed at roughly the same time as knsq9
* 12:52 Tim: started squid-frontend on knsq9, died at ~17:30 on the 3rd. Syslog shows many crashes, followed by "out of socket memory" a couple of hundred times, then silence
== January 3 ==
* 20:17 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'Deploying r60588, r60590'
* 17:29 logmsgbot: midom synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'added die(); somewher in there'
* 14:38 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 14:30 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 'ixia is being put on a bench'
== January 2 ==
* 19:13 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21954: Enable import from Commons on brwikimedia'
== December 31 ==
* 19:45 logmsgbot: tfinc synchronized php-1.5/wmf-config/reporting-setup.php 'Moving tracking back to db9 as it requires insert'
* 18:38 logmsgbot: tfinc synchronized php-1.5/wmf-config/reporting-setup.php 'Switching to query the slave db10 instead of the master db9'
== December 30 ==
* 23:37 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/FundraiserStatistics_body.php 
* 23:35 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.i18n.php 
* 12:51 mark: Set /proc/sys/vm/min_free_kbytes to 65535 on brewster and streber, to see if it helps with the swapper page allocation failure bug
* 07:57 Tim: fixed private+secure file access, URLs were broken for months at least
* 07:50 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php 
* 07:48 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php 'possible resolution for boardwiki problem'
* 07:45 logmsgbot: tstarling synchronized php-1.5/wmf-config/secure.php 'debugging hack'
* 07:34 Tim: created an account for myself on boardwiki to debug file upload issue
== December 29 ==
* 23:42 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'added tracking for 2009_notice51.'
* 18:27 tomaszf: dropping retention on storage2 to 4 xml snapshots
* 13:04 logmsgbot: catrope synchronized php-1.5/includes/api/ApiQueryAllUsers.php 'Deploying r60468'
* 12:50 RoanKattouw: Started logmsgbot (running as catrope instead of nobody)
* 12:32 RoanKattouw: logmsgbot down since fenari reboot, needs to be restarted by a root
== December 28 ==
* 21:02 RoanKattouw: Restarting refreshLinks for enwikibooks on hume
* 20:49 mark: Created 10G of swap space on fenari
* 20:07 domas: powercycled fenari, console not responsive
* 15:00 mark: Installed Karmic on LVS1, setting up a test network for LVS performance testing
* 14:55 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 17338: Disable $wgRestrictDisplayTitle and enable subpages in the main namespace on rmwiki'
* 13:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21566: Enable Collection on viwiki'
* 13:42 RoanKattouw: Running refreshLinks on enwikinews per bug 19404
* 13:41 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19404: Set $wgCategoryPrefixedDefaultSortkey=false on enwikinews'
* 13:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21958: Create Book namespace on enwiki'
== December 27 ==
* 19:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21954: Import sources, subpages for brwikimedia'
* 11:26 domas: db20 offlined disk array, after reboot booted into netinstall (but saw the array), after reset /SYS, and some operator's no-operation in BIOS and RAID controller setup screens, it booted up properly
== December 26 ==
* 19:59 DaBPunkt: (non-dev-entry) Many apaches in the US died, some db-server reported overusing. Problem fixed itself after some minutes.
== December 25 ==
* 20:24 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 
* 20:09 logmsgbot: aaron synchronized php-1.5/wmf-config/InitialiseSettings.php 'fixed frwiktionary config (sysops should be able to remove patroller)'
== December 24 ==
* 22:52 RoanKattouw: Running updateArticleCount.php on arwiki and arwiktionary
* 20:21 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'wrong var AGAIN'
* 20:19 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'wrong var'
* 20:18 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 21859: Enable Doublewiki on frwiktionary'
* 19:03 logmsgbot: catrope ran sync-common-all 
* 18:56 RoanKattouw: Running sync-docroot, sync-apache, sync-common-all, apache-graceful-all
* 18:45 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Explicitly add langcodes for recently created wikis'
* 18:27 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 19534: Add Page and Index namespaces on sourceswiki'
* 17:02 logmsgbot: catrope ran sync-common-all 
* 16:57 RoanKattouw: Syncing updates for wiki creations
* 16:57 RoanKattouw: Not creating fiwikimedia, already created
* 16:54 RoanKattouw: Adding fiwikimedia (sans DNS entry) per bug 20502
* 16:41 RoanKattouw: Adding arbcom_fiwiki (sans DNS entry) per bug 21375
* 16:32 RoanKattouw: Adding dkwikimedia (sans DNS entry) per bug 21009
* 16:23 RoanKattouw: Adding brwikimedia (sans DNS entry) per bug 21149
* 16:14 RoanKattouw: Adding pcdwiki (sans DNS entry) per bug 21634
* 16:06 RoanKattouw: Made raw, non-highlighted versions of .php files visible through noc.wikimedia.org/conf
* 15:55 RoanKattouw: Made abusefilter.php visible through noc.wikimedia.org/conf
* 15:34 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20515, bug 20551, bug 20923, bug 21214: Enable AbuseFilter on barwiki, eswikibooks, ltwiktionary, ptwikibooks'
* 15:34 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'bug 20515, bug 20551, bug 20923, bug 21214: Enable AbuseFilter on barwiki, eswikibooks, ltwiktionary, ptwikibooks'
* 15:15 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix name of oldwikisource -> sourceswiki'
* 15:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Enable ProofreadPage for oldwikisource, which strangely is not part of the wikisource group'
* 15:06 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20402: Enable Collection extension on mswiki'
* 15:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21389: Set $wgDisabledVariants on Chinese wikis'
* 14:55 logmsgbot: catrope synchronized php-1.5/wmf-config/CommonSettings.php 'bug 21527: Enable Special:Cite on itwiktionary'
* 14:28 RoanKattouw: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21100: Add Tac_gia namespace to $wgNamespacesToBeSearchedDefault on viwikisource'
* 14:19 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21011: Restrict uploads, change $wgUploadNavigationUrl on enwikibooks'
* 14:10 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21236: Set $wgEnableNewpagesUserFilter=true; on plwiki'
* 14:06 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21370: Create custom namespaces on etwikisource, fix namespace alias on arwikisource'
* 13:51 RoanKattouw: Running updateArticleCount.php on zhwikisource for bug 20998
* 13:48 domas: s1 and s2 had rolling restarts to mysql#3193 build
* 13:48 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21109: Rename autopatroller group to autoreviewer on zhwiki'
* 13:44 domas: s2 master switched to db15-bin.000001:106
* 13:44 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 13:44 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 13:32 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'misspelled right name'
* 13:29 logmsgbot: catrope synchronized php-1.5/wmf-config/abusefilter.php 'bug 20721: Add abusefilter-viewprivate right for sysops on enwiki'
* 13:11 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20561: Change sitename on tkwiki'
* 12:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 15237: Localize sitename and Wikipedia namespace on nahwiki'
== December 23 ==
* 16:02 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 14:57 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 14:35 Rob: db27 back online with new fan controller board.
* 13:53 Rob: shutting down mysql on db27 for hardware replacement.
* 12:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Actually enable patrolling on frwikibooks and frwiktionary'
* 08:22 Tim: on isidore, also disabled the CentralNotice job that was running every 20 minutes from /etc/crontab
* 05:16 Tim: disabled CentralNotice rebuilds for donate.dev.wikimedia.org, was overloading isidore (which is only a single-core pentium)
== December 22 ==
* 23:45 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Added 2009_Notice49 to tracking.'
== December 21 ==
* 19:25 Rob: ms8 still borked.
* 19:24 Rob: restarted mysql on db27
* 19:12 Rob: removed the fan boards in db27 and cleaned connections (were filthy with toner or dust or something) and replaced them. system booting back online. (Will watch it for errors for the next week.)
* 18:36 Rob: shutting down db27 mysql manually for troubleshooting hardware on the system
* 18:04 Rob: db28 mainboard and fans replaced, booting back online.
* 17:35 Rob: restarted mysql on srv185, since it appears to be a ext. storage slave.
* 17:32 Rob: srv185 memory replaced, back online.
* 17:20 Rob: shutting down srv185 to swap bad dimm1
* 16:52 Rob: swapped out bad disk in db30, all leds are green now.
* 16:22 Rob: swapped out bad memory in sq32, booting it back up.
* 16:13 Rob: shutting down sq32 to swap out bad memory
* 07:29 logmsgbot: tstarling synchronized php-1.5/extensions/intersection/DynamicPageList.php 'deploying r60254'
== December 20 ==
* 21:29 RoanKattouw: Clean up entries with rev_user_text='conversion script' (capitalize C) on dewiki for bug 21910
* 21:24 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'forgot $wgRemoveGroups'
* 21:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21517: Add patroller group on frwikibooks and frwiktionary'
* 21:03 RoanKattouw: Running namespaceDupes on simplewiktionary for bug 21906
* 21:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21906: Add WT: -> Wiktionary: alias on simplewiktionary'
* 10:25 domas: um, no more stalls?
== December 19 ==
* 20:53 RoanKattouw: Running namespaceDupes on zhwiki
* 00:41 logmsgbot: fvassard synchronized php-1.5/wmf-config/CommonSettings.php 'Adding JimmyAppeal[8-9] to tracking'
== December 18 ==
* 22:25 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'revert my last: logo too large'
* 22:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21213: Logo for acewiki'
* 21:57 logmsgbot: catrope ran sync-common-all 
* 21:56 RoanKattouw: Closing nlwikinews per bug 20325
* 21:45 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'Fix mistake (autopatrol vs. autopatroller)'
* 20:24 mark: Updated firewall for allowing ssh access from fenari on loudon
* 13:21 mark: Stopped oprofile on db22
* 11:22 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21109: Add autopatroller group, allow sysops to add/remove confirmed on zhwiki'
* 11:18 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20910: Set $wgUploadNavigationUrl on nlwiki'
* 11:18 RoanKattouw: srv205 has full disk
* 11:09 domas: s1 switched master to db16-bin.000001
* 11:08 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 11:08 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 11:07 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 11:07 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 11:00 logmsgbot: midom synchronized php-1.5/wmf-config/db.php 
* 00:14 aZa_: restarted srv141. Puppet went crazy cpu hungry.
== December 17 ==
* 23:54 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionTrackingStatistics_body.php 
* 23:53 logmsgbot: tfinc synchronized php-1.5/extensions/ContributionReporting/ContributionReporting.i18n.php 
* 23:15 RoanKattouw: Strike that, frwikinews
* 23:07 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21436: Activate flood flag on frwikibooks'
* 20:32 Rob: sq20 reinstalled, squid package installed. Not back in service until I check with Mark about some puppet settings.
* 20:07 Rob: resynced nagios to remove sq27
* 20:03 Rob: sq20 disk died, replaced and reinstalling.
* 20:01 Rob: sq27 decomissioned (stole drive to fix sq20)
* 19:59 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21071: Allow sysops to add/remove patroller and autopatroller on hewiki'
* 18:54 logmsgbot: tfinc synchronized php-1.5/wmf-config/CommonSettings.php 
* 17:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21158: Enable $wgBlockAllowsUTEdit on enwikiquote'
* 16:54 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21121: Enable flood flag on simplewikibooks'
* 16:22 RoanKattouw: Running namespaceDupes on iawiktionary
* 16:21 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21241: Add Appendice namespace on iawiktionary'
* 16:17 Rob: pushing update to dns for search13 dns
* 16:09 RoanKattouw: Running namespaceDupes on iawiki for bug 21241
* 16:08 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21241: Add Appendice namespace on iawiki'
* 16:04 RoanKattouw: Running namespaceDupes on brwiki for bug 21417
* 16:03 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21417: Change name of project namespace on brwiki, add old name as alias'
* 15:52 mark: Killed memcached on browne
* 15:51 mark: Started ircd on browne
* 15:51 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21041: Allow sysops to grant and bcrats to remove transwiki on nowiki'
* 15:46 Rob: browne back online after move.
* 15:46 Rob: nagios synced to node group files without will.
* 15:45 Rob: will decommissioned, pulled from rack
* 15:40 Rob: shutting down browne to move its rack location, will be back online shortly.
* 15:36 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21071: Add patroller and autopatroller groups on hewiki'
* 15:26 mark: Moved udpmcast from browne to dobson
* 13:11 mark: Fixed Racktables rack thumb problem by installing php5-gd; it was just serving cached thumbs
* 11:06 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21306: Allow sysops to add/remove patroller and autopatrol on hrwiki'
* 11:00 RoanKattouw: Running namespaceDueps on mlwiki for bug 21277
* 10:56 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21277: Add Portal and Portal talk as aliases on mlwiki'
* 10:53 RoanKattouw: Running namespaceDupes.php on cawikibooks for bug 20980
* 10:53 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 20980: Add Viquiprojecte namespace on cawikibooks'
* 10:47 RoanKattouw: Resyncing srv194 and rebuilding caches
* 10:46 domas: srv194 had 5G-sized oprofile error log, hehe hehehe
* 10:39 RoanKattouw: srv194 disk still full
* 10:39 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21134: Allow sysops to add/remove rollbacker on gawiki'
== December 16 ==
* 20:41 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'fix mistake'
* 20:32 RoanKattouw: srv194 disk full
* 20:32 logmsgbot: catrope synchronized php-1.5/wmf-config/InitialiseSettings.php 'bug 21134: Add rollbacker group on gawiki'
* 19:56 logmsgbot: andrew synchronized php-1.5/extensions/UsabilityInitiative/js/js2/jquery-ui-1.7.2.js 'Deploying r60130'
* 19:55 logmsgbot: andrew synchronized php-1.5/extensions/UsabilityInitiative/js/js2.combined.min.js 'Deploying r60130'
* 19:55 logmsgbot: andrew synchronized php-1.5/extensions/UsabilityInitiative/js/js2.combined.js 'Deploying r60130'
* 19:53 logmsgbot: fvassard synchronized php-1.5/wmf-config/InitialiseSettings.php 'changed groupOvrrides for fiwiki for arbcom.'
* 17:44 logmsgbot: aaron synchronized php-1.5/wmf-config/flaggedrevs.php 'FR labs config - remove excess level'
* 16:36 hcatlin: new and improved mobile with compressed memcached + new homepags + better utf-8 handling
* 16:24 hcatlin: taking down mobile1 for a large software update
* 15:59 mark: Disabled NIS on zwinger
* 15:54 mark: Disabled NIS on ms1
* 15:53 mark: Disabled NIS on ms4
* 15:47 mark: Installed puppet on ms1 and ms7
* 15:27 Andrew: srv129, srv123, srv120, srv95 seem to be in swapdeath
* 15:25 logmsgbot: andrew ran sync-common-all 
* 15:24 Andrew: sync-common-all caused memory spike on apaches again, site seems to still be up though
* 15:17 Andrew: Updating LiquidThreads to trunk state, using sync-common-all
* 12:58 RoanKattouw: Tim ran rebuildTemplates.php on hume for all languages
* 12:52 Andrew: Fixed morebots, was choking because the Server Admin Log was archived.
* 12:50 Andrew: Moved Morebots init script into a line in rc.local. Morebots seems to have been down due to Freenode's DDoS problem, maybe it isn't exiting properly when disconnected from the server
* 12:46 Tim: ran rebuildTemplates.php on hume for all languages