Release Engineering/SAL/Archive 4

== 2018-12-31 ==
* 16:45 Hauskatze: gerrit/github: created https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/CiteDrawer, w/ github and diffusion mirror; cf. mw.org request
* 14:33 Hauskatze: gerrit: created https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/MissedPages & associated extension-MissedPages group; as requested on mw.org
* 14:05 Hauskatze: github: created https://github.com/wikimedia/mediawiki-extensions-MissedPages to mirror similar new Gerrit repo
== 2018-12-22 ==
* 13:30 Hauskatze: github: deleting https://github.com/wikimedia/labs-tools-signpost mirror T212559
== 2018-12-21 ==
* 11:17 hashar: new doc job is https://integration.wikimedia.org/ci/job/publish-to-doc1001/ and runs just after https://integration.wikimedia.org/ci/job/publish-on-contint1001/
* 11:17 hashar: updating all publish jobs to also publish to doc1001.eqiad.wmnet https://gerrit.wikimedia.org/r/#/c/integration/config/+/480657/ T137890
== 2018-12-20 ==
* 18:53 thcipriani: unstick post-merge pipeline (beta-scap-eqiad stuck for 9 hours), [x] beta-scap-eqiad job clogging the works
* 05:10 legoktm: deployed https://gerrit.wikimedia.org/r/480908
* 05:04 legoktm: deployed https://gerrit.wikimedia.org/r/480904
== 2018-12-19 ==
* 23:27 thcipriani: integration-slave-jessie-1003:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online
* 18:05 Hauskatze: Created mediawiki/extensions/ExternalGuidance.git repos on Gerrit, set-up mirrors on Diffusion and GitHub; per mediawiki.org request.
* 15:20 hashar: Testing spicerack postmerge job for T205894: contint1001$ zuul enqueue --trigger gerrit --pipeline postmerge --project operations/software/spicerack --change 480724,2
== 2018-12-18 ==
* 10:56 legoktm: rebuilding docker tox-* images
* 08:17 legoktm: rebuilding docker tox image https://gerrit.wikimedia.org/r/480428
== 2018-12-17 ==
* 22:50 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/480256
* 22:26 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/480165/
* 11:43 Amir1: ores:18d3657 is going beta
== 2018-12-14 ==
* 16:19 Krenair: T204500 Powering off the remaining trusty and main deployment instance, deployment-zotero01, which is no longer in use anyway
* 13:08 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintEntities.php --wiki=wikidatawiki --config-format=wgConf tee WikibaseQualityConstraints-config.php # T209957
* 06:23 kart_: Beta: Updated cxserver to 
== 2018-12-13 ==
* 21:56 thcipriani: integration-slave-jessie-1002:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online
* 18:31 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/479501/
* 17:20 Amir1: ores:1a3de73 is going beta
* 09:32 hashar: Adding integration-slave-jessie-1001 back online
* 09:30 hashar: cleaned up disk on integration-slave-jessie-1001
* 01:05 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/479158
== 2018-12-12 ==
* 21:50 Hauskatze: github: deleting https://github.com/wikimedia/mediawiki-extensions-ConditionalShowSection refs. T211821
* 21:20 bearND: (beta): Update mobileapps to 
* 19:45 hashar: contint1001: sudo chown -R zuul:zuul /etc/zuul/wikimedia/.git
* 19:37 greg-g: enabled 2fa on the wmfgerrit github user account, recovery codes in releng's pwstore
* 17:24 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintEntities.php --wiki=wikidatawiki --config-format=wgConf tee WikibaseQualityConstraints-config.php # T209957
== 2018-12-11 ==
* 20:03 hashar: building image docker-registry.wikimedia.org/releng/commit-message-validator:0.6.0 T210278
* 10:09 hashar: jenkins: removed some legacy/unused user token via https://integration.wikimedia.org/ci/administrativeMonitor/legacyApiToken/manage T206234
* 10:03 hashar: Deleting unused instance jenkinstest.integration.eqiad.wmflabs T208803
* 09:42 godog: enable profile::rsyslog::udp_localhost_compat
* 09:04 hashar: gerrit: added _joe_ to 'integration' project so he can +2 changes notably to integration/config ( https://gerrit.wikimedia.org/r/#/admin/groups/10,members )
== 2018-12-09 ==
* 23:38 Krenair: restarted apache on deployment-mediawiki-07 for T211524
== 2018-12-05 ==
* 14:58 thcipriani: bring integration-slave-jessie-1004 back online (disks look fine)
* 02:57 kart_: Beta: Updated cxserver to 
== 2018-12-04 ==
* 06:04 legoktm: deploying https://gerrit.wikimedia.org/r/477115
* 04:08 legoktm: deploying https://gerrit.wikimedia.org/r/473724 https://gerrit.wikimedia.org/r/477416
== 2018-12-03 ==
* 22:05 Krenair: T210214 deleting deployment-cache-text04
* 15:01 Amir1: ores:03b9c98 is going beta
== 2018-11-30 ==
* 22:32 thcipriani: removing deployment-redis0{5,6} since I can't ssh in, and they don't seem to have any puppet roles applied
== 2018-11-29 ==
* 14:26 godog: switch mediawiki logging to use localhost syslog -> kafka -> logstash
* 13:54 hashar: Building docker-registry.discovery.wmnet/releng/java8-sonar-scanner:0.1.0 for https://gerrit.wikimedia.org/r/475496 T209849
* 08:35 hashar: Deleted instances castor02 and integration-publishing , replaced by new instances in the new WMCS region T208803
== 2018-11-28 ==
* 08:45 hashar: Switching Jenkins job cache to integration-castor03 with AN EMPTY CACHE T208803
* 08:18 godog: test mediawiki kafka logging on deployment-mediawiki-07
== 2018-11-27 ==
* 15:55 Amir1: giving User:Fisch-WMDE and User:Thiemo Mättig (WMDE) admin rights on enwiki, dewiki and arwiki (WMDE staff, beta cluster)
* 13:59 godog: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/475352
* 11:38 Amir1: Deploying ores:9b9ba06 to beta
== 2018-11-26 ==
* 22:22 Krenair: shutoff deployment-cache-text04, now replaced with deployment-cache-text05 in the new region - T210214
* 18:02 Amir1: deploy ores:3cdaaa6 to beta
== 2018-11-23 ==
* 13:55 dcausse: restarted elasticsearch on all deployement-elastic0X nodes (search broken on the beta cluster)
* 11:09 hashar: Jenkins: removing plugins "Single Use Slave" and "Event Publisher (via ZMQ PUB SUB)". Were used for Nodepool T209361
* 10:29 hashar: Building releng/java8-wikidata-query-rdf:0.2.1 container for https://gerrit.wikimedia.org/r/#/c/integration/config/+/475428/ T209776
== 2018-11-22 ==
* 10:34 hashar: Building container releng/java8-wikidata-query-rdf:0.2.0 to support Chrome ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/475298/ T209776 )
== 2018-11-21 ==
* 21:44 Krenair: Changed beta.wmflabs.org and *.beta.wmflabs.org A records from 208.80.155.135 (deployment-cache-text04) to 185.15.56.36 (deployment-cache-text05)
* 20:49 hashar: gerrit: added WMDE-leszek to group "Gerrit Managers" https://gerrit.wikimedia.org/r/#/admin/groups/119,members T200311
* 03:25 bd808: Forced puppet run on deployment-mediawiki-0[79] to pick up new redis::shards settings T210030
* 03:02 bd808: Repeated application of role::mediawiki::memcached on deployment-memc04, deployment-memc06, and deployment-memc07 for T210030
* 01:22 bd808: Applied role::mediawiki::memcached on deployment-memc05.deployment-prep.eqiad.wmflabs to provision redis T210030
== 2018-11-20 ==
* 23:45 mutante: deployment-deploy01 edited /srv/deployment/iegreview/iegreview/.git/DEPLOY_HEAD - - replaced deployment-tin with deployment-deploy1 to fix scap cloning / puppet
* 23:01 mutante: deployment-deploy01 edited /srv/deployment/scholarships/scholarships/.git - replaced deployment-tin with deployment-deploy1 to fix scap / cloning of scholarships app
* 22:42 thcipriani: repooling integration-slave-jessie-1003 after cleaning mvn and gradle cache
* 20:23 Krenair: Changed deployment-prep mysql repl password in attempt to get replication working again, have stored it at deployment-puppetmaster03:/var/lib/git/labs/private/modules/secret/secrets/mysql/repl_password
* 18:18 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@7553087]
* 14:59 hashar: created integration-castor03.integration.eqiad.wmflabs intended as a replacement for castor02 T208803
* 14:34 hashar: beta-scap job fixed by deleting a file on deployment-mediawiki-07
* 14:27 hashar: deployment-mediawiki-07 : sudo rm -fR /srv/mediawiki/.~tmp~/
* 13:55 andrewbogott: deleting deployment-redis05 and deployment-redis06 as per Giuseppe, "we're not using the old jobqueue, we should remove those vms"
* 13:08 twentyafterfour: removed local unix user mwdeploy from deployment-mediawiki-07 because it was shadowing the real mwdeploy user in ldap
* 13:01 twentyafterfour: PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/luasandbox.so' - /usr/lib/php/20151012/luasandbox.so: cannot open shared object file: No such file or directory T208101
* 12:11 twentyafterfour: scap failures on deployment-mediawiki-07 are related to uid/gid mismatch of the mwdeploy user, specifically the owner of that user's home dir is uid 603 but /etc/passwdgroup have a different uid/gid for the same username. T208101
* 11:47 hashar: Armed keyholder on deployment-deploy01 Got shutdown while being migrated a new cloud region # T208101
* 11:00 hashar: deployment-deploy01 got migrated to a new region but the Jenkins configuration had not been updated. Adjusting IP address from 10.68.23.38 to 172.16.4.18 T208101
* 10:58 hashar: Clearing out deployment-deploy01 disk space. Went offline due to disk space consumption
* 10:04 Krenair: manually fixed deployment-mediawiki-09:/srv/mediawiki/wmf-config/db-labs.php to match deployment copy, not sure why it didn't deploy properly yet
== 2018-11-19 ==
* 22:40 Krenair: manually sorted certs for deployment-puppetdb02
* 22:23 Krenair: duplicated wiki(useradmin) mysql users on deployment-db0[34] - previous hosts 10.%, new hosts 172.16.%
* 18:32 Krenair: creating deployment-cache-text05 to replace deployment-cache-text04
* 15:53 Amir1: cherry-picking gerrit:474694/1 in beta puppetmaster
* 15:06 Amir1: ores:e957b24 is going beta
* 13:47 hashar: Shutdown integration-publishing , replaced by integration-publishing02 # T208803
* 13:21 hashar: Created integration-publishing02 172.16.4.5 for WMCS region migration # T208803
* 11:13 hashar: updating jobs wikidata-query-rdf-maven-java8-docker wikidata-query-rdf-maven-java8-docker-site-publish for https://gerrit.wikimedia.org/r/#/c/integration/config/+/474660/
* 10:23 gehel: Updating docker-pkg files on contint1001 for wikdiata-query-rdf image
== 2018-11-17 ==
* 06:26 legoktm: deployed https://gerrit.wikimedia.org/r/474415
== 2018-11-16 ==
* 21:01 legoktm: deploying https://gerrit.wikimedia.org/r/473718
* 20:51 legoktm: deploying https://gerrit.wikimedia.org/r/473671 https://gerrit.wikimedia.org/r/474330
* 20:36 legoktm: deployed https://gerrit.wikimedia.org/r/474306
* 17:37 hashar: Added integration-slave-docker-1046 and integration-slave-docker-1047
* 17:20 hashar: Deleted integration-slave-docker-1045 (wrong OS stretch, must be jessie)
* 17:19 hashar: Added two more Jenkins docker slaves: https://integration.wikimedia.org/ci/computer/integration-slave-docker-1044/ and https://integration.wikimedia.org/ci/computer/integration-slave-docker-1045/
* 14:35 hashar: Building docker image releng/java8:0.4.2
* 13:33 hashar: building docker image releng/java8:0.4.1 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/472661/
== 2018-11-15 ==
* 21:09 hashar: Last Nodepool instance had id (yeah more than a million)
* 17:15 Amir1: ores:51cdf6b is going beta
== 2018-11-14 ==
* 18:48 andrewbogott: moving deployment-mediawiki-07 to labvirt1008
* 18:31 andrewbogott: moving deployment-chromium01 to labvirt1009
* 18:06 andrewbogott: moving deployment-mx02 to labvirt1003
* 18:05 andrewbogott: migrating deployment-snapshot01 to labvirt1001
* 10:39 gtirloni: added gtirloni as projectadmin in deployment-prep project
== 2018-11-13 ==
* 22:39 thcipriani: reenabling and running puppet on deployment-deploy01
* 22:19 andrewbogott: moving deployment-urldownloader02 to labvirt1012
* 22:02 thcipriani: disable puppet on deployment-deploy01 temporarily while deployment-deploy02 is migrating to preserve dsh files
* 21:59 andrewbogott: moving deployment-deploy02 to another labvirt
* 21:55 andrewbogott: moving deployment-webperf12 to a new labvirt
* 21:50 andrewbogott: moving deployment-dumps-puppetmaster02 to a new labvirt
* 21:43 andrewbogott: moving deployment-elastic05 to a new labvirt to clear out labvirt1016
* 13:01 arturo: a puppet refactor for the aptly module may have caused some puppet issues. Should be solved now
* 09:31 addshore: manually brought integration-slave-docker-1021 back online
* 09:29 addshore: integration-slave-docker-1021:/# docker rmi $(docker images grep " months " grep -v " [1-2] months " awk '{print $3}')
* 09:26 addshore: integration-slave-docker-1021:/# docker rmi $(docker images grep " months " grep -v " [1-5] months " awk '{print $3}')
== 2018-11-12 ==
* 12:30 hashar: mass purging legacy Jenkins jobs from contint1001. They were still deployed but no more in JJB, that causes the logs / artifacts to be left behind. T209123
* 11:26 hashar: building container releng/operations-puppet:0.5.1 to bump mtail version ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/472960/ by godog )
== 2018-11-11 ==
* 23:47 addshore: manually set quibble-vendor-mysql-hhvm-docker timeout to be 60 (i should really get my jjb stuff working again)
== 2018-11-09 ==
* 13:43 Amir1: ores:0728805 is going beta
* 12:13 kart_: Update cxserver to 
== 2018-11-08 ==
* 22:55 thcipriani: contint1001: cleanup with (essentially, but not exactly) https://phabricator.wikimedia.org/P7781
* 22:21 thcipriani: contint1001:sudo find /srv/jenkins/builds -maxdepth 2 -mindepth 2 -type d -regex '.*/[1-9][0-9]+' -mtime +500 -exec rm -r "{}" \;
* 21:50 thcipriani: contint1001:rm apps-android-wikipedia-test builds > 30 days old
* 15:42 Krenair: dropped https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/434055/ cherry-pick and rebased to fix puppet updates
* 15:37 addshore: for https://gerrit.wikimedia.org/r/472476
* 15:36 addshore: addshore@deployment-deploy01:~$ sudo puppet agent -tv
* 14:30 hashar: updated search-highlighter-maven-java8-docker-site-publish to decouple install and site goals T208496
== 2018-11-07 ==
* 23:00 thcipriani: repooling integration-slave-docker-1017 after cleaning up docker images
* 19:09 Amir1: ores:25dfa4f is going to beta cluster
* 09:06 hashar: building container releng/operations-puppet:0.5.0 for python3 T208873
== 2018-11-06 ==
* 19:56 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/472044
* 15:25 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/blubber/+/471277/
== 2018-11-04 ==
* 17:53 legoktm: deploying https://gerrit.wikimedia.org/r/471251
* 17:50 legoktm: deployed https://gerrit.wikimedia.org/r/471283
== 2018-11-02 ==
* 19:54 Amir1: stopping redis-server on deployment-ores01, this would make ores go down in beta cluster. Testing for T181632
* 14:04 Krenair: made onimisionipe a projectadmin per request in -cloud
* 02:30 legoktm: deploying https://gerrit.wikimedia.org/r/471198
* 00:38 legoktm: deploying https://gerrit.wikimedia.org/r/471186 https://gerrit.wikimedia.org/r/471181
== 2018-11-01 ==
* 17:12 legoktm: deploying https://gerrit.wikimedia.org/r/471047
* 15:55 legoktm: deploying https://gerrit.wikimedia.org/r/471033 https://gerrit.wikimedia.org/r/471034
* 15:41 legoktm: deploying https://gerrit.wikimedia.org/r/471032
* 15:29 legoktm: deploying https://gerrit.wikimedia.org/r/471029
* 15:11 legoktm: building docker for https://gerrit.wikimedia.org/r/471025
* 14:48 andrewbogott: moving deployment-redis05 to labvirt1012
* 14:47 Krenair: shut off deployment-redis05 for migration to new physical host
* 14:45 legoktm: building mediawiki-tarball docker image https://gerrit.wikimedia.org/r/471019
== 2018-10-31 ==
* 21:16 Krenair: remove horizon hiera config for deployment-redis0[56] to unbreak puppet and remove old redis0[12] instance IPs T208040
* 19:55 andrewbogott: moving deployment-elastic06 to labvirt1012
* 19:40 andrewbogott: moving deployment-cpjobqueue to labvirt1012 to help clear out labvirt1017
* 19:11 andrewbogott: moving deployment-kafka-jumbo-1 to labvirt1012 to help clear out labvirt1017
* 18:54 andrewbogott: moving deployment-kafka-main-2 to labvirt1012 to help clear out labvirt1017
* 17:21 Amir1: ores:70ba14b is going to beta
* 13:23 godog: enable statsd reporting for swift
== 2018-10-30 ==
* 16:51 hashar: building docker container releng/operations-puppet:0.4.1 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/470574/
* 14:48 addshore: beta T208307: wikiadmin@10.68.23.30(wikidatawiki)> delete from wb_changes_dispatch where chd_site = 'specieswiki'
* 11:15 hashar: contint1001: cleaned up old Docker images
* 10:13 hashar: Building Docker container releng/operations-puppet:0.4.0 for https://gerrit.wikimedia.org/r/470419 by _joe_
== 2018-10-29 ==
* 22:56 legoktm: deployed https://gerrit.wikimedia.org/r/470453 https://gerrit.wikimedia.org/r/469954 https://gerrit.wikimedia.org/r/470241 https://gerrit.wikimedia.org/r/468900 https://gerrit.wikimedia.org/r/469181
* 22:54 James_F: Manually triggered a beta-update-databases-eqiad run to see if our fix for T208043 has worked.
* 10:04 hashar: cleaned disk on integration-slave-jessie-1002
* 04:07 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n_*
* 00:44 legoktm: also deploying https://gerrit.wikimedia.org/r/470059
* 00:44 legoktm: deploying https://gerrit.wikimedia.org/r/469948
== 2018-10-26 ==
* 09:04 hashar: jenkins: update IRC config at https://integration.wikimedia.org/ci/configure to replace #wikimedia-ri-ci by #wikimedia-infrastructure ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/469034/ )
== 2018-10-25 ==
* 17:21 bearND: (beta): Update mobileapps to 
* 07:50 hashar: enabling puppet again on deployment-deploy01 . Was disabled by _joe_ for apache-fast-test hacking
== 2018-10-24 ==
* 20:44 hasharDinner: Rebuilding CI containers for Quibble 0.0.28
* 20:27 hasharDinner: tagged Quibble 0.0.28 at 
* 08:57 hashar: gerrit: added Lars "liw" Wirzenius to the Administrators group T207830
* 07:27 Krenair: T207825 reapplied role::jobqueue_redis::master to deployment-redis prefix
* 07:01 Krenair: T207825 replacing deployment-redis3-changeprop with deployment-redis3-changeprop02 (jessie m1.small)
* 06:59 Krenair: T207825 moved role::jobqueue_redis::master role from deployment-redis prefix to deployment-redis0[56]
* 05:22 kart_: Beta: Updated cxserver to 
== 2018-10-23 ==
* 23:59 legoktm: deploying https://gerrit.wikimedia.org/r/468171
== 2018-10-22 ==
* 13:01 kart_: Beta: Update cxserver to 
* 09:17 _joe_: deployment-prep: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/468583/, disable puppet on deployment-mediawiki-09
* 00:27 Krenair: Added gtirloni as a member per T207474 - I imagine he'll want to get in to look at shinken-related things
== 2018-10-21 ==
* 21:27 Krenair: rm'd scap l10n stuff on deployment-deploy01 as usual
* 12:25 legoktm: deployed https://gerrit.wikimedia.org/r/468816
== 2018-10-19 ==
* 11:36 hashar: nuking some caches on castor02.integration.eqiad.wmflabs to free up disk space
* 07:08 legoktm: also deploying https://gerrit.wikimedia.org/r/c/integration/config/+/468022
* 07:07 legoktm: deploying https://gerrit.wikimedia.org/r/467329 https://gerrit.wikimedia.org/r/468417 https://gerrit.wikimedia.org/r/468393 https://gerrit.wikimedia.org/r/467724
== 2018-10-18 ==
* 13:27 Amir1: deploying ores:d724d20
== 2018-10-16 ==
* 19:09 twentyafterfour: Branching 1.32.0-wmf.26
* 19:08 legoktm: integration-slave-docker-1017 back online
* 19:07 legoktm: root@integration-slave-docker-1017:/# docker rmi $(docker images -q) to clean up disk space
* 04:22 legoktm: deploying https://gerrit.wikimedia.org/r/467563
* 04:14 legoktm: deploying https://gerrit.wikimedia.org/r/467562
* 04:04 legoktm: deploying https://gerrit.wikimedia.org/r/467560
* 03:06 Krinkle: Rebooted deployment-deploy01 Jenkins slave (stuck for 8 hours)
== 2018-10-15 ==
* 22:28 legoktm: deployed https://gerrit.wikimedia.org/r/467536
* 22:20 legoktm: fixed disk space issue with integration-slave-jessie-1004 and brought it back online
* 22:16 legoktm: legoktm@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 22:09 legoktm: triggering jobs on contint1001 via gear_client, kill if necessary
* 22:03 legoktm: building Quibble PHP 7.3 images (https://gerrit.wikimedia.org/r/467259)
* 20:31 bearND: Beta cluster: Update mobileapps to 
* 19:53 legoktm: rebuilding npm* docker images for https://gerrit.wikimedia.org/r/467451
* 18:13 legoktm: deploying https://gerrit.wikimedia.org/r/467434
* 17:56 legoktm: building php73(-compile)? images https://gerrit.wikimedia.org/r/465204
* 15:14 thcipriani: bring integration-slave-jessie-1001 back online
* 13:30 hashar: Cleaned up disk space on integration-slave-jessie-1001
* 05:10 kart_: Update cxserver to 
* 04:00 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n*
== 2018-10-13 ==
* 04:45 legoktm: rebuilding docker images for https://gerrit.wikimedia.org/r/467023
* 04:34 legoktm: deploying https://gerrit.wikimedia.org/r/467022
* 03:20 Krinkle: Fixed console section pattern for "setup quibble mw-update"
* 03:19 Krinkle: Fixed console section pattern for "setup quibble npm-ls"
* 03:04 legoktm: deploying https://gerrit.wikimedia.org/r/467016
* 02:15 legoktm: deploying https://gerrit.wikimedia.org/r/467012
* 01:55 legoktm: building quibble-stretch-php72 image https://gerrit.wikimedia.org/r/466991
* 01:25 legoktm: rolling out https://gerrit.wikimedia.org/r/c/integration/config/+/466950
== 2018-10-12 ==
* 21:13 legoktm: deployed https://gerrit.wikimedia.org/r/465049
* 20:32 Krinkle: deploy01 /usr/local/bin/mwscript update.php --wiki=simplewiki --quick
* 20:32 Krinkle: deploy01 /usr/local/bin/mwscript update.php --wiki=ruwiki --quick
* 20:14 mutante: contint1001 - gzip merge-debug.log.2018-* debug.log.2018-* etc.. all in /var/log/zuul/ there seems to be no compression (but max age is 1 month)
* 20:11 mutante: contint1001 - gzip zuul.log.2018-* in /var/log/ to prevent running out of disk
* 20:08 mutante: contint1001 - apt-get clean - frees 1GB to bring it _just_ under the Icinga warning threshold :p
* 19:23 legoktm: rebuilding docker images for https://gerrit.wikimedia.org/r/465606
* 17:18 marxarelli: bringing integration-slave-docker-1021 back online
* 17:16 marxarelli: killing 7 long-running containers (> 1 hour) on docker integration nodes (T198517)
* 16:42 marxarelli: killing 9 long-running containers (> 1 hour) on docker integration nodes (T198517)
* 16:31 marxarelli: taking integration-slave-docker-1021 offline (host is unresponsive)
* 16:25 marxarelli: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465671 for 321 affected jobs (T198517)
* 16:05 marxarelli: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465671 for 1 job mediawiki-quibble-vendor-mysql-php70-docker (T198517)
* 15:30 marxarelli: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465668 for 320 affected jobs
* 15:26 marxarelli: killing 51 long-running containers (> 1 hour) on docker integration nodes (T198517)
* 13:22 Reedy: running foreachwiki update.pph
* 11:48 Amir1: ores:ba4ea40 is going beta
* 05:14 legoktm: deployed https://gerrit.wikimedia.org/r/466821
* 04:40 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/466812
* 03:52 legoktm: deployed https://gerrit.wikimedia.org/r/466811
* 03:46 legoktm: tagged 0.0.27 of quibble ()
== 2018-10-11 ==
* 23:50 marxarelli: rolling back deployment of https://gerrit.wikimedia.org/r/c/integration/config/+/465668 (full deployment delayed until around 1600 tomorrow)
* 23:45 marxarelli: killed 6 long-running containers (> 3 hours) on docker integration nodes (T198517)
* 23:34 marxarelli: redeploying https://gerrit.wikimedia.org/r/c/integration/config/+/465668 for 1 job mediawiki-quibble-vendor-mysql-php70-docker
* 23:14 marxarelli: rolling back deployment of https://gerrit.wikimedia.org/r/c/integration/config/+/465668 for 1 job
* 23:08 marxarelli: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465668 for 1 job mediawiki-quibble-vendor-mysql-php70-docker
* 19:28 thcipriani: integration-slave-jessie-1003 back online
* 19:28 thcipriani: integration-slave-jessie-1003:sudo rm -rf /srv/jenkins-workspace/workspace/*
* 18:18 Amir1: ores:1a5630f is going beta
* 18:17 marxarelli: killed 42 long-running (> 3 hours) containers on docker integration nodes (T198517)
* 18:16 marxarelli: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/463852/ to all 320 affected jobs
* 17:47 marxarelli: killing long-running containers (> 3 hours) on docker integration nodes (T198517)
* 17:37 marxarelli: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/463852/ to 8 wmf-quibble-* jobs
* 02:03 Krinkle: Fix Jenkins' section pattern for 'setup-docker-quibble-castor-load'
* 02:03 Krinkle: Fix Jenkins' collapsible sections for "npm-install" and "mw-install" under quibble to match current patterns
== 2018-10-10 ==
* 19:38 awight: restarted ORES celery workers on ores2003 (~17:00), ores200* (17:05)
* 17:15 twentyafterfour: cowboy-coding on deployment-deploy01 to solve scap fatal error dcheck failing to catch fatals
== 2018-10-09 ==
* 19:00 Krinkle: Re-enable beta-scap-eqiad job
* 18:19 Krinkle: Messing with scap in beta to test T121597 / D1114
* 15:32 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n_*
== 2018-10-08 ==
* 03:31 kart_: Update(d) cxserver to 
* 03:02 legoktm: onlined integration-slave-jessie-1002
* 02:20 legoktm: legoktm@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-10-06 ==
* 13:59 Reedy: cleared some large folders out of /tmp on deployment-deploy01
== 2018-10-05 ==
* 19:47 marxarelli: bringing integration-slave-docker-1040 back online
* 19:05 marxarelli: taking integration-docker-slave-1040 offline for docker daemon restart
* 19:04 marxarelli: bringing integration-slave-docker-1038/1041/1043 back online
* 19:02 marxarelli: taking integration-docker-slave-1038/1041/1043 offline for docker daemon restart
* 19:01 marxarelli: bringing integration-slave-docker-1033/1037 back online
* 18:58 marxarelli: taking integration-docker-slave-1033/1037 offline for docker daemon restart
* 18:56 marxarelli: bringing integration-slave-docker-1034 back online
* 18:56 marxarelli: integration-puppetmaster01:/var/lib/git/operations/puppet is up-to-date again after manually updating submodules and subsequent automated git-sync-upstream
* 18:33 marxarelli: taking integration-slave-docker-1034 offline for docker daemon restart
* 11:18 Krenair: rm -rf /tmp/scap_l10n_* on deployment-deploy01
* 06:00 legoktm: deployed https://gerrit.wikimedia.org/r/464757
* 04:35 legoktm: deploying https://gerrit.wikimedia.org/r/464747
* 01:17 legoktm: triggering php71 quibble jobs manually via contint1001
== 2018-10-04 ==
* 22:23 thcipriani: updated jenkins jobs: https://phabricator.wikimedia.org/P7634
* 22:19 thcipriani: updating 103 jenkins jobs that use ci-src-setup-simple
* 21:45 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/464719/
* 21:13 marxarelli: bringing integration-slave-jessie-1003 back online
* 21:12 marxarelli: deleted /mnt/home/jenkins-deploy/{.m2,.gradle} on integration-slave-jessie-1003
* 21:10 marxarelli: deleting cache directories in /mnt/home/jenkins-deploy on integration-slave-jessie-1003 to free up disk space
* 20:31 marxarelli: terminating long-running (> 3 hours) CI docker containers (T198517)
== 2018-10-03 ==
* 18:45 marxarelli: deploying for all 313 affected jobs
* 18:16 marxarelli: deploying for 12 mediawiki-quibble-vendor-* jobs
* 17:46 marxarelli: removing old workspaces on integration-slave-docker-1034 (`rm -rf /srv/jenkins-workspace/workspace/*`) and bringing back online
* 16:52 thcipriani: reloading zuul to deploy Change gate-and-submit-l10n to low precedence
* 15:46 marxarelli: bringing integration-slave-docker-1040/1043 nodes back online after killing long running docker job and freeing up /var/lib/docker space (T206134)
* 15:40 marxarelli: killing long running docker jobs on integration-slave-docker-1040/1043 which are filling up /var/lib/docker with log output
* 15:12 thcipriani: reloading zuul to deploy l10 pipeline
* 14:53 thcipriani: integration-slave-docker-1038 removed workspaces, brought back online integration-slave-docker-104{0,3} need more investigation since docker is full(?)
* 07:07 mdholloway: deployment-maps04 deployed [kartotherian/deploy@27062b4]: Specify WDQS endpoint in the service config (T205607)
* 06:22 _joe_: cherry-picking 455154 (php-fpm installation) on deployment-prep
== 2018-10-02 ==
* 21:34 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/464048/
* 18:22 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n_*
* 18:15 legoktm: deploying https://gerrit.wikimedia.org/r/464000
* 16:45 thcipriani: bring integration-slave-docker-1037 back online after rm -rf /srv/jenkins-workspace/workspace/*
* 16:20 thcipriani: investigating integration-slave-docker-1037
* 08:56 godog: bounce logstash
* 07:59 legoktm: deployed https://gerrit.wikimedia.org/r/463905
* 07:38 legoktm: building quibble-stretch-php71 docker image
* 05:32 legoktm: rebuilding quibble-stretch images https://gerrit.wikimedia.org/r/463883
* 03:26 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/463886
== 2018-10-01 ==
* 20:09 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n_* to remove stale l10n json and free up space
* 17:13 marxarelli: bringing integration-slave-docker-1041 back online following source directory clean up (T205902)
* 16:52 marxarelli: removing old workspace src directories left by non-quibble docker jobs on integration-slave-docker-1041
* 07:10 mdholloway: deployment-maps04 updated kartotherian and tilerator to latest
* 05:24 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@07cbfb4]: Update mobileapps to 
== 2018-09-29 ==
* 14:48 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/463615
* 01:22 Krinkle: Marking integration-slave-docker-1041 as offline (again). Why/How did it come back?
* 00:24 Krinkle: Marking integration-slave-docker-1041 as offline. Various odd build failures, including https://integration.wikimedia.org/ci/job/mediawiki-quibble-composer-mysql-php70-docker/6913/console
== 2018-09-28 ==
* 15:31 Amir1: ladsgroup@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/deleteLocalPasswords.php --wiki=fawiki --prefix (T201009)
* 14:50 thcipriani: investigating integration-slave-docker-1041
* 07:35 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@bf09080]: Update mobileapps to 
== 2018-09-27 ==
* 17:53 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@a0054ba]: Update mobileapps to 
* 07:38 mdholloway: deployment-maps04 updated tilerator and kartotherian node modules (T195513, T200594)
== 2018-09-26 ==
* 15:13 thcipriani: integration-slave-docker-1034:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online -- https://phabricator.wikimedia.org/P7592
* 15:05 thcipriani: integration-slave-docker-1033:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online
* 14:47 thcipriani: investigating integration-slave-docker-103{3,4}
* 11:57 Amir1: gerrit:462927 (ores) is going to beta
* 08:24 hashar: Restarting CI Jenkins on contint1001 [#2]
* 08:14 hashar: Restarting CI Jenkins on contint1001
== 2018-09-25 ==
* 23:01 marxarelli: configured new jenkins node integration-slave-docker-1043 with 6 executors
* 23:01 marxarelli: replaced integration-slave-docker-1042 with new integration-slave-docker-1043 instance
* 22:39 marxarelli: launching new integration-slave-docker-1042 bigram instance
* 22:33 marxarelli: deleting remaining m1.medium instances used as m4executors (T205362)
* 22:15 marxarelli: taking remaining m1.medium m4executor jenkins nodes offline (T205362)
* 18:16 marxarelli: reconfiguring bigram jenkins nodes to use 6 executors. 7 were configured by mistake (T205362)
* 18:00 marxarelli: configuring new integration-slave-docker-1041 jenkins node with 7 executors (T205362)
* 17:42 marxarelli: configuring new jenkins node integration-slave-docker-1040 with 7 executors (T205362)
* 17:38 marxarelli: launching integration-slave-docker-1041 bigram instance (T205362)
* 17:30 marxarelli: the puppet parameter for docker_lvm_volume specified in horizon was not applied correctly on the first puppet run for some reason. tearing down integration-slave-docker-1039...
* 17:25 marxarelli: launching integration-slave-docker-1040 bigram instance (T205362)
* 17:24 marxarelli: deleting instances integration-slave-docker-1007/1008 (T205362)
* 17:13 marxarelli: launching new integration-slave-docker-1039 bigram instance
* 17:12 marxarelli: taking integration-slave-docker-1007/1008 offline for replacement (T205362)
* 17:09 marxarelli: deleting integration-slave-docker-1030/1031 instances (T205362)
* 17:05 marxarelli: taking integration-slave-docker-1030/1031 offline for replacement
* 16:47 marxarelli: increasing executors to 7 for jenkins nodes integration-slave-docker-1033/1034
* 16:46 marxarelli: new instance creation delayed due to quota
* 16:45 marxarelli: launching new integration-slave-docker-1039/1040 bigram instances
* 01:21 legoktm: deployed https://gerrit.wikimedia.org/r/450508
* 00:36 legoktm: deploying https://gerrit.wikimedia.org/r/462609
* 00:22 legoktm: deploying https://gerrit.wikimedia.org/r/453447
== 2018-09-24 ==
* 20:21 bearND: (beta): Update mobileapps to 
* 10:55 hashar: gerrit: granting labs/tools/* project owners the ability to submit changes https://gerrit.wikimedia.org/r/#/c/labs/tools/+/462420/
* 09:51 hashar: deployment-deploy01 : backed up /srv/mediawiki-staging/php-master/cache/gitinfo and created a new. Its size of 69632 bytes might cause slow writes?? T204762
* 09:24 hashar: Live hacked scap code on deployment-deploy01 for T204762 and reverted hack changes
* 08:32 hashar: deployment-deploy01 rm -fR /tmp/scap_l10n_*
* 06:41 legoktm: deploying https://gerrit.wikimedia.org/r/462341
* 03:45 kart_: Update cxserver to 
== 2018-09-23 ==
* 14:03 Krenair: rm stuff in deployment-deploy01:/tmp to try to clear space and stop shinken whining
* 01:05 andrewbogott: rebooted deployment-maps03; OOM and also T205195
== 2018-09-22 ==
* 20:51 Hauskatze: github: deleting several wikimedia/mediawiki-extensions-Collection-.* mirror repos for T183891
* 20:05 Hauskatze: github: deleted mirror wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-zim_renderer T183891; moving to the next one
* 18:21 Krenair: went to do the same with deployment-maps03 and accidentally broke SSH access to the server
* 18:21 Krenair: removed ferm package from deployment-snapshot01 as it appeared unmanaged by puppet and was causing problems with SSH access from the current deployment hosts (previous logs referenced T153468, this just explains why puppet hadn't purged stuff)
* 18:01 Krenair: rm deployment-maps03:/etc/ferm/conf.d/10_redis_exporter_6379 as it was breaking ferm from starting (T153468), puppet has not re-created it so I assume it was historical (shouldn't puppet be purging such files?)
* 18:00 Krenair: rm deployment-snapshot01:/etc/ferm/conf.d/10_prometheus-nutcracker-exporter as it was breaking ferm from starting (T153468), puppet has not re-created it so I assume it was historical (shouldn't puppet be purging such files?)
== 2018-09-21 ==
* 17:26 marxarelli: adding jenkins node integration-slave-docker-1038 with 7 executors
* 16:47 marxarelli: added new jenkins node integration-slave-docker-1037 with 7 executors
* 15:49 marxarelli: replacing integration-slave-docker-1036 with new bigram instance
* 15:48 marxarelli: taking node integration-slave-docker-1035 offline due to unusually high steal cpu time and long build durations
* 15:17 marxarelli: integration-slave-docker-1035/1036 showing unusually high cpu steal and unusually long mean build durations
* 15:15 marxarelli: taking integration-slave-docker-1036 offline due to unusually high cpu steal % trend
* 15:13 marxarelli: launching integration-slave-docker-1037 bigram instance
* 13:03 Amir1: ores:7b987a7 is going beta
* 05:32 legoktm: deployed https://gerrit.wikimedia.org/r/461510
== 2018-09-20 ==
* 23:48 marxarelli: adding new integration-slave-docker-1035/1036 jenkins nodes, each with 7 executors
* 23:23 marxarelli: launching integration-slave-docker-1035/1036 bigram instances
* 23:20 marxarelli: taking integration-slave-docker-1004/1005 offline for replacement (T202160)
* 16:52 Amir1: deploy ores:ee2d28b
* 11:21 hashar: Refreshing jenkins jobs to get rid of docker run option "--tmp /tmpfs" . It is mounted with 'noexec' which causes various jobs to fail. T203181 and T204919
* 11:17 hashar: deployment-deploy01: removed /srv/deployment/analytics/refinery-cache (8GBytes)
* 11:07 hashar: deployment-deploy01 is out of disk space (again)
== 2018-09-19 ==
* 21:18 Hauskatze: github: deleted https://github.com/wikimedia/mediawiki-services-ocg-collection T183891
* 20:07 bearND: (beta): Update mobileapps to 
* 18:56 Amir1: ores:76fe25a goes to beta (T204862)
== 2018-09-18 ==
* 10:04 hashar: Updating Quibble Jenkins jobs to 0.0.26
* 09:46 hashar: updating mwselenium-quibble-docker	 to Quibble 0.0.26
* 07:48 hashar: Updating jenkins jobs to use Quibble 0.0.25
* 07:34 hashar: deployment-sca01: rm -fR /srv/ores /srv/deployment/cxserver.jenkins # untouched since 2016
* 07:31 hashar: cleaning disk on deployment-sca01
* 00:30 marxarelli: configuring integration-slave-docker-1034 jenkins node to use 6 executors
== 2018-09-17 ==
* 22:06 Hauskatze: added missing log entries (actor, etc.) on AbuseFilter (addMissingLoggingEntries.php) for beta
* 22:00 Reedy: update.php in screen is done
* 21:53 Hauskatze: maurelio@deployment-deploy01:~$ mwscript extensions/TorBlock/maintenance/loadExitNodes.php --wiki=deploymentwiki --force (998 nodes loaded)
* 21:12 Reedy: running `foreachwiki update.php --quick` in screen on deployment-deploy01
* 20:17 bearND: (beta): Update mobileapps to 
* 20:14 marxarelli: running `jenkins-jobs update config/ 'service-pipeline*'` to deploy 
* 19:37 hasharAway: jenkins: remove compiler02.puppet3-diffs.eqiad.wmflabs and compiler03.puppet3-diffs.eqiad.wmflabs from jenkins config. Instances deletd
* 17:25 mdholloway: deployment-maps04 updated kartotherian and tilerator to latest (T109776)
* 13:24 hashar: Building docker images for Quibble 0.0.25 https://gerrit.wikimedia.org/r/460536
* 12:58 hashar: mwselenium-quibble-docker to Chromium 69 https://gerrit.wikimedia.org/r/#/c/integration/config/+/460512/ T204214
* 12:27 phuedx: phuedx@deployment-deploy01:~$ foreachwikiindblist gettingstarted-with-category-suggestions.dblist extensions/GettingStarted/maintenance/populate_categories.php
* 09:15 Amir1: ladsgroup@deployment-deploy01:~$ foreachwikiindblist all-labs populateChangeTagDef.php --set-user-tags-only --force
* 07:04 legoktm: building npm6 docker images: https://gerrit.wikimedia.org/r/453441 https://gerrit.wikimedia.org/r/451690 https://gerrit.wikimedia.org/r/453445
* 00:10 legoktm: deployed https://gerrit.wikimedia.org/r/460775
== 2018-09-16 ==
* 23:01 legoktm: deployed https://gerrit.wikimedia.org/r/460774 https://gerrit.wikimedia.org/r/#/c/integration/config/+/460578/
* 16:26 Hauskatze: phabricator: Account https://phabricator.wikimedia.org/p/Ponani/ disabled by me due to vandalism T200856
* 12:47 Hauskatze: github: deleting <https://github.com/wikimedia/mediawiki-extensions-CommunityTwitter> T204372
== 2018-09-14 ==
* 17:47 marxarelli: adding instance-type-* labels to m4executor nodes in jenkins
* 14:27 hashar: deployment-deploy01 /srv/mediawiki synced
* 14:16 hashar: deployment-deploy01: sudo rm -fR /srv/mediawiki && mkdir /srv/mediawiki && chown mwdeploy:mwdeploy /srv/mediawiki && scap pull
* 13:57 hashar: deployment-deploy01: mv /srv/mediawiki/php-master/cache/l10n/*.php /srv/mediawiki/php-master/cache/l10n/backup-php/
* 13:55 hashar: deployment-deploy01 13:53:58 rsync: write failed on "/srv/mediawiki/php-master/cache/l10n/upstream/l10n_cache-ab.cdb.json": No space left on device (28)
* 13:48 hashar: Tagged Quibble 0.0.25 
* 11:49 hashar: Cleaned /tmp on deployment-deploy01
* 11:31 hashar: Manually running https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28313/ T204340
* 05:50 legoktm: deployed https://gerrit.wikimedia.org/r/460475
* 05:40 legoktm: deployed https://gerrit.wikimedia.org/r/460473
== 2018-09-13 ==
* 19:20 legoktm: deployed https://gerrit.wikimedia.org/r/457109
* 16:22 hashar: Cleaning disk space on deployment-deploy01.deployment-prep.eqiad.wmflabs
* 13:06 hashar: Nuking castor cache /srv/jenkins-workspace/caches/mediawiki-vendor mediawiki/vendor fails composer --ansi validate --no-check-publish
* 08:57 hashar: Updating jenkins job mwselenium-quibble-docker to use Chromium 69/Firefox 60 T203902
* 08:42 hashar: aborted maintenance-disconnect-full-disks job T204077
* 05:58 legoktm: updating docker for https://gerrit.wikimedia.org/r/460199
* 05:55 legoktm: deployed https://gerrit.wikimedia.org/r/460197 (no-op)
* 05:13 legoktm: onlined integration-slave-docker-1021
* 05:12 legoktm: legoktm@integration-slave-docker-1021:~$ sudo docker rmi $(sudo docker images -q)
* 05:08 legoktm: deployed
* 02:23 legoktm: deploying https://gerrit.wikimedia.org/r/460183 / https://gerrit.wikimedia.org/r/460184
== 2018-09-12 ==
* 21:17 marxarelli: adding new jenkins node integration-slave-docker-1034 with 4 executors
* 21:01 marxarelli: launching integration-slave-docker-1034 bigram instance
* 20:59 marxarelli: deleting integration-slave-docker-1003/-1004 instances
* 20:57 marxarelli: taking integration-slave-docker-1003/-1004 offline for replacement
* 20:51 marxarelli: deleting integration-slave-docker-1002 instance
* 20:50 marxarelli: taking integration-slave-docker-1002 offline for replacement
* 20:16 marxarelli: adding newly provisioned integration-slave-docker-1033 jenkins node with 4 executors
* 20:01 marxarelli: launching new integration-slave-docker-1033 bigram instance
* 19:58 marxarelli: replacing integration-slave-docker-1032 offline 85/15% split for docker/workspace left too little space for workspace. puppet change has been updated to use 70/30% volume space ratio
* 18:44 marxarelli: adding jenkins node integration-slave-docker-1032 with 4 executors
* 18:25 marxarelli: launching new bigram instance integration-slave-docker-1032
* 18:21 marxarelli: deleting integration-slave-docker-1027 instance
* 18:20 marxarelli: deleting jenkins node integration-slave-docker-1027 due to insufficient /var/lib/docker space (replaced with 1031 which has dedicated /var/lib/docker volume)
* 18:18 marxarelli: added new jenkins node integration-slave-docker-1031 with 4 executors
* 18:00 marxarelli: launching new xlarge instance integration-slave-docker-1031
* 17:59 marxarelli: deleting integration-slave-docker-1001 instance
* 17:58 marxarelli: deleting now idle node integration-slave-docker-1001
* 17:52 marxarelli: removing integration-slave-docker-1001 jenkins node for replacement with a xlarge instance
* 17:40 marxarelli: adding Jenkins node integration-slave-docker-1030
* 17:36 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/460067/
* 15:49 marxarelli: provisioning new xlarge integration-slave-docker-1030
* 15:47 marxarelli: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/459875 on integration-puppetmaster01 for testing
* 09:38 hashar: Updating jobs oojs-ui-docker-publish oojs-ui-npm-run-jenkins-node-6-docker for Chromium 69 and Firefox 60 - T203902
* 00:14 marxarelli: deleting instance integration-slave-docker-1029
== 2018-09-11 ==
* 22:43 marxarelli: launching m1.xlarge integration-slave-docker-1029 using stretch image
* 22:40 marxarelli: deleting integration-slave-docker-1028 in favor of trying a stretch instance
* 22:05 marxarelli: launching replacement instance integration-slave-docker-1028
* 21:49 marxarelli: removing unresponsive jenkins node integration-slave-docker-1025
* 21:39 marxarelli: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/459850 on integration-puppetmaster01
* 09:38 hashar: sudo cumin --force 'name:docker' 'rm -fR /srv/jenkins-workspace/workspace/selenium-daily-beta-Wikibase*' # T188742
== 2018-09-10 ==
* 14:17 hashar: Update CI Quibble jobs to explicitly set the log directory to /log (--log-dir /log) https://gerrit.wikimedia.org/r/#/c/integration/config/+/458788/
* 08:34 Amir1: ores is going to beta
== 2018-09-09 ==
* 19:47 legoktm: deploying https://gerrit.wikimedia.org/r/459401
* 02:32 legoktm: deploying https://gerrit.wikimedia.org/r/458980
* 00:11 legoktm: updating docker-pkg for https://gerrit.wikimedia.org/r/459265
== 2018-09-08 ==
* 20:22 legoktm: killed https://integration.wikimedia.org/ci/job/maintenance-disconnect-full-disks/1503/ because it was taking 2+ hours
* 20:20 legoktm: deployed https://gerrit.wikimedia.org/r/457077
* 20:19 legoktm: deleted mwext-PoolCounter-* jobs since they're now unused
* 03:06 legoktm: deploying https://gerrit.wikimedia.org/r/458990 https://gerrit.wikimedia.org/r/458996
* 01:51 legoktm: deploying https://gerrit.wikimedia.org/r/458956 https://gerrit.wikimedia.org/r/458957
* 00:18 legoktm: deployed https://gerrit.wikimedia.org/r/458882
== 2018-09-07 ==
* 20:01 marxarelli: bringing integration-slave-docker-1006 online again since disk space has been reclaimed
* 19:15 Krinkle: marked integration-slave-docker-1025 offline (no space), aborted builds manualy
* 18:30 legoktm: started gear_client on contint1001
* 18:19 marxarelli: setting integration-slave-docker-1026 executors to 4 to avoid disk space exhaustion due to concurrent builds
* 18:12 Krinkle: Marking integration-slave-docker-1026 offline (ENOSPC)
* 09:30 hashar: integration-slave-docker-1025 lower number of executors from 5 to 4. 8 CPUS can not sustain 5 concurrent Quibble builds T201972
== 2018-09-06 ==
* 18:54 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/458553
* 18:48 legoktm: reverted tmpfs change for search-mjolnir-tox-docker
* 17:37 thcipriani: integration-slave-docker-1026 sudo docker kill -> stopped a container that had been running for 5 hours
* 16:33 thcipriani: mark integration-slave-docker-1026 back online after diskspace recovery
* 12:08 hashar: cleaned integration-slave-docker-1012 and integration-slave-docker-1026
== 2018-09-05 ==
* 18:31 thcipriani: bring integration-slave-docker-1026 back online since disk space is normal again
* 07:47 elukey: tested and removed a patch to operations/puppet on puppetmaster03. Solved a git rebase conflict between two changes (hope I did it well) and updated the nginx submodule
* 04:08 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/457070 (tmpfs for /tmp) to all *tox*docker and *composer*docker jobs
* 03:35 legoktm: deploying https://gerrit.wikimedia.org/r/458095
== 2018-09-04 ==
* 20:40 Hauskatze: github: created https://github.com/wikimedia/mediawiki-extensions-CreatedPagesList for T203431
* 17:16 legoktm: deploying https://gerrit.wikimedia.org/r/457905
* 04:51 legoktm: deployed https://gerrit.wikimedia.org/r/457808
== 2018-09-03 ==
* 21:16 legoktm: deploying https://gerrit.wikimedia.org/r/457070 (tmpfs for /tmp) to all *quibble* jobs
* 08:39 hashar: builder releng/quibble-stretch-bundle:0.0.24-4
* 08:21 Hauskatze: github: added "this is a mirror" notice (as nearly all other repos have) to https://github.com/wikimedia/mediawiki-extensions-UserExport
* 07:54 hashar: deployment-deploy01 scap sync-l10n master
* 07:52 hashar: deployment-deploy01 chown -R l10nupdate:l10nupdate /srv/mediawiki-staging/php-master/cache/l10n
* 07:44 hashar: fixing beta-scap-eqiad failure: Operation not permitted: '/srv/mediawiki-staging/php-master/cache/l10n/upstream/l10n_cache-sah.cdb.json'
* 07:05 legoktm: deployed https://gerrit.wikimedia.org/r/457360
* 05:35 legoktm: deployed https://gerrit.wikimedia.org/r/457293
* 05:20 legoktm: building docker images for https://gerrit.wikimedia.org/r/457278
== 2018-09-02 ==
* 22:25 Hauskatze: maurelio@deployment-sca01:~$ sudo puppet agent -tv attempting to fix "<shinken-wm> PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]"
* 13:26 Hauskatze: maurelio@deployment-deploy01:/srv/mediawiki-staging/php-master/cache$ sudo -u jenkins-deploy chmod -R 777 l10n/ more permissions fixes for beta-scap-eqiad
* 12:48 Krenair: deployment-deploy01: `sudo rm /srv/mediawiki/.git/gc.log` to clear error about permissions problems accessing nonexistent file.
* 12:28 Hauskatze: root@deployment-deploy01:/srv/mediawiki-staging# chown -R jenkins-deploy:wikidev portals scap fixing beta-scap-update-equiad failures
== 2018-09-01 ==
* 23:52 legoktm: deploying https://gerrit.wikimedia.org/r/457070 for mediawiki-quibble-* jobs
* 22:08 legoktm: deploying https://gerrit.wikimedia.org/r/457068
* 06:53 legoktm: deployed https://gerrit.wikimedia.org/r/456812
* 05:05 legoktm: deployed https://gerrit.wikimedia.org/r/456782
* 02:46 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf mediawiki-core-*lint apps-android-wikipedia-test # /srv was getting full
* 00:40 Krinkle: Disconnect Jenkins slave integration-slave-docker-1026 (ENOSPC)
* 00:02 Krenair: armed keyholder on deployment-cumin using deployment-puppetmaster:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
== 2018-08-31 ==
* 11:34 hashar: cleaned workspace on integration.integration-slave-jessie-1002
* 06:41 legoktm: deployed https://gerrit.wikimedia.org/r/456547
* 04:26 legoktm: deploying https://gerrit.wikimedia.org/r/456535
* 04:09 legoktm: deploying https://gerrit.wikimedia.org/r/456534
* 03:09 legoktm: deploying https://gerrit.wikimedia.org/r/456531
* 02:42 legoktm: deploying https://gerrit.wikimedia.org/r/456530
* 02:28 legoktm: deployed https://gerrit.wikimedia.org/r/456528
== 2018-08-30 ==
* 20:47 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/455269 for 18 affected jobs
* 20:34 Hauskatze: github: deleting https://github.com/wikimedia/mediawiki-extensions-SemanticExpressiveness mirror for T199417
* 20:31 Hauskatze: github: deleting https://github.com/wikimedia/mediawiki-extensions-SemanticLinks mirror for T199422
* 20:21 Hauskatze: github: added standard description to https://github.com/wikimedia/mediawiki-extensions-FundraisingEmailUnsubscribe
* 19:38 thcipriani: bring integration-slave-docker-1025 back online
* 19:38 Hauskatze: github: deleted mirror of extension-MwEmbedSupport per T197918
* 16:14 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456362/
* 07:53 legoktm: deploying https://gerrit.wikimedia.org/r/456343
* 07:39 legoktm: deployed https://gerrit.wikimedia.org/r/456341
* 07:19 legoktm: deploying https://gerrit.wikimedia.org/r/456340
* 07:04 legoktm: deploying https://gerrit.wikimedia.org/r/456337
* 05:57 legoktm: manually triggering jobs on contint1001 (this time with 8 parallel jobs, up from 6 previously)
* 03:44 legoktm: deployed https://gerrit.wikimedia.org/r/456328
* 02:55 legoktm: rebuilding docker for https://gerrit.wikimedia.org/r/456073
== 2018-08-29 ==
* 18:43 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456168/ (T203093)
* 18:10 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/456189 (T203110)
* 17:23 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456180/
* 12:35 hashar: fixed keyholder on deployment-deploy01
* 08:51 hashar: cleaned up disk on integration-slave-docker-1001 and integration-slave-docker-1002
== 2018-08-28 ==
* 18:55 hasharAway: Bump Jenkins Quibble jobs to 0.0.23 https://gerrit.wikimedia.org/r/#/c/integration/config/+/455881/ T203009
* 18:04 hasharAway: Rebuilding Docker quibble containers to use Quibble 0.0.23 T203009
* 17:30 legoktm: reverted back to previous quibble version (T203009)
* 16:59 hashar: Switched Jenkins jobs to Quibble 0.0.22 https://gerrit.wikimedia.org/r/455861
* 16:13 hashar: building docker containers for quibble 0.0.22
* 16:06 hashar: tagged quibble 0.0.22 
== 2018-08-27 ==
* 19:45 legoktm: deployed https://gerrit.wikimedia.org/r/446185 https://gerrit.wikimedia.org/r/455226
* 18:49 Amir1: ores:0e8cc73 is going to beta cluster
* 13:47 hashar: updating phplint jobs to use shallow clone AND wipe the workspace https://gerrit.wikimedia.org/r/#/c/integration/config/+/405722/ T179963
* 13:25 hashar: Delete gerrit repo operations/debs/pkg-php/php-ast , was for php-ast 1.6 which is now in Debian T174338
== 2018-08-25 ==
* 06:09 legoktm: deployed https://gerrit.wikimedia.org/r/455282 https://gerrit.wikimedia.org/r/455285
* 04:34 legoktm: deployed https://gerrit.wikimedia.org/r/455279
* 04:26 legoktm: deployed https://gerrit.wikimedia.org/r/454776
* 04:23 legoktm: deployed https://gerrit.wikimedia.org/r/455107
== 2018-08-24 ==
* 19:47 Krenair: set profile::elasticsearch::cirrus::tls_port: 9243 to appease puppet on deployment-elastic* hosts following https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/447568/
* 12:01 addshore: manaully install nmon on integration-slave-docker-1025 to inspect IO
* 11:55 addshore: manaully install iotop on integration-slave-docker-1025 to inspect IO
== 2018-08-23 ==
* 23:27 Krenair: Fixed puppet rebasing on deployment-puppetmaster03 by removing old cherry-picked version as of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/444610/ - it broke puppet rebasing since Wednesday
* 17:28 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/454706 for 270 affected jobs
* 16:43 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/454706 for mediawiki-quibble-vendor-mysql-php70-docker only
* 13:47 _joe_: cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/453093 on deployment-prep again
* 11:06 _joe_: reverting my cherry-pick
* 11:03 _joe_: cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/453093 on deployment-prep
* 04:42 TimStarling: on deployment-puppetmaster03 running puppet-merge
== 2018-08-22 ==
* 20:41 mdholloway: deployment-maps04 deployed kartotherian and tilerator to latest
* 20:05 bearND: (beta): Update mobileapps to (T202105 T202237)
* 12:49 Krenair: deleted deployment-deploy01:/tmp/scap_l10n_2097651226 from 31st July to free up space
* 08:08 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-PdfExport T202220
* 06:24 legoktm: deploying https://gerrit.wikimedia.org/r/454471
* 06:20 legoktm: deploying https://gerrit.wikimedia.org/r/454202 plus 3 commits hashar didn't deploy :< (<3)
* 05:37 TimStarling: on deployment-db03 installing pt-heartbeat
== 2018-08-21 ==
* 18:39 mateusbs17: deployment-maps04 kartotherian/deploy@2047778 "Updating snapshot package on kartotherian"
* 10:37 Amir1: ladsgroup@deployment-deploy01:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki --rebuild-all-terms (T202260)
* 08:09 hashar: castor: removing cache for mediawiki-extensions-* jobs (no more used) rm -fR /srv/jenkins-workspace/caches/*/*/mediawiki-extensions* T202341
* 07:46 legoktm: deployed https://gerrit.wikimedia.org/r/454201
* 04:20 legoktm: deployed https://gerrit.wikimedia.org/r/454192
* 04:05 legoktm: deploying https://gerrit.wikimedia.org/r/453975
* 01:13 legoktm: triggering seccheck jobs manually on contint1001
== 2018-08-20 ==
* 20:37 marxarelli: Reconfigured integration-slave-docker-1026 back to 8 executors (at least 1 vcpu per concurrent job)
* 20:30 marxarelli: Correction: integration-slave-docker-1026 configured w/ 9 executors
* 20:18 marxarelli: Adding jenkins node integration-slave-docker-1026 (bigram instance) w/ 8 executors
* 20:16 mateusbs17: (beta): Update mobileapps to (T202105)
* 19:53 marxarelli: Deleting instance integration-slave-docker-1016
* 19:52 marxarelli: Taking integration-slave-docker-1016 node offline for replacement
* 18:24 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/454038/
* 18:19 marxarelli: Deleting integration-slave-docker-1020 instance
* 18:16 marxarelli: Taking integration-slave-docker-1020 node offline for replacement
* 15:58 hashar: deleting Jenkins slave deployment-tin.eqiad the instance has been replaced T192561
* 02:36 legoktm: deploying https://gerrit.wikimedia.org/r/453899
* 01:42 legoktm: running jobs manually on contint1001 with gear_client
* 01:34 legoktm: deploying https://gerrit.wikimedia.org/r/453895
* 01:24 legoktm: deploying https://gerrit.wikimedia.org/r/453586 https://gerrit.wikimedia.org/r/453894
* 01:08 legoktm: deploying https://gerrit.wikimedia.org/r/452580 https://gerrit.wikimedia.org/r/452617 https://gerrit.wikimedia.org/r/447758
* 01:02 legoktm: deployed https://gerrit.wikimedia.org/r/453419 https://gerrit.wikimedia.org/r/453884
== 2018-08-18 ==
* 00:45 mdholloway: moved maps-beta.wmflabs.org proxy from deployment-maps03 to deployment-maps04
== 2018-08-17 ==
* 21:10 marxarelli: removing local docker images that are not tagged 'latest' to free up space on contint1001
* 19:19 marxarelli: pruning stopped containers and dangling images on contint1001 to free up space
* 16:47 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/450635/
* 13:23 addshore: reload zuul for: Add WikibaseMediaInfo to Wikibase dependencies [integration/config] - https://gerrit.wikimedia.org/r/453400 (https://phabricator.wikimedia.org/T202066) (owner: Addshore)
* 13:17 addshore: reload zuul for: Use Quibble for WikibaseMediaInfo [integration/config] - https://gerrit.wikimedia.org/r/453396 (https://phabricator.wikimedia.org/T202067) (owner: Addshore)
* 12:32 addshore: reload zuul for: Add extension-quibble template to Wikibase [integration/config] - https://gerrit.wikimedia.org/r/453385 (https://phabricator.wikimedia.org/T200508) (owner: Addshore)
* 08:46 addshore: reloaded zuul
* 04:29 legoktm: deployed https://gerrit.wikimedia.org/r/452619 https://gerrit.wikimedia.org/r/452732 https://gerrit.wikimedia.org/r/452618
== 2018-08-16 ==
* 14:37 mdholloway: deployed [mobileapps/deploy@166eafa]: Update mobileapps to (T201979)
== 2018-08-15 ==
* 21:21 Krinkle: krinkle@deployment-deploy01: Removing php-master/StartProfiler.php for T201782.
* 16:24 Reedy: beta-update-databases-eqiad broken due to ooui patch not being merged by jerkins
== 2018-08-14 ==
* 20:23 Krenair: deactivated and cleaned node for deployment-elastic09 which vanished mysteriously
* 17:10 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/452714
* 16:58 mateusbs17: deployment-maps04 kartotherian/deploy@a0f7111 Update node mapnik to 3.7.2 - Fixing submodule deploy
* 16:58 mateusbs17: deployment-maps04 tilerator/deploy@f4c4359 Update node mapnik to 3.7.2 - Fixing submodule deploy
* 16:00 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/452634/
* 14:33 mateusbs17: deployment-maps04 kartotherian/deploy@fed001c Update node mapnik to 3.7.2
* 14:33 mateusbs17: deployment-maps04 tilerator/deploy@31e8585 Update node mapnik to 3.7.2
== 2018-08-13 ==
* 
* 16:54 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/452361/
* 16:03 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/451247
* 16:01 addshore: reloaded zuul 3 hours ago for https://gerrit.wikimedia.org/r/452356 ......
== 2018-08-12 ==
* 23:56 legoktm: deploying https://gerrit.wikimedia.org/r/452277
* 23:13 legoktm: deployed https://gerrit.wikimedia.org/r/452276
* 22:43 legoktm: deployed https://gerrit.wikimedia.org/r/444350 https://gerrit.wikimedia.org/r/447119 https://gerrit.wikimedia.org/r/451525
* 22:29 legoktm: deployed https://gerrit.wikimedia.org/r/451059 https://gerrit.wikimedia.org/r/451058
* 22:18 legoktm: deployed https://gerrit.wikimedia.org/r/451319
* 07:48 legoktm: building doxygen image for https://gerrit.wikimedia.org/r/452179
== 2018-08-10 ==
* 17:58 marxarelli: Reloading zuul to deploy 
* 16:47 marxarelli: Adding "Blubber" label to all integration-slave-docker-* nodes in jenkins
== 2018-08-09 ==
* 22:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451798
* 20:09 Krenair: deleted deployment-urldownloader (the oldest host still running, on trusty, from feb 2015) and replaced with deployment-urldownloader02
* 17:13 bearND: (beta): Update mobileapps to 
* 01:42 awight: T201518: ORES, fawiki wp10, misc updates
== 2018-08-08 ==
* 20:39 James_F: Updated mobileapps to in Beta Cluster
* 15:50 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451212
* 00:24 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451055 / T199850
== 2018-08-07 ==
* 21:37 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/450157/
* 19:59 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/450435 / T201107)
* 19:52 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/450644
* 19:35 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/451005
== 2018-08-06 ==
* 16:11 addshore: reload zuul for https://gerrit.wikimedia.org/r/450604
* 15:42 addshore: reload zuul for https://gerrit.wikimedia.org/r/450590
* 07:53 addshore: reload zuul (I think, without using fab) for https://gerrit.wikimedia.org/r/450514
== 2018-08-04 ==
* 02:40 legoktm: deployed https://gerrit.wikimedia.org/r/450296
== 2018-08-02 ==
* 20:59 legoktm: deployed https://gerrit.wikimedia.org/r/450149
* 20:12 legoktm: deployed https://gerrit.wikimedia.org/r/450085
* 19:01 legoktm: removed all docker images from integration-slave-docker-1004 to free up root partition, cleaned up all workspaces while I was at it (T201077)
* 16:27 gehel: re-imaging depoyment-elastic* to stretch completed
* 14:07 gehel: re-imaging depoyment-elastic* to stretch
* 07:47 legoktm: triggering jobs directly on contint1001 w/ gear_client.py
== 2018-08-01 ==
* 23:00 legoktm: deploying https://gerrit.wikimedia.org/r/449898 https://gerrit.wikimedia.org/r/c/integration/config/+/447454
* 14:56 hashar: Phasing out mediawiki-extensions jobs in favor of wmf-quibble ones T197469
* 13:49 moritzm: upgrade application servers in beta to wikidiff 1.7,2 (T199801)
* 10:08 dcausse: deployment-prep T199270: fix dirty diff on wmf-config/event-schema submodule on deployment-tin and deployment-deploy01
== 2018-07-31 ==
* 22:04 thcipriani: moving active deployment-prep deployment server to deployment-deploy01
* 19:50 marxarelli: Configuring Jenkins to include integration/pipelinelib as an available pipeline library
* 17:50 Krinkle: Apply role::webperf::profiling_tools to deployment-webperf12; T195312 / T180761
* 10:15 hashar: gerrit: deleting branch wmf/es6 on mediawiki/vendor . We use 'es6' branch instead. (made for dcausse )
* 06:05 legoktm: deployed https://gerrit.wikimedia.org/r/449402
* 05:56 legoktm: deploying https://gerrit.wikimedia.org/r/449399
== 2018-07-30 ==
* 23:28 Krinkle: Setting up puppet cert for deployment-webperf12; T195312 / T180761
* 23:06 Krinkle: Perform node deactivate/clean for deleted deployment-webperf13
* 23:04 Krinkle: Create instance deployment-webperf13 (deployment-webperf13 ) - T195312 / T180761
* 23:03 Krinkle: Delete and recreate deployment-webperf13 (T195312 / T180761)
* 21:38 Krenair: ran package upgrades on deployment-imagescaler02 to fix thumbor
* 20:57 mdholloway: deployment-maps04 restarted postgresql to pick up the correct data_directory value (T200634)
* 19:44 hasharAway: renaming Jenkins job "mediawiki-core-code-coverage" to "mediawiki-core-code-coverage-docker" to keep history. It is migrated by https://gerrit.wikimedia.org/r/#/c/integration/config/+/449273/ T195918
* 19:34 Krenair: copied /etc/pam.d/sshd from deployment-restbase01 to deployment-restbase02 to fix missing MOTD, T195709
* 17:04 hashar: github: granted @thcipriani owner rights
* 00:03 legoktm: deploying https://gerrit.wikimedia.org/r/449035 https://gerrit.wikimedia.org/r/449107 https://gerrit.wikimedia.org/r/449087
== 2018-07-29 ==
* 05:34 legoktm: deployed https://gerrit.wikimedia.org/r/449002
== 2018-07-28 ==
* 22:08 legoktm: deployed https://gerrit.wikimedia.org/r/448858
* 03:24 legoktm: legoktm@contint1001:/srv/org/wikimedia/doc/cover-extensions$ sudo -u jenkins-slave rm -rf EtherEditor/ # T197698
== 2018-07-27 ==
* 21:11 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/448762/
* 14:58 hashar: building releng/quibble-stretch:0.0.21-1 container T195918
* 14:43 hashar: Building tox-certcentral container for vgutierrez !
* 13:13 hashar: Refreshing Quibble Jenkins jobs to use 0.0.21 https://gerrit.wikimedia.org/r/#/c/integration/config/+/448479/
* 12:05 hashar: Building releng/quibble images 0.0.21
* 11:40 hashar: Tagged quibble 0.0.21 
* 11:38 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/448468/ T200508
* 11:38 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/448468/
* 01:11 legoktm: deployed https://gerrit.wikimedia.org/r/443433 https://gerrit.wikimedia.org/r/447337
== 2018-07-26 ==
* 16:08 legoktm: deployed https://gerrit.wikimedia.org/r/447227
* 15:03 zeljkof: Reloading Zuul to deploy 
* 04:00 kart_: Updated cxserver to 
== 2018-07-24 ==
* 05:22 kart_: Updated cxserver to 
== 2018-07-23 ==
* 20:06 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@565b41a]: Update mobileapps to 
* 19:16 mdholloway: deployment-maps04 deployed dependency updates for agreement on mapnik v3.5.14
* 16:25 Krinkle: Applying role::webperf::profiling_tools class to deployment-webperf13, T195312
* 16:04 Krinkle: Set up puppet cert stuff on deployment-webperf13 T195312
* 15:57 Krinkle: Set 'puppetmaster' Hiera for deployment-webperf13 / T195312
* 15:34 Krinkle: Creating deployment-webperf13 - T195312
* 15:34 Krinkle: Deleting deployment-webperf12 - T195312
* 10:26 hashar: Regenerate all debian-glue jobs from JJB. Just to be sure.
* 10:24 hashar: Regenerate https://integration.wikimedia.org/ci/job/debian-glue/ from JJB. The timeout is forced to 3 when it should rely on the BUILD_TIMEOUT environment variable
== 2018-07-19 ==
* 21:47 hashar: CI: Force npm loglevel to 'info' via a Zuul parameter T198348
* 21:27 hashar: nodepool: deleted a few lost instances due to jenkins arbitrary reboots
* 15:53 hashar: CI: npmjs is slow. Bumping Quibble jobs timeout from 30 to 45 minutes https://gerrit.wikimedia.org/r/#/c/446863 T198348
* 08:52 hashar: Switching back timeout of Quibble jobs from 30 to 45 minutes https://gerrit.wikimedia.org/r/442989 T198348
== 2018-07-17 ==
* 01:36 Krinkle: Applying role::webperf::profiling_tools class to webperf12 in Beta Cluster - T195312, T180761.
== 2018-07-16 ==
* 21:10 awight: ran namespaceDupes.php on beta enwiki
* 19:56 bearND: (beta): Update mobileapps to 
== 2018-07-14 ==
* 21:20 Krenair: redirected security_audit traffic (see T72181) traffic from deployment-mediawiki06 to deployment-mediawiki-09 to fix puppet on varnish (06 was deleted in T192996)
* 03:27 Krinkle: Clearing various workspaces on integration-slave-jessie-1001 to fix operations-mw-config-php55lint Jenkins builds - T179963
== 2018-07-13 ==
* 17:31 Reedy: Reloading Zuul to deploy stuff
== 2018-07-12 ==
* 22:45 Krenair: deployment-maps04 groupadd -g 1000 maps-admins
* 19:04 mdholloway: deployment-prep: deleting new instance deployment-maps04 (initial puppet run failed) and creating deployment-maps05
== 2018-07-11 ==
* 20:07 bearND: (beta): Update mobileapps to 
* 19:53 Reedy: next set of db updates for beta might be a bit slow. Expected!
* 13:35 mdholloway: deployment-prep: launched new instance deployment-maps04 for maps testing on stretch
== 2018-07-10 ==
* 21:48 hashar: manually cleaning up disk space on integration-slave-jessie-1002 ( apt-get clean , apt-get autoremove --purge , nuked /srv/jenkins-workspace/workspace/*lint
* 21:27 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-Transliterator T199103
* 18:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/443006 (and apparently https://gerrit.wikimedia.org/r/444884 as well)
* 18:31 Krenair: deleted deployment-dumps-puppetmaster earlier per ariel in -releng (some details in T72792)
* 13:05 hashar: Building container releng/quibble-stretch:0.0.20 https://gerrit.wikimedia.org/r/444876 T198171
== 2018-07-09 ==
* 23:24 Krinkle: Ran `puppet node deactivate` and `puppet node clean` for deployment-apertium02.deployment-prep.eqiad.wmflabs (T142152)
* 22:09 Krinkle: Deleting instance deployment-apertium02 - T142152
* 19:10 Krenair: deleted deployment-mediawiki06 T192996
* 17:35 zeljkof: Reloading Zuul to deploy 
* 10:48 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-Premoderation T198952
* 10:47 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-SwarmExport T198957
* 10:41 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-VisualChat T198955
* 10:40 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-ContributorsAddon T198864
== 2018-07-08 ==
* 20:17 Krinkle: Shutting off deployment-apertium02 (T142152)
* 18:31 Krenair: deleted deployment-mx T184244
* 16:55 Krenair: deleted deployment-redis01 T179371
* 16:54 Krenair: deleted deployment-redis02 T179371
* 16:42 Krenair: deleted deployment-puppetmaster02. root and home files are in archives under deployment-puppetmaster03:/root/
== 2018-07-07 ==
* 03:38 thcipriani: deployment-tin:sudo rm -rf /srv/mediawiki/.git
== 2018-07-06 ==
* 03:53 kart_: Updated cxserver to 
== 2018-07-05 ==
* 06:27 kart_: Updated cxserver to 
== 2018-07-04 ==
* 20:15 Reedy: beta unbroke beta code autodeploy T72597
* 05:31 Krinkle: Fix beta-scap-eqiad by cherry-picking https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/443767/ T192996
* 05:31 Krinkle: beta-scap-eqiad failing due to mediawiki06 still being a scap target (despite being shutdown per T192996)
== 2018-07-03 ==
* 22:11 Krenair: rebased stuff on deployment-puppetmaster and ran puppet everywhere through cumin
* 21:48 Krenair: cleaned up cherry-pick conflict to try to fix puppet
* 08:44 hashar: Building docker container releng/tox-labs-striker to add libssl-dev T198076
* 04:31 Krinkle: Setting up puppetmaster/cerf for deployment-webperf12 (T195312)
* 03:49 Krinkle: Create deployment-webperf12 as equivalent of webperf1002/webperf2002 in prod (T195312, T194390)
* 03:41 Krinkle: Shut off deployment-mediawiki06 from Horizon (but not yet deleted) - T192996
== 2018-07-02 ==
* 20:28 hashar: building docker image releng/tox:0.1.1 , that adds libssl-dev T198076
* 16:35 hashar: github: delete wikimedia/mediawiki-extensions-PubSubHubbub T198322
* 00:25 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/443023
== 2018-06-30 ==
* 20:40 Krenair: ran git gc on deployment-tin:/srv/mediawiki to free up space
== 2018-06-28 ==
* 21:17 hasharAway: castor02: nuking cache of npm/node jobs via rm -fR /srv/jenkins-workspace/caches/*/*/*node* (note: other jobs might still have a npm cache) T198348
* 21:09 hasharAway: castor: nuking caches castor-mw-ext-and-skins/master/wmf-quibble-vendor-mysql-hhvm-docker/npm and castor-mw-ext-and-skins/master/wmf-quibble-vendor-mysql-php70-docker/npm T198348
* 21:07 hasharAway: castor: nuking caches mediawiki-core/master/mediawiki-quibble-vendor-mysql-php70-docker and mediawiki-core/master/mediawiki-quibble-vendor-mysql-hhvm-docker T198348
* 11:18 Hauskatze: Ran namespaceDupes for eswiki and eswikibooks following namespace changes on JADE.
== 2018-06-27 ==
* 16:27 hashar: Building Docker containers releng/quibble-jessie-php55:0.0.19-1 and releng/quibble-stretch:0.0.19-1 T196346 T198336
* 04:45 mdholloway: deployed to beta: [mobileapps/deploy@2207b66]: Update mobileapps to 
== 2018-06-26 ==
* 22:17 Krenair: arming keyholder on deployment-deploy01
* 20:16 Krenair: done the same on -sca02 and -apertium02
* 20:09 Krenair: package upgrades on -sca01 to try to fix apertium stuff
* 15:32 hashar: repooling integration-slave-docker-1013 integration-slave-docker-1014 and integration-slave-docker-1015 (converted to m1.medium instances)
* 15:11 hashar: Deleting integration-slave-docker-1013 integration-slave-docker-1014 and integration-slave-docker-1015 . Recreating them as m1.medium instances
* 15:11 hashar: Changing integration-slave-docker-1012 in jenkins m1executor -> m4executor. It is a m1.medium instance and can thus run mediawiki jobs
* 13:18 hashar: cleaned containers on integration-slave-docker-1006
== 2018-06-25 ==
* 21:19 mdholloway: deployed to beta: [mobileapps/deploy@770cdb0]: Update mobileapps to 
* 17:45 Krenair: fixed puppet repo rebasing
* 16:20 hashar: deployment-prep: git gc on a few repositories under /srv/mediawiki-staging/php-master
* 12:19 kart_: "Beta: Update cxserver to ece5e7a"
* 07:49 hashar: github: deleting archived repository https://github.com/wikimedia/mediawiki-extensions-CommunityVoice T196618
== 2018-06-23 ==
* 19:19 Krenair: restarted hhvm on -mediawiki-07 then apache2 to bring beta back up
== 2018-06-22 ==
* 21:37 hashar: deployment-prep: sudo cumin --force '*' 'apt-get clean' # some instances are causing disk warnings
* 14:46 zeljkof: Reloading Zuul to deploy 
* 12:34 hashar: Building docker images for Quibble 0.0.19 T197687
* 12:10 hashar: Tagging quibble 0.0.19 which pass "--autoplay-policy=no-user-gesture-required" to Chromium T197687
== 2018-06-21 ==
* 12:04 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-EtherEditor T197698
* 10:40 hashar: Buidling docker image releng/operations-puppet:0.3.2 https://gerrit.wikimedia.org/r/c/integration/config/+/441349/
== 2018-06-20 ==
* 19:47 hasharAway: Removing old Quibble images from the CI Docker slaves
* 17:33 hashar: Refreshing Quibble Jenkins jobs to use 0.0.18-2 docker image. That adds JSDuck T197806
* 16:32 hashar: Rebuilding Quibble images to get jsduck included in the image T197806
== 2018-06-19 ==
* 15:01 hashar: CI docker slaves: docker container prune --force ; docker image prune --force
* 05:12 Amir1: ladsgroup@deployment-tin:~$ foreachwikiindblist all-labs deleteAutoPatrolLogs.php
== 2018-06-18 ==
* 17:25 Amir1: ladsgroup@deployment-tin:~$ mwscript populateChangeTagDef.php --wiki=enwiki
* 16:15 Amir1: ladsgroup@deployment-tin:~$ mwscript populateChangeTagDef.php --wiki=enwiki
* 16:07 Amir1: change_tag_def is back now
* 16:05 Amir1: making enwiki in beta clsuter readonly
== 2018-06-15 ==
* 12:19 zeljkof: Reloading Zuul to deploy 
* 08:30 hashar: github: deleted lot of archived/read-only extensions and skins T180864
* 07:54 hashar: apt-get upgrade on integration-slave-jessie-android
* 07:54 hashar: Fixing puppet on integration-slave-jessie-android . Has been stall for a month due to jenkins-debian-glue-buildenv.deb which could not be upgraded
* 07:51 hashar: cleaning disk space on integration-slave-jessie-1001
== 2018-06-14 ==
* 21:47 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/439572/
* 14:26 zeljkof: Reloading Zuul to deploy 
* 08:45 Hauskatze: maurelio@deployment-tin:~$ mwscript namespaceDupes.php --wiki=eswiki --fix (64 broken pagelinks resolved, no remaining conflicts)
* 07:40 hashar: Cleaned up bunch of Docker containers and images from the CI slave-docker-* instances
* 07:40 hashar: Armed keyholder on integration-cumin using passphrase from integration-puppetmaster01 T197207
== 2018-06-13 ==
* 19:47 Krenair: powered off deployment-mx T184244
* 07:21 hashar: github: deleting archived repo wikimedia/operations-software-tessera TT186096
* 05:55 legoktm: deployed https://gerrit.wikimedia.org/r/440054
* 01:10 legoktm: deploying https://gerrit.wikimedia.org/r/440027
== 2018-06-12 ==
* 17:40 halfak: deploying ores 
* 01:53 legoktm: deployed https://gerrit.wikimedia.org/r/439829
== 2018-06-11 ==
* 21:47 halfak: deploying ores 
* 18:13 legoktm: deployed https://gerrit.wikimedia.org/r/436957 https://gerrit.wikimedia.org/r/439547
* 11:59 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-DidYouMean T196430
* 00:12 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-06-10 ==
* 17:03 Krenair: also fixed scap install on -zotero01
* 04:09 Krenair: added SPF and DMARC records to beta.wmflabs.org
* 00:42 Krenair: cumin 'P{R:Package = scap} and P{F:lsbdistcodename = jessie}' 'apt-get install scap=3.8.2-1+0~20180607230422.353~1.gbp2bb4cc -y --force-yes'
== 2018-06-09 ==
* 21:00 Krenair: Temporarily substituting certificates on deployment-cache-text04 for certs generated from T182927 to test
* 02:20 Krenair: stopping deployment-puppetmaster02 again, looks like it was automatically booted by novaadmin after security patches a couple days ago
* 02:17 Krenair: shut down old deployment-dumps-puppetmaster instance (replaced with a newer stretch instance), emailed ariel
* 02:13 Krenair: shut down old deployment-redis01 and deployment-redis02 instances T179371
== 2018-06-08 ==
* 18:05 legoktm: updating docker files for https://gerrit.wikimedia.org/r/438152
* 09:48 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-EditOwn T196433
* 07:50 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-InterwikiIntegration T196442
* 07:46 hashar: nuking integration-slave-jessie-1002 jenkins workspaces
== 2018-06-07 ==
* 23:13 Krenair: armed keyholder on deployment-cumin after reboot, found passphrase at deployment-puppetmaster03:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
* 18:51 Krenair: deployment-deploy-01 rejecting all connections, rebooting
* 17:24 Reedy: unstuck jenkins deployments to beta
* 14:27 _joe_: force-run puppet on deployment-prep appservers, restarted nutcracker after reconfigruation
* 14:16 _joe_: restarted apache2 in deployment-puppetmaster02 in deployment-prep, which is the correct way to run the puppetmaster there
* 14:16 _joe_: killed "puppet master" process in depoloyment-prep
* 14:03 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-WikivoteMapsYandex T193844
* 11:22 hashar: Switching quibble jenkins jobs to 0.0.18 images
* 10:41 hashar: Building Docker images for Quibble 0.0.18
* 09:15 addshore: added wmde-leszek to deployment-prep
== 2018-06-06 ==
* 21:40 bearND: BC: Update mobileapps to 
* 20:36 Amir1: deploying on ores
* 20:25 Amir1: deploying on ores
* 19:59 bearND: Update mobileapps to on BC
* 19:47 hashar: github: delete https://github.com/wikimedia/mediawiki-extensions-Genderize (archived repository) T196108
* 19:14 hashar: github: deleted https://github.com/wikimedia/mediawiki-extensions-SolrStore (archived repo) T196513
* 18:58 thcipriani: @deployment-tin: sudo keyholder arm
* 17:09 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-CategorySlideShow T186969
* 17:03 hashar: Phabricator: marked SkelJS Diffusion repository inactive https://phabricator.wikimedia.org/diffusion/ESJS/ T196509
* 17:01 hashar: github: deleted archived repository SkelJS https://github.com/wikimedia/mediawiki-extensions-SkelJS T196509
== 2018-06-05 ==
* 19:59 mdholloway: deployed to BC: [mobileapps/deploy@7ecc3b6]: Update mobileapps to 
* 12:07 kart_: Update cxserver to 
== 2018-06-04 ==
* 20:37 mdholloway: Deployed to BC: [mobileapps/deploy@276ea43]: Update mobileapps to 
* 19:37 Reedy: running foreachwiki maintenance/deduplicateArchiveRevId.php --force T196401
* 15:44 awight: ORES: Fix T194322
== 2018-06-02 ==
* 21:20 legoktm: legoktm@integration-slave-docker-1003:~$ sudo docker rmi $(sudo docker images -q)
* 19:57 greg-g: gjg@integration-slave-docker-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 19:31 Krenair: restarted parsoid on deployment-parsoid09 to try to fix stuff
* 18:07 Krinkle: Beta Cluster's RESTBase or Parsoid is broken. Saving VE times out, logstash-beta contain restbase: "internal_http_error" / "Error: ESOCKETTIMEDOUT"
* 01:16 legoktm: running docker-pkg in a screen because my connection is super flaky
== 2018-06-01 ==
* 22:17 Reedy: https://gerrit.wikimedia.org/r/#/c/436902/ finished deploying
* 21:37 Krinkle: Re-create performance-beta.wmflabs.org webproxy (wired to webperf01) - T195314
* 21:29 Krinkle: Re-creating webperf01 in deploymet-prep, T195314
* 20:57 legoktm: deploying docker-pkg with https://gerrit.wikimedia.org/r/436859 for reals this time (again)
* 20:51 legoktm: deleting old versions of docker images
* 20:48 hashar: contint1001: deleting some old wikimedia/mediawiki-services-mathoid docker images
* 20:40 mutante: contint1001 - mkdir /srv/zuul-debug-logs ; mv debug.log.2018-05-* from /var/log/zuul/ over there to free up disk space on / VG
* 20:26 mutante: contint1001 - apt-get clean got a little bit more disk space
* 20:07 legoktm: really Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/436852
* 20:02 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/436852
* 13:52 hashar: Tagged Quibble 0.0.17, rebuilding Docker images and bumping jenkins jobs
* 08:45 hashar: Bumping Quibble jobs to 0.0.16
* 02:39 legoktm: deployed https://gerrit.wikimedia.org/r/436716
== 2018-05-31 ==
* 23:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/436701
* 18:13 hashar: deployment-prep integration : revert puppet change https://gerrit.wikimedia.org/r/#/c/436600/ T187622
* 18:04 Krinkle: un-cherry-pick https://gerrit.wikimedia.org/r/#/c/436581/
* 18:03 Krinkle: removed scap::sources override from Horizon Puppet config for deployment-prep - T195314
* 11:41 hashar: Fixed EXT_DEPENDENCIES no more being injected on quibble jobs T196062
* 08:43 legoktm: deployed https://gerrit.wikimedia.org/r/436482
* 08:21 legoktm: deployed https://gerrit.wikimedia.org/r/436403
* 07:54 legoktm: deployed https://gerrit.wikimedia.org/r/436477
* 07:42 legoktm: deployed https://gerrit.wikimedia.org/r/436471
* 07:16 legoktm: repooled integration-slave-docker-1010
* 06:41 legoktm: depooled integration-slave-docker-1010 in jenkins for testing
* 01:19 Krinkle: Add web proxy in Horizon for 'performance-beta', mapping to deployment-webperf01 port 80 - T195314
* 00:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/389412
* 00:40 Krinkle: Apply 'role::webperf' to deployment-webperf01, T195314
* 00:32 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434722
== 2018-05-30 ==
* 21:07 legoktm: deployed https://gerrit.wikimedia.org/r/436404, reverted quibble upgrade
* 20:37 legoktm: deployed https://gerrit.wikimedia.org/r/436351
* 20:28 hasharAway: contint1001: triggered a few quibble runs from contint1001. Running in a screen
* 20:27 hasharAway: contint1001: (for ext in BlueSpiceAbout BlueSpiceArticleInfo BlueSpiceAuthors BlueSpiceAvatars BlueSpiceBlog BlueSpiceCategoryManager BlueSpiceChecklist BlueSpiceConfigManager BlueSpiceContextMenu BlueSpiceCountThings BlueSpiceEditNotifyConnector BlueSpiceEmoticons BlueSpiceExtendedFilelist BlueSpiceExtendedSearch BlueSpiceExtendedStatistics BlueSpiceExtensions BlueSpiceFoundation BlueSpiceGroupManager BlueSpiceHideTitle
* 18:32 mutante: created instance deployment-deploy-01 with stretch and flavor x-large (T192561)
== 2018-05-29 ==
* 23:41 legoktm: deployed https://gerrit.wikimedia.org/r/436190
* 18:21 legoktm: deployed https://gerrit.wikimedia.org/r/436020
* 14:25 gehel: rolling restart of elasticsearch deployment-prep for plugin upgrade - T193734
* 12:42 zeljkof: Reloading Zuul to deploy 
* 07:36 hashar: cleaning up some caches on castor02.integration.eqiad.wmflabs (under /srv/jenkins-workspace/caches/ )
* 06:12 legoktm: deployed https://gerrit.wikimedia.org/r/435961
* 04:47 legoktm: building https://gerrit.wikimedia.org/r/430037
* 01:57 legoktm: deployed https://gerrit.wikimedia.org/r/435952
* 01:34 legoktm: deployed https://gerrit.wikimedia.org/r/435950
* 01:31 legoktm: deployed https://gerrit.wikimedia.org/r/435949
* 01:27 legoktm: deployed https://gerrit.wikimedia.org/r/435948
* 01:16 legoktm: deployed https://gerrit.wikimedia.org/r/435706 https://gerrit.wikimedia.org/r/435707 https://gerrit.wikimedia.org/r/434799
* 01:02 legoktm: deployed https://gerrit.wikimedia.org/r/435946
* 01:00 legoktm: deployed https://gerrit.wikimedia.org/r/435945
* 00:44 legoktm: deployed https://gerrit.wikimedia.org/r/435943
* 00:14 legoktm: deployed https://gerrit.wikimedia.org/r/435939
* 00:11 legoktm: deployed https://gerrit.wikimedia.org/r/435938
== 2018-05-28 ==
* 11:09 hashar: Building Quibble Docker images 0.0.13 T195634
* 11:01 hashar: tagging quibble 0.0.13 hhvm server should set .svg Content-Type T195634
* 04:47 legoktm: running modified version of hashar's gear_client.py on contint1001, feel free to kill if it causes problems
== 2018-05-27 ==
* 06:02 legoktm: deployed https://gerrit.wikimedia.org/r/435681
* 00:24 legoktm: deployed https://gerrit.wikimedia.org/r/435673
== 2018-05-26 ==
* 23:09 Krinkle: Killed a bunch of stuck beta-mediawiki-config-update-eqiad jobs in Jenkins
* 23:06 Krinkle: beta-mediawiki-config-update-eqiad jobs have been stuck in Zuul for 17 hours
== 2018-05-25 ==
* 23:08 legoktm: deployed https://gerrit.wikimedia.org/r/435278 https://gerrit.wikimedia.org/r/433197 https://gerrit.wikimedia.org/r/434950 https://gerrit.wikimedia.org/r/434727 https://gerrit.wikimedia.org/r/435221
* 21:41 hashar: cleaned up disk space on deployment-tin . Reenabled jenkins slave.
== 2018-05-24 ==
* 19:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434990
* 19:04 legoktm: deleted https://integration.wikimedia.org/ci/view/Labs/job/labs-tools-ZppixBot-php55lint/
* 19:03 legoktm: deployed https://gerrit.wikimedia.org/r/434982 https://gerrit.wikimedia.org/r/434983 https://gerrit.wikimedia.org/r/434985 https://gerrit.wikimedia.org/r/434986
* 18:47 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434980
* 17:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434966 https://gerrit.wikimedia.org/r/434954
* 16:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434956
* 15:56 zeljkof: Reloading Zuul to deploy 
* 15:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434922
* 14:17 zeljkof: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/434917/
* 03:49 legoktm: deployed https://gerrit.wikimedia.org/r/434846
== 2018-05-23 ==
* 22:24 legoktm: deploying https://gerrit.wikimedia.org/r/434815
* 20:07 bearND: Beta Cluster: Update mobileapps to 
* 14:22 maurelio@deployment-tin:~$ foreachwiki extensions/TorBlock/maintenance/loadExitNodes.php --force (completed: successfully loaded 941 exit nodes on all Beta Cluster wikis)
* 05:39 legoktm: deployed https://gerrit.wikimedia.org/r/434626
* 05:21 legoktm: deployed https://gerrit.wikimedia.org/r/433960 https://gerrit.wikimedia.org/r/433363
== 2018-05-22 ==
* 20:42 eddiegp: eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo puppet agent -tv; done (for T194926, finished without errors on any of the hosts)
* 20:38 eddiegp: eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils; done T194926
* 20:15 eddiegp: eddie@deployment-db03:~$ sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils
* 16:53 Krinkle: Created deployment-webperf01 instance (m1.small) - ref T195312
== 2018-05-21 ==
* 16:06 Hauskatze: Added `User:Kullfar` to 'confirmed' for two weeks for OAuth-consumer registration, per request from Lucas Werkmeister; cfr. <https://meta.wikimedia.beta.wmflabs.org/w/index.php?oldid=7031#Admin_request>.
== 2018-05-20 ==
* 16:55 addshore: reload zuul for (Merged) jenkins-bot: Add quibble for Wikibase experimental [integration/config] - https://gerrit.wikimedia.org/r/434198 (owner: Addshore)
* 09:18 greg-g: gjg@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-05-19 ==
* 14:56 zeljkof: Reloading Zuul to deploy 
* 10:56 Krenair: amended uncommitted changes into HEAD commit (notified author) so I can unbreak puppet updates, also removed my old POC secure redirect puppet patch
* 05:07 legoktm: deployed https://gerrit.wikimedia.org/r/433918
* 04:55 legoktm: deployed https://gerrit.wikimedia.org/r/433911
* 04:48 legoktm: deployed https://gerrit.wikimedia.org/r/433845
== 2018-05-18 ==
* 18:31 zeljkof: Reloading Zuul to deploy 
* 01:26 twentyafterfour: syncing wmf.4 again to deploy https://gerrit.wikimedia.org/r/#/c/433673/ refs T194900 T191050
== 2018-05-17 ==
* 07:18 legoktm: deploying mediawiki-phan-seccheck:0.2.1 image
== 2018-05-16 ==
* 17:51 twentyafterfour: deployed mobileapps-periodic-test to jenkins with jenkins-job-builder. refs T177896
* 07:09 thcipriani: removed shadow mwdeploy users on deployment-mediawiki-07
* 02:32 legoktm: deployed https://gerrit.wikimedia.org/r/433308
== 2018-05-15 ==
* 23:39 legoktm: deployed https://gerrit.wikimedia.org/r/433297
* 23:22 legoktm: live hacking zuul config on contint1001
* 05:29 legoktm: deployed https://gerrit.wikimedia.org/r/432947 https://gerrit.wikimedia.org/r/433017 https://gerrit.wikimedia.org/r/433020
== 2018-05-14 ==
* 19:41 bearND: BC: Update mobileapps to 
* 17:16 legoktm: deployed https://gerrit.wikimedia.org/r/432764
* 03:40 legoktm: Building image docker-registry.discovery.wmnet/releng/wikimedia-audit-resources:0.1.1
* 03:17 legoktm: Building image docker-registry.discovery.wmnet/releng/wikimedia-audit-resources:0.1.0
== 2018-05-11 ==
* 00:08 Krinkle: Add Console Section patterns to Jenkins Config for "Selenium"
* 00:08 Krinkle: Update Console Sections in Jenkins Config to collapse setup-* sections by default
== 2018-05-10 ==
* 22:59 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/432165/
* 22:07 Krinkle: Update Console Section patterns to Jenkins Config for "setup-composer" (didn't match anymore)
* 22:03 Krinkle: Update Console Section patterns to Jenkins Config for "QUnit"
* 22:02 Krinkle: Add Console Section patterns to Jenkins Config for "JS Documentation"
* 22:02 Krinkle: Add Console Section patterns to Jenkins Config for "setup-docker"
* 21:58 MaxSem: Touched everything in beta cluster's wmf-config to fix some files not being synced
* 20:47 duploktm: deployed https://gerrit.wikimedia.org/r/432526
* 20:15 legoktm: deployed https://gerrit.wikimedia.org/r/432051 https://gerrit.wikimedia.org/r/431806 https://gerrit.wikimedia.org/r/431853 https://gerrit.wikimedia.org/r/431610 https://gerrit.wikimedia.org/r/431554 https://gerrit.wikimedia.org/r/432426
* 19:36 legoktm: kicked postmerge queue in jenkins
* 19:10 twentyafterfour: reset jenkins slave on deployment-tin
* 13:19 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Hosiryuhosi' 'Rxy' T194376
* 12:49 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Hosiryuhosi' 'Rxy'
== 2018-05-09 ==
* 22:53 awight: ORES: wheels fixups
* 21:31 awight: Bump ORES wheels
* 21:21 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/432150/
* 21:04 awight: ORES: drafttopic in beta
* 16:17 thcipriani: deployment-tin:apt-get clean && rm -rf /srv/mediawiki/.git to free space
== 2018-05-08 ==
* 19:49 legoktm: aborted mwext-phpunit-coverage-patch #2819
* 12:19 moritzm: upgrading app servers in beta to wikidiff 1.6.0 (T190717)
== 2018-05-07 ==
* 20:19 bearND: Update mobileapps to in BC
* 16:30 greg-g: doing the disconnect/reconnect dance as a group in our team meeting: https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update (tyler's doing it)
* 04:01 legoktm: deployed https://gerrit.wikimedia.org/r/431503
* 03:21 legoktm: deployed https://gerrit.wikimedia.org/r/431500 https://gerrit.wikimedia.org/r/431501
* 01:30 legoktm: deployed https://gerrit.wikimedia.org/r/431494 https://gerrit.wikimedia.org/r/431495
== 2018-05-06 ==
* 20:59 legoktm: deployed https://gerrit.wikimedia.org/r/431308
* 20:39 legoktm: deployed https://gerrit.wikimedia.org/r/431339
* 04:26 legoktm: deployed https://gerrit.wikimedia.org/r/430290
== 2018-05-05 ==
* afk: revert https://phabricator.wikimedia.org/D1047 from scap to fix beta-scap-eqiad (hopefully)
* 02:27 legoktm: deployed https://gerrit.wikimedia.org/r/431003
* 01:36 legoktm: deploying https://gerrit.wikimedia.org/r/430287
* 01:26 legoktm: updating docker for https://gerrit.wikimedia.org/r/431055
== 2018-05-04 ==
* 18:55 legoktm: deployed https://gerrit.wikimedia.org/r/430946 https://gerrit.wikimedia.org/r/428068
* 12:59 hashar: ssh deployment-tin.deployment-prep.eqiad.wmflabs sudo rm /var/lock/scap.operations_mediawiki-config.lock
== 2018-05-03 ==
* 19:20 hashar: Updating Jenkins jobs for quibble-stretch:0.0.11-3 T193767
* 14:45 greg-g: gjg@deployment-tin:~$ sudo rm /var/lock/scap.operations_mediawiki-config.lock
* 04:09 legoktm: deployed https://gerrit.wikimedia.org/r/430536 https://gerrit.wikimedia.org/r/430537 https://gerrit.wikimedia.org/r/400159
== 2018-05-02 ==
* 17:42 greg-g: gjg@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 17:19 greg-g: gjg@deployment-tin:/var/lock$ sudo rm scap.operations_mediawiki-config.lock
* 16:25 hasharAway: cleaning up disk on integration-slave-docker-1006
* 10:52 hashar: gerrit: added Hauskatze / MarcoAurelio to group "mediawiki-skins-Synagonism" ( https://gerrit.wikimedia.org/r/#/admin/groups/838,members ) for T193611
* 10:43 hashar: Building quibble docker iamges 0.0.11-2 T118371 T193222
* 07:04 legoktm: deployed https://gerrit.wikimedia.org/r/430295
* 06:52 legoktm: building/deploying docker-registry.discovery.wmnet/releng/ci-src-setup:0.2.5 https://gerrit.wikimedia.org/r/430292
* 06:49 legoktm: deployed https://gerrit.wikimedia.org/r/430291
== 2018-05-01 ==
* 16:51 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Spambot~BobFeliciano' 'BobFeliciano'
== 2018-04-30 ==
* 09:09 hashar: Quibble announce on Phame blog thing https://phabricator.wikimedia.org/phame/post/view/99/introducing_quibble/ (thanks zeljkof )
* 08:32 legoktm: deployed https://gerrit.wikimedia.org/r/429638 https://gerrit.wikimedia.org/r/429740
== 2018-04-27 ==
* 22:07 hashar: Running quibble-vendor-mysql-php70-docker against ~ 900 MediaWiki extensions. Triggered with a custom gear-client.py script from contint1001. PID 29710
* 21:31 hashar: Running quibble-vendor-mysql-php70-docker against 43 mediawiki skins. Triggered with a custom gear_client.py script from contint1001 . PID 13134
== 2018-04-26 ==
* 20:19 hashar: bumped quibble Jenkins jobs to 0.0.11
* 19:00 hashar: Building releng/quibble 0.0.11 docker images
* 17:41 mobrovac: stopped cpjobqueue and purging logs
* 16:51 thcipriani: mark integration-slave-jessie-1001 online
* 16:51 thcipriani: thcipriani@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 16:49 greg-g: tyler marked integration-slave-jessie-1001 as offline
== 2018-04-25 ==
* 21:27 hasharAway: building releng/quibble docker images @0.0.10
* 21:13 hasharAway: tagged quibble 0.0.10 
* 20:13 bearND: deployed mobileapps to BC: Config: Start up to 4 workers in parallel during start-up
* 18:55 awight: ORES: Revscoring 2.2.2
* 10:08 _joe_: deleted deployment-mediawiki0[45] and deployment-jobrunner02
* 00:14 legoktm: deployed https://gerrit.wikimedia.org/r/415769
== 2018-04-24 ==
* 23:49 legoktm: deployed https://gerrit.wikimedia.org/r/428858 https://gerrit.wikimedia.org/r/428859
== 2018-04-23 ==
* 23:27 ejegg: cleared out /srv/jenkins-workspace/workspace on jenkins-1003
* 21:20 thcipriani: cleaned up 1.5G of space on deployment-tin:/tmp hopefully fixes beta-scap-eqiad
* 20:29 mdholloway: deployed to BC: [mobileapps/deploy@5650605]: Update mobileapps to 
* 19:09 ottomata: replacing deployment-kafka0[45] with deployment-kafka-main-[12]
* 07:27 hashar: Upgrading Blue Ocean on CI Jenkins 1.4.2 -> 1.5.0
== 2018-04-20 ==
* 21:32 eddiegp: deployment-kafka04: full disk, deleted /var/log/syslog.1 (3G) to make interactive sessions work again
* 17:30 thcipriani: rebase integration-puppetmaster01:/var/lib/puppet/git conflicting on https://gerrit.wikimedia.org/r/#/c/348236/
* 08:38 hashar: Cleaned integration-slave-docker-1005 disk (deleting workspace and all docker images)
* 08:27 hashar: Created integration-slave-docker-1016 and integration-slave-docker-1017 (2G RAM / 2 executors )
* 08:26 hashar: replaced integration-slave-docker-1010 and integration-slave-docker-1011 with bigger instances (4GB RAM)
* 07:03 hashar: Replaced integration-slave-docker-1001 to get 4GB of RAM
* 00:46 awight: roll back ORES beta to master
* 00:08 awight: Push ORES git-lfs to look at stuff
== 2018-04-19 ==
* 21:23 hashar: Pooling in the new integration-slave-docker-1002 and integration-slave-docker-1003
* 21:08 hashar: rebuilding integration-slave-docker-1002 and integration-slave-docker-1003 ci1.medium > m1.medium (+2G RAM)
* 21:01 hashar: Bringing back quibble (0.0.9), this time with COMPOSER_PROCESS_TIMEOUT=600 T192576
* 19:22 hashar: integration-slave-docker-1004 : wiping all jenkins workspace and all docker images
* 18:00 thcipriani: cleared a bunch of old docker images from integration-slave-docker-1004, freed 4.4GB of space
* 17:43 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/427717/
* 10:34 Hauskatze: fixStuckGlobalRename.php ran to unblock rename jobs never arriving to jobqueue
* 09:41 hashar: building releng/quibble-stretch:0.0.8-2
* 08:19 eddiegp: eddie@deployment-tin:~$ for wiki in deploymentwiki enwiki enwikinews loginwiki metawiki simplewiki; do mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=$wiki --logwiki=loginwiki 'Samtar' "There'sNoTime"; done T192476
== 2018-04-18 ==
* 22:53 eddiegp: started nutcracker on deployment-jobrunner03
* 22:53 eddiegp: root@deployment-jobrunner03:/var/run# mkdir nutcracker && chown nutcracker.nutcracker nutcracker ref T178457
* 21:16 Hauskatze: (addition to previous message) Cmd ran on deployment-jobrunner03; zillions of messages complaining nutcracker was down
* 21:15 Hauskatze: Ran sudo puppet agent -tv to restart nutcracker, which was "Notice: /Stage[main]/Nutcracker/Service[nutcracker]/ensure: ensure changed 'stopped' to 'running'"
* 20:32 hashar: reloaded zuul for https://gerrit.wikimedia.org/r/427534
* 20:29 Reedy: reloaded zuul for 427469
* 20:11 mdholloway: deployed to BC: [mobileapps/deploy@9328a7d]: Update mobileapps to 
* 18:03 hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=loginwiki 'Artix Kreiger 2' 'Artix Krieger 2' - T192471
* 17:34 legoktm: deployed https://gerrit.wikimedia.org/r/427421 https://gerrit.wikimedia.org/r/427422 https://gerrit.wikimedia.org/r/427423 a while back
* 13:23 addshore: added Thiemo Kreuz (WMDE) to deployment-prep
* 13:18 addshore: added WMDE-Fisch to deployment-prep
* 09:41 hashar: Triggered https://integration.wikimedia.org/ci/job/quibble-integration/ for REL1_31
* 09:40 hashar: Buiding quibble jobs with (master, mysql, php5.5) https://integration.wikimedia.org/ci/job/quibble-integration/7/ T192432
* 07:17 legoktm: deploying https://gerrit.wikimedia.org/r/426115
* 04:52 legoktm: live hacked /etc/nodepool/wikimedia, trying some things to get rid of mod_php5
* 04:26 legoktm: try once more to refresh nodepool snapshot
* 03:22 legoktm: manually refreshing nodepool ci-jessie image for https://gerrit.wikimedia.org/r/427309
== 2018-04-17 ==
* 23:40 Krinkle: Rebuilding Nodepool snapshot for https://gerrit.wikimedia.org/r/415312
* 21:10 hasharAway: Running https://integration.wikimedia.org/ci/job/quibble-integration/6/ overnight
* 20:28 hasharAway: building releng/quibble.*:0.0.8 images
* 18:16 Krinkle: Deleted performance-webpagetest-wmf job (removed from config.git)
* 14:32 hashar: Added integration-slave-docker-1008 and integration-slave-docker-1009 (m4executor label)
* 13:33 moritzm: removed role::mediawiki::imagescaler from deployment-mediawiki05, per watroles the only use of that role in WMCS
* 13:32 moritzm: removed role::mediawiki::imagescaler from deployment-prep, per watroles the only use of that role in WMCS
* 02:41 Krinkle: Fix Jenkins config for console section "Composer" end pattern (rm mandatory .+ match from end)
== 2018-04-16 ==
* 23:13 legoktm: removing obsolete coverage report legoktm@contint1001:/srv/org/wikimedia/doc/cover$ sudo -u jenkins-slave rm -rf mediawiki-core-php7
* 22:32 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/427027
* 22:25 legoktm: kick stuch beta-update jenkins jobs
* 20:35 mdholloway: restarted restbase and mobileapps services for testing (T192287)
* 18:58 awight: Update ORES editquality; T185903
* 13:13 _joe_: also shutting down deployment-jobrunner02 T192071
* 13:12 _joe_: manually fixing ssh hostkey for mediawiki-jobrunner03 in scap on deployment-prep
* 12:48 _joe_: turning off deployment-mediawiki{04,05} T192071, will be deleted by EOW
* 02:51 legoktm: deployed https://gerrit.wikimedia.org/r/426837
== 2018-04-13 ==
* 17:23 marxarelli: Enabling Pipeline Utility Steps plugin in jenkins for changes to service pipeline in https://gerrit.wikimedia.org/r/#/c/425936/
* 16:55 eddiegp: cherry-pick https://gerrit.wikimedia.org/r/#/c/426104/ (test 3)
* 10:02 eddiegp: root@deployment-tin:/mnt/home/jenkins-deploy# sudo -u jenkins-deploy -- sh -c 'ssh-keyscan "deployment-mediawiki-09.deployment-prep.eqiad.wmflabs" >> .ssh/known_hosts' to fix beta-scap-eqiad
* 08:48 eddiegp: debugging T189493 on beta
* 08:20 _joe_: creating deployment-mediawiki-09 with stretch, eliminating -08 which was left in an unusable state T192071
* 07:22 legoktm: restarting jenkins
* 07:21 legoktm: uploading xunit 1.103-wmf.1 to jenkins
* 07:01 hashar: rebuilding quibble containers to use 0.0.7
* 00:53 awight: ORES: Test large file in LFS
== 2018-04-12 ==
* 23:01 awight: Try gerrit-based submodules for ORES, T180627
* 17:25 thcipriani: running scap pull on deployment-mediawiki07 to catch up from missed beta-scap-eqiad deploys
* 17:23 thcipriani: add ssh host key for deployment-mediawiki07 to /mnt/home/jenkins-deploy/.ssh/known_hosts so that beta-scap-eqiad will work again
* 15:57 thcipriani: integration-slave-docker-1003 had to reinstall python-pbr to fix puppet complaining about tzdata update
* 14:24 _joe_: installing deployment-mediawiki{08,09} for the beta upgrade to stretch of deployment-prep (T192071)
* 14:24 _joe_: installing deployment-mediawiki{08,09} for the beta upgrade to stretch
* 12:51 hashar: building releng/quibble-jessie and releng/quibble-jessie-php55
* 11:52 _joe_: creating deployment-mediawiki-07, first stretch appserver T192071
* 00:14 twentyafterfour: preparing to deploy phabricator rPHDEP/release/2018-04-12/1 https://phabricator.wikimedia.org/project/view/3335/
== 2018-04-11 ==
* 21:04 awight: ORES experiment with git-lfs, T180627
* 16:34 Krinkle: `fab deploy_zuul` for https://gerrit.wikimedia.org/r/425551
* 00:03 legoktm: deploying https://gerrit.wikimedia.org/r/425446
== 2018-04-10 ==
* 13:41 moritzm: upgraded HHVM on mediawiki-deployment04/05/06 to a build with a patch for the MEMC_VAL_COMPRESSION_ZLIB flag in the memcached module (T184854)
* 12:38 hashar: gerrit: created repo operations/debs/tidy-0.99 , a for of tidy Jessie package T191771
== 2018-04-09 ==
* 19:50 awight: Redundant virtualenv for ORES
* 18:06 awight: Restore to ORES master branch
* 17:17 awight: Test git-lfs in ORES
* 11:46 Hauskatze: maurelio@deployment-mira:~$ sudo puppet agent -tv to fix T191786 (success: Notice: Applied catalog in 27.11 seconds)
== 2018-04-06 ==
* 21:51 eddiegp: beta: Cherry-picking https://gerrit.wikimedia.org/r/c/424707/ , test for T173887
* 11:41 moritzm: upgrading deployment-prep to wikidiff2 1.6.0 (T190717)
* 07:40 moritzm: removed mediawiki-deployment07 from deployment-prep (T191578)
== 2018-04-05 ==
* 21:34 eddiegp: updated deployment-prep cherry-pick of https://gerrit.wikimedia.org/r/c/392221/ to PS38
* 18:08 eddiegp: ran 'cat portals/urls-to-purge.txt mwscript purgeList.php' on deployment-tin:/srv/mediawiki-staging to clear portals cache
* 17:50 eddiegp: Cherry-picking https://gerrit.wikimedia.org/r/c/424361/ on deployment-puppetmaster02 to try unbreak T173887
* 17:08 bearND: Update mobileapps to 
* 17:05 hashar: docker: building releng/quibble:0.0.5
* 17:00 bearND: Update mobileapps to on beta
* 11:32 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/424271
* 07:52 moritzm: removed unused/defunct deployment-videoscaler01 from deployment-prep (T191293)
* 07:52 moritzm: removed unused/defunct deployment-tmh01 from deployment-prep (T191293)
* 07:47 hashar: deployment-prep: made Muehlenhoff an admin
== 2018-04-04 ==
* 22:17 thcipriani: deployment-mediawiki0{4,5} clear apt-cache, restart clear hhvm cache, restart hhvm
* 21:24 awight: Roll back beta ORES
* 20:19 mdholloway: deployed to BC: [mobileapps/deploy@0460519]: Update mobileapps to 
* 20:12 awight: Try dsh scap config for ORES
* 20:07 addshore: reload zuul for https://gerrit.wikimedia.org/r/424045 (and some quibble experimental stuff)
* 16:09 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/423958
* 15:30 addshore: reload zuul for https://gerrit.wikimedia.org/r/423652
* 12:17 hashar: added experimental quibble job to mediawiki core / vendor / skins/Vector
* 11:58 hashar: Building releng/quibble-stretch:0.0.4
* 11:58 hashar: Building releng/quibble:0.0.4
* 07:11 hashar: deployment-prep: adding EddieGP as a member
== 2018-04-03 ==
* 09:49 hashar: building releng/quibble:0.0.3
* 00:09 paladox: created new repo "All-Avatars" which will be used to host avatars used by gerrit. Setting owner as Gerrit Managers will allow merging in the repo fro all users soon :)
== 2018-04-02 ==
* 20:19 mdholloway: deployed to BC: [mobileapps/deploy@940bd48]: Update mobileapps to 
== 2018-03-31 ==
* 21:42 Hauskatze: Ran sudo puppet agent --enable and sudo puppet agent -tv on deployment-maps03 to fix puppet staleness
== 2018-03-30 ==
* 11:38 dcausse: deployment-prep reindexing with forceSearchIndex all beta wikis (T189694)
* 09:56 hashar: Nuking /srv/zuul/git/labs/tools/stewardbots on zuul-merger hosts (contint1001 and contint2001). Fetch fails with org.eclipse.jgit.transport.UploadPackInternalServerErrorException T191077
* 09:56 hashar: Nuking /srv/zuul/git/labs/tools/stewardbots on zuul-merger hosts (contint1001 and contint2001). Fetch fails with org.eclipse.jgit.transport.UploadPackInternalServerErrorException TT191077
== 2018-03-29 ==
* 19:39 Amir1: ladsgroup@deployment-tin:~$ mwscript updateCollation.php --wiki=fawiki (T190965)
* 19:33 hashar: docker build image releng/quibble-stretch:0.0.2 https://gerrit.wikimedia.org/r/#/c/423025/
* 18:08 greg-g: ran scap task-series blockers --count 26 --date 2018-04-23 1.32.0-wmf.1
* 16:44 hashar: rebuilding docker images for releng/quibble:0.0.1 https://gerrit.wikimedia.org/r/#/c/422959/
* 16:38 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/422482/
* 16:35 greg-g: brought compiler02.puppet3-diffs.eqiad.wmflabs back online (herron asked)
* 15:13 paladox: creating new repo mediawiki/extensions/SecureAuth per https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests
* 14:55 mutante: removing deploy1001 from scap masters, keeping it as scap host. to avoid issues syncing (slower) to 3 masters but keep it synced as a regular host (T191029)
* 14:26 elukey: removed from deployment-puppetmaster02's ops repo "1ffd6e72a9 WIP Bug T184234" due to git sync failures
== 2018-03-28 ==
* 23:42 legoktm: deployed https://gerrit.wikimedia.org/r/422590
* 23:00 legoktm: deployed https://gerrit.wikimedia.org/r/422566 https://gerrit.wikimedia.org/r/422418
* 20:45 bearND: Update mobileapps to on BC
* 19:13 legoktm: killed stuck docker container on 1003 to free up root partition, and then deleted old/all images to free up the rest of the space
* 19:07 legoktm: legoktm@integration-slave-docker-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf * # full disk
* 16:04 hasharAway: nodepool: deleting 4 instances that are no more used but that Nodepool failed to detect as no omre used (due to some reboots in the openstack infra)
== 2018-03-27 ==
* 15:23 hashar: building docker-registry.wikimedia.org/releng/operations-puppet:0.3.1 https://gerrit.wikimedia.org/r/#/c/422168/
* 08:30 kart_: Update cxserver to 
* 03:10 legoktm: deleted mediawiki-core-code-coverage* workspaces to work around git/gerrit issue and retriggered jobs
== 2018-03-26 ==
* 20:43 mdholloway: deployed to BC: [mobileapps/deploy@e223f51]: Update mobileapps to 
* 13:55 hashar: restarting CI Jenkins . Upgrades Mail plugin from 1.20 to 1.21 T190393
* 03:35 bd808: Marked integration-slave-docker-1003 offline because / is full. Needs cleanup. (cc hashar, thcipriani)
== 2018-03-23 ==
* 21:19 greg-g: gjg@deployment-tin:/srv/mediawiki-staging/php-master$ git remote prune origin
== 2018-03-22 ==
* 12:28 elukey: removed two patches from deployment-puppetmaster02's operations/puppet repo because git-sync was failing (https://gerrit.wikimedia.org/r/#/c/361648/ and https://gerrit.wikimedia.org/r/#/c/402758/)
== 2018-03-21 ==
* 23:25 RoanKattouw: Created maps security group for port 6533; removed port 6533 from sca security group
* 23:22 bd808: Raised security group quota from 20 to 40
* 22:40 hashar: cancel beta-code-update build in Jenkins (deadlock)
* 22:40 hashar: cancel beta-update-database-eqiad build in Jenkins (deadlock)
* 20:19 mdholloway: deployed to BC: [mobileapps/deploy@675837f]: Update mobileapps to 
* 18:40 bd808: Re-added ArielGlenn as deployment-prep project admin
* 18:18 elukey: fixing operations/puppet on deployment-puppetmaster02, git-sync stuck due to a old patch still there
* 17:52 hashar: deployment-prep: scap sync-file docroot/wwwportal/portal 'T173887'
* 00:43 Reedy: enabled compiler03.puppet3-diffs.eqiad.wmflabs and disabled compiler02.puppet3-diffs.eqiad.wmflabs in jenkins
== 2018-03-20 ==
* 19:04 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/420777
* 17:27 mdholloway: deployed to BC: [mobileapps/deploy@fad1009]: Update mobileapps to 
* 12:25 hashar: deployment-prep: added Muehlenhoff (moritzm) as a member
* 10:35 elukey: stop eventlogging + mysql on deployment-eventlog05 for maintenance
* 09:58 hashar: deployment-mira: deleted iegreview/scholarships files scap/checks.yaml (invalid yaml somehow), ran puppet and restore the files
* 09:38 hashar: deployment-tin: deleted iegreview/scholarships files scap/checks.yaml (invalid yaml somehow), ran puppet and restore the files
== 2018-03-19 ==
* 18:34 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/420398/
* 16:57 awight: ORES beta service is restored.
* 16:45 awight: Put ORES-beta back onto master branch
* 16:43 awight: ORES-beta has been down since January.
* 16:29 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/420373/
* 16:11 bd808: jenkins: compiler03.puppet3-diffs offline for https://phabricator.wikimedia.org/T187258
* 15:20 thcipriani: reinit /srv/mediawiki-staging/portals on deployment-tin to fix beta-mediawiki-config-update-eqiad
* 11:56 hashar: Creating docker container docker-registry.wikimedia.org/releng/java8-xgboost:0.1.1 https://gerrit.wikimedia.org/r/#/c/420311/ T190032
* 10:50 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/419479/
* 07:43 kart_: Add cx_version to cx_translations table (T190009)
== 2018-03-18 ==
* 22:22 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/420230
* 04:42 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/417344
== 2018-03-16 ==
* 12:56 hashar: deployment-tin: ran git remote prune origin / git gc on all /srv/deployment git repositories
* 12:56 hashar: deployment-tin: setting /srv/deployment files/dir to be group owned by wikidev and group writable
* 12:50 hashar: deployment-tin: deleting /srv/grafana (no more in Gerrit)
* 12:46 hashar: deployment-tin: sudo chown mwdeploy:mwdeploy /srv/mediawiki/.git/objects/pack/* # some pack of 6GB belonged to root
* 11:11 hashar: zuul: reenqueue all coverage jobs lost when restarting Zuul
* 10:53 hashar: Upgrading zuul to zuul_2.5.1-wmf4 to resolve a mutex deadlock T189859
* 02:44 thcipriani: beta-scap-eqiad stuck 14 hours, had to do disconnect reconnect dance
== 2018-03-15 ==
* 17:15 bearND: Update mobileapps to 
* 16:43 Amir1: created deployment-ores01 to use stretch
* 16:36 Amir1: deleting deployment-ores01 to reimage
* 15:47 awight: ORES with git-lfs, scap config
* 09:24 hashar: deployment-prep: added gehel and dcausse has members/admins of the labs project
* 00:21 awight: ORES with git-lfs
== 2018-03-14 ==
* 23:30 awight: new ORES submodule, pre-git-lfs
* 20:23 mdholloway: deployed to BC: [mobileapps/deploy@0f9625a]: Update mobileapps to 
* 19:43 no_justification: deployment-prep: fixed permissions on i18n cache to unbreak scap job
* 16:36 Hauskatze: maurelio@deployment-tin:~$ sudo chown -R jenkins-deploy:wikidev /srv/mediawiki-staging/php-master
* 15:44 addshore: killed https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3382/ and https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/143/ to unblock integration slaves
* 14:23 Reedy: beta-scap-eqiad disabled while freeing up space on deployment-tin
* 12:16 kart_: update cxserver to 
* 10:07 hashar: nodepool: Image snapshot-ci-jessie-1521021138 in wmflabs-eqiad is ready
* 09:52 hashar: Refreshing Nodepool snapshot for "mw-apply-settings: get LocalSettings.php archived" https://gerrit.wikimedia.org/r/#/c/419375/
* 08:05 hashar: Move Jenkins jobs mwext-phpunit-coverage-patch mwext-phpunit-coverage-publish mediawiki-phpunit-coverage-patch to Nodepool (they use too much disk space) T189365
== 2018-03-13 ==
* 21:47 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 18:08 awight: Enable Extension:JADE, T176333
* 11:26 kart_: Update cxserver to 
* 10:39 kart_: Update cxserver to 
* 09:50 hashar: integration: cherry pick https://gerrit.wikimedia.org/r/#/c/419131/1 T188112
* 09:46 hashar: integration: cherry pick PS3 of https://gerrit.wikimedia.org/r/#/c/412894/ T188112
* 08:13 hashar: integration-cumin: ln -s /usr/local/lib/python3.4 /usr/local/lib/python3 (HACK) T188112
* 08:09 hashar: integration: cherry picking https://gerrit.wikimedia.org/r/c/412894/ to fix cumin 3.0.1 T188112
== 2018-03-12 ==
* 20:13 mdholloway: deployed to BC: [mobileapps/deploy@c764714]: Update mobileapps to 
* 20:09 bd808: Forced puppet run on deployment-logstash2 for 
* 20:09 bd808: Removed role::logstash::eventlogging from deployment-logstash2 because the hiera config is failing (undefined method `[]' for nil:NilClass at /etc/puppet/modules/role/manifests/logstash/eventlogging.pp:11)
* 19:55 bd808: Forced puppet run on deployment-logstash2. Failed due to bad logstash::eventlogging hiera data
* 19:54 bd808: Cherry picked https://gerrit.wikimedia.org/r/#/c/418986/ to deployment-puppetmaster02
* 12:33 hashar: Jenkins: installed Blue Ocean plugin. Eg: https://integration.wikimedia.org/ci/blue/ T155840
* 12:24 hashar: Jenkins: uninstalled the "cvs" plugin
== 2018-03-10 ==
* 07:03 greg-g: gjg@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf * - T189365
== 2018-03-09 ==
* 22:01 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
* 20:10 legoktm: deployed https://gerrit.wikimedia.org/r/418024
* 16:06 hashar: Deploying docker releng/npm-test:0.5.0 https://gerrit.wikimedia.org/r/#/c/417960/
* 10:58 hashar: Polling 6 new Docker instances to jenkins: integration-slave-docker 1010 to 1015. They are ci.medium (2G RAM / 2vcpu) each with 2 executors and labels DebianJessieDocker, m1executor
* 10:37 hashar: Update *-maven-java8-docker-site-publish jobs which were not mounting /src into the container T188686
* 02:31 legoktm: deployed https://gerrit.wikimedia.org/r/417343
== 2018-03-08 ==
* 22:19 hasharDinner: cleaned up /srv on integration-slave-jessie-1001 . Upgrade packages and reboot.
* 21:59 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf * # out of disk space
* 18:36 bearND: Update mobileapps to 
* 17:13 hashar: deleting a few nodepool instances that are no more registered in Jenkins
* 14:12 hashar: deployment-tin: rm -fR /srv/ocg
* 14:03 hashar: deployment-tin: rm /srv/jenkins/home/jenkins-deploy/workspace/beta-scap-eqiad/central.hhbc # 1.4GBytes
* 14:02 hashar: deployment-tin is out of disk space on /srv
* 10:27 hashar: Deploy docker images for /deploy repositories https://gerrit.wikimedia.org/r/#/c/417217/
== 2018-03-07 ==
* 11:08 hashar: reloading Zuul for "Add search/analysis-extra to jenkins" T188686
* 03:50 legoktm: deployed https://gerrit.wikimedia.org/r/416886 https://gerrit.wikimedia.org/r/415588
* 02:28 legoktm: manually triggering jenkins jobs
== 2018-03-06 ==
* 20:59 hashar: gerrit: changed scoring/ores/assets parent permission group to scoring/ores
* 20:59 hashar: gerrit: created scoring/ores/draftquality scoring/ores/drafttopic scoring/ores/articlequality scoring/ores/editquality scoring/ores/deploy all inherit permissions from scoring/ores itself inheriting from scoring/
* 20:56 hashar: gerrit: created scoring/ parent project with owner being research-ores ( https://gerrit.wikimedia.org/r/#/admin/projects/scoring,access )
* 20:52 MaxSem: refreshing spoofuser on beta
* 19:34 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/TorBlock/maintenance/loadExitNodes.php --force
* 19:31 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 18:34 mdholloway: deployed to beta: [mobileapps/deploy@5986ab7]: Update mobileapps to 
* 03:43 Krinkle: Jenkins postmerge queue has 'beta-scap-eqiad' and 'beta-update-databases-eqiad' stuck "Waitinf for execute" for over 3h
== 2018-03-05 ==
* 16:38 Reedy: deleted the stack traces too
* 16:37 Reedy: removed pre 2018 hhvm error logs from deployment-mediawiki04
* 16:37 Reedy: that was from deployment-mediawiki04
* 16:35 Reedy: removed 2G temp folder from /srv/mediawiki/php-master/cache/l10n/upstream
* 06:14 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-03-04 ==
* 19:00 thcipriani: cleared /tmp, apt-cache deployment-mediawiki04
* 06:02 Krinkle: Re-create php-master/StartProfiler.php on deployment-tin in Beta Cluster, similar to the one scap auto-creates each week in prod, except to include StartProfiler-labs.php instead.
* 02:12 Krenair: Regenerated captcha images for T164047
== 2018-03-02 ==
* 20:49 legoktm: deployed https://gerrit.wikimedia.org/r/411185
* 20:07 legoktm: deployed https://gerrit.wikimedia.org/r/415925 https://gerrit.wikimedia.org/r/414069
* 10:21 legoktm: deployed https://gerrit.wikimedia.org/r/415823 https://gerrit.wikimedia.org/r/415824 https://gerrit.wikimedia.org/r/415825
* 10:16 legoktm: deployed https://gerrit.wikimedia.org/r/415819
* 09:09 legoktm: deployed https://gerrit.wikimedia.org/r/415553
* 08:47 legoktm: deploying https://gerrit.wikimedia.org/r/415809
* 08:09 legoktm: manually queing jenkins jobs
== 2018-03-01 ==
* 20:48 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
* 14:55 elukey: delete deployment-eventlog02 ubuntu instance in favor of the brand new deployment-eventlog05 (stretch)
* 02:11 legoktm: manually queued jenkins jobs
== 2018-02-28 ==
* 16:31 legoktm: manually queued jenkins jobs
* 10:51 hashar: integration-slave-jessie-android killed stall qemu-system-i386 process
* 10:42 hashar: build docker-registry.discovery.wmnet/releng/npm-browser-test:0.1.2 and docker-registry.discovery.wmnet/releng/npm-test-oojsui:0.1.1
* 08:42 legoktm: queued more jenkins jobs (last for tonight)
* 07:29 legoktm: mass queuing jenkins jobs again
== 2018-02-27 ==
* 21:17 hashar: Building docker image releng/npm-test-oojsui:0.1.0 - https://gerrit.wikimedia.org/r/#/c/415102/
* 10:15 zeljkof: Reloading Zuul to deploy 
* 09:35 hashar: deployment-mediawiki05: out of disk space. Ran apt-get clean, cleaned old kernels/packages and dropped hhvm bytecode cache
* 08:44 hashar: deployment-mediawiki06: out of disk space. Ran apt-get clean
* 07:40 legoktm: deployed https://gerrit.wikimedia.org/r/414957
* 06:11 legoktm: manually triggering a bunch of jenkins jobs
* 03:29 legoktm: deployed https://gerrit.wikimedia.org/r/414896
* 03:02 Krinkle: Deleted beta-* related job builds in Jenkins that were stuck >1hr
* 03:01 Krinkle: Jenkins slave connection to deployment-tin is broken again. No error. Script console works. Disconnect/Relaunch doesn't resolve. 6 idle executors but jobs are no starting for some reason.
== 2018-02-26 ==
* 22:40 Hauskatze: updating list of Tor nodes for TorBlock on Beta Cluster wikis
* 22:39 Hauskatze: purging old abusefilter IP data from Beta Cluster wikis while we wait for a cron job to do this automatically
* 18:39 mutante: deployment-cache-text04 - manually creating Letsencrypt SSL cert for fr.wikipedia.beta.wmflabs.org (acme-setup -i "fr_wikipedia_beta_wmflabs_org" -s "fr.wikipedia.beta.wmflabs.org" --key-user root --key-group root), restarted nginx (T188288)
== 2018-02-24 ==
* 02:21 legoktm: deployed https://gerrit.wikimedia.org/r/413964
== 2018-02-23 ==
* 16:52 elukey: created deployment-eventlogging05 to test eventlogging on Debian in deployment-prep
* 11:50 hashar: deployment-mediawiki04 : apt-get clean && apt-get autoremove --purge
* 11:50 hashar: deployment-mediawiki04 : rm /var/cache/hhvm/*.sq3 and restarting hhvm
* 11:48 hashar: deployment-mediawiki04 is out of disk space on / causing beta-scap-eqiad to fail
* 08:19 hashar: gerrit: marked apps/android/java-mwapi.git read-only T187995
== 2018-02-22 ==
* 14:58 hashar: build docker-registry.discovery.wmnet/releng/tox-mysqld:0.1.0 https://gerrit.wikimedia.org/r/#/c/413374/
* 10:23 hashar: Building docker-registry.discovery.wmnet/releng/npm-test-3d2png:0.1.2 for https://gerrit.wikimedia.org/r/#/c/413336/
* 09:43 hashar: Building docker-registry.discovery.wmnet/releng/npm-test-3d2png:0.1.1 for https://gerrit.wikimedia.org/r/#/c/413286/
* 00:27 mdholloway: deployed to BC [mobileapps/deploy@8ffb03b]: Update mobileapps to 
== 2018-02-21 ==
* 22:31 hashar: Building docker image releng/npm-test-3d2png:0.1.0 and reloading Zuul https://gerrit.wikimedia.org/r/413227
* 20:51 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/413060
* 16:15 zeljkof: Reloading Zuul to deploy 
* 11:26 Hauskatze: Refreshed TOR exit nodes & cleaned old abusefilter log data for all Beta Cluster wikis.
* 02:39 no_justification: beta: ran initSiteStats.php --update on all wikis
== 2018-02-20 ==
* 20:57 Hauskatze: Refreshed tor exit node lists for all Beta Cluster wikis
* 20:50 legoktm: refreshing nodepool image: nodepool@labnodepool1001:~$ nodepool image-update wmflabs-eqiad snapshot-ci-jessie
* 20:47 HausAFKatze: mwscript extensions/TorBlock/maintenance/loadExitNodes.php --wiki=deploymentwiki --force
* 19:41 Hauskatze: maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
== 2018-02-19 ==
* 23:40 Krinkle: Running `nodepool image-update wmflabs-eqiad snapshot-ci-jessie` to deploy https://gerrit.wikimedia.org/r/412825
* 18:52 legoktm: deleted all current nodepool instances for ci-jessie
* 18:36 legoktm: manually refreshing nodepool images (nodepool image-update wmflabs-eqiad snapshot-ci-jessie)
* 10:03 hashar_: deployment-tin: git gc in /srv/mediawiki-staging/php-master and /srv/mediawiki-staging/php-master/extensions
== 2018-02-18 ==
* 20:37 Hauskatze: Ran foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php on Beta
== 2018-02-16 ==
* 21:26 legoktm: deploying https://gerrit.wikimedia.org/r/411186
* 21:12 hashar: Upgraded Zuul to https://gerrit.wikimedia.org/r/#/c/411322/3
* 15:39 hashar: castor02: sudo rm -fR /srv/jenkins-workspace/caches/*/*/*trusty
* 13:56 hashar: Jenkins: disabled JNLP protocol entirely https://integration.wikimedia.org/ci/configureSecurity/
* 11:04 hashar: updating all docker related jobs https://gerrit.wikimedia.org/r/#/c/411098/
* 07:21 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-02-15 ==
* 23:30 Hauskatze: Ran cleanupSpam.php on deploymentwiki to get rid of a bunch of crap.
* 20:34 hashar: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/410990/ (no jobs touched)
* 10:35 hashar: gerrit: marking search/ltr and search/repository-swift as read-only T187428
* 07:26 legoktm: cancelled a job of mediawiki-core-doxygen-publish to unstuck them
* 07:13 legoktm: deployed https://gerrit.wikimedia.org/r/410654
* 01:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/410255
== 2018-02-14 ==
* 21:41 mdholloway: Deployed to beta cluster: [mobileapps/deploy@9bad612]: Update mobileapps to 
* 20:23 legoktm: deploying https://gerrit.wikimedia.org/r/410554
* 20:21 hashar: building docker-registry.wikimedia.org/releng/tox-pyspark:0.1.1
* 17:37 hashar: Manually deleting some Nodepool instances following Jenkins restart
* 17:37 hashar: Upgraded Jenkins to 2.89.4
* 13:31 Hauskatze: maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php > purge_abusefilter_20180214.log 2>&1
* 09:41 hashar: restart Docker on all Jenkins agents
* 08:26 legoktm: deployed https://gerrit.wikimedia.org/r/410397
* 04:28 legoktm: CodeEditor and WikiEditor are now part of shared MediaWiki gate job
* 04:03 legoktm: deployed https://gerrit.wikimedia.org/r/410254
* 03:50 legoktm: deployed https://gerrit.wikimedia.org/r/410194
* 03:33 legoktm: deploying https://gerrit.wikimedia.org/r/410372
* 03:12 legoktm: deploying https://gerrit.wikimedia.org/r/410005
* 02:23 MaxSem: manually created the global_preferences table on betalabs' centralauth DB
== 2018-02-13 ==
* 18:19 mdholloway: deployed mobileapps@5851dfc to beta cluster
* 16:00 legoktm: deployed https://gerrit.wikimedia.org/r/410173 https://gerrit.wikimedia.org/r/410192
* 10:51 moritzm: shutdown deployment-videoscaler01 (T187063)
* 10:49 moritzm: shutdown deployment-tmh01 (T187063)
* 10:21 Hauskatze: purged old IP data from abuse_filter_log
* 09:15 hashar: beta: scap fixed by deleting a leftover /srv/mediawiki/.~tmp~/
* 09:12 hashar: deployment-jobrunner02: sudo rm -fR /srv/mediawiki/.~tmp~/ # might be causing the beta-scap-eqiad Jenkins job to fail
* 03:44 legoktm: deployed https://gerrit.wikimedia.org/r/410100
* 01:52 legoktm: deploying https://gerrit.wikimedia.org/r/410089
== 2018-02-12 ==
* 23:06 legoktm: deployed https://gerrit.wikimedia.org/r/410075 https://gerrit.wikimedia.org/r/410076
* 20:53 mdholloway: deployed mobileapps@f14bdd5 to beta cluster
* 18:55 Hauskatze: Running maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldIPLogData.php for T186870
* 18:48 Hauskatze: Restoring first missing log entries on Beta refs. T54919
* 18:46 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=arwiki (37 rows purged - T186870)
* 18:45 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged - T186870)
* 18:44 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged)
* 18:43 Hauskatze: Starting to purge old afl_ip data from abuse_filter_log on Beta Cluster - T186870
* 15:59 hashar: Deploying java8 docker image https://gerrit.wikimedia.org/r/#/c/409881/
== 2018-02-10 ==
* 18:58 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=eswiki (1695 rows purged - T186870)
* 18:49 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php --wiki=zhwiki (22 rows missing inserted)
* 18:46 Hauskatze: Ran mwscript extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php --wiki=deploymentwiki (17 rows inserted)
== 2018-02-09 ==
* 23:05 legoktm: deploying https://gerrit.wikimedia.org/r/409461
* 22:59 legoktm: deployed https://gerrit.wikimedia.org/r/409459
* 22:40 legoktm: deploying https://gerrit.wikimedia.org/r/409458
* 21:53 legoktm: deploying https://gerrit.wikimedia.org/r/408760
* 21:16 legoktm: deleted test-csteipp-sensiolabs-securityadvisorieschecker from jenkins, obsolete
* 13:43 hashar: Deploying docker-registry.wikimedia.org/releng/jsduck:0.1.0 https://gerrit.wikimedia.org/r/#/c/409320/
* 10:16 hashar: Launching new integration-slave-docker-1003 T186856
* 09:48 hashar: Deleting integration-slave-docker-1003 and recreating it. It is running on an overloaded labvirt machine. T179378 T186856
* 09:25 legoktm: deploying https://gerrit.wikimedia.org/r/409297
* 08:53 hashar: Reloading zuul for 
* 07:13 legoktm: marked integration-slave-docker-1003 as offline - T186856
* 00:59 bd808: Removed Yuvipanda at user request (T186289)
* 00:58 bd808: Removed Yuvipanda at user request (T186289)
== 2018-02-08 ==
* 13:58 Hauskatze: maurelio@deployment-tin:~$ mwscript initSiteStats.php --wiki=deploymentwiki --update --active --use-master
* 10:02 hashar: Rebuilding docker-pkg images on contint1001. Would get chromium 64 into npm-browser-test T179552
== 2018-02-07 ==
* 21:56 legoktm: deploying https://gerrit.wikimedia.org/r/408781
* 03:15 legoktm: deployed https://gerrit.wikimedia.org/r/408746
* 02:57 legoktm: deploying https://gerrit.wikimedia.org/r/408368
* 01:19 legoktm: deploying https://gerrit.wikimedia.org/r/408735
* 01:00 legoktm: deploying https://gerrit.wikimedia.org/r/408230
== 2018-02-06 ==
* 21:41 hashar: Rebuilding Zuul package to hotfix T186381
* 21:14 legoktm: restarted zuul due to patch being stuck (T186381)
* 19:25 hashar: Restarted Zuul due to T186381
* 18:14 thcipriani: removing /srv/mediawiki/.git on deployment-tin to clear space
* 02:33 legoktm: deploying https://gerrit.wikimedia.org/r/408480
== 2018-02-05 ==
* 22:09 mdholloway: deployed mobileapps@3140b1a to BC
* 16:09 mdholloway: mobileapps deployment to BC failed with error (T186532)
* 12:37 Hauskatze: deployment-prep maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki *.doxawatches.com --delete
* 11:02 hashar: Upgrading jenkins-debian-glue to 0.18.4-wmf1 T186494
* 09:48 hashar: operations/debs/jenkins-debian-glue create branches debian/jessie-wikimedia and patch-queue/debian/jessie-wikimedia based on v0.17.0 T186494
== 2018-02-04 ==
* 23:47 legoktm[NE]: deploying/building https://gerrit.wikimedia.org/r/408228
== 2018-02-03 ==
* 21:05 legoktm: manually deleted /srv/zuul/git/mediawiki/tools/phan on contint1001 so zuul could clone the new repo
* 21:02 legoktm: deployed https://gerrit.wikimedia.org/r/407991
* 04:01 legoktm: disable/enabled gearman in jenkins
* 03:55 legoktm: restarting zuul to drop 407165,3 from the queue
* 03:48 legoktm: disabled/enabled gearman in jenkins
== 2018-02-01 ==
* 16:14 Amir1: deleting deployment-sca03 (T184501)
* 07:09 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-01-31 ==
* 22:01 mdholloway: updated mobileapps to on beta cluster
* 05:29 legoktm: brought integration-slave-jessie-1003 back online after clearing disk space
* 05:28 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
== 2018-01-30 ==
* 20:09 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/404778
* 13:23 Amir1: made User:Ladsgroup admin and 'crat in wikidatawiki
== 2018-01-29 ==
* 23:24 awight: Experiment with versioned ORES venv, T181071
== 2018-01-24 ==
* 23:14 Krenair: armed keyholder on deployment-cumin using deployment-puppetmaster02:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase - this seems to have fixed cumin
== 2018-01-23 ==
* 20:50 legoktm: deployed https://gerrit.wikimedia.org/r/405941
* 19:56 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/405930
* 18:25 Hauskatze: Ran fixDoubleRedirects.php on deploymentwiki
* 18:20 Hauskatze: maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki --delete *.bestrealmoneygames.com
* 15:58 hashar: integration: armed keyholder on integration-cumin using passphrase from integration-puppetmaster01.integration.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/cumin.passphrase https://wikitech.wikimedia.org/wiki/Keyholder
== 2018-01-22 ==
* afk: restarting jenkins
== 2018-01-20 ==
* 09:09 legoktm: reloading zuul to deploy https://gerrit.wikimedia.org/r/405404 https://gerrit.wikimedia.org/r/405352 https://gerrit.wikimedia.org/r/404972 https://gerrit.wikimedia.org/r/404229 https://gerrit.wikimedia.org/r/404230 https://gerrit.wikimedia.org/r/354667
== 2018-01-19 ==
* 17:23 zeljkof: Reloading Zuul to deploy 
* 17:10 zeljkof: Reloading Zuul to deploy 
* 15:34 elukey: added deployment-eventlog02.deployment-prep.eqiad.wmflabs to /etc/ssh/ssh_known_hosts on deployment-tin (following https://phabricator.wikimedia.org/T116206#2251441) to unblock "Host key verification failed" for Analytics
== 2018-01-18 ==
* 18:29 bearND: (beta): Update mobileapps to 
* 17:00 ottomata: stashing local changes to deployment-puppetmaster02 in /var/lib/git/operations/puppet (mail/mx.pp and exim/exim4.conf.mx.erb)
* afk: cleared some space on deployment-mediawiki05 (apt-cache and old logs) so scap had room to work again. Although space is tight.
* 11:49 legoktm: mediawiki-core-doxygen-publish jobs are stuck
* 00:21 bd808: Deleted 6-7 nodepool instances in an alive but offline for running jobs state
== 2018-01-17 ==
* 16:24 zeljkof: Reloading Zuul to deploy 
* 13:11 hashar: nodepool: updating snapshot to get hhvm +wmf4 for T185024 : nodepool image-update wmflabs-eqiad snapshot-ci-jessie
== 2018-01-16 ==
* 21:29 thcipriani: armed keyholder using instructions on https://wikitech.wikimedia.org/wiki/Keyholder
* 20:20 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/403904/1
* 18:16 greg-g: muted shinken in -releng while the reboots are on-going
* 09:38 hashar: Deleting all legacy wmfreleng/ docker images from the Jenkins slaves : sudo cumin --force 'name:docker' "docker imagesgrep wmfrelengawk '{print \$3}'sortuniqxargs docker rmi --force"
* 09:27 hashar: deploy rake/rake-vagrant docker images https://gerrit.wikimedia.org/r/#/c/404422/
== 2018-01-15 ==
* 21:59 hashar: deployment-mx echo -n > /var/log/mtail/mtail.log
* 21:59 hashar: deployment-mx rm /var/log/git-sync-upstream.log*
* 17:25 zeljkof: Reloading Zuul to deploy 
* 17:14 zeljkof: Reloading Zuul to deploy 
* 13:59 kart_: Ran: "mwscript extensions/Translate/scripts/createMessageIndex.php --wiki=metawiki" for T180841
* 13:37 kart_: Ran update.php on metawiki betacluster (T180841)
* 11:23 hashar: Mirroring git://anonscm.debian.org/pkg-php/php-ast.git to operations/debs/pkg-php/php-ast.git T174338
* 09:50 hashar: integration/zuul pushed upstream git tags to our repo
== 2018-01-12 ==
* 14:11 zeljkof: Reloading Zuul to deploy 
== 2018-01-11 ==
* 17:50 tgr: added Groovier1 to project members for T158909
* 14:05 hashar: Migrate composer-php70-docker mwgate-composer-php70-docker to a new docker image https://gerrit.wikimedia.org/r/403654
* 13:45 hashar: Migrate composer-package-php70-docker mwgate-composer-package-php70-docker to a new docker image https://gerrit.wikimedia.org/r/403647
== 2018-01-10 ==
* 20:55 hasharAway: Updating '*rake-docker' jobs https://gerrit.wikimedia.org/r/#/c/403471/
* 20:54 hasharAway: Updating rake / rake-vagrant docker containers https://gerrit.wikimedia.org/r/#/c/403471/
* 20:45 apergos: that should be of course deployment-puppetmaster02
* 20:43 apergos: replaced /var/lib/git/labs/private on deployment-puppetmaster0 with a clean copy of the repo with all uncommitted changes checked in, all extra files checked in, all local commits rebased on top; old repo is in /var/lib/git/labs/private_icky
* 19:38 hasharAway: updating docker-pkg images for rake-vagrant https://gerrit.wikimedia.org/r/#/c/403462/
* 14:48 _joe_: shutting down deployment-puppetdb01.deployment-prep.eqiad.wmflabs, unused
* 09:40 hashar: update docker-pkg images for releng/rake https://gerrit.wikimedia.org/r/#/c/403311/
* 07:50 legoktm: deployed https://gerrit.wikimedia.org/r/402826
== 2018-01-09 ==
* 14:53 hashar: Change integration/zuul.git HEAD from 'master' to 'patch-queue/debian/jessie-wikimedia' T158243 T162191
* 13:20 addshore: reloaded zuul to deploy https://gerrit.wikimedia.org/r/402862
* 10:54 hashar: gerrit: created operations/debs/node-tunnel-agent a fork of git://anonscm.debian.org/collab-maint/node-tunnel-agent.git T183569
* 08:51 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki (T184276)
* 08:49 Amir1: ladsgroup@deployment-sca03:/srv/deployment$ sudo rm -rf ores (T184282)
* 08:48 Amir1: stopping ores services in deployment-sca03 (T184282)
* 08:43 Amir1: changed DNS that ores-beta.wmflabs.org points to deployment-ores01 instead of deployment-sca03
* 08:42 Amir1: deleted deployment-ores-redis-01 in favor of deployment-ores01 (T184282)
== 2018-01-08 ==
* 18:32 thcipriani: doing https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Troubleshooting#Jenkins_executioner_lock to fix deployment-tin executioner lock stalling postmerge.
* 16:32 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/383872/
== 2018-01-05 ==
* 21:02 legoktm: legoktm@contint1001:/srv/org/wikimedia/doc/cover$ sudo -u jenkins-slave rm -rf extensions
* 14:26 halfak: restarted celery-ores-worker on deployment-sca03
== 2018-01-03 ==
* 19:55 hashar: manually upgrading puppet to 4.8 on deployment-mx / deployment-redis01 / deployment-redis02 T184114
* 19:52 hashar: purging old kernels on deployment-mx / deployment-redis01 / deployment-redis02 T184114
* 19:24 hashar: deployment-prep: fix puppet run broken by a duplicate definition due to profile::base::firewall vs base::firewall
* 19:20 hashar: deployment-tin "upgrade" scap to 3.7.4-3 the version in apt.wm.o
* 19:09 hashar: apt-get upgrade on deployment-tin . "downgrade" scap from 3.7.4-3 (apt.wm.o) to 3.7.4-1~20180103034049.266 (from CI)
* 19:07 hashar: deployment-prep: restored all the cherry picks on the puppet master
* 06:01 legoktm: manually installing php-xdebug on integration-slave-jessie-1004 to make sure this works (temporary)
== 2018-01-01 ==
* 23:46 Krenair: ran `mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=sqwiki` on deployment-tin for T183862
* 19:56 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=nlwiki (T183862)
* 19:56 Amir1: restarting ores services in deployment-sca03 (T183862)