Release Engineering/SAL/Archive 8

== 2022-12-30 ==
* 15:34 hasharAway: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/873779/ Add FlaggedRevs to the dependencies list of CheckUser # T61677
== 2022-12-27 ==
* 00:48 Reedy: restarted zuul using `sudo /usr/sbin/service zuul restart` for T325946
== 2022-12-20 ==
* 23:58 dancy: reboot deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud
* 21:52 Reedy: reboot deployment-mwmaint02.deployment-prep.eqiad1.wikimedia.cloud
== 2022-12-19 ==
* 23:09 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/869330
== 2022-12-17 ==
* 22:44 Reedy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/868802
== 2022-12-15 ==
* 17:20 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/868399
* 16:51 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/868437/ "inference-services: remove old revscoring pipelines" T323586
== 2022-12-14 ==
* 22:54 hashar: Updated PHP compile jobs to compile with `CFLAGS=-Wall -Werror` https://gerrit.wikimedia.org/r/c/integration/config/+/868184/
* 18:18 hashar: Reloaded Zuul for "Zuul: [cloud/wmcs-cookbooks] Setup CI" https://gerrit.wikimedia.org/r/c/integration/config/+/868134
* 08:43 hashar: deployment-prep: fixed beta-code-update-eqiad Jenkins job by cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/868002 T325128
* 01:31 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/867734
== 2022-12-13 ==
* 16:59 James_F: Zuul: Add experimental PHP 8.2 jobs to quibble things for T314093
* 16:40 James_F: Docker: Publishing initial quibble-buster-php82 for T314093
* 15:08 Krinkle: delete 'vscode' branch from mediawiki/core. Tagged at (8 Dec 2021) with 0 commits ahead.
== 2022-12-12 ==
* 14:31 James_F: Docker: Publishing PHP 8.2 images now based on 8.2.0 GM for T314093
* 09:55 hashar: Switching PHP based jobs to container images without XDebug https://gerrit.wikimedia.org/r/c/integration/config/+/866618 T319495
== 2022-12-09 ==
* 16:46 brennen: temporarily disabling gitlab-phabricator push notifications
* 16:36 James_F: Docker: Publishing CI images without the xdebug extension per T319495
* 14:07 hashar: CI Jenkins: granted "Job Config History" permissions to members of `ciadmin` LDAP group
* 14:06 jbond: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/866584
* 13:40 hashar: ssh -p 29418 gerrit.wikimedia.org gerrit flush-caches --cache ldap_groups
== 2022-12-08 ==
* 16:27 hashar: After six years of services `contint1001.wikimedia.org` has been shutdown. *waves*
* 08:20 hashar: Attached contint1002 as an agent of the CI Jenkins # T313832
== 2022-12-07 ==
* 22:59 brennen: Updating development images on contint primary to release php 8.1 images for T319432
* 21:25 hashar: gerrit: on next restart it will started with Java property `-Dh2.maxCompactTime=15000` and on the next shutdown that would cause it to compact the oversized H2 database files https://gerrit.wikimedia.org/r/c/operations/puppet/+/865023/ T323754
* 21:17 hashar: Add contint1002 as an agent to the CI Jenkins, albeit in offline mode cause it is being provisioned https://integration.wikimedia.org/ci/computer/contint1002/ T313832
* 17:40 hashar: Restarted zuul-merger on contint2001.wikimedia.org
* 12:01 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/865612
== 2022-12-06 ==
* 08:30 taavi: reloading zuul for 865026
== 2022-12-02 ==
* 16:07 James_F: Zuul: [mediawiki/extensions/PagePermissions] Add basic tests
* 10:59 slyngs: Updating development images bullseye-openldap and puppet-debugger for merge request 22 and 24
* 01:51 James_F: Zuul: [mediawiki/extensions/WatchAnalytics] Fix name
* 01:04 James_F: Zuul: [mediawiki/extensions/ImageSuggestions] Add phan CirrusSearch and Elastica dependencies for T302711
* 01:02 James_F: Zuul: [mediawiki/libs/Bcp47Code] Add basic test
== 2022-12-01 ==
* 17:25 James_F: Zuul: [mediawiki/extensions/Wikisource] Add ProofreadPage dependency
* 17:22 James_F: Zuul: [operations/software/bitu-ldap] Add basic tox CI
* 14:27 James_F: Zuul: [mediawiki/extensions/WatchAnalyics] Add basic CI
* 14:20 James_F: Zuul: [css-sanitizer] Re-disable PHP 8.1 job, still broken T311451
== 2022-11-30 ==
* 20:42 James_F: Zuul: Drop all CI for the REL1_37 branch, now EOL for T324133
== 2022-11-29 ==
* 21:35 brennen: gitlab repos/releng/scap: added direct membership for some non-releng maintainers who show up frequently/recently in commit log
* 20:33 James_F: Zuul: [wikimedia/wikimania-scholarships] Set as archived for T243037
== 2022-11-27 ==
* 21:27 James_F: Docker: Publishing new php82 images with rc.7 for T314093
== 2022-11-25 ==
* 12:45 hashar: Reloaded Zuul for "add pipelinename to autogenerated:ci tags" # T214068
== 2022-11-23 ==
* 22:41 urandom: accidentally deleted deployment-sessionstore04
* 15:07 James_F: Zuul: configure CI for operations/debs/varnish-modules for T321309
== 2022-11-22 ==
* 21:51 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/859615
* 21:06 TheresNoTime: samtar@deployment-mwmaint02:~$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php zhwiki pagetriage T323378
* 14:11 TheresNoTime: [samtar@deployment-deploy03 ~]$ sudo keyholder arm
== 2022-11-21 ==
* 14:54 James_F: Zuul: [mediawiki/extensions/WikiLambda] Disable selenium tests for T294388
* 14:41 vgutierrez: move deployment-cache-(textupload)07 from role::cache::(textupload)_haproxy to role::cache::(textupload) - T323365
== 2022-11-18 ==
* 10:05 hashar: gerrit: change HEAD branch to point to `deploy/wmf/stable-3.5` # T307334
== 2022-11-17 ==
* 17:44 taavi: reloading zuul to deploy https://gerrit.wikimedia.org/r/858391
== 2022-11-16 ==
* 20:53 thcipriani: restarting jenkins for update
* 08:46 hashar: gerrit: reindexed accounts `ssh -p 29418 gerrit.wikimedia.org -- gerrit index start accounts --force` # T323135
* 08:45 hashar: gerrit: deleted 192 LDAP accounts (scheme `gerrit:`) containing upper case characters which had an exact equivalent in an all lower case form. `All-Users.git` commit is # T323135
* 08:45 hashar: gerrit: deleted 192 LDAP accounts (scheme `gerrit:`) containing upper case characters which had an exact equivalent in an all lower case form #
== 2022-11-15 ==
* 20:21 hashar: gerrit: removed legacy mixed case accounts and moved the extra secondary email to a mailto id for `gerrit:krinkle`, `gerrit:revi`, `gerrit:daniel kinzler`, `gerrit:harej` and `gerrit:samanthanguyen` T323135#8397539
* 20:20 hashar: gerrit: removed legacy mixed case accounts for `gerrit:Fomafix` and `gerrit:Ricordisamoa` T323135#8397539
* 16:25 James_F: Zuul: [mediawiki/services/parsoid] Make MW jobs voting in test
* 15:57 James_F: Zuul: [mediawiki/extensions/CampaignEvents] Add Echo as phan dependency for T317231
* 15:24 hashar: gerrit: converted, to all lower case, the Gerrit accounts `username:Kaldari`, `username:Fran McCrory` and `username:SamanthaNguyen` # T323097
== 2022-11-14 ==
* 17:36 hashar: Nuking unused Castor cached files in `/srv/jenkins-workspace/caches` # T323051
* 17:35 hashar: Changing Castor cache saving from `/srv/jenkins-workspace/caches/` to `/srv/cache/caches/` which is the one served by rsync T323051
* 17:34 hashar: Changing Castor cache saving from `/srv/jenkins-workspace/caches/` to `/srv/cache/caches/` which is the one served by rsync.
* 14:19 James_F: Zuul: [mediawiki/services/function-schemata] Move from node 12 to 16
== 2022-11-10 ==
* 21:33 James_F: Docker: Upgrading quibble-buster-php74-coverage with a new vesion of phpunit-patch-coverage for T322864
* 08:37 hashar: Rebuilding https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ , it probably failed due to Gerrit being restarted
* 01:09 James_F: Zuul: Make PHP 8.1 voting for all quibble items for T316078
* 01:05 James_F: Zuul: Drop mwext-php74-phan-docker from experimental for gate
== 2022-11-09 ==
* 23:02 James_F: Zuul: [mediawiki/core] Add PHP 8.1 phan job for T322278
* 14:56 andrewbogott: fixed puppet breakage on several instances
== 2022-11-08 ==
* 20:17 dduvall: puppet re-enabled on gitlab-runner hosts (T322453) normal log level will be restored on next puppet run
* 20:01 dduvall: temporarily enabling buildkitd debug logging on gitlab-runner hosts (T322453)
* 15:58 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/854535
* 15:26 vgutierrez: delete deployment-ms-be06 - T322231
* 15:21 vgutierrez: shutdown deployment-ms-be06 - T322231
* 06:39 vgutierrez: delete deployment-ms-be05 - T322231
* 06:36 vgutierrez: delete deployment-ms-fe03 - T322554
* 06:30 vgutierrez: downgrade to firejail 0.9.44.8-2 on deployment-imagescaler03
* 05:51 vgutierrez: shutdown deployment-ms-fe03 - T322554
== 2022-11-07 ==
* 18:19 vgutierrez: let deployment-cache-upload07 use deployment-ms-fe04 - T322554
* 15:57 vgutierrez: shutting down deployment-ms-be05 - T322231
== 2022-11-03 ==
* 20:31 hashar: Reloaded Zuul for 
* 19:19 brennen: attempting initial phab1004 phabricator deploy
* 17:45 James_F: Zuul: Add CI for CategoryExplorer and EmailDeletedPages extensions and Cavendish and Pivot skins
* 17:15 James_F: Zuul: Add experimental PHP 8.2 jobs for PHP extensions for T314093
* 16:53 James_F: Docker: Publishing initial PHP 8.2 CI test images for T314093
* 13:44 TheresNoTime: add `cxserver-beta` (port 8080) proxy for deployment-prep, T322323
== 2022-11-02 ==
* 22:44 James_F: Zuul: [mediawiki/tools/scap] Mark as archived for T322269
* 09:56 vgutierrez: update to HAProxy 2.6.6 in deployment-cache-(textupload)07 - T321775
== 2022-10-31 ==
* 15:56 andrewbogott: shutting down deployment-echostore01, deployment-ms-be0[56], deployment-mdb01, deployment-prometheus02, deployment-wikifeeds01 as per https://phabricator.wikimedia.org/T306068
* 15:50 James_F: Zuul: [mediawiki/libs/RemexHtml] Re-enable PHP 8.1 CI for T311450
== 2022-10-28 ==
* 14:14 zabe: delete deployment-db07 and deployment-db08
* 06:24 hashar: devtools: phabricator-prod-1001: `rmdir /etc/envoy/clusters.d /etc/envoy/listeners.d`
* 06:24 hashar: devtools: `rmdir /etc/envoy/clusters.d /etc/envoy/listeners.d`
* 06:23 hashar: devtools: set `profile::phabricator::main::dumps_rsync_clients: []` project wide to fix up Puppet. Settings got moved to a `role` ( https://gerrit.wikimedia.org/r/c/operations/puppet/+/842875 T313360 )
== 2022-10-27 ==
* 21:38 James_F: Zuul: [mediawiki/core] Run standalone jobs T203694
* 20:58 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/850198 # T321769
== 2022-10-26 ==
* 23:12 dancy: Restarted Zuul CI server due to stall ssh connections which went against the max per user connection limit in Gerrit #T308943
* 18:28 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/849638 # T321668
* 08:58 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/849480 # T321594
== 2022-10-25 ==
* 16:33 hashar: Updating Jenkins jobs for Quibble 1.4.7 # T320935 T318029
* 15:36 hashar: Tag Quibble 1.4.7 @ # T320935 T318029
* 14:30 hashar: Manually cleaned /srv/jenkins/workspace on integration-agent-docker-1024
* 07:24 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/848514 # T317378
== 2022-10-24 ==
* 17:42 James_F: Zuul: Add new e-mail for Hoo man to allow list
== 2022-10-21 ==
* 08:46 hashar: Created https://gerrit.wikimedia.org/r/admin/repos/phabricator/translations # T321350
== 2022-10-20 ==
* 13:04 hashar: Updating Jenkins jobs to add `AllowEncodedSlashes On` to Apache config https://gerrit.wikimedia.org/r/c/integration/config/+/844974 T321278
* 12:40 hashar: Building Quibble Docker images to add `AllowEncodedSlashes On` to Apache configuration https://gerrit.wikimedia.org/r/c/integration/config/+/844937 T321278
* 07:23 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/844540
== 2022-10-19 ==
* 22:27 dduvall: deleted 'trigger-blubber-pipeline-*' 'blubber-pipeline-*' jobs to deploy https://gerrit.wikimedia.org/r/844529
* 22:22 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/844528
* 09:56 hashar: Reloaded Zuul for noop change https://gerrit.wikimedia.org/r/c/integration/config/+/830584 Zuul: [mediawiki/extensions/SearchVue] Mark as in production
== 2022-10-18 ==
* 19:13 hashar: devtools: unbreak puppet on `deploy-1004.devtools.eqiad1.wikimedia.cloud` by applying `profile::mediawiki::scap_client::is_master: true` # T319681
* 17:51 James_F: Zuul: [wikimedia/fundraising/SmashPig] Use composer-test-php74-only template
* 08:03 vgutierrez: wipe deployment-cache-(textupload)06 - T320930
== 2022-10-17 ==
* 16:21 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/843499 T309600
* 14:01 vgutierrez: shutdown deployment-cache-(textupload)06 - T320930
* 13:56 vgutierrez: switch 185.15.56.36 from deployment-cache-text06 to deployment-cache-text07 - T320930
* 13:54 vgutierrez: switch 185.15.56.35 from deployment-cache-upload06 to deployment-cache-upload07 - T320930
* 11:02 urbanecm: deployment-prep: wikiadmin@172.16.0.238(wikishared)> source /srv/mediawiki-staging/php-master/extensions/ContentTranslation/sql/significant-edits.sql; # cswiki beta was failing with cx_significant_edits table not found
* 09:41 wm-bot2: Increased quotas by 4 cores (T320932) - cookbook ran by arturo@nostromo
== 2022-10-14 ==
* 20:57 James_F: Zuul: Fix dependencies for BlueSpice extensions that depend on VisualEditor
* 20:49 James_F: Docker: Publishing helm-linter without deprecated kubeyaml for T316348
* 20:06 James_F: Docker: Publish images with php-ast upgraded from v1.0.14 to v1.1.0
* 18:22 dduvall: upgrade of docker on contint hosts aborted due to missing buster package. agents are back online
* 18:01 dduvall: upgrading docker on contint servers. agents will be available for a short time
* 16:07 James_F: Zuul: [mediawiki/libs/Zest] Re-enable PHP 8.1 tests for T311463
* 15:54 James_F: Zuul: [mediawiki/vendor] Add experimental job to check composer.lock for T74952
* 13:48 James_F: Zuul: [css-sanitizer] Re-enable PHP 8.1 jobs for T311451
== 2022-10-13 ==
* 21:16 dduvall: all integration-agent-docker-* hosts have been upgraded to docker 20.10.18
* 20:37 dduvall: starting rolling upgrade of docker on integration-agent-docker-* hosts to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/834399
== 2022-10-12 ==
* 20:09 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/841996
* 17:07 dduvall: deployed blubberoid using docker-registry.discovery.wmnet/wikimedia/blubber:2022-10-12-162839-production
== 2022-10-11 ==
* 15:49 dduvall: manually (re-)re-running `sudo -u mwpresync /usr/bin/scap stage-train --yes auto` after patch cleanup
* 15:29 dduvall: correction ^ full command is `sudo -u mwpresync /usr/bin/scap stage-train --yes auto`
* 15:28 dduvall: manually (re)running `stage-train --yes auto` following cron job failure
* 10:53 TheresNoTime: add MVernon to deployment-prep, T316845#8307183
== 2022-10-10 ==
* 12:04 TheresNoTime: cherry 836953 picking for T316845 to deployment-prep/Swift
== 2022-10-08 ==
* 21:00 Reedy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/840337
== 2022-10-07 ==
* 13:27 James_F: Zuul: Add two former contractors to the CI allowlist
== 2022-10-06 ==
* 13:17 hashar: Mass updating Jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/839520
* 13:12 taavi: reloading zuul for https://gerrit.wikimedia.org/r/839472
* 13:02 jelto: update gitlab-settings to enable admin_mode on gitlab production instances - T316419
* 13:00 James_F: Docker: Building and publishing php74:0.3.2 and cascade for T318918
* 12:59 jelto: update gitlab-settings to enable admin_mode on gitlab replica instances - T316419
* 12:55 jelto: update gitlab-settings to enable admin_mode on gitlab test instance - T316419
== 2022-10-05 ==
* 22:03 James_F: layout: [mediawiki/tools/phan/SecurityCheckPlugin] Publish PHP coverage for T279423
* 17:29 hashar: Building docker images for https://gerrit.wikimedia.org/r/814154
== 2022-10-04 ==
* 19:46 dduvall: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/838249
== 2022-10-03 ==
* 14:22 TheresNoTime: set `ring_manager` host to `deployment-ms-fe03` in deployment-prep's _.yaml. T316845
* 13:22 hashar: Triggering CI for design/codex@v0.2.1 using `zuul enqueue-ref --trigger gerrit --pipeline publish --project design/codex --ref refs/tags/v0.2.1 --newrev 4abb7677b3ea076bbd6778977d9a9374cf45015c` # T313767
* 13:15 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/822144 # T313767
* 12:47 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/833061
* 09:02 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/837494/
== 2022-09-30 ==
* 18:43 James_F: Triggering graceful restart of zuul to see if that fixes on-going merge/gerrit connection issues.
* 17:07 James_F: Zuul: Make PHP 8.1 non-voting for all skins and extensions T316078
* 16:46 James_F: Zuul: Make PHP 8.0 and PHP 8.1 voting for all skins and extensions in master for T300463 and T316078
* 15:12 James_F: Docker: Building and publishing PHP 8.0.24 images for T315167
* 02:33 James_F: Zuul: [mediawiki/core] Clean up REL1_35 and REL1_37 PHP 8 jobs
* 02:30 James_F: Zuul: [mediawiki/core] Upgrade PHP 8.0 and 8.1 jobs to full vendor jobs for T300463 and T316078
* 02:27 James_F: Zuul: Drop FIXME messages for T318093, being Declined
== 2022-09-29 ==
* 23:54 TheresNoTime: samtar@deployment-jobrunner04:~$ sudo systemctl stop php7.2-fpm.service && sudo systemctl start php7.4-fpm.service
* 23:47 TheresNoTime: cherry pick 836953 to deployment-prep
* 23:09 TheresNoTime: [samtar@deployment-deploy03 ~]$ sudo puppet agent -tv
* 23:08 TheresNoTime: deployment-deploy03, `sudo systemctl stop php7.2-fpm.service`, `sudo systemctl start php7.4-fpm.service`
* 23:03 TheresNoTime: ran `sudo puppet agent -tv` on deployment -deploy03, -mediawiki11, -mediawiki12
* 13:56 James_F: Zuul: Drop PHP72 jobs everywhere, and PHP73 everywhere except old branches
* 13:41 James_F: Zuul: [mediawiki/core] Drop PHP 7.2 and PHP 7.3 testing for master and wmf for T261872
* 13:34 James_F: Zuul: [mediawiki/vendor] Drop PHP72 jobs, use only PHP74 ones
* 12:30 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/836696
== 2022-09-28 ==
* 17:32 brennen: trying a re-publish of dev-images in case https://gitlab.wikimedia.org/repos/releng/dev-images/-/commit/eb82162b4bf443df20998a53bfb06460bfc6a365 didn't get picked up
* 00:30 James_F: Zuul: [mediawiki/services/function-orchestrator] Use direct coverage job here too
* 00:20 James_F: Zuul: [mediawiki/services/function-evaluator] Use direct coverage job, for T302608
== 2022-09-27 ==
* 08:11 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/835182
== 2022-09-26 ==
* 21:46 Daimona: Applying schema changes to the wikishared DB on beta for the CampaignEvents extension # T318379 T318120
* 21:31 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (1/2) # T318120
* 20:00 dduvall: regenerating 314 jobs for deployment of https://gerrit.wikimedia.org/r/835262
* 11:40 James_F: Docker: Building and publishing quibble-buster-php74-bundle
* 11:40 James_F: Docker
* 10:52 hashar: Rolling quibble/ruby jobs from php 7.4 to 7.2: `mediawiki-selenium-integration-docker` `legacy-quibble-rubyselenium-docker` # T318525
* 09:35 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/834717/
== 2022-09-23 ==
* 18:09 James_F: Zuul: [wikimedia-cz/web-*] Migrate tests from php73+ to php74+
* 18:06 James_F: Zuul: [labs/tools/guc] Migrate tests from php73+ to php74+
* 18:04 James_F: Zuul: [labs/tools/coverme] Migrate tests from php73+ to php74+
* 15:55 James_F: Docker: Building and publishing php74 versions of composer-security-check, mediawiki-phan, mediawiki-phan-testrun, and phpmetrics
* 13:26 James_F: Zuul: Run php 7.4 phan for extensions and skins
== 2022-09-22 ==
* 20:40 zabe: shutoff deployment-db07 # T318126
* 20:36 zabe: take deployment-prep out of read-only # T318126
* 20:32 zabe: failover deployment-prep master from deployment-db07 to deployment-db09 # T318126
* 20:25 zabe: set deployment-prep as read-only # T318126
* 16:26 dancy: Upgrading scap to latest code revision in beta cluster
* 10:38 zabe: deployment-db10: start replication # T318126
== 2022-09-21 ==
* 23:34 zabe: shutoff deployment-db08 # T318126
* 23:00 jeena: restarting zuul to try and fix CI issues
* 20:46 zabe: clone deployment-db10 from deployment-db08 # T318126
* 18:49 TheresNoTime: cherry-picked gerrit:833839 to deployment-puppetmaster04, testing T317417
* 18:19 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db10 # T318126
* 17:55 zabe: create volume db10 and attach to deployment-db10 # T318126
* 17:54 zabe: create deployment-db10 as g3.cores8.ram16.disk20 # T318126
* 14:21 zabe: deployment-db09: restart mariadb # T318126
* 13:55 TheresNoTime: modified deployment-prep "prometheus" security group - port 80, T315699
* 13:18 James_F: Jenkins: Dropped 16 more old node jobs left on the server.
* 13:11 James_F: Jenkins: Dropped four old node10 jobs left on the server (oojs-core-node10-browser-docker, ooui-special-node10-plus-php80-composer-docker, wikipeg-special-node10-plus-php72-composer-docker, wikipeg-special-node10-plus-php80-composer-docker)
* 13:05 James_F: Jenkins: Dropped scap-pipeline-stretch and trigger-scap-pipeline-stretch following 
* 12:36 hashar: Reloaded Zuul for Remove Stretch from mediawiki/tools/scap - https://gerrit.wikimedia.org/r/833705
* 09:46 andrewbogott: removed some stray whitespace in /var/lib/git/operations/puppet that was preventing rebase on deployment-puppetmaster04.deployment-prep.eqiad.wmflabs
== 2022-09-20 ==
* 22:00 zabe: deployment-db09: start replication # T318126
* 20:06 zabe: deployment-db09: import dump into mariadb # T318126
* 20:04 zabe: rsynced dump from deployment-db08 to deployment-db09 # T318126
* 08:08 hashar: Upgrading CI and releases Jenkins plugins notably to update the git client T315897
* 02:06 zabe: created backup of all databases on deployment-db08 # T318126
== 2022-09-19 ==
* 23:58 zabe: install mariadb 10.6 via role::mariadb::beta on deployment-db09 # T318126
* 23:57 zabe: create volume db09 and attach to deployment-db09 # T318126
* 23:57 zabe: create deployment-db09 as g3.cores8.ram16.disk20 # T318126
* 20:24 dduvall: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/833066
* 16:54 James_F: Zuul: [operations/mediawiki-config] Switch to PHP 7.4 jobs
* 16:24 James_F: Zuul: [mediawiki/core] Add php80 and php81 to `check php` command
* 15:36 James_F: Zuul: [mediawiki/core] run phan on PHP 7.4 for T316518
* 13:50 James_F: Zuul: [mediawiki/core] Add a non-vendor php81 job for main branch for T316078
* 12:06 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (2/2) # T316128
* 11:57 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (1/2) # T316128
== 2022-09-16 ==
* 15:47 dancy: Upgrading scap to latest code revision in beta cluster
== 2022-09-15 ==
* 19:56 thcipriani: Updating development images on contint primary
* 17:24 dduvall: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/832374
* 11:03 TheresNoTime: soft reboot deployment-parsoid12, unresponsive
== 2022-09-13 ==
* 22:14 zabe: delete deployment-urldownloader02
== 2022-09-12 ==
* 22:15 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/19
* 20:38 RhinosF1: (for lack of a better place) added Cyberpower678 to acl*userdisable. has enough clue, fairly active and trusted.
* 08:14 James_F: Zuul: [mediawiki/extensions/Realnames] Enable quibble composer jobs
== 2022-09-09 ==
* 17:46 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (2/2) # T311126
* 17:25 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (1/2) # T311126
* 17:08 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (2/2) # T316409
* 16:36 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (1/2) # T316409
* 10:49 hashar: devtools: fixed fqdn of instances puppetmaster-1001 and gerrit-prod-1001 by manually editing `/etc/hosts` # T317404
== 2022-09-08 ==
* 20:50 dduvall: running `./fab deploy_docker` to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/830919
* 15:47 dancy: Upgrading scap to latest code revision in beta cluster
== 2022-09-07 ==
* 15:05 TheresNoTime: making hack changes to beta to test T317195 resolution
== 2022-09-06 ==
* 15:42 bd808: Promoted user 'StrikerBot' to admin on gitlab.wikimedia.org so that Striker can use the account to attach Developer accounts to gitlab via API.
* 02:05 James_F: Running REL1_39 branch commands for T313920
* 00:20 Krinkle: Prune various old mediawiki/core wmf branches for Gerrit usability, ref T303828
== 2022-09-02 ==
* 15:59 zabe: added vwalters as member of the deployment-prep project T316943
* 13:40 Krinkle: " ENOENT: no such file or directory, lstat " failing quibble jobs on integration-agent-docker-1024
== 2022-09-01 ==
* 09:36 zabe: shutoff deployment-urldownloader02
* 07:43 hashar: Updating Jenkins jobs for Quibble 1.4.5 > 1.4.6 + php 7.4 update T305525 T314586 T316601 https://gerrit.wikimedia.org/r/c/integration/config/+/828611
== 2022-08-31 ==
* 23:41 zabe: deleted shutoff deployment-restbase03
* 16:39 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension # T308738
* 16:37 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (2/2) # T312870
* 16:21 Daimona: Applying schema change to the wikishared DB on beta for the CampaignEvents extension (1/2) # T312870
* 16:18 hashar: Tag Quibble 1.4.6 @ # T305525 T314586
* 15:27 James_F: Docker: Building and publishing quibble-fresnel image based on php74 for T316525
== 2022-08-30 ==
* 16:00 James_F: Zuul: [labs/tools/heritage] Switch postmerge job to tox-py37-coverage-publish for T316627
* 09:32 hashar: doc: on doc1002: `sudo -u doc-uploader rm -fR /srv/doc/mw-tools-scap/` That got moved to `/srv/doc` and a redirect has been set. # T315541
== 2022-08-29 ==
* 21:25 inflatador: ES6->7 upgrade in beta-cluster T315604
* 13:39 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/826985
* 12:34 James_F: Zuul: [mediawiki/core] Don't run phan on PHP 7.4, it doesn't pass; for T316518
* 12:08 James_F: Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [Re-re-try] for T293924
* 11:28 James_F: Zuul: [mediawiki/core] Make PHP 8.1 voting on REL1_38 and REL1_39 for T316080
* 09:00 hashar: Updated Jenkins job mediawiki-quibble-composer-mysql-php80-docker to capture core dumps using https://gerrit.wikimedia.org/r/c/integration/config/+/496392 # T315167
* 00:28 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/827015
== 2022-08-25 ==
* 15:53 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/826593
* 07:37 James_F: Zuul: [mediawiki/extensions/UnlinkedWikibase] Add quibble for T316183
== 2022-08-23 ==
* 17:40 hashar: Stopping Gerrit
* 11:54 hashar: Manually applied a `docker-pkg` fix on contint2001 to prevent it from downloading unrelated images T310458
== 2022-08-22 ==
* 07:17 taavi: trying to disconnect jenkins from gearman and then re-connect to see if it helps with T315818
* 07:12 taavi: restart zuul-merger on contint2001 T315818
== 2022-08-21 ==
* 13:07 Reedy: looks live various CI jobs (coverage etc) have been stuck for about 8.5 hours
* 13:00 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/824862
== 2022-08-19 ==
* 23:04 TheresNoTime: resized deployment-mwlog01's /srv volume, restarted
* 22:57 TheresNoTime: shutting down deployment-mwlog01 for T315707
* 15:50 James_F: Docker: Build stalled out for 30 minutes; terminated and re-started.
* 15:15 dancy: Upgrading scap to latest code revision in beta cluster
* 15:11 James_F: Docker: Building and publishing images with PHP 8.0.22 for T315167
== 2022-08-18 ==
* 17:01 hashar: Restarted zuul-merger on contint1001 # T315586
* 16:42 hashar: Reloaded Zuul for 
* 13:14 awight: [beta] Deploying new kartotherian version
== 2022-08-17 ==
* 14:18 zabe: fix merge conflicts in deployment-prep private repo # T315394
* 10:27 hashar: Built image docker-registry.discovery.wmnet/releng/commit-message-validator:1.0.0 # T315159
== 2022-08-16 ==
* 20:51 RhinosF1: beta: is down see wikitech-l and https://phabricator.wikimedia.org/T315350
* 20:30 hashar: Repooled integration-agent-docker-1028 , it was mysteriously unreachable T315372
* 19:18 Krinkle: mediawiki/extensions/EventLogging$ git remote-wildcard-br-d 'wmf/1.35*' 'wmf/1.36*' 'wmf/1.37*' 'wmf/1.38*'
* 19:17 Krinkle: mediawiki/extensions/Scribunto$ git remote-wildcard-br-d 'wmf/1.35*' # ref T303828
* 19:16 TheresNoTime: manually running `/usr/local/bin/wmf-beta-update-databases.py` on `deployment-deploy03`
* 17:16 TheresNoTime: soft-rebooting deployment-mediawiki12
== 2022-08-12 ==
* 17:47 dancy: Restarting zuul
* 17:42 dancy: Restarting Jenkins in an attempt to get CI jobs running again
* 00:54 ori: On deployment-cache-{text,upload}06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service . Certificate was close to expiry
== 2022-08-11 ==
* 21:11 mutante: restarted phd service on phab2001
* 19:12 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/16
* 12:26 jnuche: Reenabled CI beta sync jobs after cluster incident
* 11:48 jnuche: Temporarily disabled CI beta sync jobs until issue in cluster is resolved
* 10:25 zabe: take deployment-prep out of read-only mode
== 2022-08-10 ==
* 11:36 jnuche: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/822052
== 2022-08-09 ==
* 22:11 James_F: Docker: Building and publishing quibble-buster-php74-coverage for PHP7.4+ coverage
* 21:56 James_F: Two failures in devimage build: releng/eventlogging and releng/buster-swift53 – nothing new from me, looks like they've been broken for a bit?
* 21:17 James_F: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/17
* 21:07 James_F: Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [Re-try] for T293924
* 19:42 James_F: Docker: Re-build and publish quibble-buster-php74 based on Wikimedia PHP not sury-php for T293851
== 2022-08-08 ==
* 15:56 taavi: gerrit: used `ssh gerrit.wikimedia.org -p 29418 gerrit close-connection` to disconnect four of sgimeno's stuck sessions
* 14:43 James_F: jforrester@doc1002:~$ sudo -u doc-uploader rm -rf /srv/doc/wikibase-vuejs-components/ for T309872
* 13:23 James_F: Zuul: [mediawiki/libs/metrics-platform] Run Java jobs on maven file paths for T314630
* 10:28 jnuche: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/821166
== 2022-08-05 ==
* 16:02 James_F: Docker: Building and publishing composer-security-check:1.1.1 for T296967
* 15:40 James_F: Zuul: [mediawiki/services/function-*] Switch coverage to node16
* 15:33 James_F: Zuul: [mediawiki/libs/metrics-platform] Add experimental regular java jobs for T314630
* 14:48 James_F: Zuul: Add WelpThatWorked to allow list
* 14:48 James_F: Zuul: [mediawiki/extensions/MenuEditor] BlueSpiceDiscovery dependency is a skin
== 2022-08-04 ==
* 15:21 dancy: Deleting beta-mediawiki-config-update-eqiad job
* 15:16 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/820405
* 10:01 TheresNoTime: clearing out stuck beta deployment jobs T314378 T72597
== 2022-08-03 ==
* 21:05 James_F: Zuul: Doing a graceful restart to see if this clears the fork-bombed CI jobs.
* 20:13 taavi: reloading zuul for https://gerrit.wikimedia.org/r/820212
* 17:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/820171
* 14:57 brennen: gitlab: flipping admin bit for bd808 for API testing purposes
* 14:11 James_F: Zuul: [wikimedia/vuejs-components] Mark as archived for T309872
* 12:00 James_F: Ran `zuul-test-repo design/codex postmerge` on contint2001 to finally run coverage for Codex
* 11:58 James_F: Zuul: Run publish jobs on branches called 'main' too
== 2022-08-02 ==
* 19:26 James_F: Zuul: [design/codex] Switch coverage job back to -direct
* 15:23 dancy: Deleted beta-build-scap-deb and beta-publish-deb Jenkins jobs. (https://gerrit.wikimedia.org/r/c/integration/config/+/819028)
* 15:22 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/819028
* 07:55 TheresNoTime: cleared stuck beta deployment jobs T72597
== 2022-08-01 ==
* 23:16 James_F: Zuul: [design/codex] Switch to node16
* 23:16 James_F: 16:15:59 <+wikibugs> (Merged) jenkins-bot: Zuul: [design/codex] Switch to node16 [integration/config] - https://gerrit.wikimedia.org/r/819185 (owner: Jforrester)
* 22:53 TheresNoTime: remove stuck beta deployment jobs
* 22:51 dduvall: re-armed keyholder on deploy-1004.devtools following reboot
* 22:50 James_F: Zuul: Don't use browser-direct-coverage where browser-coverage will do
* 22:49 dduvall: modified `deployment_hosts` puppet config for devtools project to allow deployments from `deploy-1004`
* 22:24 dduvall: armed keyholder with phabricator key on deploy-1004.devtools
* 22:11 dduvall: setting puppetmaster to project standalone for deploy-1004.devtools
* 21:01 James_F: Zuul: [mediawiki/extensions/Phonos] Add comment about deployment timing for T314306
* 21:00 James_F: Zuul: [mediawiki/extensions/BlueSpiceCustomMenu] Add MenuEditor dependency
* 15:53 taavi: reloading zuul for https://gerrit.wikimedia.org/r/819097
* 09:14 TheresNoTime: clearing stuck beta CI jobs
== 2022-07-29 ==
* 22:16 James_F: Zuul: Configure CI for the forthcoming REL1_39 branches for T313919
* 18:00 brennen: using standalone puppetmaster in devtools to test phabricator scap3 changes
== 2022-07-28 ==
* 17:54 brennen: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/818189/
== 2022-07-27 ==
* 13:55 James_F: Zuul: [mediawiki/core] Add a non-vendor php80 job for main branch T300463
* 13:08 James_F: Zuul: [mediawiki/core] Make php80 voting on REL1_38 for T274965
* 13:04 James_F: Zuul: Add php81 experimental job everywhere we have php80
* 12:39 James_F: Zuul: [mediawiki/extensions/WikibaseLexeme] Add WikibaseLexemeCirrusSearch dep
* 03:48 Krinkle: Click "Disable publishing" for a dozen repos created recently, including OAuthRateLimiter, ref T143162, T193565
== 2022-07-25 ==
* 22:16 dduvall: re-enabled puppet on untrusted runners following testing of https://gerrit.wikimedia.org/r/c/operations/puppet/+/815769
* 21:25 dduvall: disabling puppet on untrusted gitlab-runners to test deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/815769
== 2022-07-23 ==
* 17:43 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/816251
== 2022-07-21 ==
* 21:55 dancy: Upgrading scap to 4.11.2-1+0~20220720160115.349~1.gbpd4a6cb in beta cluster
== 2022-07-20 ==
* 15:43 dancy: Upgrading scap to 4.11.1-1+0~20220720154238.348~1.gbp94de82 in beta cluster
* 13:19 James_F: Zuul: [mediawiki/extensions/VueTest] Add extension-codehealth pipeline
== 2022-07-19 ==
* 17:40 dancy: Upgrading scap to 4.11.0-1+0~20220719173732.346~1.gbpe07bc9 in beta cluster
* 17:00 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ mwscript extensions/GrowthExperiments/maintenance/migrateWikitextMentorList.php --wiki=arwiki # T310905
== 2022-07-18 ==
* 19:43 dancy: Upgrading scap to 4.10.0-1+0~20220718175214.344~1.gbpe518a1 in beta cluster
* 13:40 Lucas_WMDE: lucaswerkmeister-wmde@deployment-deploy03:~$ sql wikishared --write < /srv/mediawiki-staging/php-master/extensions/CampaignEvents/db_patches/mysql/tables-generated.sql # T311752
* 10:40 hashar: Refreshing Jenkins jobs for https://gerrit.wikimedia.org/r/814745
* 09:58 hashar: Refreshing Jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/814730 jjb: update php jobs to have php-pcov included
* 09:46 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/814728
== 2022-07-17 ==
* 13:00 taavi: reloading zuul for https://gerrit.wikimedia.org/r/814356
== 2022-07-16 ==
* 00:10 mutante: doc1002 - sudo systemctl start rsync-doc-doc2001.codfw.wmnet - Icinga alerted after an 'rsync warning: some files vanished before they could be transferred (code 24)' - but all is ok on next attempt
== 2022-07-15 ==
* 15:59 hashar: Built pcov php docker images T280170
* 15:46 hashar: contint2001: `docker-system-prune-dangling.service` it failed overnight cause Docker was not running. That should clear Icinga state # T313119
* 14:05 James_F: Zuul: [mediawiki/tools/wikilambda-cli] Switch to node16 jobs
* 13:05 James_F: Docker: Building node16 images for CI for T313075, this time actually.
* 12:30 hashar: Starting docker on contint2001.wikimedia.org # T313119
* 12:20 hashar: rebuilding `php??` images for pcov https://gerrit.wikimedia.org/r/c/integration/config/+/694621 # T280170
* 10:55 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/813967
* 10:49 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/813932
== 2022-07-14 ==
* 18:50 James_F: Docker: Building node16 images for CI for T313075
* 14:52 James_F: Zuul: [mediawiki/skins/BlueSpiceSkin] Archive for T203215
* 14:48 James_F: Zuul: [mediawiki/extensions/BlueSpiceExtensions] Archive
* 14:42 James_F: Zuul: [mediawiki/extensions/BlueSpiceBookshelfUI] Archive for T268085
* 14:38 James_F: Zuul: [mediawiki/tools/wikilambda-cli] Install node14 CI
== 2022-07-13 ==
* 23:23 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/813720
* 20:31 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/813707
== 2022-07-12 ==
* 17:29 Amir1: dropping tl_namespace and tl_title from templatelinks in fawiki (T312865)
== 2022-07-11 ==
* 22:55 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/812934
* 19:46 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/812467
== 2022-07-10 ==
* 00:07 Krinkle: krinkle@mediawiki12$ sudo enable-puppet
== 2022-07-09 ==
* 20:39 ori: ori@deployment-mediawiki12:~$ sudo apt install php-tideways-xhprof-dbgsym
* 17:25 ori: Cherry-picked (varnish: use libvmod-querysort on Beta Cluster) on deployment-prep Puppetmaster. Can be reverted if there are any issues.
* 06:16 Krinkle: krinkle@mediawiki12$ sudo disable-puppet
* 06:08 ori: ori@deployment-mediawiki12: userdel systemd-coredump, followed by apt install systemd-coredump
* 05:50 Krinkle: krinkle@deployment-mediawiki-12$ sudo apt-get install systemd-coredump # ref T312689
== 2022-07-07 ==
* 22:42 TheresNoTime: clear stuck beta deployment jobs (again), T72597
* 21:10 TheresNoTime: clear stuck beta deployment jobs, T72597
* 16:47 urbanecm: deployment-prep: wikiadmin@172.16.3.206(enwiki)> delete from growthexperiments_mentor_mentee where gemm_mentor_id=93651; # testing a specific workflow in Special:MentorDashboard
* 12:22 hashar: integration: rebooting `integration-agent-docker-1039` T312534
== 2022-07-05 ==
* 14:17 dwalden: restarted mathoid service on deployment-docker-mathoid01
* 11:39 hashar: Reloaded Zuul for `skip selenium for Wikibase repo/rest-api` https://gerrit.wikimedia.org/r/c/integration/config/+/811258
* 08:49 hauskatze: Diffusion rORES repository. Changed URI settings: enabled SSH push for mirroring; disabled HTTP T311390
== 2022-06-30 ==
* 22:02 TheresNoTime: unstuck beta-mediawiki-config-update-eqiad jobs, will comment at T72597
* 21:05 TheresNoTime: cancelled beta-code-update-eqiad#398138 to make way for pending beta-scap-sync-world#57641, queued another beta-code-update-eqiad
* 16:47 taavi: reloading zuul to deploy https://gerrit.wikimedia.org/r/810053
== 2022-06-29 ==
* 14:48 ori: Clearing data from incomplete migration on Wikifunctionswiki via sql.php
* 13:39 TheresNoTime: clearing stuck beta deployment jobs, watching to ensure they catch up :')
== 2022-06-28 ==
* 14:45 TheresNoTime: clear stuck beta deployment jobs, now running & will keep an eye
* 13:39 hashar: gerrit: added `Cindy-the-browser-test-bot` to the `Service Users` group https://gerrit.wikimedia.org/r/admin/groups/d39fe9cefd40ca1a07e372c0d7bd7e72ce2e4a2f,members T311370
* 09:37 hashar: phabricator: changed username of rORES Phab>Gerrit replication from `phab` to `phabricator` # T311390
== 2022-06-27 ==
* 21:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/809022
* 19:28 Reedy: Reloading Zuul to deploy https://phabricator.wikimedia.org/T308406
== 2022-06-24 ==
* 20:52 taavi: added `denisse` as a member
== 2022-06-23 ==
* 15:59 taavi: reload zuul for https://gerrit.wikimedia.org/r/808021
== 2022-06-22 ==
* 17:36 taavi: gerrit: add tfellows to the extension-OpenBadges group per request in T308278
* 17:35 taavi: gerrit: create group extension-JsonData with robla in it, make it an owner of mediawiki/extensions/JsonData per request in T303147
* 16:19 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/807586
* 09:35 hashar: Switched `gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud` instance to use the project Puppet master `puppetmaster-1001.devtools.eqiad1.wikimedia.cloud`
* 09:08 hashar: contint1001 , contint2002: deleting `.git/logs` from all zuul-merger repositories. We do not need the reflog `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir rm -fR .git/logs \;` # T307620
* 09:00 hashar: contint1001 , contint2002: setting `core.logallrefupdates=false` on all Zuul merger git repositories: `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir git config core.logallrefupdates false \;` # T307620
* 07:46 hashar: Building operations-puppet docker image for https://gerrit.wikimedia.org/r/c/integration/config/+/807180
== 2022-06-21 ==
* 22:01 brennen: gitlab-runners: re-registering all shared runners
* 17:55 dancy: Upgrading scap to 4.9.4-1+0~20220621174226.320~1.gbp56e4d4 in beta cluster
== 2022-06-20 ==
* 16:30 urbanecm: add sgimeno as a project member (Growth engineer with need for access)
* 15:50 ori: On deployment-cache-{text,upload}06, ran: touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service (T310957)
* 14:07 ori: restarted acme-chief on deployment-acme-chief03
== 2022-06-17 ==
* 17:15 ori: provisioned deployment-cache-text07 in deployment-prep to test query normalization via VCL
* 01:08 TimStarling: on deployment-docker-cpjobqueue01 and deployment-docker-changeprop01 I redeployed the changeprop configuration, reverting the PHP 7.4 hack
== 2022-06-16 ==
* 12:24 hashar: gitlab: runner-1030: `docker volume prune -f`
* 12:24 hashar: gitlab: runner-1026: `docker volume prune -f`
* 10:02 elukey: ran `scap install-world --batch` to allow scap/puppet to work on ml-cache100[2,3]
== 2022-06-15 ==
* 22:39 brennen: phabricator: tagged release/2022-06-15/1 (T310742)
* 16:31 hashar: integration-agent-docker-1035: docker image prune
* 15:26 dancy: Upgrading scap to 4.9.4-1+0~20220615151557.315~1.gbped3b8d in beta cluster
== 2022-06-14 ==
* 21:30 TheresNoTime: clear out stuck `beta-scap-sync-world` jobs (repeatedly per each queued `beta-mediawiki-config-update-eqiad` job), queued jobs now running. monitored for until each job had run successfully. jobs up to date
* 17:18 brennen: starting 1.39.0-wmf.16 (T308069) transcript in deploy1002:~brennen/1.39.0-wmf.16.log
* 13:35 TheresNoTime: clear stuck `beta-scap-sync-world` job, other queued jobs now running. Cancel running `beta-update-databases-eqiad` job, will ensure it runs on the next timer
* 00:42 TimStarling: on deployment-deploy03 removed helm2, as was done in production
== 2022-06-13 ==
* 22:04 TheresNoTime: cleared out stalled Jenkins beta jobs on `deployment-deploy03`, manually started `beta-code-update-eqiad` job & watched to completion. all caught up
* 04:33 hashar: Restarting Docker on contint1001.wikimedia.org , apparently can't build images anymore
== 2022-06-12 ==
* 21:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/804777
== 2022-06-10 ==
* 15:20 James_F: Zuul: [mediawiki/extensions/SearchVue] Add initial CI jobs for T309932
* 08:28 hashar: Reloaded Zuul to remove mediawiki/services/parsoid from CI dependencies # https://gerrit.wikimedia.org/r/c/integration/config/+/803990
* 04:27 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.4 for T295578
* 04:03 TimStarling: on deployment-deploy03 running scap sync-world -v with PHP 7.2 for T295578 sanity check
== 2022-06-09 ==
* 22:49 dancy: Upgrading scap to 4.9.1-1+0~20220609211227.304~1.gbpe48c42 in beta cluster
* 16:39 brennen: gitlab shared runners: re-registering to apply image allowlist configuration
== 2022-06-08 ==
* 17:14 hashar: Reloaded Zuul for 
* 15:57 dancy: Set `profile::mediawiki::php::restarts::ensure: present` in deployment-prep hiera config for T237033
* 09:28 hashar: Reloaded Zuul for "Add doc publish for Translate" https://gerrit.wikimedia.org/r/792134
== 2022-06-06 ==
* 14:37 James_F: Zuul: [mediawiki/extensions/ImageSuggestions] Mark as in production for T302711
== 2022-06-02 ==
* 15:33 dancy: Upgrading scap to 4.8.1-1+0~20220602153109.295~1.gbp318d9c in beta cluster
* 11:26 hashar: Restarting Jenkins on contint2001
* 11:19 hashar: Restarting Jenkins on releases1002
== 2022-05-31 ==
* 21:16 dancy: Upgrading scap to 4.8.0-1+0~20220531211114.292~1.gbp8dbbcf in beta cluster
* 17:40 dancy: Upgrading scap to 4.8.0-1+0~20220531173912.291~1.gbp21a7ef in beta cluster
* 17:33 dancy: Reverted to scap 4.8.0-1+0~20220524160924.288~1.gbp794a08 in beta cluster
* 17:07 dancy: Upgrading scap to 4.8.0-1+0~20220531170512.289~1.gbp143729 in beta cluster
== 2022-05-30 ==
* 11:47 jelto: apply gitlab-settings to gitlab1004 - T307142
* 11:46 jelto: apply gitlab-settings to gitlab1003 - T307142
== 2022-05-28 ==
* 19:09 TheresNoTime: deployment-deploy04 live, not referenced by anything T309437
== 2022-05-27 ==
* 22:55 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/WikiLambda/maintenance/updateTypedLists.php --wiki=wikifunctionswiki --db # started ~20 min ago
* 22:49 TheresNoTime: manually running database update script: samtar@deployment-deploy03:~$ /usr/local/bin/wmf-beta-update-databases.py
* 22:09 TheresNoTime: samtar@deployment-deploy03:~$ sudo keyholder arm
* 21:44 TheresNoTime: hard rebooted deployment-deploy03 as soft reboot unresponsive
* 21:44 bd808: `sudo wmcs-openstack role add --user zabe --project deployment-prep projectadmin` (T309419)
* 21:10 zabe: zabe@deployment-deploy03:~$ sudo keyholder arm
* 20:53 bd808: `sudo wmcs-openstack role add --user samtar --project deployment-prep projectadmin` (T309415)
* 20:49 dancy: Initiated hard reboot of deployment-deploy03.deployment-prep
== 2022-05-26 ==
* 18:33 dancy: Updated Jenkins beta-* job configs
* 16:51 TheresNoTime: manually triggered beta-update-databases-eqiad post-merge of 
* 16:51 brennen: puppetmaster-1001.devtools: resetting ops/puppet checkout to production branch
== 2022-05-25 ==
* 18:38 TheresNoTime: (@ ~18:20UTC) samtar@deployment-mwmaint02:~$ mwscript resetUserEmail.php --wiki=wikidatawiki Mahir256 [snip] phab:T309230T309230
* 15:46 dancy: Restarted apache2 on gerrit1001
== 2022-05-24 ==
* 15:15 dancy: Upgrading scap to 4.7.1-1+0~20220524151055.286~1.gbpe809e8 in beta cluster
* 13:35 James_F: Zuul: [mediawiki/tools/code-utils] Add composer test CI for T309099
* 11:36 TheresNoTime: cleared stuck beta deployment jobs per https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update
== 2022-05-23 ==
* 19:21 inflatador: Deleted deployment-elastic0[5-7] in favor of newer bullseye hosts T299797
* 18:37 dancy: Reverted to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
* 18:35 dancy: Upgrading beta cluster scap to 4.7.1-1+0~20220523183110.280~1.gbpaa0826
* 14:49 James_F: Zuul: Enforce Postgres and SQLite support via in-mediawiki-tarball
* 08:37 elukey: move kafka jumbo in deployment-prep to fixed uid/gid - T296982
* 08:29 elukey: move kafka main in deployment-prep to fixed uid/gid - T296982
* 08:06 elukey: move kafka logging in deployment-prep to fixed uid/gid - T296982
== 2022-05-22 ==
* 18:39 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/795818/
== 2022-05-21 ==
* 23:05 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/794756/
* 14:11 hashar: Icinga reports `Gerrit Health Check SSL Expiry` errors filed as T308908
== 2022-05-20 ==
* 16:21 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793809
== 2022-05-19 ==
* 19:34 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/793527
* 14:31 hashar: Reloaded zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793458 Don't re-trigger the test pipeline on patches with C+2 already
== 2022-05-18 ==
* 19:31 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/793028
* 18:45 brennen: gitlab: created placeholder /repos/mediawiki group for squatting purposes
* 08:29 hashar: Updating SSH Build agent from 1.31.5 to 1.32.0 on CI Jenkins to prevent an issue when uploading `remoting.jar` # T307339#7937268
* 07:32 hashar: Deleting Jenkins agent configuration for `integration-castor03` # T252071
== 2022-05-17 ==
* 23:26 James_F: Zuul: [mediawiki/extensions/Phonos] Install basic quibble CI for T308558
== 2022-05-16 ==
* 19:31 inflatador: bking@deployment-elastic07 halted deployment-elastic07 in beta ES cluster; will decom on Friday T299797
* 19:02 inflatador: bking@deployment-elastic06 halted deployment-elastic06 in beta ES cluster; will decom on Friday T299797
* 08:33 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791809
== 2022-05-14 ==
* 23:19 James_F: Zuul: Add Dreamy_Jazz to CI allow list
* 23:17 James_F: Zuul: [mediawiki/extensions/LocalisationUpdate] Move out of production section
* 20:25 urbanecm: add TheresNoTime (samtar) as a project member per request
== 2022-05-13 ==
* 22:59 James_F: Zuul: [mediawiki/extensions/SocialProfile] Add WikiEditor as a CI dependency
* 22:52 James_F: Zuul: Add Tranve to CI allow list
* 22:01 hashar: reloaded zuul for https://gerrit.wikimedia.org/r/791688
* 18:58 inflatador: bking@deployment-elastic05 halted deployment-elastic05 in beta ES cluster; will decom in 1 wk T299797
* 17:18 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/791644/
* 13:16 taavi: added user Zoranzoki21 to extension-HidePrefix gerrit group T305317
== 2022-05-12 ==
* 22:09 inflatador: bking@deployment-elastic05 banned deployment-elastic05 from beta ES cluster in preparation for decom T299797
* 19:53 hashar: gerrit: triggering full replication to gerrit2001 to test T307137
* 16:00 hashar: contint2001 and contint1001 now automatically run `docker system prune --force` every day and `docker system prune --force` on Sunday https://gerrit.wikimedia.org/r/c/operations/puppet/+/773784/
* 15:05 brennen: gitlab-prod-1001.devtools: soft reboot
* 00:46 brennen: gitlab: disabling container registries on all existing projects (T307537)
== 2022-05-11 ==
* 23:20 brennen: gitlab-prod-1001.devtools: container registry currently enabled
* 18:58 brennen: gitlab-prod-1001.devtools: setting to use devtools standalone puppetmaster
== 2022-05-10 ==
* 12:06 hashar: Updating Quibble jobs to image 1.4.5 with Memcached enabled https://gerrit.wikimedia.org/r/c/integration/config/+/790641 T300340
* 10:55 hashar: Updating `wmf-quibble-*` jobs to Quibble 1.4.5 # https://gerrit.wikimedia.org/r/c/integration/config/+/790638/
* 08:36 hashar: Updating wikibase-client-docker and wikibase-repo-docker to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790621
* 08:30 hashar: Updating MediaWiki coverage jobs to Quibble image 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790381
* 08:24 hashar: Updating codehealth jobs to Quibble 1.4.5 + supervisord https://gerrit.wikimedia.org/r/c/integration/config/+/790380/
* 08:23 hashar: Updating MediaWiki Phan jobs to Quibble 1.4.5 https://gerrit.wikimedia.org/r/c/integration/config/+/790377
== 2022-05-09 ==
* 21:43 James_F: Beta Cluster: Shutting down old deployment-restbase03 instance for T295375
* 20:33 hashar: Manually cancelling deadlock build jobs for beta https://integration.wikimedia.org/ci/view/Beta/ # T307963
== 2022-05-08 ==
* 12:33 urbanecm: deployment-prep: urbanecm@deployment-mwmaint02:~$ foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/migrateMenteeOverviewFiltersToPresets.php --update # T304057
== 2022-05-06 ==
* 12:55 hashar: Migrated Castor service from integration-castor03 to integration-castor05 # T252071
== 2022-05-05 ==
* 22:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789723
* 22:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789721
* 22:28 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789720
* 22:24 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789718
* 22:21 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/789717
* 22:15 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789714
* 22:13 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789713
* 22:09 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789711
* 22:07 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/789710
* 21:57 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789707/1
* 21:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789706
* 21:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789704
* 21:44 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789703
* 21:38 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789698
* 21:35 dduvall: created 4 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789697
* 21:26 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789694
* 21:22 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789693
* 18:27 dduvall: reenabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud
* 18:25 dancy: Update to scap 4.7.1-1+0~20220505181519.270~1.gbpeb47ae in beta cluster
* 18:16 dduvall: disabled puppet on integration-agent-docker-1023.integration.eqiad1.wikimedia.cloud for deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/768774
* 16:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789650
* 16:26 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789649
* 14:25 hashar: Created integration-castor05
* 12:28 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/789179 and https://gerrit.wikimedia.org/r/789232
* 07:45 hashar: deployment-prep: removed a few queued Jenkins builds from https://integration.wikimedia.org/ci/view/Beta/
== 2022-05-04 ==
* 21:29 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789285
* 21:16 dduvall: created 1 new job to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789284
* 21:07 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789278
* 21:00 dduvall: created 2 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789277
* 20:48 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789274
* 20:44 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789273
* 20:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789265
* 20:25 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789264
* 20:22 urbanecm: urbanecm@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki "There'sNoTime" "TheresNoTime" # T307590
* 20:14 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789259/1
* 20:11 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789258
* 18:54 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789245
* 18:47 dduvall: creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789244
* 18:31 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789238
* 18:24 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789237
* 17:51 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789225
* 17:22 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789218
* 17:12 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789217
* 16:11 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789204
* 16:01 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789203
* 16:01 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789195
* 15:42 dduvall: created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789194
* 13:44 James_F: Zuul: [mediawiki/services/function-evaluator] Use bespoke pipeline jobs only T307507
== 2022-05-03 ==
* 23:35 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788871
* 23:23 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788868
* 22:03 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
* 22:01 dduvall: created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788806
* 21:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/788798
* 21:27 dduvall: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/788799
* 21:25 dduvall: created trigger-pipelinelib-pipeline-test and pipelinelib-pipeline-test jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/788799
* 11:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/788682
== 2022-05-02 ==
* 15:09 dancy: Updating beta cluster scap to 4.7.1-1+0~20220502085300.264~1.gbp367de7?
* 10:06 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/786934 # T301766
== 2022-04-29 ==
* 21:49 brennen: created https://gitlab.wikimedia.org/toolforge-repos and https://gitlab.wikimedia.org/cloudvps-repos for cloud tenants (T305301)
* 18:37 James_F: Zuul: Add SimilarEditors dependency on QuickSurveys extension for T297687
== 2022-04-28 ==
* 20:31 James_F: Zuul: Add PHP81 as voting for libraries, PHP extensions etc. for T293509
* 18:57 brennen: finished editing mediawiki-new-errors
* 18:50 brennen: adding some filters to mediawiki-new-errors, including one based on https://wikitech.wikimedia.org/wiki/Performance/Runbook/Kibana_monitoring#Filtering_by_query_string
* 09:03 hashar: Gerrit upgraded to 3.4.4 at roughly 8:00 UTC
== 2022-04-27 ==
* 19:06 hashar: Updating operations/software/gerrit branches and tags from upstream # T292759
* 15:20 hashar: Updating non-quibble jobs to composer 2.3.3 T303867 https://gerrit.wikimedia.org/r/c/integration/config/+/777029
== 2022-04-26 ==
* 15:40 brennen: train 1.39.0-wmf.9 (T305215): no current blockers - expect to start train ops after the toolhub deployment window wraps, so some time after 17:00 UTC; taking a pre-train stroll-around-the-block break before that.
* 13:46 James_F: Deleting deployment-mx02.deployment-prep.eqiad1.wikimedia.cloud for T306068
* 13:38 James_F: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI for T306897
* 12:33 hashar: Manually pruned dangling docker images on contint1001 and contint2001
* 08:30 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/780824
* 08:09 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/785204
== 2022-04-25 ==
* 17:29 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/779450
* 15:31 James_F: Zuul: [mediawiki/extensions/RegularTooltips] Add basic quibble CI
== 2022-04-20 ==
* 16:25 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service
== 2022-04-18 ==
* 19:27 brennen: gitlab runners: deleting a number of stale runners with no contacts in > 2 months which are most likely no longer extant
* 16:49 brennen: phabricator: created phame blog https://phabricator.wikimedia.org/phame/blog/view/22/ for T306329
* 16:48 brennen: phabricator: adding self to acl*blog-admins
* 15:33 James_F: Shutting off deployment-wdqs01 from the Beta Cluster project per T306054; it's apparently unused, so this shouldn't break anything.
== 2022-04-14 ==
* 22:30 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779969
* 16:09 brennen: removed or renamed 4 filters from mediawiki-new-errors per check-new-error-tasks/check.sh
== 2022-04-12 ==
* 21:49 brennen: Updating dev-images docker-pkg files on primary contint for elastic 7.10.2
* 21:46 brennen: Updating dev-images docker-pkg files on primary contint for elastic 6.8.23
* 21:37 brennen: Updating dev-images docker-pkg files on primary contint for apache & elasticsearch changes (T304290, T305143)
* 16:05 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779500
* 15:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/779498 https://gerrit.wikimedia.org/r/779141
== 2022-04-08 ==
* 11:08 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/778287
== 2022-04-07 ==
* 06:07 urbanecm: deployment-prep: foreachwiki extensions/GrowthExperiments/maintenance/T304461.php --delete # T304461, output is at P24204
* 05:54 urbanecm: deployment-prep: mwscript extensions/GrowthExperiments/maintenance/T304461.php --wiki={enwiki,cswiki} --delete # T304461
== 2022-04-06 ==
* 20:03 thcipriani: rebooting phabricator
* 11:44 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add BetaFeatures to phan deps for T304596
== 2022-04-04 ==
* 22:43 James_F: dockerfiles: [composer-scratch] Upgrade composer to 2.3.3 and cascade for T294260
* 18:49 hashar: Reloading Zuul to revert https://gerrit.wikimedia.org/r/776179
* 18:23 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776179
* 17:50 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/775796
* 12:12 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/776723
* 10:28 James_F: Zuul: [mediawiki/extensions/WikiLambda] Publish PHP and JS documentation
* 08:54 jnuche: redeploying Zuul
== 2022-04-02 ==
* 12:00 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/773903 on deployment-prep centralauth databases
== 2022-03-31 ==
* 20:58 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/775957
== 2022-03-29 ==
* 14:20 James_F: Zuul: [mediawiki/extensions/IPInfo] Add EventLogging phan dependency for T304948
* 12:32 hashar: integration-agent-docker-1039: clearing leftover pipelinelib builds: `sudo rm -fR /srv/jenkins/workspace/workspace/*` T304932 T302477
* 05:35 hashar: Relocate castor directory on integration-castor03 from `/srv/jenkins-workspace/caches` to `/srv/castor` https://gerrit.wikimedia.org/r/c/operations/puppet/+/774771
== 2022-03-28 ==
* 16:55 hashar: integration: created instance integration-castor04 with flavor `g3.cores8.ram32.disk20` (twice more ram than integration-castor03) # T252071
* 16:49 hashar: integration: created 320G volume https://horizon.wikimedia.org/project/volumes/3f90c3f2-158d-4e45-a919-0f048f47c3b6/ . Intended to migrate integration-castor03 T252071
* 10:34 hashar: contint2001 and contint1001: pruning obsolete branches from the zuul-merger: `sudo -H -u zuul find /srv/zuul/git -type d -name .git -print -execdir git -c url."https://gerrit.wikimedia.org/r/".insteadOf="ssh://jenkins-bot@gerrit.wikimedia.org:29418/" remote prune origin \;` T220606
* 10:25 hashar: Changed `Trainsperiment Survey Questions` surveys permissions to be open outside of WMF and limited to 1 answer (forcing signin) https://docs.google.com/forms/u/0/d/e/1FAIpQLSd0Nc2jGkAGW-5rTiKN2EHWzfw2HeHm13N-ZCw1xUdE3z6woQ/formrestricted
* 10:18 hashar: contint2001 and contint1001: pruning all git reflog entries from the zuul-merger: `sudo -u zuul find /srv/zuul/git -name .git -type d -execdir git reflog expire --expire=all --all`. They are useless and no more generated since https://gerrit.wikimedia.org/r/c/operations/puppet/+/757943
* 09:53 hashar: Tag Quibble 1.4.5 @ T291549
== 2022-03-27 ==
* 13:23 James_F: Zuul: [releng/phatality] Make the node14 CI job voting T304736
== 2022-03-26 ==
* 02:37 Reedy: beta-update-databases-eqiad is back to @hourly
== 2022-03-25 ==
* 23:51 Reedy: temporarily turning off period building of beta-update-databases-eqiad until it's run to completion
* 23:21 Reedy: running /usr/local/bin/wmf-beta-update-databases.py manually
* 20:22 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/773866
* 20:02 brennen: mediawiki-new-errors: ran check-new-error-tasks/check.sh and cleared "resolved" filters
* 09:43 hashar: Building Quibble Docker images to rename quibble-with-apache to quibble-with-supervisord
== 2022-03-24 ==
* 20:00 hashar: reloading Zuul for # T299320
* 20:00 James_F: Clearing integration-castor03:/srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node14-docker/_cacache/content-v2/sha512/22/ for T304652
* 15:00 James_F: Zuul: [design/codex] Publish code coverage reports for T303899
* 09:37 Lucas_WMDE: killed a beta-scap-sync-world job manually, let’s see if that helps getting beta updates unstuck
== 2022-03-23 ==
* 17:35 brennen: restarting phabricator for T304540, brief downtime expected
* 14:56 dancy: Updating scap to 4.5.0-1+0~20220321191814.216~1.gbp24bc64 in beta cluster
== 2022-03-22 ==
* 14:44 hashar: gerrit: `./deploy_artifacts.py --version=3.3.10 gerrit.war` T304226
* 13:50 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/771945
== 2022-03-21 ==
* 08:35 hashar: The castor cache for mediawiki/core wmf/1.39-wmf.1 is actually empty!
* 08:32 hashar: Nuking npm castor cache /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/wmf-quibble-selenium-php72-docker/npm/ # T300203
== 2022-03-18 ==
* 14:18 elukey: restart testing of kafka logging TLS certificates (may affect logstash in beta, ping me in case it is a problem)
* 13:22 hashar: Rolling back Quibble jobs from 1.4.4 T304147
* 07:41 elukey: experimenting with PKI and kafka logging on deployment-prep, logstash dashboard/traffic may be down (please ping me in case it is a problem)
== 2022-03-17 ==
* 19:11 hashar: Building Docker images for Quibble 1.4.4
* 19:06 hashar: Tag Quibble 1.4.4 @ # T300340
* 16:25 hashar: Switching Quibble jobs to use memcached rather than APCu https://gerrit.wikimedia.org/r/c/integration/config/+/770468 T300340
* 14:11 hashar: Update all jobs to support `CASTOR_HOST` env variable https://gerrit.wikimedia.org/r/770921 T216244 T252071
* 14:07 hashar: Building Docker image to support `CASTOR_HOST` https://gerrit.wikimedia.org/r/770921 T216244
== 2022-03-16 ==
* 22:00 James_F: Docker: Publishing sonar-scanner:4.6.0.2311-3 for T303958
* 20:13 James_F: Zuul: [mediawiki/services/function-evaluator and …/function-orchestrator] Switch to npm coverage job for T302607 and T302608
* 19:48 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769424/ on deployment-prep
* 19:43 taavi: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/771347/ on deployment-prep
== 2022-03-15 ==
* 18:26 brennen: gitlab: removed most existing /people groups
* 18:10 brennen: gitlab: finished migrating access for all existing people groups to direct project membership (T274461, T300935)
* 16:49 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/770963
* 14:30 hashar: CI Jenkins: globally defined CASTOR_HOST=integration-castor03.integration.eqiad.wmflabs via https://integration.wikimedia.org/ci/configure # T216244
* 14:17 hashar: Apply label `castor` to node https://integration.wikimedia.org/ci/computer/integration-castor03/ # T216244
* 01:37 James_F: Zuul: Switch services/function* publish job from node12 to node14
* 01:14 James_F: Zuul: [wikidata/query-builder] Switch branchdeploy from node12 to node14
* 00:08 James_F: Zuul: [wikipeg] Switch from node12 to node14 special job
== 2022-03-14 ==
* 23:57 James_F: Zuul: [ooui] Switch from node12 to node14
* 23:46 James_F: Docker: Publishing node14-test-browser-php80-composer:0.1.0
* 23:27 James_F: Zuul: Drop legacy node12 templates except the one for Services
* 23:10 James_F: Zuul: [oojs/router] Drop custom job and just use the generic node14 one
* 23:08 James_F: Zuul: [oojs/core] Switch from node12 to node14 jobs
* 22:46 James_F: Zuul: [unicodejs] Switch from node12 to node14
* 22:25 James_F: Zuul: [VisualEditor/VisualEditor] Switch from node12 to node14
* 19:51 James_F: Zuul: Migrate almost all libraries and tools from node12 to node14 for T267890
* 15:36 James_F: Zuul: Switch extension-javascript-documentation from node12 to node14 for T267890
* 15:21 James_F: Zuul: Switch all mwgate jobs from node12 to node14 for T267890
* 09:52 hashar: Building Quibble Docker images for https://gerrit.wikimedia.org/r/757867 T300340
* 08:54 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/770079
== 2022-03-11 ==
* 04:02 zabe: zabe@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/populateGlobalEditCount.php --wiki=metawiki
== 2022-03-10 ==
* 20:45 zabe: apply https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/769416 on deployment-prep centralauth databases
* 20:25 James_F: Zuul: [mediawiki/extensions/VueTest] Add basic quibble CI
* 20:03 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/768843
* 15:12 hashar: updating Quibble jenkins jobs
* 14:26 James_F: Docker: Publishing new versions of quibble-buster and cascade adding unzip for T250496 / T303417.
* 11:43 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/769668
* 09:59 dwalden: restarted apache on deployment-mediawiki11 # T302699
== 2022-03-09 ==
* 17:08 hashar: Updating Gerrit Comment.soy to get rid of a literal `null` string being inserted in notification emails https://gerrit.wikimedia.org/r/c/operations/puppet/+/768005 https://phabricator.wikimedia.org/T288312
== 2022-03-08 ==
* 20:31 brennen: requiring 2fa for all users under /repos
== 2022-03-07 ==
* 10:53 zabe: restarted apache on deployment-mediawiki11 # T302699
== 2022-03-04 ==
* 20:29 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/768146
* 19:13 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/768068
== 2022-03-03 ==
* 19:13 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767864
* 15:37 James_F: Docker: Publishing sury-php images based on bullseye not stretch and cascade for T278203
* 14:43 hashar: Reloading Zuul for 
* 12:47 hashar: Upgrading Quibble on CI Jenkins jobs from 1.3.0 to 1.4.3 https://gerrit.wikimedia.org/r/c/integration/config/+/767749/
* 10:30 hashar: Building Docker images for Quibble 1.4.3
* 10:22 hashar: Tagged Quibble 1.4.3 @ 
* 09:24 hashar: Building Docker images for Quibble 1.4.2
* 09:20 hashar: Tag Quibble 1.4.2 @ # T302226 T302707
== 2022-03-02 ==
* 19:53 James_F: Zuul: Configure CI for the forthcoming REL1_38 branches for T302908
* 15:56 dancy: Updating scap to 4.4.1-1+0~20220302155149.192~1.gbpe351d6 in beta
* 15:27 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/767493
* 15:04 taavi: resolve merge conflicts on deployment-puppetmaster04
== 2022-02-28 ==
* 19:29 brennen: removing mutante (dzahn) as application-level gitlab admin; adding as owner of /repos for the time being to facilitate some migrations
* 19:22 dancy: Update scap to 4.4.0-1+0~20220228192031.189~1.gbp0a8436 in beta
* 19:17 brennen: adding mutante (dzahn) as application-level gitlab admin
== 2022-02-26 ==
* 20:05 zabe: apply T302658 on deployment-prep centralauth databases
* 13:24 zabe: apply T302660 on deployment-prep centralauth databases
* 13:19 zabe: apply T302659 on deployment-prep centralauth databases
== 2022-02-24 ==
* 16:02 dancy: Updating beta cluster scap to 4.4.0-1+0~20220224155429.187~1.gbp66c5c2
* 13:44 hashar: integration/config now fully enforces shellcheck https://gerrit.wikimedia.org/r/756088
* 13:13 hashar: Built image docker-registry.discovery.wmnet/releng/castor:0.2.5
* 13:10 hashar: Updating castor-save-workspace-cache job https://gerrit.wikimedia.org/r/764817
* 11:54 hashar: Built image docker-registry.discovery.wmnet/releng/shellcheck:0.1.1
* 11:41 hashar: Built image docker-registry.discovery.wmnet/releng/sonar-scanner:4.6.0.2311-2
* 11:04 hashar: Built image docker-registry.discovery.wmnet/releng/operations-puppet:0.8.6
* 08:58 hashar: Built image docker-registry.discovery.wmnet/releng/mediawiki-phan-testrun:0.2.1
== 2022-02-23 ==
* 23:21 dancy: Update beta cluster scap to 4.3.1-1+0~20220223231645.183~1.gbp8ddb60
* 20:10 dancy: Updating scap in beta
* 19:23 hashar: Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.3
* 12:41 hashar: Depooling integration-agent-puppet-docker-1002 , pooling integration-agent-puppet-docker-1003 # T252071
* 10:21 hashar: Created Bullseye instance integration-agent-puppet-docker-1003 https://horizon.wikimedia.org/project/instances/96cf9ddc-daa3-4c9f-8c21-cdd58e95973e/ # T252071
* 08:37 hashar: Removing Stretch based integration-agent-qemu-1001 # T284774
== 2022-02-22 ==
* 16:41 zabe: zabe@deployment-mwmaint02:~$ foreachwiki migrateUserGroup.php oversight suppress # T112147
* 13:28 urbanecm: deployment-prep: Create database for incubatorwiki (T210492)
== 2022-02-21 ==
* 14:58 hashar: Reverting Quibble jobs from 1.4.0 to 1.3.0 # T302226
* 07:31 hashar: Switching Quibble jobs from Quibble 1.3.0 to 1.4.0 # T300340 T291549 T225730
* 07:27 hashar: Refreshing all Jenkins jobs
== 2022-02-20 ==
* 10:32 qchris: Manually triggering replication run of Gerrit's analytics/datahub to populate newly created analytics-datahub GitHub repo
== 2022-02-19 ==
* 12:19 taavi: restart trafficserver-tls on deployment-cache-text06
* 02:15 James_F: Zuul: [design/codex] Publish the Netlify preview on every patch for T293705
* 00:35 James_F: Manually re-triggered a build of the docs of Codex (via `zuul-test-repo design/codex postmerge`) now that we actually set the environment vars for T293705
== 2022-02-18 ==
* 22:54 James_F: Zuul: [branchdeploy-codex-node14-npm-docker] Create as experimental for T293705
* 22:14 James_F: Jenkins: Defined BRANCHDEPLOY_AUTH_TOKEN_codex and BRANCHDEPLOY_SITE_ID_codex secrets for T293705
* 13:44 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763724 T301453
* 09:21 hashar: Reloading Zuul for 
== 2022-02-17 ==
* 21:48 brennen: added Dzahn (mutante) to acl*repository-admins on phabricator
* 15:58 zabe: root@deployment-cache-upload06:~# touch /srv/trafficserver/tls/etc/ssl_multicert.config && systemctl reload trafficserver-tls.service # T301995
* 13:35 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/763207
* 13:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/763458
* 11:12 hashar: Bringing deployment-deploy03 back
* 11:07 hashar: Disabled deployment-deploy03 Jenkins agent in order to revert some mediawiki/core patch and test the outcome
== 2022-02-16 ==
* 18:20 hashar: Tag Quibble 1.4.1 @ # T300301
* 16:42 dancy: Updating to scap 4.3.1-1+0~20220216163646.173~1.gbp823710?in beta
* 12:55 jelto: apply gitlab-settings to gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud
* 10:09 hashar: Reloading Zuul for 
* 09:59 hashar: Reloading Zuul for 
== 2022-02-15 ==
* 21:12 dancy: rebooting deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud to try to revive beta wikis
* 20:59 dancy: Killed runaway puppet agent on deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
* 16:24 hashar: Restarting CI Jenkins for plugins updates
* 16:21 hashar: Upgrading Jenkins plugins on releases Jenkins
* 16:06 hashar: Rollback fresh-test Jenkins job to the version intended to run on integration-agent-qemu-1001
* 15:26 hashar: Reloading Zuul for 
== 2022-02-14 ==
* 16:28 dancy: Updating scap in beta cluster to 4.3.1-1+0~20220211225318.167~1.gbp315b2c
* 16:16 Amir1: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/762471
* 15:41 hashar: Messing up with fresh-test Jenkns job to polish up Qemu / qcow2 integration
* 14:26 jnuche: Jenkins upgrade complete T301361
* 13:54 jnuche: Jenkins contint instances are going to be restarted soon
== 2022-02-12 ==
* 18:22 urbanecm: deployment-prep: reboot deployment-eventgate-3 (T289029)
== 2022-02-10 ==
* 17:29 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/761602
== 2022-02-09 ==
* 15:22 taavi: deleted shutoff deployment-mx02
== 2022-02-08 ==
* 17:34 taavi: remove scap from deployment-kafka-main/jumbo
* 16:23 taavi: hard reboot misbehaving deployment-echostore01
* 13:39 taavi: delete /srv/mediawiki-staging.save on deployment-deploy03
== 2022-02-07 ==
* 20:55 taavi: added Zabe as member of the deployment-prep project T301179
* 18:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/760550
== 2022-02-04 ==
* 00:21 Krinkle: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/759622
== 2022-02-03 ==
* 18:41 taavi: deployment-prep: route /w/api.php to deployment-mediawiki11, trying to reduce load on a single server
* 14:53 hashar: Building Docker images for Quibble 1.4.0 (prepared by kostajh)
* 13:51 kostajh: Tag Quibble 1.4.0 @ # T300340 T291549 T225730
== 2022-02-02 ==
* 16:50 dancy: Upgrading scap to 4.2.2-1+0~20220202164708.157~1.gbp376a16 in beta.
* 16:12 dancy: Upgrading scap to 4.2.2-1+0~20220201161808.156~1.gbp1c1c64 in beta
== 2022-02-01 ==
* 17:27 addshore: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/734654
* 00:34 tgr: deployment-pre un-cherry-picked gerrit 758584 from beta puppetmaster, patch is now merged T300591
* 00:12 tgr: deployment-prep cherry-picked gerrit 758584 to beta puppetmaster T300591
== 2022-01-31 ==
* 19:01 James_F: Re-configured Jenkins job mediawiki-i18n-check-docker to for T222216
* 10:49 hashar: Added integration-agent-qemu-1003 with label `Qemu` # T284774
== 2022-01-28 ==
* 21:45 taavi: running recountCategories.php on all beta wikis per T299823#7652496
* 14:27 hashar: taking heapdump of CI Jenkins `sudo -u jenkins /usr/lib/jvm/java-11-openjdk-amd64/bin/jmap -dump:live,format=b,file=/var/lib/jenkins/202201281527.hprof xxxx`
== 2022-01-27 ==
* 20:26 hashar: Successfully published image docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.2 # T299431
* 19:34 Amir1: Reloading Zuul to deploy 757464
* 16:00 hashar: Pooling back agents 1035 1036 1037 1038 , they could not connect due to ssh host mismatch since yesterday they all got attached to instance 1033 and accepted that host key # T300214
* 09:16 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # T300236
* 09:05 hashar: integration: cumin --force 'name:docker' 'apt install rsync' # T300214
* 00:24 thcipriani: restarting jenkins
== 2022-01-26 ==
* 20:29 hashar: Completed migration of integration-agent-docker-XXXX instances from Stretch to Bullseye - T252071
* 19:55 hashar: deleting integration-agent-docker-1014 which only has the `codehealth` label. A short live experiment no more used since October 2nd 2019 - https://gerrit.wikimedia.org/r/c/integration/config/+/540362 - T234259
* 18:56 hashar: integration: pooled in Jenkins a few more Bullseye docker agents for T252071
* 18:17 hashar: integration: pooled in Jenkins a few Bullseye docker agent for T252071
* 16:45 hashar: integration: creating integration-agent-docker-1023 based on buster with new flavor `g3.cores8.ram24.disk20.ephemeral60.4xiops` # T290783
== 2022-01-25 ==
* 20:17 James_F: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency
* 16:39 James_F: Zuul: Mark Math extension as now tarballed in parameter_functions for T232948
* 15:57 James_F: Zuul: [mediawiki/extensions/Math] Add Math to the main gate for T232948
* 13:44 hashar: Jenkins CI: added Logger https://integration.wikimedia.org/ci/log/ProcessTree%20-%20T299995/ to watch `hudson.util.ProcessTree` for T299995
* 10:02 hashar: integration: removing usage of `role::ci::slave::labs::docker::docker_lvm_volume` in Horizon following https://gerrit.wikimedia.org/r/c/operations/puppet/+/755948 . Docker role instances now always have a 24G partition for Docker
* 09:59 hashar: integration-agent-qemu-1001: resized /srv to 100% disk free: `lvextend -r -l +100%FREE /dev/mapper/vd-second--local--disk` # T299996
* 09:59 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` # T299996
* 09:51 hashar: integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G`
* 09:51 hashar: integration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet T299996
* 09:22 Reedy: unblocked beta again
* 07:32 Krinkle: integration-castor03:/srv/jenkins-workspace/caches$ sudo rm -rf castor-mw-ext-and-skins/
== 2022-01-24 ==
* 21:44 Reedy: unstick beta ci jobs
* 21:19 jeena: reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756523
* 20:36 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/756139
* 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/{quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker} # T299933
* 17:28 hashar: Nuke castor caches on integration-castor03 : sudo rm -fR /srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/{quibble-vendor-mysql-php72-selenium-docker,wmf-quibble-selenium-php72-docker}
== 2022-01-22 ==
* 13:40 taavi: apply T299827 on deployment-prep centralauth database
* 11:44 taavi: restart varnish-frontend.service on deployment-cache-upload06 to clear puppet agent failure alerts
== 2022-01-21 ==
* 18:12 taavi: resolved merge conflicts on deployment-puppetmaster04
* 15:50 hashar: integration-puppetmaster-02: deleted 2021 snapshot tags in puppet repo and ran `git gc --prune=now`
== 2022-01-20 ==
* 20:24 James_F: Zuul: [Kartographer] Add parsoid as dependency for CI jobs
* 20:22 James_F: Zuul: [DiscussionTools] Add Gadgets as dependency for Phan jobs
* 20:04 dancy: Jenkins beta jobs are back online, using scap prep auto now.
* 19:19 dancy: Pausing beta Jenkins jobs to make a copy of /srv/mediawiki-staging in preparation for testing
* 19:10 dancy: Unpacking scap (4.1.1-1+0~20220120175448.144~1.gbp517f9d) over (4.1.1-1+0~20220113154148.133~1.gbp6e3a17) on deploy03
* 18:07 hashar: Updating Quibble jobs to have MediaWiki files written on the hosts /srv partition (38G) instead of inside the container which ends in /var/lib/docker (24G) https://gerrit.wikimedia.org/r/755743 # T292729
* 16:31 hashar: Rebalancing /var/lib/docker and /srv partitions on CI agents https://gerrit.wikimedia.org/r/755713
* 12:12 hashar: contint2001 deleting all the Docker images (they will be pulled as needed)
* 12:10 hashar: contint2001 : docker container prune && docker image prune
* 12:07 hashar: contint1001 deleting all the Docker images (they will be pulled as needed)
* 12:04 hashar: contint1001 `docker image prune`
* 11:51 hashar: Cleaning very old Docker images on contint1001.wikimedia.Org
== 2022-01-19 ==
* 18:20 hashar: Adding https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool again
* 17:31 hashar: Adding https://integration.wikimedia.org/ci/computer/contint1001/ back to the pool after the machine got powercycled # T299542
* 10:38 Reedy: kill some stuck jobs T299485
== 2022-01-18 ==
* 19:56 hashar: building Docker images for https://gerrit.wikimedia.org/r/754951
* 18:01 taavi: added ryankemper as a member of the deployment-prep project
* 15:00 hashar: Updating Jenkins jobs for Quibble 1.3.0 with proper PHP version in the images # T299389
* 11:39 hashar: Rolling back Quibble 1.3.0 jobs due to php configuration files with at least releng/quibble-buster73:1.3.0 # T299389
* 08:07 hashar: Updating Jenkins jobs for Quibble to pass `--parallel-npm-install` https://gerrit.wikimedia.org/r/c/integration/config/+/754569
* 08:02 hashar: Updating Jenkins jobs for Quibble 1.3.0
== 2022-01-17 ==
* 16:28 hashar: Building Quibble 1.3.0 Docker images
* 16:16 hashar: Tagged Quibble 1.3.0 @ # T297480 T226869 T294931
* 08:32 hashar: Refreshing all Jenkins jobs with jjb to take in account recent changes related to the Jinja2 docker macro
== 2022-01-14 ==
* 15:56 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/753981
* 14:59 hashar: Starting VM integration-agent-docker-1022 which was in shutdown state since December and is Bullseye based # T290783
* 13:49 hashar: Restarting all CI Docker agents via Horizon to apply new flavor settings T265615 T299211
* 01:47 dancy: revert to scap 4.1.1-1+0~20220113154148.133~1.gbp6e3a17 in beta
== 2022-01-13 ==
* 18:02 dancy: Updating scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on all beta hosts
* 17:54 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/753792
* 16:27 dancy: testing scap prep auto on deployment-deploy03
* 15:52 dancy: Update scap to 4.1.1-1+0~20220113154506.135~1.gbp523480 on deployment-deploy03
* 11:27 hashar: Updating Jenkins job to normalize usage of `docker run --workdir` https://gerrit.wikimedia.org/r/c/integration/config/+/753457
* 10:52 hashar: Restarting Jenkins CI for plugins update
* 10:42 hashar: Applied Jenkins built-in node migration to CI Jenkins (`master` > `built-in` renaming) # T298691
* 10:14 taavi: cancelled stuck deployment-prep jobs on jenkins
== 2022-01-12 ==
* 18:58 hashar: Applied plugins update to https://releases-jenkins.wikimedia.org/
== 2022-01-11 ==
* 09:18 hashar: Updating all Jenkins jobs following recent "noop" refactorings
== 2022-01-10 ==
* 17:13 dancy: Update beta scap to 4.1.0-1+0~20220107203309.130~1.gbpcd0ace
* 14:01 James_F: Zuul: Add gate-and-submit-l10n to Isa for T222291
== 2022-01-05 ==
* 19:15 taavi: run `sudo chown -R jenkins-deploy:wikidev public/dists/bullseye-deployment-prep/` on deployment-deploy03
* 17:31 hashar: Deploying Zuul change https://gerrit.wikimedia.org/r/c/integration/config/+/751697 to get rid of the wmf-quibble-apache jobs # T285649
* 10:48 hashar: CI: switching MediaWiki selenium from php built-in server to Apache # https://gerrit.wikimedia.org/r/751697
* 09:24 hashar: Updating Quibble jobs to use latest image (provides `quibble-with-apache` entrypoint) https://gerrit.wikimedia.org/r/c/integration/config/+/751685/
== 2022-01-04 ==
* 12:49 hashar: Reloading Zuul for "api-testing: rename jobs to shorter forms" https://gerrit.wikimedia.org/r/751422
* 09:48 hashar: Builder Quibble Docker images with Apache included https://gerrit.wikimedia.org/r/c/integration/config/+/748104
* 09:47 hashar: Reloading Zuul for "Add CentralAuth to phan dependency list for GrowthExperiments" https://gerrit.wikimedia.org/r/751383
== 2022-01-03 ==
* 14:37 hashar: Upgraded Java 11 on contint2001 && contint1001. Restarted CI Jenkins.
* 14:35 hashar: Upgraded Java 11 on releases1002 && releases2002
Category:SAL